Posts Tagged ‘ScalableArchitecture’
[DevoxxPL2019] Semantic Event Sourcing: A Case Study of Transitioning from CRUD to Log-Based State Management
Lecturer
Neil Boddy is an engineer at Goldman Sachs, working in master data management. His expertise focuses on scalable data architectures, including shifts from relational CRUD to NoSQL and event-sourcing techniques for handling complex, high-volume financial data.
Abstract
This article delves into the transition from traditional CRUD operations in relational databases to log-based state management using event sourcing in a master data management context at Goldman Sachs. It analyzes the motivations, engineering challenges, and solutions for managing large-scale, fragmented data flows. Concepts like immutable events, semantic metadata, and sharded clusters are examined, with emphasis on resilience, recovery, and standardized access. Implications for fault tolerance and scalability in data-intensive applications are explored through practical observations.
Challenges in Traditional CRUD-Based Master Data Management
Master data management (MDM) involves processes, tools, and policies to create and maintain critical data shared across an organization, such as financial instruments (bonds, equities) or static references (currencies, countries). Neil describes Goldman Sachs’ environment with hundreds of heterogeneous data flows, schemas, and databases processing millions of structurally complex records daily—e.g., a bond with 300 attributes expanding to thousands at runtime due to relationships.
Traditional CRUD persists only the latest state, leading to issues: schema proliferation, domain-specific expertise needs, and poor scalability for large set operations (e.g., universe comparisons). Sequential processing fragility exacerbates problems; out-of-order batches corrupt states, requiring forensic analysis, migrations, and business disruptions. Human errors, buggy code, or upstream issues cascade, with recovery being error-prone and stressful.
Neil contrasts this with accounting’s compensatory transactions, preserving audit trails—paralleling event sourcing’s immutable events to avoid irreversible damage.
Principles of Event Sourcing and Log-Based Management
Event sourcing reconstructs state from sequenced events, inspired by domain-driven design. Unlike CRUD’s mutable updates, events are append-only, capturing changes immutably. Neil illustrates with a shopping cart: CRUD shows final quantities, but events reveal full history (additions, removals), enabling replays and audits.
In MDM, data arrives fragmented (e.g., car analogy: engine, chassis batches). Aggregates ensure logical consistency, atomic storage, and intuitive structures. Using MongoDB as a document store and log, events are stored in collections, supporting sharding for scalability.
Two-dimensional time—business (effective date) and system (processing timestamp)—enables point-in-time queries. For Wednesday’s data viewed Friday, include all events; rewind system time for as-at views.
Technical Implementation with MongoDB
Goldman Sachs employs a pipe-and-filters architecture: staging, transformation, resolution, persistence. MongoDB’s document model accommodates multi-structured data without custom schemas, embracing event sourcing semantics (full universes, deltas, corrections).
Dual logs: one for business data (minimal indexes, large sets), another for semantic metadata (rich indexes, small footprint). Metadata captures batch types, times, and segments, facilitating precise queries.
Sharded clusters partition logs, enabling concurrent queries. Process: query metadata log for pointers, then concurrently fetch from business log across nodes. Deduplicate by grouping identities, selecting latest observing 2D time, and dropping tombstones.
Projections support use cases: full universes, changes across times/segments, replays with deduplication, historic analyses.
Code sample for MongoDB aggregation (simplified projection):
db.businessLog.aggregate([
{ $match: { identity: "100", businessTime: { $lte: ISODate("2023-03-03") } } },
{ $sort: { systemTime: -1 } },
{ $group: { _id: "$identity", latest: { $first: "$$ROOT" } } },
{ $match: { "latest.tombstone": { $ne: true } } }
]);
This fetches latest non-tombstoned record for an identity up to a business date.
Resilience, Recovery, and Performance Gains
Log-based approaches enhance fault tolerance: stale data redefines points without corruption; rogue data exclusion is straightforward. Parallel recovery reduces times (e.g., 30 hours to 4), supporting out-of-order processing.
Low-friction persistence eases backups, indexing, and maintenance. Concurrency spreads read/write loads; rich metadata enables hygiene controls (e.g., preventing base data on processed days).
Standardization reduces variation, improving scalability over CRUD’s custom solutions.
Broader Implications and Observations
This shift yields richer data utility for computations (deltas, reconciliations) without data copying. Retired records are handled via log points, avoiding deletions. While new concepts and tech require learning, benefits include standardized access, tolerance to errors, and high availability via redundancy.
Neil notes applicability beyond MDM to batch-oriented, large-scale state management, though not universal. In conclusion, semantic event sourcing fortifies systems against fragility, promoting scalable, recoverable architectures in data-centric domains.