Posts Tagged ‘EventSourcing’
[DevoxxPL2019] Semantic Event Sourcing: A Case Study of Transitioning from CRUD to Log-Based State Management
Lecturer
Neil Boddy is an engineer at Goldman Sachs, working in master data management. His expertise focuses on scalable data architectures, including shifts from relational CRUD to NoSQL and event-sourcing techniques for handling complex, high-volume financial data.
Abstract
This article delves into the transition from traditional CRUD operations in relational databases to log-based state management using event sourcing in a master data management context at Goldman Sachs. It analyzes the motivations, engineering challenges, and solutions for managing large-scale, fragmented data flows. Concepts like immutable events, semantic metadata, and sharded clusters are examined, with emphasis on resilience, recovery, and standardized access. Implications for fault tolerance and scalability in data-intensive applications are explored through practical observations.
Challenges in Traditional CRUD-Based Master Data Management
Master data management (MDM) involves processes, tools, and policies to create and maintain critical data shared across an organization, such as financial instruments (bonds, equities) or static references (currencies, countries). Neil describes Goldman Sachs’ environment with hundreds of heterogeneous data flows, schemas, and databases processing millions of structurally complex records daily—e.g., a bond with 300 attributes expanding to thousands at runtime due to relationships.
Traditional CRUD persists only the latest state, leading to issues: schema proliferation, domain-specific expertise needs, and poor scalability for large set operations (e.g., universe comparisons). Sequential processing fragility exacerbates problems; out-of-order batches corrupt states, requiring forensic analysis, migrations, and business disruptions. Human errors, buggy code, or upstream issues cascade, with recovery being error-prone and stressful.
Neil contrasts this with accounting’s compensatory transactions, preserving audit trails—paralleling event sourcing’s immutable events to avoid irreversible damage.
Principles of Event Sourcing and Log-Based Management
Event sourcing reconstructs state from sequenced events, inspired by domain-driven design. Unlike CRUD’s mutable updates, events are append-only, capturing changes immutably. Neil illustrates with a shopping cart: CRUD shows final quantities, but events reveal full history (additions, removals), enabling replays and audits.
In MDM, data arrives fragmented (e.g., car analogy: engine, chassis batches). Aggregates ensure logical consistency, atomic storage, and intuitive structures. Using MongoDB as a document store and log, events are stored in collections, supporting sharding for scalability.
Two-dimensional time—business (effective date) and system (processing timestamp)—enables point-in-time queries. For Wednesday’s data viewed Friday, include all events; rewind system time for as-at views.
Technical Implementation with MongoDB
Goldman Sachs employs a pipe-and-filters architecture: staging, transformation, resolution, persistence. MongoDB’s document model accommodates multi-structured data without custom schemas, embracing event sourcing semantics (full universes, deltas, corrections).
Dual logs: one for business data (minimal indexes, large sets), another for semantic metadata (rich indexes, small footprint). Metadata captures batch types, times, and segments, facilitating precise queries.
Sharded clusters partition logs, enabling concurrent queries. Process: query metadata log for pointers, then concurrently fetch from business log across nodes. Deduplicate by grouping identities, selecting latest observing 2D time, and dropping tombstones.
Projections support use cases: full universes, changes across times/segments, replays with deduplication, historic analyses.
Code sample for MongoDB aggregation (simplified projection):
db.businessLog.aggregate([
{ $match: { identity: "100", businessTime: { $lte: ISODate("2023-03-03") } } },
{ $sort: { systemTime: -1 } },
{ $group: { _id: "$identity", latest: { $first: "$$ROOT" } } },
{ $match: { "latest.tombstone": { $ne: true } } }
]);
This fetches latest non-tombstoned record for an identity up to a business date.
Resilience, Recovery, and Performance Gains
Log-based approaches enhance fault tolerance: stale data redefines points without corruption; rogue data exclusion is straightforward. Parallel recovery reduces times (e.g., 30 hours to 4), supporting out-of-order processing.
Low-friction persistence eases backups, indexing, and maintenance. Concurrency spreads read/write loads; rich metadata enables hygiene controls (e.g., preventing base data on processed days).
Standardization reduces variation, improving scalability over CRUD’s custom solutions.
Broader Implications and Observations
This shift yields richer data utility for computations (deltas, reconciliations) without data copying. Retired records are handled via log points, avoiding deletions. While new concepts and tech require learning, benefits include standardized access, tolerance to errors, and high availability via redundancy.
Neil notes applicability beyond MDM to batch-oriented, large-scale state management, though not universal. In conclusion, semantic event sourcing fortifies systems against fragility, promoting scalable, recoverable architectures in data-centric domains.
Links:
[DevoxxPL2019] Practical Event Sourcing: Avoiding Common Errors in Implementation
Lecturer
David Schmitz functions as a key architect at Senacor Technologies, with deep roots in rebuilding financial platforms using event-driven designs. His work spans multiple large-scale projects in banking and insurance, emphasizing sustainable architectures.
Abstract
This review critiques simplistic event sourcing adoptions, advocating refined tactics derived from extensive enterprise deployments. It defines events as unalterable facts in streams, integrates CQRS for separated concerns, and tackles hurdles in transactions, compensations, and privacy compliance. Via financial illustrations, it inspects techniques for consistency, deletion via tombstones, and polyglot persistence, reflecting on outcomes for durability and adaptability in regulated domains.
Understanding Event-Driven Foundations: Events as Core Artifacts
Event sourcing posits systems as event appendages, reconstructing states from chronological logs. David analogizes to routine transactions, like cafe orders, where each step—ordering, paying—yields events forming the narrative.
In enterprises, particularly finance, this supplants CRUD with append-only logs, yielding audit trails. Streams, via Kafka, order events, partitions grouping related ones (e.g., per account) for sequential processing.
Context: monolithic databases yield to distributed microservices, but naive per-service databases fragment truth. Event sourcing unifies via shared streams, though over-partitioning bloats management.
Analytically, events as facts enable replays for debugging or migrations, but demand schema versioning. Implications: bolsters traceability, yet escalates storage, mitigated by compaction removing intermediates.
CQRS Integration: Decoupling Commands and Queries
CQRS bifurcates modifications (commands appending events) from retrievals (queries on projections). David exemplifies transfers: commands emit events to sender/receiver streams; processors project balances.
Methodologically, Kafka Streams aggregate, ensuring eventual consistency sans distributed transactions. For guarantees, single-writer patterns serialize per-entity events.
Challenges: optimistic concurrency via versions prevents overwrites; failures invoke compensations—explicit reversals preserving history.
In insurance, initial oversights led to redesigns; polyglot views (e.g., Elasticsearch for searches) enhance flexibility.
Analytically, this scales sides independently, but lags demand monitoring. Consequences: agile evolutions, though compensations complicate logic, necessitating clear business rules.
Compliance Handling: Deletion and Rectification in Immutable Logs
GDPR mandates erasure/rectification, clashing with immutability. David proposes tombstones—events signaling deletions, prompting anonymization downstream.
For users, tombstones propagate, purging identifiers while retaining structures for audits. Rectifications append compensations, adjusting views idempotently.
Methodologically, upcasters during replays adapt old events to new schemas, ensuring compatibility.
Analytically, this reconciles permanence with privacy, but requires cross-domain coordination. Implications: legal alignment, though tombstones inflate streams, offset by compaction.
Maintainability and Tradeoffs: Insights from Deployments
Event sourcing decouples, but excessive streams hinder navigation. David advises aligning with bounded contexts, mixing with relational stores for queries.
From banking, optimistic conflicts resolved via retries; security via encryption guards sensitive events.
Methodologically, tools like KSQL query streams declaratively. Implications: audit-ready systems, but complexity in debugging distributed flows.
In reflection, event sourcing thrives with deliberate design, yielding robust, evolvable platforms in stringent sectors.
Links:
[DevoxxBE2013] CQRS for Great Good
Oliver Wolf, principal consultant and executive board member at INNOQ, challenges conventional architectures with CQRS (Command-Query Responsibility Segregation). A SOA and Java expert, Oliver traces CQRS’s evolution from CQS, demonstrating incremental adoption—from read-write separation to event sourcing. His session, enriched with examples, equips developers to rethink data flows, optimizing for asymmetric workloads in banking and beyond.
CQRS decouples commands (writes) from queries (reads), enabling tailored models. Oliver illustrates phased implementation, culminating in event-sourced systems for auditability and scalability.
From CQS to CQRS: Foundational Concepts
Oliver recalls CQS—Bertrand Meyer’s principle segregating mutators from inspectors. CQRS extends this, allowing distinct read/write models. He demos a simple e-commerce app, splitting a unified model into command (order placement) and query (inventory views).
This separation, Oliver explains, resolves impedance mismatches, enhancing performance.
Incremental Adoption Strategies
Phased rollout minimizes risk: start with asymmetric databases, Oliver advises, using separate stores for reads/writes. He showcases materialized views, syncing via background jobs.
Advanced steps introduce event sourcing: commands emit events, replayed for state reconstruction, ensuring immutability.
Event Sourcing and Distribution
Event sourcing captures changes as immutable logs, Oliver illustrates, rebuilding state on demand. Distribution follows: client/server variants, with web frontends querying dedicated services.
In banking, Oliver notes, CQRS optimizes configurable systems, balancing risk with extensibility.
Guidelines for Application
Oliver urges starting small: identify read-heavy operations, segregate gradually. Avoid over-engineering; CQRS suits complex domains, not simple CRUD.
Community examples, he shares, validate phased approaches, with INNOQ projects exploring hybrid models.
Links:
[DevoxxFR2013] Distributed DDD, CQRS, and Event Sourcing – Part 1/3: Time as a Business Core
Lecturer
Jérémie Chassaing is an architect at Siriona, focusing on scalable systems for hotel channel management. Author of thinkbeforecoding.com, a blog on Domain-Driven Design, CQRS, and Event Sourcing, he founded Hypnotizer (1999) for interactive video and BBCG (2004) for P2P photo sharing. His work emphasizes time-centric modeling in complex domains.
Abstract
Jérémie Chassaing posits time as central to business logic, advocating Event Sourcing to capture temporal dynamics in Domain-Driven Design. He integrates Distributed DDD, CQRS, and Event Sourcing to tackle scalability, concurrency, and complexity. Through examples like order management, Chassaing analyzes event streams over relational models, demonstrating eventual consistency and projection patterns. The first part establishes foundational shifts from CRUD to event-driven architectures, setting the stage for distributed implementations.
Time’s Primacy in Business Domains
Chassaing asserts time underpins business: reacting to events, analyzing history, forecasting futures. Traditional CRUD ignores temporality, leading to lost context. Event Sourcing records immutable facts—e.g., OrderPlaced, ItemAdded—enabling full reconstruction.
This contrasts relational databases’ mutable state, where updates erase history. Events form audit logs, facilitating debugging and compliance.
Domain-Driven Design Foundations: Aggregates and Bounded Contexts
DDD models domains via aggregates—consistent units like Order with line items. Bounded contexts delimit scopes, preventing model pollution.
Distributed DDD extends this to microservices, each owning a context. CQRS separates commands (writes) from queries (reads), enabling independent scaling.
CQRS Mechanics: Commands, Events, and Projections
Commands mutate state, emitting events. Handlers project events to read models:
case class OrderPlaced(orderId: UUID, customer: String)
case class ItemAdded(orderId: UUID, item: String, qty: Int)
// Command handler
def handle(command: AddItem): Unit = {
// Validate
emit(ItemAdded(command.orderId, command.item, command.qty))
}
// Projection
def project(event: ItemAdded): Unit = {
updateReadModel(event)
}
Projections denormalize for query efficiency, accepting eventual consistency.
Event Sourcing Advantages: Auditability and Scalability
Events form immutable logs, replayable for state recovery or new projections. This decouples reads/writes, allowing specialized stores—SQL for reporting, NoSQL for search.
Chassaing addresses concurrency via optimistic locking on aggregate versions. Distributed events use pub/sub (Kafka) for loose coupling.
Challenges and Patterns: Idempotency and Saga Management
Duplicates require idempotent handlers—e.g., check event IDs. Sagas coordinate cross-aggregate workflows, reacting to events and issuing commands.
Chassaing warns of “lasagna architectures”—layered complexity—and advocates event-driven simplicity over tiered monoliths.
Implications for Resilient Systems: Embracing Eventual Consistency
Event Sourcing yields antifragile designs: failures replay from logs. Distributed CQRS scales horizontally, handling “winter is coming” loads.
Chassaing urges rethinking time in models, shifting from mutable entities to immutable facts.