Posts Tagged ‘DevoxxPL2019’
[DevoxxPL2019] Enhancing Career Through Effective Communication: Strategies for Developers
Lecturer
Piotr Stawirej operates as a software engineer at Revolut, with a background from Politechnika Łódzka. As a clean code enthusiast and TDD/BDD advocate, he trains on architecture and people skills, drawing from 12+ years in development.
Abstract
This inquiry addresses communication’s pivotal role in developers’ success, beyond technical prowess. It identifies common pitfalls like poor first impressions or ineffective feedback, proposing techniques from Dale Carnegie for building rapport, influencing ideas, and handling conflicts. Through personal anecdotes, it analyzes emotional intelligence’s impact on career progression, team dynamics, and personal growth, while advocating experimentation for practical mastery.
Communication Barriers: Common Developer Pitfalls
Developers often prioritize code over interpersonal skills, leading to frustrations in idea adoption or managerial interactions. Piotr shares experiences: aggressive presentations alienate audiences, underscoring preparation’s importance.
Context: in tech, 85% success ties to emotions; unchecked behaviors block advancements. Analytically, this manifests in overlooked suggestions or code review disputes.
Implications: fosters awareness, encouraging shifts from confrontation to collaboration.
Building Rapport: Techniques for Likability
Carnegie’s principles guide: smile genuinely, remember names, listen actively. Piotr exemplifies: using names personalizes, active listening validates.
Methodologically, avoid arguments—seek common ground. Analytically, this defuses tensions, promoting constructive dialogues.
Implications: strengthens relationships, easing idea acceptance.
Influencing Positively: Winning Without Conflict
Praise publicly, criticize privately; address faults indirectly. Piotr recounts: framing suggestions as questions empowers recipients.
Analytically, this leverages psychology for motivation. Implications: accelerates career by aligning teams.
Changing Behaviors: Gentle Approaches
Admit faults first, use questions over commands. Piotr’s story: self-deprecation eases corrections.
Analytically, reduces defensiveness. Implications: cultivates growth mindsets.
Practical Experimentation: Applying Skills Daily
Start small: experiment with techniques, observe outcomes. Piotr urges: internalize through practice.
Implications: transforms workplaces, enhancing satisfaction.
Links:
[DevoxxPL2019] User Login
Successful Login
- Navigate to login page
- Enter username “user” and password “pass”
- Verify dashboard visible
Steps code in Java:
@Step(“Navigate to login page”)
public void navigate() {
// code
}
Analytically, markdown's familiarity lowers barriers. Implications: accelerates onboarding, but lacks Gherkin's structure for some.
## Taiko Integration: Scriptless Web Automation
Taiko, a Chrome DevTools wrapper, enables intuitive interactions sans selectors. Dmitry demos on TodoMVC: open URLs, write inputs, click elements.
Code:
const { openBrowser, goto, write, click } = require(‘taiko’);
await openBrowser();
await goto(“todomvc.com”);
await write(“Buy milk”);
await click(“Add”);
“`
Analytically, natural APIs reduce fragility from UI changes. Implications: speeds test creation, though Chrome exclusivity limits cross-browser testing.
Practical Application: End-to-End Testing Flows
Combining Gauge specs with Taiko steps, Dmitry automates todo flows: create, complete, filter.
Specs drive discussions; Taiko handles executions reliably.
Analytically, this revives BDD by minimizing overheads. Implications: boosts test reliability, encouraging widespread use.
Organizational Adoption: Challenges and Strategies
Despite simplicity, resistance persists from legacy habits. Dmitry advises gradual introductions, emphasizing communication benefits.
Implications: transforms testing from chore to collaborative asset, enhancing product quality.
Links:
[DevoxxPL2019] Revitalizing Behavior-Driven Development for Web Testing with Gauge and Taiko
Lecturer
Dmitry Vinnik functions as an Engineering Manager at Meta, specializing in AI/ML, where he leads teams on cutting-edge solutions. As a Lead Developer Advocate for Open Source and Business Messaging, he promotes tools enhancing productivity, with a passion for software quality and international speaking engagements.
Abstract
This review revitalizes Behavior-Driven Development for web testing, contrasting traditional frameworks with Gauge and Taiko. It explores BDD principles, Gauge’s markdown-based specifications versus Cucumber’s Gherkin, and Taiko’s scriptless automation. Through demonstrations on a todo app, it assesses methodologies for cross-team communication, test maintenance, and browser interactions, while contemplating impacts on adoption and organizational dynamics.
BDD Fundamentals: Communication Over Automation
Behavior-Driven Development bridges stakeholders by expressing behaviors in natural language, yet adoption falters from tool complexities. Dmitry contextualizes this: BDD aims for shared understanding, but frameworks like Cucumber introduce regex-heavy steps, alienating non-technical participants.
Gauge counters with markdown specs, free from rigid syntax, allowing plain English descriptions. Steps implement in languages like Java or JavaScript, fostering flexibility.
Analytically, this decouples narratives from executions, easing maintenance. Implications: enhances collaboration, though requires discipline to avoid verbosity.
Gauge Versus Cucumber: Simplifying Specifications
Cucumber’s Gherkin mandates “Given-When-Then,” parsed via regex, complicating updates. Gauge uses markdown, supporting tables and parameters natively.
For a login scenario:
“`
[DevoxxPL2019] Mastering Kubernetes Development Within IntelliJ: Seamless Integration and Debugging
Lecturer
Ivan Portyankin works as a software engineer at Google, contributing to Google Cloud Platform and Cloud Code for IntelliJ. Based in New York City, he focuses on tools that simplify cloud-native development, with a background in enhancing developer productivity through IDE integrations.
Abstract
This discussion examines the capabilities of Google Cloud Tools for IntelliJ in streamlining Kubernetes development workflows. It covers motivations for IDE-centric approaches, conversions from plain Java apps to containerized deployments, and features like live debugging and continuous updates. Through demonstrations, it analyzes methodologies for YAML minimization, cluster interactions, and hot-swapping, while reflecting on implications for developer efficiency and Kubernetes adoption barriers.
Motivational Landscape: Bridging Code and Cluster Management
Kubernetes’ complexity often deters developers, as traditional workflows involve extensive CLI commands and YAML configurations, diverting focus from core coding. Ivan addresses this by showcasing tools that embed orchestration directly into IDEs like IntelliJ, allowing seamless transitions from local development to production deployments.
Contextually, this aligns with the rise of cloud-native paradigms, where teams seek to abstract infrastructure. Google’s Cloud Code plugin exemplifies this, supporting Java, Kotlin, Go, and other languages across JetBrains IDEs and VS Code.
Analytically, the approach reduces cognitive load: developers remain in familiar environments, avoiding context switches. Implications: accelerates iterations, lowers entry barriers for Kubernetes newcomers, fostering broader adoption in enterprises.
Application Conversion: From Monolith to Microservices
Starting with a plain Java app, Ivan demonstrates scaffolding Kubernetes manifests via Cloud Code. For a voting service, the plugin generates deployments, services, and ingresses, minimizing manual YAML edits.
Methodologically, select templates for languages like Java/Spring Boot, auto-populating fields. Deploy to clusters like GKE or Minikube directly from IDE run configurations.
For multi-language setups—Java, Kotlin/Go—the tool handles diverse runtimes, ensuring consistent deployments.
Analytically, this decouples app logic from ops, but requires accurate kubeconfig setups. Implications: enables polyglot teams, though debugging multi-pod interactions demands careful logging.
Live Debugging and Continuous Deployment: Enhancing Iteration
Cloud Code enables remote debugging on Kubernetes pods without config changes. Ivan attaches debuggers to running containers, setting breakpoints in code.
For updates, continuous mode rebuilds and redeploys on saves, hot-swapping classes where possible.
Methodologically, use Skaffold under the hood for builds; configure via skaffold.yaml for custom pipelines.
Analytically, this mirrors local debugging, bridging dev-prod gaps. Implications: shortens feedback loops, boosting productivity, though network latency can affect remote sessions.
Ecosystem Extensions and Future Directions: Beyond Basics
The plugin supports Helm for complex apps, though basic; future enhancements target better template editing.
Analytically, open-source nature invites contributions, accelerating features like multi-cluster management. Implications: democratizes Kubernetes, but skill gaps in underlying tools persist.
In essence, IDE integrations transform Kubernetes from ops burden to developer enabler.
Links:
[DevoxxPL2019] Micronaut Versus Spring Boot: Assessing Framework Alternatives
Lecturer
Vladimir Dejanović occupies the role of senior director for B2C technology at PVH, managing tech for fashion labels including Tommy Hilfiger and Calvin Klein. Leading the Amsterdam Java User Group as founder, he holds JavaOne Rockstar and CodeOne Star status, often presenting on Java ecosystems and patterns.
Abstract
This evaluation pits Micronaut against Spring Boot, exploring their strengths in Java app construction. It details comparison drivers, a CRUD repository task, and metrics like launch speed, resource consumption, and native compilation. Via coding sessions, it gauges philosophies, efficiency, and feature sets, while contemplating appropriateness for fresh initiatives versus legacy code.
Driving the Comparison: Libraries Versus Integrated Solutions
Deciding between modular libraries and all-inclusive frameworks shapes Java projects. Vladimir delineates: libraries afford customization but integration labor, frameworks like Spring Boot deliver ready solutions potentially at efficiency expense.
Background: Spring’s prowess incurs reflection-based costs, evident in clouds. Micronaut vows comparable might minus drawbacks, using build-time computations.
Analytically, suits service-oriented architectures needing swift boots. Ramifications: frameworks hasten prototypes, but burdens affect expansion; Micronaut’s method may streamline allocations.
Task Design and Execution: CRUD in Repositories
For contrast, Vladimir crafts a person-rating CRUD: compute from age/name, persist. Spring Boot uses annotations for models/repositories, leveraging CrudRepository’s auto-implementations.
Snippet:
@Entity
public class Person {
@Id
@GeneratedValue
private Long id;
private String name;
private int age;
private int rating;
// accessors
}
@Repository
public interface PersonRepository extends CrudRepository<Person, Long> {}
Micronaut necessitates explicit codings, annotating @Repository, implementing interfaces manually.
Analytically, Spring’s brevity accelerates, Micronaut’s clarity aids comprehension. Ramifications: Spring for quick builds; Micronaut for tuned performances.
Efficiency Metrics: Boot Times, Usage, Native Builds
Boot: Micronaut quicker from compile injections, Spring slower via runtime scans. Usage: Micronaut lighter, sans proxies.
Native: Micronaut natively compatible; Spring lacks direct backing.
Analytically, advantages Micronaut in ephemeral or constrained contexts. Ramifications: lowered cloud expenses, rapid initiations improving experiences.
Feature Landscape and Guides: Production Viability
Micronaut expands swiftly, backing Kafka, GraphQL, gRPC, discoveries. Guides/tutorials excel.
Spring Boot’s ripeness provides extensive links, but heavier.
Analytically, both facilitate rapid resolutions, Micronaut’s freshness attracts innovators. Ramifications: Micronaut for pioneers; keep Spring for established bases.
Final Appraisals: Judicious Choices
Both shine in output, Spring slightly in ease, Micronaut in efficacy. Maintain Spring legacies; ponder Micronaut for novices.
Ramifications: context-driven selections balance rapidity and extensibility.
Links:
[DevoxxPL2019] Resilience Patterns in Microservices: Beyond Hystrix
Lecturer
Tomasz Skowroński contributes to resilience-focused libraries and speaks on fault tolerance in distributed systems.
Abstract
This overview introduces resilience patterns for microservices, transitioning from Hystrix to Resilience4j. It explains time limiters, rate limiters, bulkheads, retries, and circuit breakers, using analogies and code to demonstrate configurations and executions. It appraises integration with existing code, order of application, and higher-layer usages, while considering effects on system stability and developer explicitness.
Introducing Resilience: Patterns for Fault Tolerance
Resilience ensures responsiveness amid failures, vital in distributed setups. Tomasz analogizes to Dunkirk evacuation, where limited boats mirror API calls—use responsibly.
Hystrix, Netflix’s library, implemented circuit breakers but ceased development. Resilience4j succeeds, embracing Java 8+ functional styles sans annotations or AOP.
Analytically, this shift favors lightweight, composable resilience over monolithic commands. Implications: easier adoption in diverse stacks, reducing overhead.
Time and Rate Limiters: Controlling Execution Durations and Frequencies
Time limiters enforce timeouts on futures or suppliers, preventing indefinite waits. Configure via builders:
TimeLimiterConfig config = TimeLimiterConfig.custom()
.timeoutDuration(Duration.ofMillis(500))
.build();
TimeLimiter timeLimiter = TimeLimiter.of(config);
Callable<String> callable = TimeLimiter.decorateFutureSupplier(timeLimiter, () -> CompletableFuture.supplyAsync(this::slowMethod));
This decorates calls, throwing on timeouts.
Rate limiters restrict invocations per period, using permissions:
RateLimiterConfig config = RateLimiterConfig.custom()
.limitForPeriod(50)
.limitRefreshPeriod(Duration.ofMinutes(1))
.timeoutDuration(Duration.ofSeconds(3))
.build();
RateLimiter rateLimiter = RateLimiter.of("backend", config);
Runnable restrictedCall = RateLimiter.decorateRunnable(rateLimiter, this::backendMethod);
Analytically, parameters like refresh periods balance throughput and protection. Implications: prevents overloads, though misconfigurations cause premature failures.
Bulkheads and Retries: Isolating and Recovering from Failures
Bulkheads isolate via thread pools or semaphores, limiting concurrent calls:
BulkheadConfig config = BulkheadConfig.custom()
.maxConcurrentCalls(100)
.maxWaitDuration(Duration.ofMillis(10))
.build();
Bulkhead bulkhead = Bulkhead.of("backend", config);
Supplier<String> decorated = Bulkhead.decorateSupplier(bulkhead, this::backendMethod);
Retries attempt failed calls, configurable for attempts and waits.
Analytically, exponential backoffs mitigate thundering herds. Implications: boosts reliability, but excessive retries amplify loads.
Circuit Breakers: Caching Failures for Protection
Circuit breakers track successes/failures in buffers, opening on thresholds to block calls, periodically probing recovery.
Configurations define states: closed (allow), open (block), half-open (test).
Analytically, sliding windows maintain recent histories. Implications: shields backends during outages, explicit via decorators.
Strategic Application: Ordering, Layers, and Myths
Order matters: circuit breakers before retries avoid futile attempts. Apply at gateways or clouds for broader protection.
Myths: not all patterns always; failure-fast over safe sans explicit fallbacks.
Implications: explicitness via decorators clarifies intent, fostering robust designs.
Links:
[DevoxxPL2019] Evaluating Micronaut Versus Spring Boot: A Framework Comparison
Lecturer
Vladimir Dejanović holds the position of senior director of B2C technology at PVH, overseeing fashion tech initiatives for brands like Tommy Hilfiger and Calvin Klein. As founder and leader of the Amsterdam Java User Group, he is a JavaOne Rockstar and CodeOne Star, frequently speaking on Java frameworks and architectures.
Abstract
This assessment contrasts Micronaut and Spring Boot, scrutinizing their capabilities in building Java applications. It outlines motivations for comparison, details a challenge involving repository implementations, and evaluates aspects like startup time, memory usage, and GraalVM compatibility. Through live demonstrations, it appraises design philosophies, performance metrics, and ecosystem maturity, while deliberating suitability for new versus existing projects.
Motivational Framework: Choosing Between Toolkits and Ecosystems
Selecting between library assemblages and comprehensive frameworks defines modern Java development. Vladimir articulates this dichotomy: libraries offer flexibility but demand integration, while frameworks like Spring Boot provide batteries-included convenience at potential runtime costs.
Context: Spring’s dominance stems from its power, yet expenses in reflection and startup manifest in cloud environments. Micronaut promises equivalent functionality sans drawbacks, leveraging compile-time processing.
Analytically, this addresses microservices’ needs for lightweight, fast-starting apps. Implications: frameworks accelerate prototyping, but overheads impact scaling; Micronaut’s approach could optimize resource utilization.
Challenge Setup and Implementation: Repository Patterns Examined
To compare, Vladimir devises a repository challenge: implement CRUD for persons with ratings from age and name. Spring Boot employs annotations for entities and repositories, extending CrudRepository for magic implementations.
Code:
@Entity
public class Person {
@Id
@GeneratedValue
private Long id;
private String name;
private int age;
private int rating;
// getters/setters
}
@Repository
public interface PersonRepository extends CrudRepository<Person, Long> {}
Micronaut requires manual implementations, using @Repository and extending interfaces, coding CRUD in classes.
Analytically, Spring’s conciseness boosts productivity, while Micronaut’s explicitness aids understanding. Implications: Spring suits rapid development; Micronaut favors control in performance-critical scenarios.
Performance Benchmarks: Startup, Memory, and Native Compilation
Startup: Micronaut launches faster due to compile-time dependency injection, versus Spring’s runtime reflection. Memory: Micronaut consumes less, avoiding proxies.
GraalVM: Micronaut compiles natively out-of-box; Spring lacks seamless support.
Analytically, these metrics favor Micronaut in serverless or resource-constrained setups. Implications: reduced costs in cloud billing, faster cold starts enhancing user experience.
Ecosystem and Documentation: Readiness for Production
Micronaut’s ecosystem grows rapidly, supporting Kafka, GraphQL, gRPC, and service discovery. Documentation excels with guides and tutorials.
Spring Boot’s maturity offers vast integrations, but at higher overheads.
Analytically, both enable quick solutions, but Micronaut’s modernity appeals for greenfield projects. Implications: Micronaut suits innovation; retain Spring for legacy stability.
Concluding Evaluations: Strategic Framework Selection
Both excel in productivity, with Spring edging in simplicity, Micronaut in efficiency. Retain existing Spring; consider Micronaut for new endeavors.
Implications: informed choices optimize for context, balancing speed and scalability.
Links:
[DevoxxPL2019] Pursuing Software Excellence: Satirical Reflections on Quality Assurance
Lecturer
Chet Haase, a comedy writer and former software engineer at Google specializing in Android, has authored several books on the platform’s history and development. With decades in Silicon Valley, he now focuses on humorous presentations and writings, drawing from his engineering background.
Abstract
This satirical examination critiques conventional quality pursuits in software, proposing absurd strategies to eliminate defects entirely. It defines quality through bug absence, introduces mock equations and metrics, and debunks testing myths via humor. Through exaggerated organizational restructurings and release tactics, it highlights methodologies’ absurdities, contextualizing within engineering cultures, and pondering satirical implications for productivity and perceptions.
Redefining Quality: Beyond Bugs and Toward Perfection
Quality in software often evokes rigorous processes, yet Chet humorously redefines it as absolute bug elimination, achievable only by ceasing code production. He parodies presentations with Greek-lettered equations, positing quality inversely proportional to defects—zero bugs yielding infinite quality.
Contextually, this lampoons metric obsessions, where charts depict declining bugs equating rising excellence. Analytically, it underscores defect inevitability in active development, as coding introduces errors.
Implications: satirizes pursuits of unattainable perfection, urging balanced approaches where quality stems from iterative improvements, not cessation.
Organizational Overhauls: Eliminating Defect Sources
Chet proposes radical restructurings: fire engineers, as they author bugs. Managers, attending meetings sans coding, produce none—hire more. Sales rebrands existing releases (e.g., 2.3 as 3.0), avoiding new defects.
Methodologically, this “bug equilibrium” maintains status quo, parodying stagnation fears. Analytically, highlights engineers’ value despite imperfections, as innovation requires risk.
Consequences: mocks efficiency quests ignoring human elements, implying true quality demands collaboration, not elimination.
Misconceptions and Practical Jests: Testing Fallacies Exposed
Common myths: testing ensures quality—Chet counters, tests merely verify existing quality, ineffective sans code. Quality as quantifiable—satirizes certificates as superficial validations.
Analytically, underscores qualitative aspects like usability over metrics. Implications: encourages holistic views, where quality encompasses user satisfaction beyond defect counts.
Satirical Ramifications: Lessons in Absurdity
Chet’s humor reveals engineering absurdities: over-reliance on processes stifles creativity. Implications: fosters self-reflection, promoting balanced methodologies valuing people over perfection.
In essence, quality eludes simplistic fixes, demanding nuanced, human-centered strategies.
Links:
[DevoxxPL2019] Crafting Effective Automated Tests: Insights Beyond Conventional Wisdom
Lecturer
Jacek Milewski serves as a senior software developer at Circle K, where he focuses on backend Java development in domains like fuel retail and electric vehicles. As a trainer at Bottega IT Minds, he conducts sessions on domain-driven design and software architecture, drawing from his experience as a consultant, speaker, and mentor in the IT community.
Abstract
This analysis investigates approaches to automated unit and integration testing in modular applications, emphasizing practical techniques for ensuring business logic integrity. It explores test builders for entity construction, in-memory versus real repositories, and the role of test-driven development in maintaining quality. Through a live-coded example of rating calculations based on age and name length, it evaluates methodologies for edge case coverage, assertion strategies, and the balance between speed and thoroughness, while considering implications for development velocity and software reliability.
Establishing Test Foundations: From Basic Assertions to Modular Design
Automated testing forms the bedrock of reliable software, yet many practitioners grapple with adapting strategies to evolving ecosystems. Jacek commences by underscoring the perpetual relevance of testing, as technological advancements continually introduce new challenges. His methodology revolves around a simple yet comprehensive example: computing a person’s rating from age and name length, where ratings range from 0 to 100, with penalties for ages under 18 or over 65, and bonuses for longer names.
Initial tests focus on isolated units, such as a rating calculator class. Here, inputs are mocked or directly provided, verifying outputs against expectations. For instance, a test might instantiate a person with age 20 and name “John Doe,” asserting the rating equals age plus name length, capped at 100. This isolates logic, ensuring purity without external dependencies.
As complexity grows, modularization becomes key. Jacek advocates separating concerns: entities hold data, services compute logic, repositories persist state. Tests then target these layers individually, using builders to construct test data fluently. A PersonBuilder might chain methods like withAge(25).withName(“Alice”).build(), promoting readability and reuse.
Contextually, this stems from real-world projects at Circle K, where business rules like vehicle charging require verifiable implementations. Analytically, such isolation accelerates feedback loops, catching defects early. However, over-isolation risks missing integration issues, necessitating complementary tests.
Implications extend to team dynamics: standardized builders reduce onboarding time, fostering consistency. Yet, excessive abstraction can obscure intent, demanding balance.
Integrating Dependencies: Balancing Unit and Integration Testing
Transitioning to dependencies, Jacek differentiates unit tests—focusing on isolated behavior—from integration tests, verifying interactions. For persistence, in-memory repositories simulate databases, allowing rapid execution without external setups.
In the rating scenario, a service saves rated persons to a repository. Unit tests inject mock repositories, asserting save invocations and contents. Code might resemble:
PersonBuilder builder = new PersonBuilder();
Person person = builder.withAge(30).withName("Bob").build();
RatingService service = new RatingService(new InMemoryRepository());
service.calculateAndSave(person);
assertEquals(1, repository.size());
assertEquals(34, repository.get(0).getRating());
This confirms logic without I/O overhead.
For integration, swap to real repositories (e.g., JPA with H2), reusing test structures. Jacek copies unit tests, altering only the injected repository, ensuring end-to-end validation with minimal duplication.
Methodologically, this dual approach leverages TDD: write failing tests, implement minimally to pass, refactor safely. Failing tests validate coverage—green from inception might overlook assertions.
Analytically, in-memory speeds iterations, while real databases catch schema mismatches. Implications: enhanced confidence in deployments, though integration suites slow CI pipelines, suggesting selective execution.
Optimizing for Development Speed: Dispelling Myths on Testing Overhead
A prevalent myth posits testing impedes velocity, yet Jacek counters with empirical observations: initial setups invest time, but yield dividends in maintainability. Without tests, early features deploy swiftly, but regressions mount, stalling progress.
Contrastingly, test-first approaches start slower—configuring builders, mocks—but sustain pace, as refactors preserve functionality. In his experience, untested codebases accrue debt, while tested ones enable fearless enhancements.
Methodologically, focus on meaningful assertions: verify behaviors, not implementations. For empty repositories, assert isEmpty() post-setup, confirming state.
Analytically, coverage metrics mislead if superficial; aim for edge cases like invalid ages or names. Implications: teams adopting this outpace untested counterparts long-term, delivering quality sustainably.
Broader Ramifications: Testing as a Catalyst for Quality Delivery
Testing transcends verification, shaping designs toward modularity. Jacek’s Circle K tenure illustrates: robust tests facilitate microservices evolution, aligning with business agility in retail.
Yet, no universal formula exists; adapt to domains—unit for logic, integration for persistence. Implications: cultivates culture valuing prevention over remediation, elevating software craftsmanship.
In summation, these practices, honed through experience, empower developers to deliver verifiable value efficiently.
Links:
[DevoxxPL2019] Semantic Event Sourcing: A Case Study of Transitioning from CRUD to Log-Based State Management
Lecturer
Neil Boddy is an engineer at Goldman Sachs, working in master data management. His expertise focuses on scalable data architectures, including shifts from relational CRUD to NoSQL and event-sourcing techniques for handling complex, high-volume financial data.
Abstract
This article delves into the transition from traditional CRUD operations in relational databases to log-based state management using event sourcing in a master data management context at Goldman Sachs. It analyzes the motivations, engineering challenges, and solutions for managing large-scale, fragmented data flows. Concepts like immutable events, semantic metadata, and sharded clusters are examined, with emphasis on resilience, recovery, and standardized access. Implications for fault tolerance and scalability in data-intensive applications are explored through practical observations.
Challenges in Traditional CRUD-Based Master Data Management
Master data management (MDM) involves processes, tools, and policies to create and maintain critical data shared across an organization, such as financial instruments (bonds, equities) or static references (currencies, countries). Neil describes Goldman Sachs’ environment with hundreds of heterogeneous data flows, schemas, and databases processing millions of structurally complex records daily—e.g., a bond with 300 attributes expanding to thousands at runtime due to relationships.
Traditional CRUD persists only the latest state, leading to issues: schema proliferation, domain-specific expertise needs, and poor scalability for large set operations (e.g., universe comparisons). Sequential processing fragility exacerbates problems; out-of-order batches corrupt states, requiring forensic analysis, migrations, and business disruptions. Human errors, buggy code, or upstream issues cascade, with recovery being error-prone and stressful.
Neil contrasts this with accounting’s compensatory transactions, preserving audit trails—paralleling event sourcing’s immutable events to avoid irreversible damage.
Principles of Event Sourcing and Log-Based Management
Event sourcing reconstructs state from sequenced events, inspired by domain-driven design. Unlike CRUD’s mutable updates, events are append-only, capturing changes immutably. Neil illustrates with a shopping cart: CRUD shows final quantities, but events reveal full history (additions, removals), enabling replays and audits.
In MDM, data arrives fragmented (e.g., car analogy: engine, chassis batches). Aggregates ensure logical consistency, atomic storage, and intuitive structures. Using MongoDB as a document store and log, events are stored in collections, supporting sharding for scalability.
Two-dimensional time—business (effective date) and system (processing timestamp)—enables point-in-time queries. For Wednesday’s data viewed Friday, include all events; rewind system time for as-at views.
Technical Implementation with MongoDB
Goldman Sachs employs a pipe-and-filters architecture: staging, transformation, resolution, persistence. MongoDB’s document model accommodates multi-structured data without custom schemas, embracing event sourcing semantics (full universes, deltas, corrections).
Dual logs: one for business data (minimal indexes, large sets), another for semantic metadata (rich indexes, small footprint). Metadata captures batch types, times, and segments, facilitating precise queries.
Sharded clusters partition logs, enabling concurrent queries. Process: query metadata log for pointers, then concurrently fetch from business log across nodes. Deduplicate by grouping identities, selecting latest observing 2D time, and dropping tombstones.
Projections support use cases: full universes, changes across times/segments, replays with deduplication, historic analyses.
Code sample for MongoDB aggregation (simplified projection):
db.businessLog.aggregate([
{ $match: { identity: "100", businessTime: { $lte: ISODate("2023-03-03") } } },
{ $sort: { systemTime: -1 } },
{ $group: { _id: "$identity", latest: { $first: "$$ROOT" } } },
{ $match: { "latest.tombstone": { $ne: true } } }
]);
This fetches latest non-tombstoned record for an identity up to a business date.
Resilience, Recovery, and Performance Gains
Log-based approaches enhance fault tolerance: stale data redefines points without corruption; rogue data exclusion is straightforward. Parallel recovery reduces times (e.g., 30 hours to 4), supporting out-of-order processing.
Low-friction persistence eases backups, indexing, and maintenance. Concurrency spreads read/write loads; rich metadata enables hygiene controls (e.g., preventing base data on processed days).
Standardization reduces variation, improving scalability over CRUD’s custom solutions.
Broader Implications and Observations
This shift yields richer data utility for computations (deltas, reconciliations) without data copying. Retired records are handled via log points, avoiding deletions. While new concepts and tech require learning, benefits include standardized access, tolerance to errors, and high availability via redundancy.
Neil notes applicability beyond MDM to batch-oriented, large-scale state management, though not universal. In conclusion, semantic event sourcing fortifies systems against fragility, promoting scalable, recoverable architectures in data-centric domains.