Jonathan Lalou's Blog

Posts Tagged ‘DataResilience’

Lecturer

The presentation features resilience specialists who architect backup and recovery solutions for SaaS DevOps platforms. Their expertise spans data protection strategies for Jira, Confluence, GitHub, and related tools that lack native recovery capabilities.

Abstract

The session reveals a critical gap in DevOps resilience: SaaS platforms that store mission-critical data without adequate backup controls. Through incident analysis and recovery patterns, it establishes that infrastructure protection alone insufficiently addresses application data loss, advocating purpose-built solutions for comprehensive business continuity.

DevOps Tools as Critical Business Assets

Modern software delivery depends on SaaS platforms:

• Jira: Product roadmaps, sprint planning
• Confluence: Technical documentation, runbooks
• GitHub: Source code, CI/CD configurations

These tools contain intellectual property and operational knowledge that infrastructure backups cannot restore. A corrupted Jira automation recently disrupted an entire product organization despite perfect infrastructure resilience.

Risk Taxonomy and Impact Analysis

Data loss manifests through multiple vectors:

1. Human Error (62%): Misconfigured automations, bulk deletes
2. Malicious Actors (24%): Compromised admin accounts
3. Application Bugs (14%): Vendor updates, API failures

Impact extends beyond availability—corrupted sprint data delays releases, lost documentation impedes incident response, deleted repositories halt deployments.

Native Backup Limitations

SaaS providers prioritize availability over recoverability:

Vendor SLA: 99.9% uptime
Vendor Backup: 30-day undo window
Point-in-time restore: Not supported

Jira retains deleted issues for 30 days; Confluence pages vanish permanently after trash emptying. GitHub offers no granular repository restore—organizations must rebuild from local clones.

Resilience Architecture Patterns

Purpose-built solutions implement:

backup_policy:
  frequency: 4_hours
  retention: 365_days
  granularity: issue_level
  encryption: customer_managed_keys

Automated backups capture metadata, attachments, and permissions. Recovery enables:

• Single issue restoration
• Project-level rollback
• Cross-instance migration

Recovery Time Objective Achievement

Traditional recovery requires vendor support tickets and partial exports. Specialized platforms achieve:

• RTO: < 5 minutes for critical items
• RPO: < 1 hour for configuration changes
• Audit trail: Immutable recovery logs

Proactive Resilience Framework

Organizations implement three pillars:

1. Risk Assessment: Map DevOps tools to business processes
2. Resilience Engineering: Automated backups with testing
3. Recovery Planning: Documented procedures and drills

Regular recovery exercises validate SLAs—75% of organizations lack tested SaaS recovery plans by 2028 projections.

Conclusion: Comprehensive Data Resilience

Infrastructure resilience protects servers; data resilience protects the business. DevOps tools represent crown jewels that native backups inadequately safeguard. Organizations that implement specialized protection achieve competitive advantage through uninterrupted delivery, regulatory compliance, and rapid incident recovery.

Links:

• Lecture Video

Lecturer

Neil Boddy is an engineer at Goldman Sachs, working in master data management. His expertise focuses on scalable data architectures, including shifts from relational CRUD to NoSQL and event-sourcing techniques for handling complex, high-volume financial data.

Abstract

This article delves into the transition from traditional CRUD operations in relational databases to log-based state management using event sourcing in a master data management context at Goldman Sachs. It analyzes the motivations, engineering challenges, and solutions for managing large-scale, fragmented data flows. Concepts like immutable events, semantic metadata, and sharded clusters are examined, with emphasis on resilience, recovery, and standardized access. Implications for fault tolerance and scalability in data-intensive applications are explored through practical observations.

Challenges in Traditional CRUD-Based Master Data Management

Master data management (MDM) involves processes, tools, and policies to create and maintain critical data shared across an organization, such as financial instruments (bonds, equities) or static references (currencies, countries). Neil describes Goldman Sachs’ environment with hundreds of heterogeneous data flows, schemas, and databases processing millions of structurally complex records daily—e.g., a bond with 300 attributes expanding to thousands at runtime due to relationships.

Traditional CRUD persists only the latest state, leading to issues: schema proliferation, domain-specific expertise needs, and poor scalability for large set operations (e.g., universe comparisons). Sequential processing fragility exacerbates problems; out-of-order batches corrupt states, requiring forensic analysis, migrations, and business disruptions. Human errors, buggy code, or upstream issues cascade, with recovery being error-prone and stressful.

Neil contrasts this with accounting’s compensatory transactions, preserving audit trails—paralleling event sourcing’s immutable events to avoid irreversible damage.

Principles of Event Sourcing and Log-Based Management

Event sourcing reconstructs state from sequenced events, inspired by domain-driven design. Unlike CRUD’s mutable updates, events are append-only, capturing changes immutably. Neil illustrates with a shopping cart: CRUD shows final quantities, but events reveal full history (additions, removals), enabling replays and audits.

In MDM, data arrives fragmented (e.g., car analogy: engine, chassis batches). Aggregates ensure logical consistency, atomic storage, and intuitive structures. Using MongoDB as a document store and log, events are stored in collections, supporting sharding for scalability.

Two-dimensional time—business (effective date) and system (processing timestamp)—enables point-in-time queries. For Wednesday’s data viewed Friday, include all events; rewind system time for as-at views.

Technical Implementation with MongoDB

Goldman Sachs employs a pipe-and-filters architecture: staging, transformation, resolution, persistence. MongoDB’s document model accommodates multi-structured data without custom schemas, embracing event sourcing semantics (full universes, deltas, corrections).

Dual logs: one for business data (minimal indexes, large sets), another for semantic metadata (rich indexes, small footprint). Metadata captures batch types, times, and segments, facilitating precise queries.

Sharded clusters partition logs, enabling concurrent queries. Process: query metadata log for pointers, then concurrently fetch from business log across nodes. Deduplicate by grouping identities, selecting latest observing 2D time, and dropping tombstones.

Projections support use cases: full universes, changes across times/segments, replays with deduplication, historic analyses.

Code sample for MongoDB aggregation (simplified projection):

db.businessLog.aggregate([
  { $match: { identity: "100", businessTime: { $lte: ISODate("2023-03-03") } } },
  { $sort: { systemTime: -1 } },
  { $group: { _id: "$identity", latest: { $first: "$$ROOT" } } },
  { $match: { "latest.tombstone": { $ne: true } } }
]);

This fetches latest non-tombstoned record for an identity up to a business date.

Resilience, Recovery, and Performance Gains

Log-based approaches enhance fault tolerance: stale data redefines points without corruption; rogue data exclusion is straightforward. Parallel recovery reduces times (e.g., 30 hours to 4), supporting out-of-order processing.

Low-friction persistence eases backups, indexing, and maintenance. Concurrency spreads read/write loads; rich metadata enables hygiene controls (e.g., preventing base data on processed days).

Standardization reduces variation, improving scalability over CRUD’s custom solutions.

Broader Implications and Observations

This shift yields richer data utility for computations (deltas, reconciliations) without data copying. Retired records are handled via log points, avoiding deletions. While new concepts and tech require learning, benefits include standardized access, tolerance to errors, and high availability via redundancy.

Neil notes applicability beyond MDM to batch-oriented, large-scale state management, though not universal. In conclusion, semantic event sourcing fortifies systems against fragility, promoting scalable, recoverable architectures in data-centric domains.

Links:

• Lecture video
• Goldman Sachs website