Jonathan Lalou's Blog

Posts Tagged ‘DistributedSystems’

In a world where systems are becoming increasingly distributed and cloud-native, microservices have emerged as the de facto architecture. But as we scale
microservices horizontally—running multiple instances for each service—one of the biggest challenges becomes inter-service communication.

How do we ensure that our services talk to each other reliably, efficiently, and in a way that’s resilient to failures?

Welcome to the world of Feign and Spring Cloud.

The Challenge: Multi-Instance Microservices

Imagine you have a user-service that needs to talk to an order-service, and your order-service runs 5 instances behind a
service registry like Eureka. Hardcoding URLs? That’s brittle. Manual load balancing? Not scalable.

You need:

• Service discovery to dynamically resolve where to send the request
• Load balancing across instances
• Resilience for timeouts, retries, and fallbacks
• Clean, maintainable code that developers love

The Solution: Feign + Spring Cloud

OpenFeign is a declarative web client. Think of it as a smart HTTP client where you only define interfaces — no more boilerplate REST calls.

When combined with Spring Cloud, Feign becomes a first-class citizen in a dynamic, scalable microservices ecosystem.

✅ Features at a Glance:

• Declarative REST client
• Automatic service discovery (Eureka, Consul)
• Client-side load balancing (Spring Cloud LoadBalancer)
• Integration with Resilience4j for circuit breaking
• Easy integration with Spring Boot config and observability tools

Step-by-Step Setup

1. Add Dependencies

If using Eureka:

       config:

           default:

                connectTimeout:3000

                readTimeout:500

[/yml]

Enable retry:

What Happens Behind the Scenes?

When user-service calls order-service:

1. Spring Cloud uses Eureka to resolve all instances of order-service.
2. Spring Cloud LoadBalancer picks an instance using round-robin (or your chosen strategy).
3. Feign sends the HTTP request to that instance.
4. If it fails, Resilience4j (or your fallback) handles it gracefully.

Observability & Debugging

Use Spring Boot Actuator to expose Feign metrics:

And tools like Spring Cloud Sleuth + Zipkin for distributed tracing across Feign calls.

Beyond the Basics

To go even further:

• Integrate with Spring Cloud Gateway for API routing and external access.
• Use Spring Cloud Config Server to centralize configuration across environments.
• Secure Feign calls with OAuth2 via Spring Security and OpenID Connect.

✨ Final Thoughts

Using Feign with Spring Cloud transforms service-to-service communication from a tedious, error-prone task into a clean, scalable, and cloud-native solution.
Whether you’re scaling services across zones or deploying in Kubernetes, Feign ensures your services communicate intelligently and resiliently.

Danica Fine, a developer advocate at Confluent, took Devoxx Belgium 2024 attendees on a captivating journey through the lifecycle of a Kafka producer’s request. Her talk demystified the complex process of getting data into Apache Kafka, often treated as a black box by developers. Using a Hobbit-themed example, Danica traced a producer.send() call from client to broker and back, detailing configurations and metrics that impact performance and reliability. By breaking down serialization, partitioning, batching, and broker-side processing, she equipped developers with tools to debug issues and optimize workflows, making Kafka less intimidating and more approachable.

Preparing the Journey: Serialization and Partitioning

Danica began with a simple schema for tracking Hobbit whereabouts, stored in a topic with six partitions and a replication factor of three. The first step in producing data is serialization, converting objects into bytes for brokers, controlled by key and value serializers. Misconfigurations here can lead to errors, so monitoring serialization metrics is crucial. Next, partitioning determines which partition receives the data. The default partitioner uses a key’s hash or sticky partitioning for keyless records to distribute data evenly. Configurations like partitioner.class, partitioner.ignore.keys, and partitioner.adaptive.partitioning.enable allow fine-tuning, with adaptive partitioning favoring faster brokers to avoid hot partitions, especially in high-throughput scenarios like financial services.

Batching for Efficiency

To optimize throughput, Kafka groups records into batches before sending them to brokers. Danica explained key configurations: batch.size (default 16KB) sets the maximum batch size, while linger.ms (default 0) controls how long to wait to fill a batch. Setting linger.ms above zero introduces latency but reduces broker load by sending fewer requests. buffer.memory (default 32MB) allocates space for batches, and misconfigurations can cause memory issues. Metrics like batch-size-avg, records-per-request-avg, and buffer-available-bytes help monitor batching efficiency, ensuring optimal throughput without overwhelming the client.

Sending the Request: Configurations and Metrics

Once batched, data is sent via a produce request over TCP, with configurations like max.request.size (default 1MB) limiting batch volume and acks determining how many replicas must acknowledge the write. Setting acks to “all” ensures high durability but increases latency, while acks=1 or 0 prioritizes speed. enable.idempotence and transactional.id prevent duplicates, with transactions ensuring consistency across sessions. Metrics like request-rate, requests-in-flight, and request-latency-avg provide visibility into request performance, helping developers identify bottlenecks or overloaded brokers.

Broker-Side Processing: From Socket to Disk

On the broker, requests enter the socket receive buffer, then are processed by network threads (default 3) and added to the request queue. IO threads (default 8) validate data with a cyclic redundancy check and write it to the page cache, later flushing to disk. Configurations like num.network.threads, num.io.threads, and queued.max.requests control thread and queue sizes, with metrics like network-processor-avg-idle-percent and request-handler-avg-idle-percent indicating thread utilization. Data is stored in a commit log with log, index, and snapshot files, supporting efficient retrieval and idempotency. The log.flush.rate and local-time-ms metrics ensure durable storage.

Replication and Response: Completing the Journey

Unfinished requests await replication in a “purgatory” data structure, with follower brokers fetching updates every 500ms (often faster). The remote-time-ms metric tracks replication duration, critical for acks=all. Once replicated, the broker builds a response, handled by network threads and queued in the response queue. Metrics like response-queue-time-ms and total-time-ms measure the full request lifecycle. Danica emphasized that understanding these stages empowers developers to collaborate with operators, tweaking configurations like default.replication.factor or topic-level settings to optimize performance.

Empowering Developers with Kafka Knowledge

Danica concluded by encouraging developers to move beyond treating Kafka as a black box. By mastering configurations and monitoring metrics, they can proactively address issues, from serialization errors to replication delays. Her talk highlighted resources like Confluent Developer for guides and courses on Kafka internals. This knowledge not only simplifies debugging but also fosters better collaboration with operators, ensuring robust, efficient data pipelines.

Links:

• Confluent Developer
• Danica Fine’s Linktree
• Apache Kafka Documentation

At Spring I/O 2022 in Barcelona, Matthias Haeussler delivered an insightful session exploring distributed systems patterns, comparing Spring Cloud, Kubernetes, service meshes, and the emerging eBPF technology. As a consultant at Novatec and a university lecturer, Matthias combined theoretical clarity with a live demo to illustrate how these technologies address challenges like service discovery, routing, and resilience in distributed architectures. His talk offered a practical guide for developers navigating modern microservice ecosystems.

Why Distributed Systems? Understanding the Motivation

Matthias began by addressing the rationale behind distributed systems, emphasizing their ability to enhance client experiences—whether for human users or other applications. By breaking systems into smaller components, developers can execute tasks in parallel, manage heterogeneous environments, and ensure scalability. For instance, running multiple Java versions (e.g., Java 11 and 17) in a single application server is impractical, but distributed systems allow such flexibility. Matthias also highlighted resilience benefits, such as load balancing, traffic throttling, and blue-green deployments, which minimize downtime and maintain system health under varying loads. Security, including authentication and authorization, further underscores the need for distributed architectures to protect and scale services effectively.

However, these benefits come with challenges. Distributed systems require robust mechanisms for service discovery, traffic management, and observability. Matthias framed his talk around comparing how Spring Cloud, Kubernetes, service meshes, and eBPF tackle these requirements, providing a roadmap for choosing the right tool for specific scenarios.

Spring Cloud and Kubernetes: Framework vs. Orchestration

Spring Cloud, dubbed the “classic” approach, integrates distributed system features directly into application code. Matthias outlined key components like Eureka (service registry), Spring Cloud Gateway (routing), and Resilience4j (circuit breaking), which rely on dependencies, annotations, and configuration properties. This in-process approach makes Spring Cloud independent of the runtime environment, allowing deployment on single machines, containers, or clouds without modification. However, changes to dependencies or code require rebuilding, which can slow iterations.

In contrast, Kubernetes offers native orchestration for distributed systems, with its own service registry (DNS), load balancing (via Kubernetes Services), and configuration (ConfigMaps/Secrets). Matthias explained how Spring Cloud Kubernetes bridges these worlds, enabling Spring applications to use Kubernetes’ registry without code changes. For example, annotating with @EnableDiscoveryClient queries Kubernetes’ DNS instead of Eureka. While Kubernetes excels at scaling and deployment, it lacks advanced traffic control (e.g., circuit breaking), where Spring Cloud shines. Matthias suggested combining both for a balanced approach, leveraging Kubernetes’ orchestration and Spring Cloud’s resilience patterns.

Service Meshes: Network-Level Control

Service meshes, such as Istio, introduce a new paradigm by injecting proxy sidecars into Kubernetes pods. Matthias described how these proxies handle network traffic—routing, encryption, and throttling—without altering application code. This separation of concerns allows developers to manage traffic policies (e.g., mutual TLS, percentage-based routing) via YAML configurations, offering granular control unavailable in base Kubernetes. A live demo showcased Istio’s traffic distribution for a Spring Pet Clinic application, visualizing load balancing between service versions.

However, service meshes add overhead. Each pod’s proxy increases latency and memory usage, and managing configurations across scaled deployments can become complex—hence the term “service mess.” Matthias cautioned against adopting service meshes unless their advanced features, like fault injection or network policies, are necessary, especially for simpler Spring Cloud Gateway setups.

eBPF: A Lightweight Future for Service Meshes

The talk’s final segment introduced eBPF (extended Berkeley Packet Filter), a Linux kernel technology enabling low-level network event processing. Unlike service meshes, eBPF injects proxies at the node level, reducing overhead compared to per-pod sidecars. Matthias likened eBPF to JavaScript for HTML, embedding sandboxed code in the kernel to monitor and manipulate traffic. Tools like Cilium leverage eBPF for Kubernetes, offering observability, encryption, and routing with minimal latency.

In his demo, Matthias contrasted Istio and Cilium, showing Cilium’s Hubble UI visualizing traffic for the Spring Pet Clinic. Though still nascent, eBPF promises a sidecar-less service mesh, simplifying deployment and reducing resource demands. Matthias noted its youth, with features like encryption still in beta, but predicted growing adoption as tools mature.

Conclusion: Choosing the Right Approach

Matthias concluded without a definitive recommendation, urging developers to assess their needs. Spring Cloud offers simplicity and runtime independence, ideal for smaller setups. Kubernetes and service meshes suit complex, containerized environments, while eBPF represents a lightweight, future-proof option. His talk underscored the importance of aligning technology choices with project requirements, leaving attendees equipped to evaluate these patterns in their own systems.

Links:

• Slides on SpeakerDeck
• Matthias Haeussler on LinkedIn
• Novatec Consulting website

Lætitia Avrot, a PostgreSQL contributor and database consultant at EnterpriseDB, delivered a compelling presentation at Forum PHP 2021, demystifying the CAP theorem and its implications for distributed systems. With a nod to Ireland’s mythical unicorns, Lætitia used humor and technical expertise to explore the trade-offs between consistency, availability, and partition tolerance. Her talk provided practical guidance for designing resilient database architectures. This post covers four key themes: understanding the CAP theorem, practical database design, managing latency, and realistic expectations.

Understanding the CAP Theorem

Lætitia Avrot opened with a clear explanation of the CAP theorem, which states that a distributed system can only guarantee two of three properties: consistency, availability, and partition tolerance. She emphasized that chasing a “unicorn” system achieving all three is futile. Drawing on her work with PostgreSQL, Lætitia illustrated how the theorem shapes database design, using real-world scenarios to highlight the trade-offs developers must navigate in distributed environments.

Practical Database Design

Focusing on practical applications, Lætitia outlined strategies for designing PostgreSQL-based systems. She described architectures using logical replication, connection pooling with HAProxy, and standby nodes to balance consistency and availability. By tailoring designs to acceptable data loss and downtime thresholds, developers can create robust systems without overengineering. Lætitia’s approach, informed by her experience at EnterpriseDB, ensures that solutions align with business needs rather than pursuing unattainable perfection.

Managing Latency

Addressing audience questions, Lætitia tackled the challenge of latency in distributed systems. She explained that latency is primarily network-driven, not hardware-dependent, and achieving sub-100ms latency between nodes is difficult. By measuring acceptable latency thresholds and using tools like logical replication, developers can optimize performance. Lætitia’s insights underscored the importance of realistic metrics, reminding attendees that most organizations don’t need Google-scale infrastructure.

Realistic Expectations

Concluding her talk, Lætitia urged developers to set pragmatic goals, quoting her colleague: “Unicorns are more mythical than the battle of China.” She emphasized that robust systems require backups, testing, and clear definitions of acceptable data loss and downtime. By avoiding overcomplexity and focusing on practical trade-offs, developers can build reliable architectures that meet real-world demands, leveraging PostgreSQL’s strengths for scalable, resilient solutions.

Links:

• EnterpriseDB website
• PostgreSQL website
• Lætitia’s blog

Lecturer

Andrii Rodionov, a Ph.D. in computer science, is an associate professor at National Technical University and a software engineer at Wix. He leads JUG UA, organizes JavaDay UA, and co-organizes Kyiv Kotlin events. Relevant links: Wix Engineering Blog (publications); LinkedIn Profile (professional page).

Abstract

This article analyzes Andrii Rodionov’s implementation of the Raft consensus protocol using Kotlin coroutines and Ktor. Set in distributed systems, it examines leader election, log replication, and fault tolerance. The analysis highlights innovations in asynchronous communication, with implications for scalable, fault-tolerant key-value stores.

Introduction and Context

Andrii Rodionov presented at KotlinConf 2018 on implementing Raft, a consensus protocol used in systems like Docker Swarm. Distributed systems face consensus challenges; Raft ensures agreement via leader election and log replication. Rodionov’s in-memory key-value store demo leveraged Kotlin’s coroutines and Ktor for lightweight networking, set against the need for robust, asynchronous distributed architectures.

Methodological Approaches to Raft Implementation

Rodionov used coroutines for non-blocking node communication, with async for leader election and channel for log replication. Ktor handled HTTP-based node interactions, replacing heavier JavaNet. The demo showcased a cluster tolerating node failures: Servers transition from follower to candidate to leader, propagating logs via POST requests. Timeouts triggered elections, ensuring fault tolerance.

Analysis of Innovations and Features

Coroutines innovate Raft’s asynchronous tasks, simplifying state machines compared to Java’s thread-heavy approaches. Ktor’s fast startup and lightweight routing outperform JavaNet, enabling efficient cluster communication. The demo’s fault tolerance—handling node crashes—demonstrates robustness. Limitations include coroutine complexity for novices and Ktor’s relative immaturity versus established frameworks.

Implications and Consequences

Rodionov’s implementation implies easier development of distributed systems, with coroutines reducing concurrency boilerplate. Ktor’s efficiency suits production clusters. Consequences include broader Kotlin adoption in systems like Consul, though mastering coroutines requires investment. The demo’s open-source nature invites community enhancements.

Conclusion

Rodionov’s Raft implementation showcases Kotlin’s strengths in distributed systems, offering a scalable, fault-tolerant model for modern consensus-driven applications.

Links

• Lecture video: https://www.youtube.com/watch?v=pNFmreSEXic
• Lecturer’s X/Twitter: @andriirodionov
• Lecturer’s LinkedIn: Andrii Rodionov
• Organization’s X/Twitter: @WixEng
• Organization’s LinkedIn: Wix

At Scala Days New York 2016, Vladimir Kostyukov, a member of Twitter’s Finagle team, provided an in-depth exploration of Finagle, a scalable and extensible RPC system written in Scala. Vladimir elucidated how Finagle simplifies the complexities of distributed systems, offering a functional programming model that enhances developer productivity while managing intricate backend operations like load balancing and circuit breaking.

The Essence of Finagle

Vladimir Kostyukov introduced Finagle as a robust RPC system used extensively at Twitter and other organizations. Unlike traditional application frameworks, Finagle focuses on facilitating communication between services, abstracting complexities such as connection pooling and load balancing. Vladimir highlighted its protocol-agnostic nature, supporting protocols like HTTP/2 and Twitter’s custom Mux, which enables efficient multiplexing. This flexibility allows developers to build services in Scala or Java, seamlessly integrating Finagle into diverse tech stacks.

Client-Server Dynamics

Delving into Finagle’s internals, Vladimir described its client-server model, where services are treated as composable functions. When a client sends a request, Finagle’s stack—comprising modules for connection pooling, load balancing, and failure handling—processes it efficiently. On the server side, incoming requests are routed through similar modules, ensuring resilience and scalability. Vladimir emphasized the “power of two choices” load balancing algorithm, which selects the least-loaded node from two randomly chosen servers, achieving near-optimal load distribution in constant time.

Advanced Features and Ecosystem

Vladimir showcased Finagle’s advanced capabilities, such as streaming support for large payloads and integration with tools like Zipkin for tracing and Twitter Server for monitoring. Libraries like Finatra and Featherbed further enhance Finagle’s utility, enabling developers to build RESTful APIs and type-safe HTTP clients. These features make Finagle a powerful choice for handling high-throughput systems, as demonstrated by its widespread adoption at Twitter for managing massive data flows.

Community and Future Prospects

Encouraging community engagement, Vladimir invited developers to contribute to Finagle’s open-source repository on GitHub. He discussed ongoing efforts to support HTTP/2 and improve performance, underscoring Finagle’s evolution toward a utopian RPC system. By offering a welcoming environment for pull requests and feedback, Vladimir emphasized the collaborative spirit driving Finagle’s development, ensuring it remains a cornerstone of scalable service architectures.

Links:

• Twitter company website
• Finagle GitHub repository

At Scala Days New York 2016, Konrad Malawski, a key member of the Akka team at Lightbend, delivered a profound exploration of the principles guiding the effective use of Akka, a toolkit for building concurrent and distributed systems. Konrad’s presentation, inspired by the philosophical lens of “The Tao of Programming,” offered practical insights into designing applications with Akka, emphasizing the shift from synchronous to asynchronous paradigms to achieve robust, scalable architectures.

Embracing the Messaging Paradigm

Konrad Malawski began by underscoring the centrality of messaging in Akka’s actor model. Drawing from Alan Kay’s vision of object-oriented programming, Konrad explained that actors encapsulate state and communicate solely through messages, mirroring real-world computing interactions. This approach fosters loose coupling, both spatially and temporally, allowing components to operate independently. A single actor, Konrad noted, is limited in utility, but when multiple actors collaborate—such as delegating tasks to specialized actors like a “yellow specialist”—powerful patterns like worker pools and sharding emerge. These patterns enable efficient workload distribution, aligning perfectly with the distributed nature of modern systems.

Structuring Actor Systems for Clarity

A common pitfall for newcomers to Akka, Konrad observed, is creating unstructured systems with actors communicating chaotically. To counter this, he advocated for hierarchical actor systems using context.actorOf to spawn child actors, ensuring a clear supervisory structure. This hierarchy not only organizes actors but also enhances fault tolerance through supervision, where parent actors manage failures of their children. Konrad cautioned against actor selection—directly addressing actors by path—as it leads to brittle designs akin to “stealing a TV from a stranger’s house.” Instead, actors should be introduced through proper references, fostering maintainable and predictable interactions.

Balancing Power and Constraints

Konrad emphasized the philosophy of “constraints liberate, liberties constrain,” a principle echoed across Scala conferences. Akka actors, being highly flexible, can perform a wide range of tasks, but this power can overwhelm developers. He contrasted actors with more constrained abstractions like futures, which handle single values, and Akka Streams, which enforce a static data flow. These constraints enable optimizations, such as transparent backpressure in streams, which are harder to implement in the dynamic actor model. However, actors excel in distributed settings, where messaging simplifies scaling across nodes, making Akka a versatile choice for complex systems.

Community and Future Directions

Konrad highlighted the vibrant Akka community, encouraging contributions through platforms like GitHub and Gitter. He noted ongoing developments, such as Akka Typed, an experimental API that enhances type safety in actor interactions. By sharing resources like the Reactive Streams TCK and community-driven initiatives, Konrad underscored Lightbend’s commitment to evolving Akka collaboratively. His call to action was clear: engage with the community, experiment with new features, and contribute to shaping Akka’s future, ensuring it remains a cornerstone of reactive programming.

Links:

• Lightbend company website
• Konrad Malawski on LinkedIn

Jonathan Winandy and Alexis Guéganno, co-founder and operations director at Valwin, respectively, presented a deep dive into advanced Apache Kafka streaming techniques at Devoxx France 2015. With expertise in distributed systems and data warehousing, they explored how Kafka enables flexible, high-performance real-time streaming beyond basic JSON payloads.

Foundations of Streaming

Jonathan opened with a concise overview of streaming, emphasizing Kafka’s role in real-time distributed systems. He explained how Kafka’s topic-based architecture supports high-throughput data pipelines. Their session moved beyond introductory concepts, focusing on advanced writing, modeling, and querying techniques to ensure robust, future-proof streaming solutions.

This foundation, Jonathan noted, sets the stage for scalability.

Advanced Modeling and Querying

Alexis detailed Kafka’s ability to handle structured data, moving past schemaless JSON. They showcased techniques for defining schemas and optimizing queries, improving performance and maintainability. Q&A revealed their use of a five-node cluster for fault tolerance, sufficient for basic journaling but scalable to hundreds for larger workloads.

These methods, Alexis highlighted, enhance data reliability.

Managing Kafka Clusters

Jonathan addressed cluster management, noting that five nodes ensure fault tolerance, while larger clusters handle extensive partitioning. They discussed load balancing and lag management, critical for high-volume environments. The session also covered Kafka’s integration with databases, enabling real-time data synchronization.

This scalability, Jonathan concluded, supports diverse use cases.

Community Engagement and Resources

The duo encouraged engagement through Scala.IO, where Jonathan organizes, and shared Valwin’s expertise in data solutions. Their insights into cluster sizing and health monitoring, particularly in regulated sectors like healthcare, underscored Kafka’s versatility.

This session equips developers for advanced streaming challenges.

Links:

• Valwin company website

Lecturers

Daniel Deogun and Daniel Sawano are senior consultants at Omega Point, a Stockholm-based consultancy with offices in Malmö and New York. Both specialize in building scalable, fault-tolerant systems, with Deogun focusing on distributed architectures and Sawano on integrating modern frameworks like Akka into enterprise environments. Their combined expertise in Java and Scala, along with practical experience in high-stakes projects, positions them as authoritative voices on leveraging Akka for real-world challenges.

Abstract

Akka, a toolkit for building concurrent, distributed, and resilient applications using the actor model, is renowned for its ability to deliver high-performance systems. However, integrating Akka into legacy environments—where entrenched codebases and conservative practices dominate—presents unique challenges. Delivered at Devoxx France 2014, this lecture shares insights from Omega Point’s experience developing an international, government-approved system using Akka in Java, despite Scala’s closer alignment with Akka’s APIs. The speakers explore how domain-specific requirements shaped their design, common pitfalls encountered, and strategies for success in both greenfield and brownfield contexts. Through detailed code examples, performance metrics, and lessons learned, the talk demonstrates Akka’s transformative potential and why Java was a strategic choice for business success. It concludes with practical advice for developers aiming to modernize legacy systems while maintaining reliability and scalability.

The Actor Model: A Foundation for Resilience

Akka’s core strength lies in its implementation of the actor model, a paradigm where lightweight actors encapsulate state and behavior, communicating solely through asynchronous messages. This eliminates shared mutable state, a common source of concurrency bugs in traditional multithreaded systems. Daniel Sawano introduces the concept with a simple Java-based Akka actor:

import akka.actor.UntypedActor;

public class GreetingActor extends UntypedActor {
    @Override
    public void onReceive(Object message) throws Exception {
        if (message instanceof String) {
            System.out.println("Hello, " + message);
            getSender().tell("Greetings received!", getSelf());
        } else {
            unhandled(message);
        }
    }
}

This actor receives a string message, processes it, and responds to the sender. Actors run in an ActorSystem, which manages their lifecycle and threading:

import akka.actor.ActorSystem;
import akka.actor.ActorRef;
import akka.actor.Props;

ActorSystem system = ActorSystem.create("MySystem");
ActorRef greeter = system.actorOf(Props.create(GreetingActor.class), "greeter");
greeter.tell("World", ActorRef.noSender());

This setup ensures isolation and fault tolerance, as actors operate independently and can be supervised to handle failures gracefully.

Designing with Domain Requirements

The project discussed was a government-approved system requiring high throughput, strict auditability, and fault tolerance to meet regulatory standards. Deogun explains that they modeled domain entities as actor hierarchies, with parent actors supervising children to recover from failures. For example, a transaction processing system used actors to represent accounts, with each actor handling a subset of operations, ensuring scalability through message-passing.

The choice of Java over Scala was driven by business needs. While Scala’s concise syntax aligns closely with Akka’s functional style, the team’s familiarity with Java reduced onboarding time and aligned with the organization’s existing skill set. Java’s Akka API, though more verbose, supports all core features, including clustering and persistence. Sawano notes that this decision accelerated adoption in a conservative environment, as developers could leverage existing Java libraries and tools.

Pitfalls and Solutions in Akka Implementations

Implementing Akka in a legacy context revealed several challenges. One common issue was message loss in high-throughput scenarios. To address this, the team implemented acknowledgment protocols, ensuring reliable delivery:

public class ReliableActor extends UntypedActor {
    @Override
    public void onReceive(Object message) throws Exception {
        if (message instanceof String) {
            // Process message
            getSender().tell("ACK", getSelf());
        } else {
            unhandled(message);
        }
    }
}

Deadlocks, another risk, were mitigated by avoiding blocking calls within actors. Instead, asynchronous futures were used for I/O operations:

import scala.concurrent.Future;
import static akka.pattern.Patterns.pipe;

Future<String> result = someAsyncOperation();
pipe(result, context().dispatcher()).to(getSender());

State management in distributed systems posed further challenges. Persistent actors ensured data durability by storing events to a journal:

import akka.persistence.UntypedPersistentActor;

public class PersistentCounter extends UntypedPersistentActor {
    private int count = 0;

    @Override
    public String persistenceId() {
        return "counter-id";
    }

    @Override
    public void onReceiveCommand(Object command) {
        if (command.equals("increment")) {
            persist(1, evt -> count += evt);
        }
    }

    @Override
    public void onReceiveRecover(Object event) {
        if (event instanceof Integer) {
            count += (Integer) event;
        }
    }
}

This approach allowed the system to recover state after crashes, critical for regulatory compliance.

Performance and Scalability Achievements

The system achieved impressive performance, handling 100,000 requests per second with 99.9% uptime. Akka’s location transparency enabled clustering across nodes, distributing workload efficiently. Deogun highlights that actors’ lightweight nature—thousands can run on a single JVM—allowed scaling without heavy resource overhead. Metrics showed consistent latency under 10ms for critical operations, even under peak load.

Integrating Akka with Legacy Systems

Legacy integration required wrapping existing services in actors to isolate faults. For instance, a monolithic database layer was accessed via actors, which managed connection pooling and retry logic. This approach minimized changes to legacy code while introducing Akka’s resilience benefits. Sawano emphasizes that incremental adoption—starting with a single actor-based module—eased the transition.

Lessons Learned and Broader Implications

The project underscored Akka’s versatility in both greenfield and brownfield contexts. Key lessons included the importance of clear message contracts to avoid runtime errors and the need for robust monitoring to track actor performance. Tools like Typesafe Console (now Lightbend Telemetry) provided insights into message throughput and bottlenecks.

For developers, the talk offers a blueprint for modernizing legacy systems: start small, leverage Java for familiarity, and use Akka’s supervision for reliability. For organizations, it highlights the business value of resilience and scalability, particularly in regulated industries.

Conclusion: Akka as a Game-Changer

Deogun and Sawano’s experience demonstrates that Akka can transform legacy environments by providing a robust framework for concurrency and fault tolerance. Choosing Java over Scala proved strategic, aligning with team skills and accelerating delivery. As distributed systems become the norm, Akka’s actor model offers a proven path to scalability, making it a vital tool for modern software engineering.

Links

• Lecture video
• Akka official site
• Omega Point company page
• Lightbend Telemetry

Lecturer

Jérémie Chassaing is an architect at Siriona, focusing on scalable systems for hotel channel management. Author of thinkbeforecoding.com, a blog on Domain-Driven Design, CQRS, and Event Sourcing, he founded Hypnotizer (1999) for interactive video and BBCG (2004) for P2P photo sharing. His work emphasizes time-centric modeling in complex domains.

Abstract

Jérémie Chassaing posits time as central to business logic, advocating Event Sourcing to capture temporal dynamics in Domain-Driven Design. He integrates Distributed DDD, CQRS, and Event Sourcing to tackle scalability, concurrency, and complexity. Through examples like order management, Chassaing analyzes event streams over relational models, demonstrating eventual consistency and projection patterns. The first part establishes foundational shifts from CRUD to event-driven architectures, setting the stage for distributed implementations.

Time’s Primacy in Business Domains

Chassaing asserts time underpins business: reacting to events, analyzing history, forecasting futures. Traditional CRUD ignores temporality, leading to lost context. Event Sourcing records immutable facts—e.g., OrderPlaced, ItemAdded—enabling full reconstruction.

This contrasts relational databases’ mutable state, where updates erase history. Events form audit logs, facilitating debugging and compliance.

Domain-Driven Design Foundations: Aggregates and Bounded Contexts

DDD models domains via aggregates—consistent units like Order with line items. Bounded contexts delimit scopes, preventing model pollution.

Distributed DDD extends this to microservices, each owning a context. CQRS separates commands (writes) from queries (reads), enabling independent scaling.

CQRS Mechanics: Commands, Events, and Projections

Commands mutate state, emitting events. Handlers project events to read models:

case class OrderPlaced(orderId: UUID, customer: String)
case class ItemAdded(orderId: UUID, item: String, qty: Int)

// Command handler
def handle(command: AddItem): Unit = {
  // Validate
  emit(ItemAdded(command.orderId, command.item, command.qty))
}

// Projection
def project(event: ItemAdded): Unit = {
  updateReadModel(event)
}

Projections denormalize for query efficiency, accepting eventual consistency.

Event Sourcing Advantages: Auditability and Scalability

Events form immutable logs, replayable for state recovery or new projections. This decouples reads/writes, allowing specialized stores—SQL for reporting, NoSQL for search.

Chassaing addresses concurrency via optimistic locking on aggregate versions. Distributed events use pub/sub (Kafka) for loose coupling.

Challenges and Patterns: Idempotency and Saga Management

Duplicates require idempotent handlers—e.g., check event IDs. Sagas coordinate cross-aggregate workflows, reacting to events and issuing commands.

Chassaing warns of “lasagna architectures”—layered complexity—and advocates event-driven simplicity over tiered monoliths.

Implications for Resilient Systems: Embracing Eventual Consistency

Event Sourcing yields antifragile designs: failures replay from logs. Distributed CQRS scales horizontally, handling “winter is coming” loads.

Chassaing urges rethinking time in models, shifting from mutable entities to immutable facts.

Links:

• Video of the lecture
• Jérémie Chassaing on LinkedIn