Jonathan Lalou's Blog

Posts Tagged ‘Akka’

Ben Fonarov, a Big Data specialist at Capital One, presented a compelling case study at Scala Days New York 2016 on building a large-scale graph analysis engine using Scala, Akka, and HBase. Ben detailed the architecture and implementation of Athena, a distributed time-series graph system designed to deliver integrated, real-time data to enterprise users, addressing the challenges of data overload in a banking environment.

Addressing Enterprise Data Needs

Ben Fonarov opened by outlining the motivation behind Athena: the need to provide integrated, real-time data to users at Capital One. Unlike traditional table-based thinking, Athena represents data as a graph, modeling entities like accounts and transactions to align with business concepts. Ben highlighted the challenges of data overload, with multiple data warehouses and ETL processes generating vast datasets. Athena’s visual interface allows users to define graph schemas, ensuring data is accessible in a format that matches their mental models.

Architectural Considerations

Ben described two architectural approaches to building Athena. The naive implementation used a single actor to process queries, which was insufficient for production-scale loads. The robust solution leveraged an Akka cluster, distributing query processing across nodes for scalability. A query parser translated user requests into graph traversals, while actors managed tasks and streamed results to users. This design ensured low latency and scalability, handling up to 200 billion nodes efficiently.

Streaming and Optimization

A key feature of Athena, Ben explained, was its ability to stream results in real time, avoiding the batch processing limitations of frameworks like TinkerPop’s Gremlin. By using Akka’s actor-based concurrency, Athena processes queries incrementally, delivering results as they are computed. Ben discussed optimizations, such as limiting the number of nodes per actor to prevent bottlenecks, and plans to integrate graph algorithms like PageRank to enhance analytical capabilities.

Future Directions and Community Engagement

Ben concluded by sharing future plans for Athena, including adopting a Gremlin-like DSL for graph traversals and integrating with tools like Spark and H2O. He emphasized the importance of community feedback, inviting developers to join Capital One’s data team to contribute to Athena’s evolution. Running on AWS EC2, Athena represents a scalable solution for enterprise graph analysis, poised to transform how banks handle complex data relationships.

Links:

• Capital One company website

Michael Nash, co-author of Applied Akka Patterns, delivered an insightful exploration of scalability at Scala Days New York 2016, distinguishing it from performance and outlining strategies to achieve near-linear scalability using the Lightbend ecosystem. Michael’s presentation delved into architectural principles, real-world patterns, and tools that enable systems to handle increasing loads without failure.

Scalability vs. Performance

Michael Nash clarified that scalability is the ability to handle greater loads without breaking, distinct from performance, which focuses on processing the same load faster. Using a simple graph, Michael illustrated how performance improvements shift response times downward, while scalability extends the system’s capacity to handle more requests. He cautioned that poorly designed systems hit scalability limits, leading to errors or degraded performance, emphasizing the need for architectures that avoid these bottlenecks.

Avoiding Scalability Pitfalls

Michael identified key enemies of scalability, such as shared databases, synchronous communication, and sequential IDs. He advocated for denormalized, isolated data stores per microservice, using event sourcing and CQRS to decouple systems. For instance, an inventory service can update based on events from a customer service without direct database access, enhancing scalability. Michael also warned against overusing Akka cluster sharding, which introduces overhead, recommending it only when consistency is critical.

Leveraging the Lightbend Ecosystem

The Lightbend ecosystem, including Scala, Akka, and Spark, provides robust tools for scalability, Michael explained. Akka’s actor model supports asynchronous messaging, ideal for distributed systems, while Spark handles large-scale data processing. Tools like Docker, Mesos, and Lightbend’s ConductR streamline deployment and orchestration, enabling rolling upgrades without downtime. Michael emphasized integrating these tools with continuous delivery and deep monitoring to maintain system health under high loads.

Real-World Applications and DevOps

Michael shared case studies from IoT wearables to high-finance systems, highlighting common patterns like event-driven architectures and microservices. He stressed the importance of DevOps in scalable systems, advocating for automated deployment pipelines and monitoring to detect issues early. By embracing failure as inevitable and designing for resilience, systems can scale across data centers, as seen in continent-spanning applications. Michael’s practical advice included starting deployment planning early to avoid scalability bottlenecks.

Links:

• Lightbend company website

At Scala Days New York 2016, Konrad Malawski, a key member of the Akka team at Lightbend, delivered a profound exploration of the principles guiding the effective use of Akka, a toolkit for building concurrent and distributed systems. Konrad’s presentation, inspired by the philosophical lens of “The Tao of Programming,” offered practical insights into designing applications with Akka, emphasizing the shift from synchronous to asynchronous paradigms to achieve robust, scalable architectures.

Embracing the Messaging Paradigm

Konrad Malawski began by underscoring the centrality of messaging in Akka’s actor model. Drawing from Alan Kay’s vision of object-oriented programming, Konrad explained that actors encapsulate state and communicate solely through messages, mirroring real-world computing interactions. This approach fosters loose coupling, both spatially and temporally, allowing components to operate independently. A single actor, Konrad noted, is limited in utility, but when multiple actors collaborate—such as delegating tasks to specialized actors like a “yellow specialist”—powerful patterns like worker pools and sharding emerge. These patterns enable efficient workload distribution, aligning perfectly with the distributed nature of modern systems.

Structuring Actor Systems for Clarity

A common pitfall for newcomers to Akka, Konrad observed, is creating unstructured systems with actors communicating chaotically. To counter this, he advocated for hierarchical actor systems using context.actorOf to spawn child actors, ensuring a clear supervisory structure. This hierarchy not only organizes actors but also enhances fault tolerance through supervision, where parent actors manage failures of their children. Konrad cautioned against actor selection—directly addressing actors by path—as it leads to brittle designs akin to “stealing a TV from a stranger’s house.” Instead, actors should be introduced through proper references, fostering maintainable and predictable interactions.

Balancing Power and Constraints

Konrad emphasized the philosophy of “constraints liberate, liberties constrain,” a principle echoed across Scala conferences. Akka actors, being highly flexible, can perform a wide range of tasks, but this power can overwhelm developers. He contrasted actors with more constrained abstractions like futures, which handle single values, and Akka Streams, which enforce a static data flow. These constraints enable optimizations, such as transparent backpressure in streams, which are harder to implement in the dynamic actor model. However, actors excel in distributed settings, where messaging simplifies scaling across nodes, making Akka a versatile choice for complex systems.

Community and Future Directions

Konrad highlighted the vibrant Akka community, encouraging contributions through platforms like GitHub and Gitter. He noted ongoing developments, such as Akka Typed, an experimental API that enhances type safety in actor interactions. By sharing resources like the Reactive Streams TCK and community-driven initiatives, Konrad underscored Lightbend’s commitment to evolving Akka collaboratively. His call to action was clear: engage with the community, experiment with new features, and contribute to shaping Akka’s future, ensuring it remains a cornerstone of reactive programming.

Links:

• Lightbend company website
• Konrad Malawski on LinkedIn

Reactive applications, built on event-driven and asynchronous foundations, require innovative monitoring strategies. At Scala Days New York 2016, Duncan DeVore and Henrik Engström, both from Lightbend, explored the challenges and solutions for monitoring such systems. They discussed how traditional monitoring falls short for reactive architectures and introduced Lightbend’s approach to addressing these challenges, emphasizing adaptability and precision in observing distributed systems.

The Shift from Traditional Monitoring

Duncan and Henrik began by outlining the limitations of traditional monitoring, which relies on stack traces in synchronous systems to diagnose issues. In reactive applications, built with frameworks like Akka and Play, the asynchronous, message-driven nature disrupts this model. Stack traces lose relevance, as actors communicate without a direct call stack. The speakers categorized monitoring into business process, functional, and technical types, highlighting the need to track metrics like actor counts, message flows, and system performance in distributed environments.

The Impact of Distributed Systems

The rise of the internet and cloud computing has transformed system design, as Duncan explained. Distributed computing, pioneered by initiatives like ARPANET, and the economic advantages of cloud platforms have enabled businesses to scale rapidly. However, this shift introduces complexities, such as network partitions and variable workloads, necessitating new monitoring approaches. Henrik noted that reactive systems, designed for scalability and resilience, require tools that can handle dynamic data flows and provide insights into system behavior without relying on traditional metrics.

Challenges in Monitoring Reactive Systems

Henrik detailed the difficulties of monitoring asynchronous systems, where data flows through push or pull models. In push-based systems, monitoring tools must handle high data volumes, risking overload, while pull-based systems allow selective querying for efficiency. The speakers emphasized anomaly detection over static thresholds, as thresholds are hard to calibrate and may miss nuanced issues. Anomaly detection, exemplified by tools like Prometheus, identifies unusual patterns by correlating metrics, reducing false alerts and enhancing system understanding.

Lightbend’s Monitoring Solution

Duncan and Henrik introduced Lightbend Monitoring, a subscription-based tool tailored for reactive applications. It integrates with Akka actors and Lagom circuit breakers, generating metrics and traces for backends like StatsD and Telegraf. The solution supports pull-based monitoring, allowing selective data collection to manage high data volumes. Future enhancements include support for distributed tracing, Prometheus integration, and improved Lagom compatibility, aiming to provide a comprehensive view of system health and performance.

Links:

• Lightbend company website
• Duncan DeVore on LinkedIn
• Henrik Engström on LinkedIn

Microservices have become a cornerstone of modern software architecture, yet their complexity often poses challenges. At Scala Days New York 2016, Mirco Dotta, a software engineer at Lightbend, introduced Lagom, an open-source framework designed to simplify the creation of reactive microservices. Mirco showcased how Lagom, meaning “just right” in Swedish, balances developer productivity with adherence to reactive principles, offering a seamless experience from development to production.

The Philosophy of Lagom

Mirco emphasized that Lagom prioritizes appropriately sized services over the “micro” aspect of microservices. By focusing on clear boundaries and isolation, Lagom ensures services are neither too small nor overly complex, aligning with the Swedish concept of sufficiency. Built on Play Framework and Akka, Lagom is inherently asynchronous and non-blocking, promoting scalability and resilience. Mirco highlighted its opinionated approach, which standardizes service structures to enhance consistency across teams, allowing developers to focus on domain logic rather than infrastructure.

Development Environment Efficiency

Lagom’s development environment, inspired by Play Framework, is a standout feature. Mirco demonstrated this with a sample application called Cheerer, a Twitter-like service. Using a single SBT command, runAll, developers can launch all services, including an embedded Cassandra server, service locator, and gateway, within one JVM. The environment supports hot reloading, automatically recompiling and restarting services upon code changes. This streamlined setup, consistent across different machines, frees developers from managing complex scripts, enhancing productivity and collaboration.

Service and Persistence APIs

Lagom’s service API is defined through a descriptor method, specifying endpoints and metadata for inter-service communication. Mirco showcased a “Hello World” service, illustrating how services expose endpoints that other services can call, facilitated by the service locator. For persistence, Lagom defaults to Cassandra, leveraging its scalability and resilience, but allows flexibility for other data stores. Mirco advocated for event sourcing and CQRS (Command Query Responsibility Segregation), noting their suitability for microservices. These patterns enable immutable event logs and optimized read views, simplifying data management and scalability.

Production-Ready Features

Transitioning to production is seamless with Lagom, as Mirco demonstrated through its integration with SBT Native Packager, supporting formats like Docker images and RPMs. Lightbend Conductor, available for free in development, simplifies orchestration, offering features like rolling upgrades and circuit breakers for fault tolerance. Mirco highlighted ongoing work to support other orchestration tools like Kubernetes, encouraging community contributions to expand Lagom’s ecosystem. Circuit breakers and monitoring capabilities further ensure service reliability in production environments.

Links:

• Lightbend company website
• Mirco Dotta on LinkedIn

Alexandre Delègue and Mathieu Ancelin, both engineers at SERLI, captivated attendees at Devoxx France 2015 with a deep dive into building reactive applications on a Raspberry Pi cluster. Leveraging their expertise in Java, Java EE, and open-source projects, they demonstrated a microservices-based system using Play, Akka, Cassandra, and Elasticsearch, testing the Reactive Manifesto’s promises on constrained hardware.

Embracing the Reactive Manifesto

Alexandre opened by contrasting monolithic enterprise stacks with the modular, scalable approach of the Reactive Manifesto. He introduced their application, built with microservices and event sourcing, designed to be responsive, resilient, and elastic. Running this on Raspberry Pi’s limited resources tested the architecture’s ability to deliver under constraints, proving its adaptability.

This philosophy, Alexandre noted, prioritizes agility and resilience.

Microservices and Event Sourcing

Mathieu detailed the application’s architecture, using Play for the web framework and Akka for actor-based concurrency. Cassandra handled data persistence, while Elasticsearch enabled fast search capabilities. Event sourcing ensured a reliable audit trail, capturing state changes as events. The duo’s live demo showcased these components interacting seamlessly, even on low-powered Raspberry Pi hardware.

This setup, Mathieu emphasized, ensures robust performance.

Challenges of Clustering on Raspberry Pi

The session highlighted configuration pitfalls encountered during clustering. Alexandre shared how initial deployments overwhelmed the Raspberry Pi’s CPU, causing nodes to disconnect and form sub-clusters. Proper configuration, tested pre-production, resolved these issues, ensuring stable heartbeats across the cluster. Their experience underscored the importance of thorough setup validation.

These lessons, Alexandre noted, are critical for constrained environments.

Alternative Reactive Approaches

Mathieu explored other reactive libraries, such as Spring Boot with reactive Java 8 features and async servlets, demonstrating versatility beyond Akka. Their demo included Gatling for load testing, though an outdated plugin caused challenges, since resolved natively. The session concluded with a nod to the fun of building such systems, encouraging experimentation.

This flexibility, Mathieu concluded, broadens reactive development options.

Links:

• Alexandre Delègue’s Google+ blog
• SERLI company website

Lecturers

Daniel Deogun and Daniel Sawano are senior consultants at Omega Point, a Stockholm-based consultancy with offices in Malmö and New York. Both specialize in building scalable, fault-tolerant systems, with Deogun focusing on distributed architectures and Sawano on integrating modern frameworks like Akka into enterprise environments. Their combined expertise in Java and Scala, along with practical experience in high-stakes projects, positions them as authoritative voices on leveraging Akka for real-world challenges.

Abstract

Akka, a toolkit for building concurrent, distributed, and resilient applications using the actor model, is renowned for its ability to deliver high-performance systems. However, integrating Akka into legacy environments—where entrenched codebases and conservative practices dominate—presents unique challenges. Delivered at Devoxx France 2014, this lecture shares insights from Omega Point’s experience developing an international, government-approved system using Akka in Java, despite Scala’s closer alignment with Akka’s APIs. The speakers explore how domain-specific requirements shaped their design, common pitfalls encountered, and strategies for success in both greenfield and brownfield contexts. Through detailed code examples, performance metrics, and lessons learned, the talk demonstrates Akka’s transformative potential and why Java was a strategic choice for business success. It concludes with practical advice for developers aiming to modernize legacy systems while maintaining reliability and scalability.

The Actor Model: A Foundation for Resilience

Akka’s core strength lies in its implementation of the actor model, a paradigm where lightweight actors encapsulate state and behavior, communicating solely through asynchronous messages. This eliminates shared mutable state, a common source of concurrency bugs in traditional multithreaded systems. Daniel Sawano introduces the concept with a simple Java-based Akka actor:

import akka.actor.UntypedActor;

public class GreetingActor extends UntypedActor {
    @Override
    public void onReceive(Object message) throws Exception {
        if (message instanceof String) {
            System.out.println("Hello, " + message);
            getSender().tell("Greetings received!", getSelf());
        } else {
            unhandled(message);
        }
    }
}

This actor receives a string message, processes it, and responds to the sender. Actors run in an ActorSystem, which manages their lifecycle and threading:

import akka.actor.ActorSystem;
import akka.actor.ActorRef;
import akka.actor.Props;

ActorSystem system = ActorSystem.create("MySystem");
ActorRef greeter = system.actorOf(Props.create(GreetingActor.class), "greeter");
greeter.tell("World", ActorRef.noSender());

This setup ensures isolation and fault tolerance, as actors operate independently and can be supervised to handle failures gracefully.

Designing with Domain Requirements

The project discussed was a government-approved system requiring high throughput, strict auditability, and fault tolerance to meet regulatory standards. Deogun explains that they modeled domain entities as actor hierarchies, with parent actors supervising children to recover from failures. For example, a transaction processing system used actors to represent accounts, with each actor handling a subset of operations, ensuring scalability through message-passing.

The choice of Java over Scala was driven by business needs. While Scala’s concise syntax aligns closely with Akka’s functional style, the team’s familiarity with Java reduced onboarding time and aligned with the organization’s existing skill set. Java’s Akka API, though more verbose, supports all core features, including clustering and persistence. Sawano notes that this decision accelerated adoption in a conservative environment, as developers could leverage existing Java libraries and tools.

Pitfalls and Solutions in Akka Implementations

Implementing Akka in a legacy context revealed several challenges. One common issue was message loss in high-throughput scenarios. To address this, the team implemented acknowledgment protocols, ensuring reliable delivery:

public class ReliableActor extends UntypedActor {
    @Override
    public void onReceive(Object message) throws Exception {
        if (message instanceof String) {
            // Process message
            getSender().tell("ACK", getSelf());
        } else {
            unhandled(message);
        }
    }
}

Deadlocks, another risk, were mitigated by avoiding blocking calls within actors. Instead, asynchronous futures were used for I/O operations:

import scala.concurrent.Future;
import static akka.pattern.Patterns.pipe;

Future<String> result = someAsyncOperation();
pipe(result, context().dispatcher()).to(getSender());

State management in distributed systems posed further challenges. Persistent actors ensured data durability by storing events to a journal:

import akka.persistence.UntypedPersistentActor;

public class PersistentCounter extends UntypedPersistentActor {
    private int count = 0;

    @Override
    public String persistenceId() {
        return "counter-id";
    }

    @Override
    public void onReceiveCommand(Object command) {
        if (command.equals("increment")) {
            persist(1, evt -> count += evt);
        }
    }

    @Override
    public void onReceiveRecover(Object event) {
        if (event instanceof Integer) {
            count += (Integer) event;
        }
    }
}

This approach allowed the system to recover state after crashes, critical for regulatory compliance.

Performance and Scalability Achievements

The system achieved impressive performance, handling 100,000 requests per second with 99.9% uptime. Akka’s location transparency enabled clustering across nodes, distributing workload efficiently. Deogun highlights that actors’ lightweight nature—thousands can run on a single JVM—allowed scaling without heavy resource overhead. Metrics showed consistent latency under 10ms for critical operations, even under peak load.

Integrating Akka with Legacy Systems

Legacy integration required wrapping existing services in actors to isolate faults. For instance, a monolithic database layer was accessed via actors, which managed connection pooling and retry logic. This approach minimized changes to legacy code while introducing Akka’s resilience benefits. Sawano emphasizes that incremental adoption—starting with a single actor-based module—eased the transition.

Lessons Learned and Broader Implications

The project underscored Akka’s versatility in both greenfield and brownfield contexts. Key lessons included the importance of clear message contracts to avoid runtime errors and the need for robust monitoring to track actor performance. Tools like Typesafe Console (now Lightbend Telemetry) provided insights into message throughput and bottlenecks.

For developers, the talk offers a blueprint for modernizing legacy systems: start small, leverage Java for familiarity, and use Akka’s supervision for reliability. For organizations, it highlights the business value of resilience and scalability, particularly in regulated industries.

Conclusion: Akka as a Game-Changer

Deogun and Sawano’s experience demonstrates that Akka can transform legacy environments by providing a robust framework for concurrency and fault tolerance. Choosing Java over Scala proved strategic, aligning with team skills and accelerating delivery. As distributed systems become the norm, Akka’s actor model offers a proven path to scalability, making it a vital tool for modern software engineering.

Links

• Lecture video
• Akka official site
• Omega Point company page
• Lightbend Telemetry

Lecturer

Mathieu Ancelin is a software engineer at SERLI, specializing in Java EE technologies with a particular focus on component frameworks. He contributes to open-source projects such as GlassFish, JOnAS, and leads initiatives like CDI-OSGi and Play CDI. A member of the JSR 346 expert group for CDI 1.1, Ancelin regularly teaches at the University of La Rochelle and Poitiers, and speaks at conferences including JavaOne and Solutions Linux. He is active in the Poitou-Charentes JUG and can be followed on Twitter as @TrevorReznik.

Abstract

Mathieu Ancelin demystifies Play 2’s real-time capabilities, answering the perennial question: “WTF are Iteratees?” Through live demonstrations of two playful applications—a multiplayer spaceship battle and a real-time roulette game—he showcases how Play 2 leverages Iteratees, Akka actors, Server-Sent Events (SSE), WebSockets, HTML5 Canvas, and even webcam input to build responsive, interactive web experiences. The session explores how these APIs integrate seamlessly with Java and Scala, enabling developers to create low-latency, event-driven systems using their preferred language. Beyond the fun, Ancelin analyzes architectural patterns for scalability, backpressure handling, and state management in real-time web applications.

Demystifying Iteratees: Functional Streams for Non-Blocking I/O

Ancelin begins by addressing the confusion surrounding Iteratees, a functional reactive programming abstraction in Play 2. Unlike traditional imperative streams, Iteratees separate data production, processing, and consumption, enabling composable, backpressure-aware pipelines.

val enumeratee: Enumeratee[Array[Byte], String] = Enumeratee.map[Array[Byte]].apply[String] { bytes =>
  new String(bytes, "UTF-8")
}

This allows safe handling of chunked HTTP input without blocking threads. When combined with Enumerators (producers) and Enumeratees (transformers), they form robust data flows:

val socket: WebSocket[JsValue, JsValue] = WebSocket.using[JsValue] { request =>
  val in = Iteratee.foreach[JsValue](msg => actor ! msg).map(_ => actor ! PoisonPill)
  val out = Enumerator.fromCallback1(_ => futurePromise.future)
  (in, out)
}

Ancelin demonstrates how this pattern prevents memory leaks and thread exhaustion under load.

Akka Actors: Coordinating Game State and Player Actions

The spaceship game uses Akka actors to manage shared game state. A central GameActor maintains positions, velocities, and collisions:

class GameActor extends Actor {
  var players = Map.empty[String, Player]
  val ticker = context.system.scheduler.schedule(0.millis, 16.millis, self, Tick)

  def receive = {
    case Join(id, out) => players += (id -> Player(out))
    case Input(id, thrust, rotate) => players(id).update(thrust, rotate)
    case Tick => broadcastState()
  }
}

Each client connects via WebSocket, sending input events and receiving rendered frames. The actor model ensures thread-safe updates and natural distribution.

Real-Time Rendering with Canvas and Webcam Integration

The game renders on HTML5 Canvas using client-side JavaScript. Server pushes state via SSE or WebSocket; client interpolates between ticks for smooth 60 FPS animation.

A bonus feature uses getUserMedia() to capture webcam input, mapping head tilt to ship rotation—an engaging demo of sensor fusion in the browser.

navigator.getUserMedia({ video: true }, stream => {
  video.src = URL.createObjectURL(stream);
  tracker.on('track', event => sendRotation(event.data.angle));
});

Play Roulette: SSE for Unidirectional Live Updates

The second demo, Play Roulette, uses Server-Sent Events to broadcast spin results to all connected clients:

def live = Action {
  Ok.chunked(results via EventSource()).as("text/event-stream")
}

Clients subscribe with:

const es = new EventSource('/live');
es.onmessage = e => updateWheel(JSON.parse(e.data));

This pattern excels for broadcast scenarios—news feeds, dashboards, live sports.

Language Interoperability: Java and Scala Working Together

Ancelin emphasizes Play 2’s dual-language support. Java developers use the same APIs via wrappers:

public static WebSocket<JsonNode> socket() {
    return WebSocket.withActor(GameActor::props);
}

This lowers the barrier for Java teams adopting reactive patterns.

Architecture Analysis: Scalability, Fault Tolerance, and Deployment

The system scales horizontally using Akka clustering. Game instances partition by room; a load balancer routes WebSocket upgrades. Failure recovery leverages supervisor strategies.

Deployment uses Play’s dist task to generate start scripts. For production, Ancelin recommends Typesafe ConductR or Docker with health checks.

Implications for Modern Web Applications

Play 2’s real-time stack enables:
– Low-latency UX without polling
– Efficient resource use via non-blocking I/O
– Graceful degradation under load
– Cross-language development in polyglot teams

From games to trading platforms, the patterns apply broadly.

Links:

• Video of the lecture
• Spaceship Game on GitHub
• Play Roulette on GitHub
• Mathieu Ancelin on Twitter