Jonathan Lalou's Blog

Posts Tagged ‘Java’

Lors de Devoxx France 2022, Brice Dutheil a présenté une conférence de 28 minutes sur le Projet Panama, une initiative visant à simplifier l’appel de fonctions natives depuis Java sans les complexités de JNI ou de bibliothèques tierces. Brice, contributeur actif à l’écosystème Java, a introduit l’API Foreign Function & Memory (JEP-419), montrant comment elle relie le monde géré de Java au code natif en C, Swift ou Rust. À travers des démonstrations de codage en direct, Brice a illustré le potentiel de Panama pour des intégrations natives fluides. Suivez Brice sur Twitter à twitter.com/Brice_Dutheil pour plus d’insights Java.

Simplifier l’intégration de code natif

Brice a débuté en expliquant la mission du Projet Panama : connecter l’environnement géré de Java, avec son garbage collector, au monde natif de C, Swift ou Rust, plus proche de la machine. Traditionnellement, JNI imposait des étapes laborieuses : écrire des classes wrapper, charger des bibliothèques et générer des headers lors des builds. Ces processus étaient sujets aux erreurs et chronophages. Des alternatives comme JNA et JNR amélioraient l’expérience développeur en générant des bindings au runtime, mais elles étaient plus lentes et moins sécurisées.

Lancé en 2014, le Projet Panama répond à ces défis avec trois composantes : les API vectorielles (non couvertes ici), les appels de fonctions étrangères et la gestion de la mémoire. Brice s’est concentré sur l’API Foreign Function & Memory (JEP-419), disponible en incubation dans JDK 18. Contrairement à JNI, Panama élimine les complexités du build et offre des performances proches du natif sur toutes les plateformes. Il introduit un modèle de sécurité robuste, limitant les opérations dangereuses et envisageant de restreindre JNI dans les futures versions de Java (par exemple, Java 25 pourrait exiger un flag pour activer JNI). Brice a souligné l’utilisation des method handles et des instructions d’invocation dynamique, inspirées des avancées du bytecode JVM, pour générer efficacement des instructions assembleur pour les appels natifs.

Démonstrations pratiques avec Panama

Brice a démontré les capacités de Panama via du codage en direct, commençant par un exemple simple appelant la fonction getpid de la bibliothèque standard C. À l’aide du SystemLinker, il a effectué une recherche de symbole pour localiser getpid, créé un method handle avec un descripteur de fonction définissant la signature (retournant un long Java), et l’a invoqué pour récupérer l’ID du processus. Ce processus a contourné les lourdeurs de JNI, nécessitant seulement quelques lignes de code Java. Brice a insisté sur l’activation de l’accès natif avec le flag –enable-native-access dans JDK 18, renforçant le modèle de sécurité de Panama en limitant l’accès à des modules spécifiques.

Il a ensuite présenté un exemple plus complexe avec la fonction crypto_box de la bibliothèque cryptographique Libsodium, portable sur des plateformes comme Android. Brice a alloué des segments de mémoire avec un ResourceScope et un NativeAllocator, garantissant la sécurité mémoire en libérant automatiquement les ressources après usage, contrairement à JNI qui dépend du garbage collector. Le ResourceScope prévient les fuites mémoire, une amélioration significative par rapport aux buffers natifs traditionnels. Brice a également abordé l’appel de code Swift via des interfaces compatibles C, démontrant la polyvalence de Panama.

Outils et potentiel futur

Brice a introduit jextract, un outil de Panama qui génère des mappings Java à partir de headers C/C++, simplifiant l’intégration de bibliothèques comme Blake3, une fonction de hachage performante écrite en Rust. Dans une démo, il a montré comment jextract créait des bindings compatibles Panama pour les structures de données et fonctions de Blake3, permettant aux développeurs Java de tirer parti des performances natives sans bindings manuels. Malgré quelques accrocs, la démo a souligné le potentiel de Panama pour des intégrations natives transparentes.

Brice a conclu en soulignant les avantages de Panama : simplicité, rapidité, compatibilité multiplateforme et sécurité mémoire renforcée. Il a noté son évolution continue, avec JEP-419 en incubation dans JDK 18 et une deuxième preview prévue pour JDK 19. Pour les développeurs d’applications desktop ou de systèmes critiques, Panama offre une solution puissante pour exploiter des fonctions spécifiques aux OS ou des bibliothèques optimisées comme Libsodium. Brice a encouragé le public à expérimenter Panama et à poser des questions, renforçant son engagement via Twitter.

The practice of running Java applications within Docker containers has become widely adopted in modern software deployment, yet it is not devoid of potential challenges, particularly when transitioning to production environments. Charles Sabourdin, a freelance architect, and Jean-Christophe Sirot, an engineer at Docker, collaborated at DevoxxFR2018 to share their valuable experiences and disseminate best practices for optimizing Java applications inside Docker containers. Their insightful talk directly addressed common and often frustrating issues, such as containers crashing unexpectedly, applications consuming excessive RAM leading to node instability, and encountering CPU throttling. They offered practical solutions and configurations aimed at ensuring smoother and more reliable production deployments for Java workloads.

Navigating Common Pitfalls: Why Operations Teams May Approach Java Containers with Caution

The presenters initiated their session with a touch of humor, explaining why operations teams might exhibit a degree of apprehension when tasked with deploying a containerized Java application into a production setting. It’s a common scenario: containers that perform flawlessly on a developer’s local machine can begin to behave erratically or fail outright in production. This discrepancy often stems from a fundamental misunderstanding of how the Java Virtual Machine (JVM) interacts with the resource limits imposed by the container’s control groups (cgroups). Several key problems frequently surface in this context. Perhaps the most common is memory mismanagement; the JVM, particularly older versions, might not be inherently aware of the memory limits defined for its container by the cgroup. This lack of awareness can lead the JVM to attempt to allocate and use more memory than has been allocated to the container by the orchestrator or runtime. Such overconsumption inevitably results in the container being abruptly terminated by the operating system’s Out-Of-Memory (OOM) killer, a situation that can be difficult to diagnose without understanding this interaction.

Similarly, CPU resource allocation can present challenges. The JVM might not accurately perceive the CPU resources available to it within the container, such as CPU shares or quotas defined by cgroups. This can lead to suboptimal decisions in sizing internal thread pools (like the common ForkJoinPool or garbage collection threads) or can cause the application to experience unexpected CPU throttling, impacting performance. Another frequent issue is Docker image bloat. Overly large Docker images not only increase deployment times across the infrastructure but also expand the potential attack surface by including unnecessary libraries or tools, thereby posing security vulnerabilities. The talk aimed to equip developers and operations personnel with the knowledge to anticipate and mitigate these common pitfalls. During the presentation, a demonstration application, humorously named “ressources-munger,” was used to simulate these problems, clearly showing how an application could consume excessive memory leading to an OOM kill by Docker, or how it might trigger excessive swapping if not configured correctly, severely degrading performance.

JVM Memory Management and CPU Considerations within Containers

A significant portion of the discussion was dedicated to the intricacies of JVM memory management within the containerized environment. Charles and Jean-Christophe elaborated that older JVM versions, specifically those prior to Java 8 update 131 and Java 9, were not inherently “cgroup-aware”. This lack of awareness meant that the JVM’s default heap sizing heuristics—for example, typically allocating up to one-quarter of the physical host’s memory for the heap—would be based on the total resources of the host machine rather than the specific limits imposed on the container by its cgroup. This behavior is a primary contributor to unexpected OOM kills when the container’s actual memory limit is much lower than what the JVM assumes based on the host.

Several best practices were shared to address these memory-related issues effectively. The foremost recommendation is to use cgroup-aware JVM versions. Modern Java releases, particularly Java 8 update 191 and later, and Java 10 and newer, incorporate significantly improved cgroup awareness. For older Java 8 updates (specifically 8u131 to 8u190), experimental flags such as -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap can be employed to enable the JVM to better respect container memory limits. In Java 10 and subsequent versions, this behavior became standard and often requires no special flags. However, even with cgroup-aware JVMs, explicitly setting the heap size using parameters like -Xms for the initial heap size and -Xmx for the maximum heap size is frequently a recommended practice for predictability and control. Newer JVMs also offer options like -XX:MaxRAMPercentage, allowing for more dynamic heap sizing relative to the container’s allocated memory. It’s crucial to understand that the JVM’s total memory footprint extends beyond just the heap; it also requires memory for metaspace (which replaced PermGen in Java 8+), thread stacks, native libraries, and direct memory buffers. Therefore, when allocating memory to a container, it is essential to account for this total footprint, not merely the -Xmx value. A common guideline suggests that the Java heap might constitute around 50-75% of the total memory allocated to the container, with the remainder reserved for these other essential JVM components and any other processes running within the container. Tuning metaspace parameters, such as -XX:MetaspaceSize and -XX:MaxMetaspaceSize, can also prevent excessive native memory consumption, particularly in applications that dynamically load many classes.

Regarding CPU resources, the presenters noted that the JVM’s perception of available processors is also influenced by its cgroup awareness. In environments where CPU resources are constrained, using flags like -XX:ActiveProcessorCount can be beneficial to explicitly inform the JVM about the number of CPUs it should consider for sizing its internal thread pools, such as the common ForkJoinPool or the threads used for garbage collection. Optimizing the Docker image itself is another critical aspect of preparing Java applications for production. This involves choosing a minimal base image, such as alpine-jre, distroless, or official “slim” JRE images, instead of a full operating system distribution, to reduce the image size and potential attack surface. Utilizing multi-stage builds in the Dockerfile is a highly recommended technique; this allows developers to use a larger image containing build tools like Maven or Gradle and a full JDK in an initial stage, and then copy only the necessary application artifacts (like the JAR file) and a minimal JRE into a final, much smaller runtime image. Furthermore, being mindful of Docker image layering by combining related commands in the Dockerfile where possible can help reduce the number of layers and optimize image size. For applications on Java 9 and later, tools like jlink can be used to create custom, minimal JVM runtimes that include only the Java modules specifically required by the application, further reducing the image footprint. The session strongly emphasized that a collaborative approach between development and operations teams, combined with a thorough understanding of both JVM internals and Docker containerization principles, is paramount for successfully and reliably running Java applications in production environments.

Links:

• Docker Official Website
• OpenJDK Docker Hub Official Images
• Understanding JVM Memory Management (Baeldung)

Hashtags: #Java #Docker #JVM #Containerization #DevOps #Performance #MemoryManagement #DevoxxFR2018 #CharlesSabourdin #JeanChristopheSirot #BestPractices #ProductionReady #CloudNative

At DevoxxUS2017, Simon Ritter, Deputy CTO at Azul Systems, delivered a detailed exploration of the 55 new features in JDK 9, with a particular focus on modularity through Project Jigsaw. Simon, a veteran Java evangelist, provided a whirlwind tour of the enhancements, categorizing them into features, standards, JVM internals, specialized updates, and housekeeping changes. His presentation equipped developers with the knowledge to leverage JDK 9’s advancements effectively. This post examines the key themes of Simon’s talk, highlighting how these features enhance Java’s flexibility, performance, and maintainability.

Modularity and Project Jigsaw

The cornerstone of JDK 9 is Project Jigsaw, which introduces modularity to the Java platform. Simon explained that the traditional rt.jar file, containing over 4,500 classes, has been replaced with 94 modular components in the jmods directory. This restructuring encapsulates private APIs, such as sun.misc.Unsafe, to improve security and maintainability, though it poses compatibility challenges for libraries relying on these APIs. To mitigate this, Simon highlighted options like the --add-exports and --add-opens flags, as well as a “big kill switch” (--permit-illegal-access) to disable modularity for legacy applications. The jlink tool further enhances modularity by creating custom runtimes with only the necessary modules, optimizing deployment for specific applications.

Enhanced APIs and Developer Productivity

JDK 9 introduces several API improvements to streamline development. Simon showcased factory methods for collections, allowing developers to create immutable collections with concise syntax, such as List.of() or Set.of(). The Streams API has been enhanced with methods like takeWhile, dropWhile, and ofNullable, improving expressiveness in data processing. Additionally, the introduction of jshell, an interactive REPL, enables rapid prototyping and experimentation. These enhancements reduce boilerplate code and enhance developer productivity, making Java more intuitive and efficient for modern application development.

JVM Internals and Performance

Simon delved into JVM enhancements, including improvements to the G1 garbage collector, which is now the default in JDK 9. The G1 collector offers better performance for large heaps, addressing limitations of the Concurrent Mark Sweep collector. Other internal improvements include a new process API for accessing operating system process details and a directive file for controlling JIT compiler behavior. These changes enhance runtime efficiency and provide developers with greater control over JVM performance, ensuring Java remains competitive for high-performance applications.

Housekeeping and Deprecations

JDK 9 includes significant housekeeping changes to streamline the platform. Simon highlighted the new version string format, adopting semantic versioning (major.minor.security.patch) for clearer identification. The directory structure has been flattened, eliminating the JRE subdirectory and tools.jar, with configuration files centralized in the conf directory. Deprecated APIs, such as the applet API and certain garbage collection options, have been removed to reduce maintenance overhead. These changes simplify the JDK’s structure, improving maintainability while requiring developers to test applications for compatibility.

Standards and Specialized Features

Simon also covered updates to standards and specialized features. The HTTP/2 client, introduced as an incubator module, allows developers to test and provide feedback before it becomes standard. Other standards updates include support for Unicode 8.0 and the deprecation of SHA-1 certificates for enhanced security. Specialized features, such as the annotations pipeline and parser API, improve the handling of complex annotations and programmatic interactions with the compiler. These updates ensure Java aligns with modern standards while offering flexibility for specialized use cases.

Links:

• Azul Systems company website
• Zulu JDK

Lecturers

Jose Antonio Lorenzo Fernandez, a PhD in physics turned software engineer, works at Lhings Technologies, specializing in Java EE and embedded programming. His transition from academia to industry reflects a passion for applying technical expertise to practical problems. Jose Pereda, a researcher at the University of Valladolid, Spain, focuses on embedded systems and IoT development, contributing to open-source projects that bridge hardware and software innovation.

Abstract

The Internet of Things (IoT) has revolutionized device connectivity, but developers often grapple with complex networking challenges, such as configuring routers, handling firewalls, and managing protocols. Presented at Devoxx France 2014, this lecture demonstrates how Lhings, a cloud-based platform, simplifies IoT development by abstracting connectivity issues, allowing developers to focus on device functionality. Through a live coding session, Jose Antonio Lorenzo Fernandez and Jose Pereda showcase how to connect Java-capable devices using open hardware and Lhings, eliminating boilerplate networking code. The talk explores Lhings’ core concepts—device management, secure communication, and web-based control panels—and highlights its scalability and reliability. By analyzing the platform’s architecture and practical applications, it provides a comprehensive guide for developers building IoT services, emphasizing rapid prototyping and real-world deployment.

The IoT Connectivity Challenge

The proliferation of affordable open hardware, such as Raspberry Pi and Arduino, has democratized IoT development, enabling rapid prototyping of smart devices. However, connectivity remains a significant hurdle. Residential routers, NAT configurations, and diverse protocols like MQTT or CoAP require extensive setup, diverting focus from core functionality. Lorenzo Fernandez explains that developers often spend disproportionate time on networking code, handling tasks like port forwarding or secure socket implementation, which can delay projects and introduce errors.

Lhings addresses this by providing a cloud-based platform that manages device communication, abstracting low-level details. Devices register with Lhings, which handles routing, security, and interoperability, allowing developers to focus on application logic. Pereda emphasizes that this approach mirrors the simplicity of web APIs, making IoT development accessible even to those without networking expertise.

Live Coding: Connecting Devices with Lhings

The speakers demonstrate Lhings through a live coding session, connecting a Java-capable Raspberry Pi to a sensor network. The setup involves minimal code, as Lhings’ SDK abstracts networking:

import com.lhings.client.LhingsDevice;

public class SensorDevice {
    public static void main(String[] args) {
        LhingsDevice device = new LhingsDevice("Sensor1", "API_KEY");
        device.connect();
        device.sendEvent("temperature", 25.5);
    }
}

This code registers a device named “Sensor1” with Lhings, connects to the cloud, and sends a temperature reading. No networking code—such as socket management or firewall configuration—is required. The platform uses encrypted WebSocket connections, ensuring security without developer intervention.

The demo extends to a web control panel, automatically generated by Lhings, where users can monitor and control devices. Pereda shows how adding a new device, such as a smart light, requires only a few lines of code, highlighting scalability. The panel supports real-time updates, allowing remote control via a browser, akin to a smart home dashboard.

Lhings’ Architecture and Features

Lhings operates as a cloud middleware, sitting between devices and end-users. Devices communicate via a lightweight SDK, available for Java, Python, and C++, supporting platforms like Raspberry Pi and Arduino. The platform handles message routing, ensuring devices behind NATs or firewalls remain accessible. Security is baked in, with all communications encrypted using TLS, addressing common IoT vulnerabilities.

Scalability is a key strength: adding devices involves registering them with unique API keys, with no upper limit on device count. The platform’s reliability stems from its distributed architecture, tested by thousands of users globally. Lorenzo Fernandez notes that Lhings supports bidirectional communication, enabling servers to push commands to devices, a feature critical for applications like home automation.

Practical Applications and Benefits

The talk showcases real-world use cases, such as a smart thermostat system where sensors report temperature and a server adjusts settings remotely. This eliminates the need for local network configuration, as devices connect to Lhings’ cloud over standard internet protocols. The web control panel provides instant access, making it ideal for rapid prototyping or production-grade systems.

Benefits include reduced development time, enhanced security, and ease of scaling. By abstracting networking, Lhings allows developers to focus on device logic—e.g., sensor algorithms or UI design—while the platform handles connectivity and management. The open-source SDK and GitHub-hosted examples further lower barriers, encouraging community contributions.

Challenges and Considerations

While powerful, Lhings requires an internet connection, limiting its use in offline scenarios. Pereda acknowledges that latency-sensitive applications, such as real-time robotics, may need local processing alongside Lhings’ cloud capabilities. The platform’s dependency on a proprietary service also raises questions about vendor lock-in, though its open SDK mitigates this by supporting custom integrations.

Conclusion: Empowering IoT Innovation

Lhings transforms IoT development by removing connectivity barriers, enabling developers to build robust services with minimal effort. The live demo at DevoxxFR2014 illustrates its practicality, from prototyping to deployment. As IoT adoption grows, platforms like Lhings will play a critical role in making smart devices accessible and secure, empowering developers to innovate without wrestling with networking complexities.

Links

• Lecture video
• Lhings official site
• GitHub repository for Lhings SDK
• Raspberry Pi official site

Lecturers

Daniel Deogun and Daniel Sawano are senior consultants at Omega Point, a Stockholm-based consultancy with offices in Malmö and New York. Both specialize in building scalable, fault-tolerant systems, with Deogun focusing on distributed architectures and Sawano on integrating modern frameworks like Akka into enterprise environments. Their combined expertise in Java and Scala, along with practical experience in high-stakes projects, positions them as authoritative voices on leveraging Akka for real-world challenges.

Abstract

Akka, a toolkit for building concurrent, distributed, and resilient applications using the actor model, is renowned for its ability to deliver high-performance systems. However, integrating Akka into legacy environments—where entrenched codebases and conservative practices dominate—presents unique challenges. Delivered at Devoxx France 2014, this lecture shares insights from Omega Point’s experience developing an international, government-approved system using Akka in Java, despite Scala’s closer alignment with Akka’s APIs. The speakers explore how domain-specific requirements shaped their design, common pitfalls encountered, and strategies for success in both greenfield and brownfield contexts. Through detailed code examples, performance metrics, and lessons learned, the talk demonstrates Akka’s transformative potential and why Java was a strategic choice for business success. It concludes with practical advice for developers aiming to modernize legacy systems while maintaining reliability and scalability.

The Actor Model: A Foundation for Resilience

Akka’s core strength lies in its implementation of the actor model, a paradigm where lightweight actors encapsulate state and behavior, communicating solely through asynchronous messages. This eliminates shared mutable state, a common source of concurrency bugs in traditional multithreaded systems. Daniel Sawano introduces the concept with a simple Java-based Akka actor:

import akka.actor.UntypedActor;

public class GreetingActor extends UntypedActor {
    @Override
    public void onReceive(Object message) throws Exception {
        if (message instanceof String) {
            System.out.println("Hello, " + message);
            getSender().tell("Greetings received!", getSelf());
        } else {
            unhandled(message);
        }
    }
}

This actor receives a string message, processes it, and responds to the sender. Actors run in an ActorSystem, which manages their lifecycle and threading:

import akka.actor.ActorSystem;
import akka.actor.ActorRef;
import akka.actor.Props;

ActorSystem system = ActorSystem.create("MySystem");
ActorRef greeter = system.actorOf(Props.create(GreetingActor.class), "greeter");
greeter.tell("World", ActorRef.noSender());

This setup ensures isolation and fault tolerance, as actors operate independently and can be supervised to handle failures gracefully.

Designing with Domain Requirements

The project discussed was a government-approved system requiring high throughput, strict auditability, and fault tolerance to meet regulatory standards. Deogun explains that they modeled domain entities as actor hierarchies, with parent actors supervising children to recover from failures. For example, a transaction processing system used actors to represent accounts, with each actor handling a subset of operations, ensuring scalability through message-passing.

The choice of Java over Scala was driven by business needs. While Scala’s concise syntax aligns closely with Akka’s functional style, the team’s familiarity with Java reduced onboarding time and aligned with the organization’s existing skill set. Java’s Akka API, though more verbose, supports all core features, including clustering and persistence. Sawano notes that this decision accelerated adoption in a conservative environment, as developers could leverage existing Java libraries and tools.

Pitfalls and Solutions in Akka Implementations

Implementing Akka in a legacy context revealed several challenges. One common issue was message loss in high-throughput scenarios. To address this, the team implemented acknowledgment protocols, ensuring reliable delivery:

public class ReliableActor extends UntypedActor {
    @Override
    public void onReceive(Object message) throws Exception {
        if (message instanceof String) {
            // Process message
            getSender().tell("ACK", getSelf());
        } else {
            unhandled(message);
        }
    }
}

Deadlocks, another risk, were mitigated by avoiding blocking calls within actors. Instead, asynchronous futures were used for I/O operations:

import scala.concurrent.Future;
import static akka.pattern.Patterns.pipe;

Future<String> result = someAsyncOperation();
pipe(result, context().dispatcher()).to(getSender());

State management in distributed systems posed further challenges. Persistent actors ensured data durability by storing events to a journal:

import akka.persistence.UntypedPersistentActor;

public class PersistentCounter extends UntypedPersistentActor {
    private int count = 0;

    @Override
    public String persistenceId() {
        return "counter-id";
    }

    @Override
    public void onReceiveCommand(Object command) {
        if (command.equals("increment")) {
            persist(1, evt -> count += evt);
        }
    }

    @Override
    public void onReceiveRecover(Object event) {
        if (event instanceof Integer) {
            count += (Integer) event;
        }
    }
}

This approach allowed the system to recover state after crashes, critical for regulatory compliance.

Performance and Scalability Achievements

The system achieved impressive performance, handling 100,000 requests per second with 99.9% uptime. Akka’s location transparency enabled clustering across nodes, distributing workload efficiently. Deogun highlights that actors’ lightweight nature—thousands can run on a single JVM—allowed scaling without heavy resource overhead. Metrics showed consistent latency under 10ms for critical operations, even under peak load.

Integrating Akka with Legacy Systems

Legacy integration required wrapping existing services in actors to isolate faults. For instance, a monolithic database layer was accessed via actors, which managed connection pooling and retry logic. This approach minimized changes to legacy code while introducing Akka’s resilience benefits. Sawano emphasizes that incremental adoption—starting with a single actor-based module—eased the transition.

Lessons Learned and Broader Implications

The project underscored Akka’s versatility in both greenfield and brownfield contexts. Key lessons included the importance of clear message contracts to avoid runtime errors and the need for robust monitoring to track actor performance. Tools like Typesafe Console (now Lightbend Telemetry) provided insights into message throughput and bottlenecks.

For developers, the talk offers a blueprint for modernizing legacy systems: start small, leverage Java for familiarity, and use Akka’s supervision for reliability. For organizations, it highlights the business value of resilience and scalability, particularly in regulated industries.

Conclusion: Akka as a Game-Changer

Deogun and Sawano’s experience demonstrates that Akka can transform legacy environments by providing a robust framework for concurrency and fault tolerance. Choosing Java over Scala proved strategic, aligning with team skills and accelerating delivery. As distributed systems become the norm, Akka’s actor model offers a proven path to scalability, making it a vital tool for modern software engineering.

Links

• Lecture video
• Akka official site
• Omega Point company page
• Lightbend Telemetry

Lecturer

Hadi Hariri has built a distinguished career as a Technical Evangelist at JetBrains, where he promotes IntelliJ IDEA and other development tools through presentations, podcasts, and community engagement. With extensive experience in software architecture and web development, he has authored numerous articles and books while contributing to open-source projects. Based in Spain, Hadi balances his professional life with family responsibilities, including raising three sons, and maintains interests in Tennis and technology evangelism.

Abstract

IntelliJ IDEA represents a pinnacle of integrated development environments, offering an extensive array of features designed to enhance developer productivity across the entire software lifecycle. This presentation delivers a fast-paced overview of 42 essential tips and tricks, though in reality incorporating over 100 individual techniques, each carefully selected to address specific challenges in code navigation, completion, refactoring, debugging, and version control. The article provides a detailed analysis of these features, explaining their implementation mechanics, practical applications, and impact on workflow efficiency. Through live demonstrations and step-by-step breakdowns, it shows how mastering these tools can transform daily development tasks from tedious obligations into streamlined processes, ultimately leading to higher quality code and faster delivery.

Navigation Mastery: Moving Through Code with Precision and Speed

Efficient navigation forms the foundation of productive development in IntelliJ IDEA, allowing users to traverse large codebases with minimal cognitive effort. The Recent Files dialog, accessed via Ctrl+E on Windows or Cmd+E on Mac, presents a chronological list of edited files, enabling quick context switching without manual searching. This feature proves particularly valuable in multi-module projects where related files span different directories, as it preserves mental flow during iterative development cycles.

The Navigate to Class command, triggered by Ctrl+N, allows instant location of any class by typing its name with support for camel-case abbreviation, such as “SC” for StringCalculator. This extends to symbols through Ctrl+Alt+Shift+N, encompassing methods, fields, and variables across the project. These capabilities rely on IntelliJ’s sophisticated indexing system, which builds comprehensive symbol tables upon project load, delivering sub-second search results even in repositories exceeding a million lines of code.

The Structure view, opened with Alt+7, offers a hierarchical outline of the current file’s elements, including methods, fields, and nested classes, with incremental search for rapid location. When combined with the File Structure Popup via Ctrl+F12, developers can navigate complex files without diverting attention from the editor window, maintaining focus during intensive coding sessions.

Code Completion and Generation: Intelligent Assistance for Faster Coding

IntelliJ’s completion system transcends basic auto-suggest by incorporating contextual awareness and type inference to propose relevant options. Basic completion, invoked with Ctrl+Space, suggests identifiers based on scope and visibility, while smart completion via Ctrl+Shift+Space filters to match expected types, preventing invalid assignments and reducing debugging time.

Postfix completion introduces a novel way to wrap expressions with common constructs; for instance, typing “.not” after a boolean generates negation logic, while “.for” creates an iteration loop over collections. This feature streamlines frequent patterns, such as null checks with “.nn” or type casting with “.cast”, integrating seamlessly with the editor’s flow.

Live templates automate repetitive structures; the built-in “sout” expands to System.out.println(), while custom templates can generate complete test methods with annotations and assertions. Hadi demonstrates creating a JUnit template that includes setup code, triggered by a user-defined abbreviation for instant productivity gains.

The generate-from-usage feature, activated with Alt+Enter on undefined elements, creates missing methods, fields, or classes on demand. This supports an intentional coding style where developers first express usage intent, then implement details, aligning perfectly with test-driven development methodologies.

Refactoring Tools: Safe Code Transformation at Scale

Refactoring in IntelliJ maintains program semantics while restructuring code for improved readability and maintainability. The rename refactoring, via Shift+F6, updates all references including comments and string literals when enabled, handling scope conflicts intelligently. Extract method (Ctrl+Alt+M) creates functions from selected code blocks, automatically determining parameters and return types based on usage analysis.

Inline refactoring (Ctrl+Alt+N) reverses extractions, useful for simplifying overly fragmented code while preserving behavior. Change signature (Ctrl+F6) modifies method parameters with propagation to callers, inserting default values for new parameters to avoid compilation errors.

Surround with (Ctrl+Alt+T) wraps selected code in control structures like try-catch or if-else, with template support for custom patterns. These tools collectively enable large-scale code reorganization without manual error-prone adjustments.

Debugging Capabilities: Deep Insight into Runtime Behavior

The debugger provides sophisticated inspection beyond basic stepping. Smart step into (Shift+F7) allows selective entry into chained method calls, focusing on relevant code paths. Evaluate expression (Alt+F8) executes arbitrary code in the current frame, supporting complex debugging scenarios like modifying variables mid-execution.

Drop frame rewinds the call stack, re-executing methods without full restart, ideal for iterative testing of logic branches. Conditional breakpoints pause only when expressions evaluate true, filtering irrelevant iterations in loops.

Lambda debugging treats expressions as methods with full variable inspection and stepping. Custom renderers format complex objects, like displaying collections as comma-separated lists.

Version Control Integration: Streamlined Collaboration

Git support includes visual diffs (Ctrl+D) for conflict resolution, branch management through intuitive dialogs, and cherry-picking from commit histories. The changes view lists modified files with diff previews; annotate shows per-line authorship and revisions.

Interactive rebase through the VCS menu simplifies history cleaning by squashing or reordering commits. Pull request workflows integrate with GitHub, displaying comments directly in the editor for contextual review.

Plugin Ecosystem: Extending Functionality

Plugins like Lombok automate boilerplate with annotations, while Key Promoter X teaches shortcuts through notifications. SonarLint integrates code quality checks, flagging issues in real-time.

Custom plugin development uses Java with SDK support for editor extensions and custom tools.

Advanced Configuration for Optimal Performance

Running on Java 8+ (edit info.plist) improves font rendering. The productivity guide tracks feature usage, helping discover underutilized tools.

Conclusion: IntelliJ as Productivity Multiplier

These techniques collectively transform IntelliJ into an indispensable tool that accelerates development while improving code quality. Consistent application leads to substantial time savings and better software outcomes.

Links:

• Lecture video
• IntelliJ IDEA official site
• JetBrains plugins marketplace

Lecturer

Ben Christensen works as a software engineer at Netflix. He leads the development of reactive libraries for the JVM. Ben serves as a core contributor to RxJava. He possesses extensive experience in constructing resilient, low-latency systems for streaming platforms. His expertise centers on applying functional reactive programming principles to microservices architectures.

Abstract

This article provides an in-depth exploration of RxJava, Netflix’s implementation of Reactive Extensions for the JVM. It analyzes the Observable pattern as a foundation for composing asynchronous and event-driven programs. The discussion covers essential operators for data transformation and composition, schedulers for concurrency management, and advanced error handling strategies. Through concrete Netflix use cases, the article demonstrates how RxJava enables non-blocking, resilient applications and contrasts this approach with traditional callback-based paradigms.

The Observable Pattern and Push vs. Pull Models

RxJava revolves around the Observable, which functions as a push-based, composable iterator. Unlike the traditional pull-based Iterable, Observables emit items asynchronously to subscribers. This fundamental duality enables uniform treatment of synchronous and asynchronous data sources:

Observable<String> greeting = Observable.just("Hello", "RxJava");
greeting.subscribe(System.out::println);

The Observer interface defines three callbacks: onNext for data emission, onError for exceptions, and onCompleted for stream termination. RxJava enforces strict contracts for backpressure—ensuring producers respect consumer consumption rates—and cancellation through unsubscribe operations.

Operator Composition and Declarative Programming

RxJava provides over 100 operators that transform, filter, and combine Observables in a declarative manner. These operators form a functional composition pipeline:

Observable.range(1, 10)
          .filter(n -> n % 2 == 0)
          .map(n -> n * n)
          .subscribe(square -> System.out.println("Square: " + square));

The flatMap operator proves particularly powerful for concurrent operations, such as parallel API calls:

Observable<User> users = getUserIds();
users.flatMap(userId -> userService.getDetails(userId), 5)
     .subscribe(user -> process(user));

This approach eliminates callback nesting (callback hell) while maintaining readability and composability. Marble diagrams visually represent operator behavior, illustrating timing, concurrency, and error propagation.

Concurrency Control with Schedulers

RxJava decouples computation from threading through Schedulers, which abstract thread pools:

Observable.just(1, 2, 3)
          .subscribeOn(Schedulers.io())
          .observeOn(Schedulers.computation())
          .map(this::cpuIntensiveTask)
          .subscribe(result -> display(result));

Common schedulers include:
– Schedulers.io() for I/O-bound operations (network, disk).
– Schedulers.computation() for CPU-bound tasks.
– Schedulers.newThread() for fire-and-forget operations.

This abstraction enables non-blocking I/O without manual thread management or blocking queues.

Error Handling and Resilience Patterns

RxJava treats errors as first-class citizens in the data stream:

Observable risky = Observable.create(subscriber -> {
    subscriber.onNext(computeRiskyValue());
    subscriber.onError(new RuntimeException("Failed"));
});
risky.onErrorResumeNext(throwable -> Observable.just("Default"))
     .subscribe(value -> System.out.println(value));

Operators like retry, retryWhen, and onErrorReturn implement resilience patterns such as exponential backoff and circuit breakers—critical for microservices in failure-prone networks.

Netflix Production Use Cases

Netflix employs RxJava across its entire stack. The UI layer composes multiple backend API calls for personalized homepages:

Observable<Recommendation> recs = userIdObservable
    .flatMap(this::fetchUserProfile)
    .flatMap(profile -> Observable.zip(
        fetchTopMovies(profile),
        fetchSimilarUsers(profile),
        this::combineRecommendations));

The API gateway uses RxJava for timeout handling, fallbacks, and request collapsing. Backend services leverage it for event processing and data aggregation.

Broader Impact on Software Architecture

RxJava embodies the Reactive Manifesto principles: responsive, resilient, elastic, and message-driven. It eliminates common concurrency bugs like race conditions and deadlocks. For JVM developers, RxJava offers a functional, declarative alternative to imperative threading models, enabling cleaner, more maintainable asynchronous code.

Links:

• Ben Christensen on LinkedIn
• RxJava GitHub
• Netflix Tech Blog

Jake Wharton, an Android engineering luminary at Square, champions Dagger, a compile-time dependency injector revolutionizing Java and Android modularity. Creator of Retrofit and Butter Knife, Jake elucidates Dagger’s divergence from reflection-heavy alternatives like Guice, emphasizing its speed and testability. His session overviews injection principles, Android-specific scoping, and advanced utilities like Lazy and Assisted Injection, arming developers with patterns for clean, verifiable code.

Dagger, Jake stresses, decouples class behaviors from dependencies, fostering reusable, injectable components. Through live examples, he builds a Twitter client, showcasing modules for API wrappers and HTTP clients, ensuring seamless integration.

Dependency Injection Fundamentals

Jake defines injection as externalizing object wiring, promoting loose coupling. He contrasts manual factories with Dagger’s annotation-driven graphs, where @Inject fields auto-resolve dependencies.

This pattern, Jake demonstrates, simplifies testing—mock modules swap implementations effortlessly, isolating units.

Dagger in Android Contexts

Android’s lifecycle demands scoping, Jake explains: @Singleton for app-wide instances, activity-bound for UI components. He constructs an app graph, injecting Twitter services into activities.

Fragments and services, he notes, inherit parent scopes, minimizing boilerplate while preserving encapsulation.

Advanced Features and Utilities

Dagger’s extras shine: @Lazy defers creation, @Assisted blends factories with injection for parameterized objects. Jake demos provider methods in modules, binding interfaces dynamically.

JSR-330 compliance, augmented by @Module, ensures portability, though Jake clarifies Dagger’s compile-time limits preclude Guice’s AOP dynamism.

Testing and Production Tips

Unit tests leverage Mockito for mocks, Jake illustrates, verifying injections without runtime costs. Production graphs, he advises, tier via subcomponents, optimizing memory.

Dagger’s reflection-free speed, Jake concludes, suits resource-constrained Android, with Square’s hiring call underscoring real-world impact.

Links:

• Jake Wharton on LinkedIn
• Square company website

Lecturers

Steven Noels stands as one of the most visionary figures in the evolution of open-source Java ecosystems, having co-founded Outerthought in the early 2000s with a mission to push the boundaries of content management, RESTful architecture, and scalable data systems. His flagship creation, Daisy CMS, became a cornerstone for large-scale, multilingual content platforms used by governments and global enterprises, demonstrating that Java could power mission-critical, document-centric applications at internet scale. But Noels’ ambition extended far beyond traditional CMS. Recognizing the seismic shift toward big data in the late 2000s, he pivoted Outerthought—and later NGDATA—toward building tools that would make the Apache Hadoop ecosystem accessible to the average enterprise Java developer. Lily, launched in 2010, was the culmination of this vision: a platform that wrapped the raw power of HBase and Solr into a cohesive, Java-friendly abstraction layer, eliminating the need for MapReduce expertise or deep systems programming.

Bruno Guedes, an enterprise Java architect at SFEIR with over a decade of experience in distributed systems and search infrastructure, brought the practitioner’s perspective to the stage. Having worked with Lily from its earliest alpha versions, Guedes had deployed it in production environments handling millions of records, integrating it with legacy Java EE applications, Spring-based services, and real-time analytics pipelines. His hands-on experience—debugging schema migrations, tuning SolrCloud clusters, and optimizing HBase compactions—gave him unique insight into both the promise and the pitfalls of big data adoption in conservative enterprise settings. Together, Noels and Guedes formed a perfect synergy: the visionary architect and the battle-tested engineer, delivering a presentation that was equal parts inspiration and practical engineering.

Abstract

This article represents an exhaustively elaborated, deeply extended, and comprehensively restructured expansion of Steven Noels and Bruno Guedes’ seminal 2012 DevoxxFR presentation, “Lily, Big Data for Dummies”, transformed into a definitive treatise on the democratization of big data technologies for the Java enterprise. Delivered in a bilingual format that reflected the global nature of the Apache community, the original talk introduced Lily as a groundbreaking platform that unified Apache HBase’s scalable, distributed storage with Apache Solr’s full-text search and analytics capabilities, all through a clean, type-safe Java API. The core promise was radical in its simplicity: enterprise Java developers could build petabyte-scale, real-time searchable data systems without writing a single line of MapReduce, without mastering Zookeeper quorum mechanics, and without abandoning the comforts of POJOs, annotations, and IDE autocompletion.

This expanded analysis delves far beyond the original demo to explore the philosophical foundations of Lily’s design, the architectural trade-offs in integrating HBase and Solr, the real-world production patterns that emerged from early adopters, and the lessons learned from scaling Lily to billions of records. It includes detailed code walkthroughs, performance benchmarks, schema evolution strategies, and failure mode analyses.

EDIT:
Updated for the 2025 landscape, this piece maps Lily’s legacy concepts to modern equivalents—Apache HBase 2.5, SolrCloud 9, OpenSearch, Delta Lake, Trino, and Spring Data Hadoop—while preserving the original vision of big data for the rest of us. Through rich narratives, architectural diagrams, and forward-looking speculation, this work serves not just as a historical archive, but as a practical guide for any Java team contemplating the leap into distributed, searchable big data systems.

The Big Data Barrier in 2012: Why Hadoop Was Hard for Java Developers

To fully grasp Lily’s significance, one must first understand the state of big data in 2012. The Apache Hadoop ecosystem—launched in 2006—was already a proven force in internet-scale companies like Yahoo, Facebook, and Twitter. HDFS provided fault-tolerant, distributed storage. MapReduce offered a programming model for batch processing. HBase, modeled after Google’s Bigtable, delivered random, real-time read/write access to massive datasets. And Solr, forked from Lucene, powered full-text search at scale.

Yet for the average enterprise Java developer, this stack was inaccessible. Writing a MapReduce job required:
– Learning a functional programming model in Java that felt alien to OO practitioners.
– Mastering job configuration, input/output formats, and partitioners.
– Debugging distributed failures across dozens of nodes.
– Waiting minutes to hours for job completion.

HBase, while promising real-time access, demanded:
– Manual row key design to avoid hotspots.
– Deep knowledge of compaction, splitting, and region server tuning.
– Integration with Zookeeper for coordination.

Solr, though more familiar, required:
– Separate schema.xml and solrconfig.xml files.
– Manual index replication and sharding.
– Complex commit and optimization strategies.

The result? Big data remained the domain of specialized data engineers, not the Java developers who built the business logic. Lily was designed to change that.

Lily’s Core Philosophy: Big Data as a First-Class Java Citizen

At its heart, Lily was built on a simple but powerful idea: big data should feel like any other Java persistence layer. Just as Spring Data made MongoDB, Cassandra, or Redis accessible via repositories and annotations, Lily aimed to make HBase and Solr feel like JPA with superpowers.

The Three Pillars of Lily

Steven Noels articulated Lily’s architecture in three interconnected layers:

1. The Storage Layer (HBase)
   Lily used HBase as its primary persistence engine, storing all data as versioned, column-family-based key-value pairs. But unlike raw HBase, Lily abstracted away row key design, column family management, and versioning policies. Developers worked with POJOs, and Lily handled the mapping.

2. The Indexing Layer (Solr)
   Every mutation in HBase triggered an asynchronous indexing event to Solr. Lily maintained tight consistency between the two systems, ensuring that search results reflected the latest data within milliseconds. This was achieved through a message queue (Kafka or RabbitMQ) and idempotent indexing.

3. The Java API Layer
   The crown jewel was Lily’s type-safe, annotation-driven API. Developers defined their data model using plain Java classes:

@LilyRecord
public class Customer {
    @LilyId
    private String id;

    @LilyField(family = "profile")
    private String name;

    @LilyField(family = "profile")
    private int age;

    @LilyField(family = "activity", indexed = true)
    private List<String> recentSearches;

    @LilyFullText
    private String bio;
}

The @LilyRecord annotation told Lily to persist this object in HBase. @LilyField specified column families and indexing behavior. @LilyFullText triggered Solr indexing. No XML. No schema files. Just Java.

The Lily Repository: Spring Data, But for Big Data

Lily’s LilyRepository interface was modeled after Spring Data’s CrudRepository, but with big data superpowers:

public interface CustomerRepository extends LilyRepository<Customer, String> {
    List<Customer> findByName(String name);

    @Query("age:[* TO 30]")
    List<Customer> findYoungCustomers();

    @Query("bio:java AND recentSearches:hadoop")
    List<Customer> findJavaHadoopEnthusiasts();
}

Behind the scenes, Lily:
– Translated method names to HBase scans.
– Converted @Query annotations to Solr queries.
– Executed searches across sharded SolrCloud clusters.
– Returned fully hydrated POJOs.

Bruno Guedes demonstrated this in a live demo:

CustomerRepository repo = lily.getRepository(CustomerRepository.class);
repo.save(new Customer("1", "Alice", 28, Arrays.asList("java", "hadoop"), "Java dev at NGDATA"));
List<Customer> results = repo.findJavaHadoopEnthusiasts();

The entire operation—save, index, search—took under 50ms on a 3-node cluster.

Under the Hood: How Lily Orchestrated HBase and Solr

Lily’s magic was in its orchestration layer. When a save() was called:
1. The POJO was serialized to HBase Put operations.
2. The mutation was written to HBase with a version timestamp.
3. A change event was published to a message queue.
4. A Solr indexer consumed the event and updated the search index.
5. Near-real-time consistency was guaranteed via HBase’s WAL and Solr’s soft commits.

For reads:
– findById → HBase Get.
– findByName → HBase scan with secondary index.
– @Query → Solr query with HBase post-filtering.

This dual-write, eventual consistency model was a deliberate trade-off for performance and scalability.

Schema Evolution and Versioning: The Enterprise Reality

One of Lily’s most enterprise-friendly features was schema evolution. In HBase, adding a column family requires manual admin intervention. In Lily, it was automatic:

// Version 1
@LilyField(family = "profile")
private String email;

// Version 2
@LilyField(family = "profile")
private String phone; // New field, no migration needed

Lily stored multiple versions of the same record, allowing old code to read new data and vice versa. This was critical for rolling deployments in large organizations.

Production Patterns and Anti-Patterns

Bruno Guedes shared war stories from production:
– Hotspot avoidance: Never use auto-incrementing IDs. Use hashed or UUID-based keys.
– Index explosion: @LilyFullText on large fields → Solr bloat. Use @LilyField(indexed = true) for structured search.
– Compaction storms: Schedule major compactions during low traffic.
– Zookeeper tuning: Increase tick time for large clusters.

The Lily Ecosystem in 2012

Lily shipped with:
– Lily CLI for schema inspection and cluster management.
– Lily Maven Plugin for deploying schemas.
– Lily SolrCloud Integration with automatic sharding.
– Lily Kafka Connect for streaming data ingestion.

Lily’s Legacy After 2018: Where the Ideas Live On

EDIT
Although Lily itself was archived in 2018, its core concepts continue to thrive in modern tools.

The original HBase POJO mapping is now embodied in Spring Data Hadoop.

Lily’s Solr integration has evolved into SolrJ + OpenSearch.

The repository pattern that Lily pioneered is carried forward by Spring Data R2DBC.

Schema evolution, once a key Lily feature, is now handled by Apache Atlas.

Finally, Lily’s near-real-time search capability lives on through the Elasticsearch Percolator.

Conclusion: Big Data Doesn’t Have to Be Hard

Steven Noels closed with a powerful message:

“Big data is not about MapReduce. It’s not about Zookeeper. It’s about solving business problems at scale. Lily proved that Java developers can do that—without becoming data engineers.”

EDIT:
In 2025, as lakehouse architectures, real-time analytics, and AI-driven search dominate, Lily’s vision of big data as a first-class Java citizen remains more relevant than ever.

Links

• Apache HBase 2.5 Documentation
• SolrCloud 9 Guide
• Spring Data Hadoop
• NGDATA Archive
• Video: Lily, Big Data for Dummies

Justin Lee, a seasoned Java developer and senior software engineer at Squarespace, guides Java EE developers through the transition to MongoDB, a leading NoSQL database. With nearly two decades of experience, including contributions to GlassFish’s WebSocket implementation and the JSR 356 expert group, Justin illuminates MongoDB’s paradigm shift from relational JPA to document-based storage. His session introduces MongoDB’s structure, explores data mapping with the Java driver and Morphia, and demonstrates adapting a JPA application to MongoDB’s flexible model.

MongoDB’s schemaless design challenges traditional JPA conventions, offering dynamic data interactions. Justin addresses performance, security, and integration, debunking myths about data loss and injection risks, making MongoDB accessible for Java developers seeking scalable, modern solutions.

Understanding MongoDB’s Document Model

Justin introduces MongoDB’s core concept: documents stored as JSON-like BSON objects, replacing JPA’s rigid tables. He demonstrates collections, where documents vary in structure, offering flexibility over fixed schemas.

This approach, Justin explains, suits dynamic applications, allowing developers to evolve data models without migrations.

Mapping JPA to MongoDB with Morphia

Using Morphia, Justin adapts a JPA application, mapping entities to documents. He shows annotating Java classes to define collections, preserving object-oriented principles. A live example converts a JPA entity to a MongoDB document, maintaining relationships via references.

Morphia, Justin notes, simplifies integration, bridging JPA’s structured queries with MongoDB’s fluidity.

Data Interaction and Performance Tuning

Justin explores MongoDB’s query engine, demonstrating CRUD operations via the Java driver. He highlights performance trade-offs: write concerns adjust speed versus durability. A demo shows fast writes with minimal safety, scaling to secure, slower operations.

No reported data loss bugs, Justin assures, bolster confidence in MongoDB’s reliability for enterprise use.

Security Considerations and Best Practices

Addressing security, Justin evaluates injection risks. MongoDB’s query engine resists SQL-like attacks, but he cautions against $where clauses executing JavaScript, which could expose vulnerabilities if misused.

Best practices include sanitizing inputs and leveraging Morphia’s type-safe queries, ensuring robust, secure applications.

Links:

• Justin Lee on LinkedIn
• Squarespace company website
• Justin Lee’s blog