Jonathan Lalou's Blog

Archive for the ‘en-US’ Category

Apache Spark, a cornerstone in big data processing, has significantly shaped the landscape of distributed computing with its functional programming paradigm rooted in Scala. In a keynote address at Scala Days New York 2016, Matei Zaharia, the creator of Spark, elucidated the evolution of Spark’s APIs, culminating in the transformative release of Spark 2.0. This presentation highlighted how Spark has progressed from its initial vision of a unified engine to a more sophisticated platform with structured APIs like DataFrames and Datasets, enabling enhanced performance and usability for developers worldwide.

The Genesis of Spark’s Vision

Spark was conceived with two primary ambitions: to create a unified engine capable of handling diverse big data workloads and to offer a concise, language-integrated API that mirrors working with local data collections. Matei explained that unlike the earlier MapReduce model, which was groundbreaking yet limited, Spark extended its capabilities to support iterative computations, streaming, and interactive data exploration. This unification was critical, as prior to Spark, developers often juggled multiple specialized systems, each with its own complexities, making integration cumbersome. By leveraging Scala’s functional constructs, Spark introduced Resilient Distributed Datasets (RDDs), allowing developers to perform operations like map, filter, and join with ease, abstracting the complexities of distributed computing.

The success of this vision is evident in Spark’s widespread adoption. With over a thousand organizations deploying it, including on clusters as large as 8,000 nodes, Spark has become the most active open-source big data project. Its libraries for SQL, streaming, machine learning, and graph processing have been embraced, with 75% of surveyed organizations using multiple components, demonstrating the power of its unified approach.

Challenges with the Functional API

Despite its strengths, the original RDD-based API presented challenges, particularly in optimization and efficiency. Matei highlighted that the functional API, while intuitive, conceals the semantics of computations, making it difficult for the engine to optimize operations automatically. For instance, operations like groupByKey can lead to inefficient memory usage, as they materialize large intermediate datasets unnecessarily. This issue is exemplified in a word count example where groupByKey creates a sequence of values before summing them, consuming excessive memory when a simpler reduceByKey could suffice.

Moreover, the reliance on Java objects for data storage introduces significant memory overhead. Matei illustrated this with a user class example, where headers, pointers, and padding consume roughly two-thirds of the allocated memory, a critical concern for in-memory computing frameworks like Spark. These challenges underscored the need for a more structured approach to data processing.

Introducing Structured APIs: DataFrames and Datasets

To address these limitations, Spark introduced DataFrames and Datasets, structured APIs built atop the Spark SQL engine. These APIs impose a defined schema on data, enabling the engine to understand and optimize computations more effectively. DataFrames, dynamically typed, resemble tables in a relational database, supporting operations like filtering and aggregation through a domain-specific language (DSL). Datasets, statically typed, extend this concept by aligning closely with Scala’s type system, allowing developers to work with case classes for type safety.

Matei demonstrated how DataFrames enable declarative programming, where operations are expressed as logical plans that Spark optimizes before execution. For example, filtering users by state generates an abstract syntax tree, allowing Spark to optimize the query plan rather than executing operations eagerly. This declarative nature, inspired by data science tools like Pandas, distinguishes Spark’s DataFrames from similar APIs in R and Python, enhancing performance through lazy evaluation and optimization.

Optimizing Performance with Project Tungsten

A significant focus of Spark 2.0 is Project Tungsten, which addresses the shifting bottlenecks in big data systems. Matei noted that while I/O was the primary constraint in 2010, advancements in storage (SSDs) and networking (10-40 gigabit) have shifted the focus to CPU efficiency. Tungsten employs three strategies: runtime code generation, cache locality exploitation, and off-heap memory management. By encoding data in a compact binary format, Spark reduces memory overhead compared to Java objects. Code generation, facilitated by the Catalyst optimizer, produces specialized bytecode that operates directly on binary data, improving CPU performance. These optimizations ensure Spark can leverage modern hardware trends, delivering significant performance gains.

Structured Streaming: A Unified Approach to Real-Time Processing

Spark 2.0 introduces structured streaming, a high-level API that extends the benefits of DataFrames and Datasets to streaming computations. Matei emphasized that real-world streaming applications often involve batch and interactive workloads, such as updating a database for a web application or applying a machine learning model. Structured streaming treats streams as infinite DataFrames, allowing developers to use familiar APIs to define computations. The engine then incrementally executes these plans, maintaining state and handling late data efficiently. For instance, a batch job grouping data by user ID can be adapted to streaming by changing the input source, with Spark automatically updating results as new data arrives.

This approach simplifies the development of continuous applications, enabling seamless integration of streaming, batch, and interactive processing within a single API, a capability that sets Spark apart from other streaming engines.

Future Directions and Community Engagement

Looking ahead, Matei outlined Spark’s commitment to evolving its APIs while maintaining compatibility. The structured APIs will serve as the foundation for new libraries, facilitating interoperability across languages like Python and R. Additionally, Spark’s data source API allows applications to seamlessly switch between storage systems like Hive, Cassandra, or JSON, enhancing flexibility. Matei also encouraged community participation, noting that Databricks offers a free Community Edition with tutorials to help developers explore Spark’s capabilities.

Links:

• Databricks company website
• Matei Zaharia on LinkedIn

Abstract

Sometimes, the filter function in Primefaces <p:datatable/> does not work when the field on which filtering is operated typed as an enum.

Explanation

Actually, in order to filter, Primefaces relies on a direct '=' comparison. The hack to fix this issue is to force Primefaces to compare on the enum name, and not by a reference check.

Quick fix

In the enum class, add the following block:

[java]public String getName(){ return name(); }[/java]

Have the datatable declaration to look like:

[xml]<p:dataTable id="castorsDT" var="castor" value="#{managedCastorListManagedBean.initiatedCastors}" widgetVar="castorsTable" filteredValue="#{managedCastorListManagedBean.filteredCastors}">
[/xml]

Declare the enum-filtered column lke this:

[xml]<p:column sortBy="#{castor.castorWorkflowStatus}" filterable="true" filterBy="#{castor.castorWorkflowStatus.name}" filterMatchMode="in">
<f:facet name="filter">
<p:selectCheckboxMenu label="#{messages[‘status’]}" onchange="PF(‘castorsTable’).filter()">

<f:selectItems value="#{transverseManagedBean.allCastorWorkflowStatuses}" var="cws" itemLabel="#{cws.name}" itemValue="#{cws.name}"/>
</p:selectCheckboxMenu>
</f:facet>
</p:column>[/xml]

Notice how the filtering attribute is declared:

[xml]filterable="true" filterBy="#{castor.castorWorkflowStatus.name}" filterMatchMode="in"[/xml]

In other terms, the comparison is forced the rely on equals() of class String, through the calls to getName() and name().

I wrote the following notice on Stephane TORTAJADA‘s profile on LinkedIn:

I had reported to Stephane for one year. I recommend Stephane for his management, that is based on a few principles:
* understand personally technical and functional problems
* allow team mates to make errors, in order to learn from them
* protect team mates from external aggressions and impediments
* escalating up and down the relevant information

I wrote the following notice on Ahmed CHAARI‘s profile on LinkedIn:

Ahmed has shown his ability to absorb and learn complex technologies, as well as improve his skills in a short time and a not-so-easy environment. This is why I consider Ahmed as a software engineer to recommend for any Java-focused team.

I wrote the following notice on Sidney COHEN‘s profile on LinkedIn:

Sidney has a deep knowledge of the technologies he uses, such as Tibco systems. In several occasions, he was able to liaise with the key people of the right teams, perform the good actions and deliver the best solutions to various issues. Moreover, he has learnt complex Java technologies in a short time.
I recommend Sidney as a good worker and a friendly team mate

 

I wrote the following notice on Yann BLAZART‘s profile on LinkedIn:

Yann has a wide knowledge in technologies, specifications and implementations of JEE. He has shown his ability to propose and implement technical solutions to various technical and functional issues and subjects. Besides, by helping his colleagues, he succeeded in raising the general level of the team. All this is why I recommend Yann as a Java developer and architect.

The blockchain emerged as a revolutionary technology, capturing significant attention with its potential to reshape industries and redefine trust. At Devoxx France 2016, Benoît Lafontaine and Yann Rouillard delivered a comprehensive university session delving into the intricacies of this much-hyped technology, moving beyond the buzzwords to explore its technical underpinnings, evolutions, and practical implications. Their detailed exposition covered the foundational principles of Bitcoin, the expanded capabilities introduced by platforms like Ethereum, the concept and implementation of smart contracts, various use cases, and the broader societal questions raised by distributed ledger technologies.

Demystifying the Blockchain: The Foundation of Bitcoin

To truly grasp the essence of blockchain, one must first understand its initial and most prominent implementation: Bitcoin. More than just a digital currency, Bitcoin introduced a novel distributed ledger technology that enables secure, transparent, and tamper-resistant record-keeping without relying on a central authority. The core of Bitcoin’s technical functioning lies in its chain of blocks. Each block contains a list of verified transactions, a timestamp, and a reference (cryptographic hash) to the preceding block, creating an immutable historical record.

The lifecycle of a Bitcoin transaction begins when a user initiates a transfer of value. This transaction is broadcast to the Bitcoin network. Nodes on the network validate the transaction based on a set of rules, ensuring the sender has sufficient funds and the transaction is correctly formatted. Once validated, the transaction is added to a pool of unconfirmed transactions.

The process of adding new blocks to the chain is handled by miners through a mechanism called Proof-of-Work (PoW). Miners compete to solve a complex computational puzzle, which essentially involves finding a number (a “nonce”) such that when added to the block data and hashed, the resulting hash meets certain criteria (e.g., starts with a specific number of zeros). This hashing process is computationally intensive but easy to verify. The first miner to find a valid nonce and create a new block broadcasts it to the network. Other nodes verify the block’s validity, including the PoW, and if correct, add it to their copy of the blockchain. This block then becomes the latest link in the chain.

Cryptography plays a vital role in ensuring the security and integrity of Bitcoin. Hashing algorithms (like SHA-256 used in Bitcoin) produce a unique fixed-size string (the hash) from input data. Even a minor change in the input data results in a completely different hash. This property is used to link blocks and verify data integrity. Digital signatures, based on public-key cryptography, are used to authorize transactions. Each user has a pair of keys: a private key (kept secret) used to sign a transaction and a public key (shared freely) used by others to verify the signature, ensuring that only the owner of the funds can authorize their transfer.

One of the fundamental challenges in a distributed system like Bitcoin is reaching consensus among participants on the correct state of the ledger, especially in the presence of potentially malicious actors. Proof-of-Work is Bitcoin’s consensus mechanism. By requiring significant computational effort to create a new block, it makes it economically infeasible for a malicious party to alter past transactions. To successfully tamper with a block, an attacker would need to redo the PoW for that block and all subsequent blocks faster than the rest of the network combined. This leads to the concept of the 51% attack, where if an entity controls more than 51% of the network’s total mining power, they could potentially manipulate transactions. However, the sheer scale of the Bitcoin network’s hashing power makes achieving a 51% attack incredibly difficult and prohibitively expensive.

Beyond Bitcoin: Exploring Ethereum and Smart Contracts

While Bitcoin demonstrated the power of a decentralized ledger for peer-to-peer currency transactions, its scripting language is intentionally limited, primarily designed for simple payment logic. The next wave of blockchain innovation arrived with platforms like Ethereum, which expanded the potential of blockchain technology far beyond just cryptocurrencies. Ethereum introduced the concept of a decentralized world computer capable of executing code, paving the way for a wide range of decentralized applications.

Ethereum’s core innovation is the Ethereum Virtual Machine (EVM), a Turing-complete virtual machine that can execute code deployed on the Ethereum blockchain. This code comes in the form of smart contracts. A smart contract is essentially a program stored on the blockchain that automatically executes predefined actions when specific conditions are met. Unlike traditional contracts, which are interpreted and enforced by legal systems, smart contracts are self-executing and enforced by the code itself, running on the decentralized and immutable ledger.

The primary language for writing smart contracts on Ethereum is Solidity, a high-level, contract-oriented language. Solidity’s syntax is influenced by languages like C++, Python, and JavaScript. Benoît and Yann provided examples of Solidity code, illustrating how to define state variables, functions, and events within a contract. These contracts can represent anything from simple tokens and voting mechanisms to complex financial agreements and decentralized autonomous organizations.

Deploying a smart contract involves compiling the Solidity code into EVM bytecode and then sending a transaction to the Ethereum network containing this bytecode. Once deployed, the contract resides at a specific address on the blockchain and its functions can be invoked by other users or contracts through transactions. The execution of smart contract functions requires gas, a unit of computation on the Ethereum network, paid for in Ether (ETH), Ethereum’s native cryptocurrency. This gas mechanism incentivizes efficient code and prevents denial-of-service attacks. The ability to write and deploy arbitrary code on a decentralized and immutable ledger opened up a vast landscape of possibilities for creating decentralized applications (DApps).

Alternative Consensus Mechanisms and Decentralized Autonomous Organizations

While Proof-of-Work (PoW) has proven effective for Bitcoin’s security, its energy consumption is a significant concern. This has led to research and development into alternative consensus mechanisms. Proof-of-Stake (PoS) is a prominent alternative where the right to create new blocks and validate transactions is determined by the amount of cryptocurrency a validator holds and is willing to “stake” as collateral. In PoS, validators are chosen to create blocks based on their stake size and other factors, rather than computational power. If a validator attempts to validate fraudulent transactions, they risk losing their staked amount. PoS is generally considered more energy-efficient than PoW. At the time of the talk, Ethereum was planning a transition from PoW to a PoS consensus mechanism called Casper, a transition that has since been completed.

The capabilities of smart contracts extend to enabling entirely new forms of organization. A Decentralized Autonomous Organization (DAO) is an organization whose rules and decision-making processes are encoded directly into smart contracts on a blockchain. Once deployed, a DAO operates autonomously according to its pre-programmed rules, without the need for central management. Funding, governance, and operations are all managed through the smart contracts and the participation of token holders who vote on proposals. The talk touched upon the concept of DAOs and their potential to create more transparent and democratic organizational structures. However, the early history of DAOs also includes cautionary tales, such as “The DAO” hack in 2016, which highlighted the critical importance of rigorous security auditing for smart contracts managing significant assets.

Real-World Applications and Societal Impact

The potential applications of blockchain technology span numerous industries beyond finance. Benoît and Yann explored various use cases that were being studied or already implemented in 2016. In finance, beyond cryptocurrencies, blockchain can streamline cross-border payments, facilitate peer-to-peer lending, and improve trade finance. In supply chain management, it can provide transparent and verifiable tracking of goods from origin to destination. For identity management, blockchain could enable self-sovereign identity solutions, giving individuals more control over their personal data. Other potential applications discussed included decentralized marketplaces, intellectual property management, voting systems, and even decentralized energy grids.

The advantages offered by blockchain technology, such as transparency (for public blockchains), immutability, security through cryptography, and disintermediation (removing the need for central authorities), make it attractive for scenarios where trust and verification are paramount. However, challenges remain, including scalability limitations of some blockchains, regulatory uncertainty, the difficulty of correcting errors on an immutable ledger, and the complexity of developing and securing smart contracts.

Beyond the technical and practical applications, blockchain introduces profound social implications. It challenges existing power structures by enabling decentralization and disintermediation. It raises questions about governance in decentralized networks, the legal status of smart contracts, and the impact on privacy in a transparent ledger. The technology empowers individuals with greater control over their assets and data but also requires a higher degree of individual responsibility. The discussion during the session underscored that blockchain is not just a technical innovation but also a socio-technical one with the potential to reshape how we organize and interact in the digital age.

In conclusion, the Devoxx France 2016 session on the blockchain provided a timely and detailed exploration of this burgeoning technology. By dissecting the mechanics of Bitcoin, presenting the advancements brought by Ethereum and smart contracts, discussing alternative consensus models and DAOs, and examining a variety of use cases and societal impacts, Benoît Lafontaine and Yann Rouillard offered attendees a clearer understanding of what lay beyond the hype and why this technology warranted serious attention from developers and businesses alike. The session emphasized that while challenges existed, the potential for blockchain to drive innovation across numerous sectors was undeniable.

Links:

• Benoît Lafontaine’s Blog Post on OCTO Talks!
• OCTO Technologies official website
• Bitcoin official website
• Ethereum official website
• Solidity official website
• The DAO hack explanation

Hashtags: #Blockchain #Bitcoin #Ethereum #SmartContracts #ProofOfWork #ProofOfStake #DAO #Cryptocurrency #Decentralization #DistributedLedger #FinTech #Web3 #Solidity #EVM #OCTOTechnologies #BenoitLafontaine #YannRouillard #DevoxxFR

Alexandre Delègue and Mathieu Ancelin, both engineers at SERLI, captivated attendees at Devoxx France 2015 with a deep dive into building reactive applications on a Raspberry Pi cluster. Leveraging their expertise in Java, Java EE, and open-source projects, they demonstrated a microservices-based system using Play, Akka, Cassandra, and Elasticsearch, testing the Reactive Manifesto’s promises on constrained hardware.

Embracing the Reactive Manifesto

Alexandre opened by contrasting monolithic enterprise stacks with the modular, scalable approach of the Reactive Manifesto. He introduced their application, built with microservices and event sourcing, designed to be responsive, resilient, and elastic. Running this on Raspberry Pi’s limited resources tested the architecture’s ability to deliver under constraints, proving its adaptability.

This philosophy, Alexandre noted, prioritizes agility and resilience.

Microservices and Event Sourcing

Mathieu detailed the application’s architecture, using Play for the web framework and Akka for actor-based concurrency. Cassandra handled data persistence, while Elasticsearch enabled fast search capabilities. Event sourcing ensured a reliable audit trail, capturing state changes as events. The duo’s live demo showcased these components interacting seamlessly, even on low-powered Raspberry Pi hardware.

This setup, Mathieu emphasized, ensures robust performance.

Challenges of Clustering on Raspberry Pi

The session highlighted configuration pitfalls encountered during clustering. Alexandre shared how initial deployments overwhelmed the Raspberry Pi’s CPU, causing nodes to disconnect and form sub-clusters. Proper configuration, tested pre-production, resolved these issues, ensuring stable heartbeats across the cluster. Their experience underscored the importance of thorough setup validation.

These lessons, Alexandre noted, are critical for constrained environments.

Alternative Reactive Approaches

Mathieu explored other reactive libraries, such as Spring Boot with reactive Java 8 features and async servlets, demonstrating versatility beyond Akka. Their demo included Gatling for load testing, though an outdated plugin caused challenges, since resolved natively. The session concluded with a nod to the fun of building such systems, encouraging experimentation.

This flexibility, Mathieu concluded, broadens reactive development options.

Links:

• Alexandre Delègue’s Google+ blog
• SERLI company website

Nicolas De Loof, an Apache Maven committer and founder of BreizhJUG, delivered an engaging session at Devoxx France 2015, stepping in for his colleague Michael Neale. Representing CloudBees, Nicolas shared the company’s evolution from a fragmented startup to a robust, globally available system, focusing on seamless infrastructure migrations without interrupting service. His narrative, infused with humor and practical insights, highlighted transitions to multi-tenant architectures and Docker-based deployments.

From Startup Chaos to Structured Systems

Nicolas began by outlining CloudBees’ early days, marked by ad-hoc technical decisions that later demanded refinement. Initial choices, such as a custom LXC-based solution, became obsolete as the company scaled. He described the challenge of maintaining zero downtime across a global user base, necessitating careful planning to evolve infrastructure while keeping services operational.

This journey, Nicolas emphasized, required strategic foresight.

Migrating to Multi-Tenant Architecture

The shift to a multi-tenant build-on-demand system was a cornerstone of CloudBees’ transformation. Nicolas detailed how this migration, spanning months, consolidated resources to improve efficiency without impacting users. By gradually phasing in the new architecture, the team ensured continuity, addressing regrets from earlier single-tenant designs that strained scalability.

This transition, he noted, enhanced resource utilization.

Adopting Docker for Containerization

Replacing LXC with Docker marked another pivotal change. Nicolas explained how Docker’s containerization simplified deployment and management, offering greater flexibility than the bespoke LXC setup. The migration, executed incrementally, maintained service uptime, with Docker’s lightweight containers streamlining operations across CloudBees’ infrastructure.

This adoption, Nicolas highlighted, modernized their platform.

Operational Best Practices

Drawing from CloudBees’ experience, Nicolas stressed the importance of health checks, monitoring, and termination strategies to prevent service disruptions. His lighthearted “Salut les Geeks” conclusion, inspired by a YouTube series, underscored practical advice: robust monitoring prevents “blonde” moments where systems fail silently. He urged teams to integrate these practices early to avoid production chaos.

These strategies, he concluded, ensure resilient operations.

Links:

• CloudBees company website

Jonathan Winandy and Alexis Guéganno, co-founder and operations director at Valwin, respectively, presented a deep dive into advanced Apache Kafka streaming techniques at Devoxx France 2015. With expertise in distributed systems and data warehousing, they explored how Kafka enables flexible, high-performance real-time streaming beyond basic JSON payloads.

Foundations of Streaming

Jonathan opened with a concise overview of streaming, emphasizing Kafka’s role in real-time distributed systems. He explained how Kafka’s topic-based architecture supports high-throughput data pipelines. Their session moved beyond introductory concepts, focusing on advanced writing, modeling, and querying techniques to ensure robust, future-proof streaming solutions.

This foundation, Jonathan noted, sets the stage for scalability.

Advanced Modeling and Querying

Alexis detailed Kafka’s ability to handle structured data, moving past schemaless JSON. They showcased techniques for defining schemas and optimizing queries, improving performance and maintainability. Q&A revealed their use of a five-node cluster for fault tolerance, sufficient for basic journaling but scalable to hundreds for larger workloads.

These methods, Alexis highlighted, enhance data reliability.

Managing Kafka Clusters

Jonathan addressed cluster management, noting that five nodes ensure fault tolerance, while larger clusters handle extensive partitioning. They discussed load balancing and lag management, critical for high-volume environments. The session also covered Kafka’s integration with databases, enabling real-time data synchronization.

This scalability, Jonathan concluded, supports diverse use cases.

Community Engagement and Resources

The duo encouraged engagement through Scala.IO, where Jonathan organizes, and shared Valwin’s expertise in data solutions. Their insights into cluster sizing and health monitoring, particularly in regulated sectors like healthcare, underscored Kafka’s versatility.

This session equips developers for advanced streaming challenges.

Links:

• Valwin company website