Archive for the ‘General’ Category
[Devoxx Poland 2022] Understanding Zero Trust Security with Service Mesh
At Devoxx Poland 2022, Viktor Gamov, a dynamic developer advocate at Kong, delivered an engaging presentation on zero trust security and its integration with service mesh technologies. With a blend of humor and technical depth, Viktor demystified the complexities of securing modern microservice architectures, emphasizing a philosophy that eliminates implicit trust to bolster system resilience. His talk, rich with practical demonstrations, offered developers and architects actionable insights into implementing zero trust principles using tools like Kong’s Kuma service mesh, making a traditionally daunting topic accessible and compelling.
The Philosophy of Zero Trust
Viktor begins by challenging the conventional notion of trust, using the poignant analogy of The Lion King to illustrate its exploitable nature. Trust, he argues, is a vulnerability when relied upon for system access, as it can be manipulated by malicious actors. Zero trust, conversely, operates on the premise that no entity—human or service—should be inherently trusted. This philosophy, not a product or framework, redefines security by requiring continuous verification of identity and access. Viktor outlines four pillars critical to zero trust in microservices: identity, automation, default denial, and observability. These principles guide the secure communication between services, ensuring robust protection in distributed environments.
Identity in Microservices
In the realm of microservices, identity is paramount. Viktor likens service identification to a passport, issued by a trusted authority, which verifies legitimacy without relying on trust. Traditional security models, akin to fortified castles with IP-based firewalls, are inadequate in dynamic cloud environments where services span multiple platforms. He introduces the concept of embedding identity within cryptographic certificates, specifically using the Subject Alternative Name (SAN) in TLS to encode service identities. This approach, facilitated by service meshes like Kuma, allows for encrypted communication and automatic identity validation, reducing the burden on individual services and enhancing security across heterogeneous systems.
Automation and Service Mesh
Automation is a cornerstone of effective zero trust implementation, particularly in managing the complexity of certificate generation and rotation. Viktor demonstrates how Kuma, a CNCF sandbox project built on Envoy, automates these tasks through its control plane. By acting as a certificate authority, Kuma provisions and rotates certificates seamlessly, ensuring encrypted mutual TLS (mTLS) communication between services. This automation alleviates manual overhead, enabling developers to focus on application logic rather than security configurations. During a live demo, Viktor showcases how Kuma integrates a gateway into the mesh, enabling mTLS from browser to service, highlighting the ease of securing traffic in real-time.
Deny by Default and Observability
The principle of denying all access by default is central to zero trust, ensuring that only explicitly authorized communications occur. Viktor illustrates how Kuma’s traffic permissions allow precise control over service interactions, preventing unauthorized access. For instance, a user service can be restricted to only communicate with an invoice service, eliminating wildcard permissions that expose vulnerabilities. Additionally, observability is critical for detecting and responding to threats. By integrating with tools like Prometheus, Loki, and Grafana, Kuma provides real-time metrics, logs, and traces, enabling developers to monitor service interactions and maintain an up-to-date system overview. Viktor’s demo of a microservices application underscores how observability enhances security and operational efficiency.
Practical Implementation with Kuma
Viktor’s hands-on approach culminates in a demonstration of deploying a containerized application within a Kuma mesh. By injecting sidecar proxies, Kuma ensures encrypted communication and centralized policy management without altering application code. He highlights advanced use cases, such as leveraging Open Policy Agent (OPA) to enforce fine-grained access controls, like restricting a service to read-only HTTP GET requests. This infrastructure-level security decouples policy enforcement from application logic, offering flexibility and scalability. Viktor’s emphasis on developer-friendly tools and real-time feedback loops empowers teams to adopt zero trust practices with minimal friction, fostering a culture of security-first development.
Links:
Hashtags: #ZeroTrust #ServiceMesh #Microservices #Security #Kuma #Kong #DevoxxPoland #ViktorGamov
[PHPForumParis2021] Exceptions: The Weak Spot in PHP’s Type System – Baptiste Langlade
Baptiste Langlade, a PHP developer at EFI Automotive, captivated the Forum PHP 2021 audience with a deep dive into the limitations of exceptions in PHP’s type system. With a decade of experience in PHP and open-source contributions, Baptiste explored how exceptions disrupt type safety and proposed functional programming-inspired solutions. His talk combined technical rigor with practical insights, urging developers to rethink error handling. This post covers four themes: the problem with exceptions, functional programming alternatives, automating error handling, and challenges with interfaces.
The Problem with Exceptions
Baptiste Langlade began by highlighting the inherent flaws in PHP’s exception system, describing it as a “hole in the type system’s racket.” Exceptions, he argued, bypass type checks, leading to unexpected runtime errors that static analysis struggles to catch. Drawing on his work at EFI Automotive, Baptiste illustrated how unchecked exceptions in complex systems, like document management, can lead to fragile code, emphasizing the need for more robust error-handling mechanisms.
Functional Programming Alternatives
Drawing inspiration from functional programming, Baptiste proposed alternatives like the Either monad to handle errors explicitly without exceptions. He demonstrated how returning values that encapsulate success or failure states can improve type safety and predictability. By sharing examples from his open-source packages, Baptiste showed how these patterns integrate with PHP, offering developers a way to write cleaner, more reliable code that aligns with modern type-safe practices.
Automating Error Handling
Baptiste emphasized the importance of automating error detection to address the limitations of manual exception testing. He noted that developers often miss edge cases when writing unit tests, leading to uncaught exceptions. Tools like static analyzers can help by enforcing explicit error handling, but Baptiste cautioned that PHP currently lacks native support for declaring thrown exceptions in method signatures, unlike languages like Java. His insights urged developers to adopt rigorous testing practices to mitigate these risks.
Challenges with Interfaces
Concluding his talk, Baptiste addressed the challenges of using exceptions with PHP interfaces. He explained that interfaces cannot enforce specific exception types, limiting their utility in ensuring type safety. By exploring workarounds, such as explicit documentation and custom error types, Baptiste provided practical solutions for developers. His talk encouraged the PHP community to push for language improvements, drawing on his experiences to advocate for a more robust type system.
Links:
[PHPForumParis2021] Automatic Type Inference in PHP – Damien Seguy
Damien Seguy, a veteran of the PHP community and a key figure in AFUP’s early days, delivered an insightful presentation at Forum PHP 2021 on the transformative potential of automatic type inference in PHP. With extensive experience in code quality, Damien explored how static analysis tools can enhance PHP’s type system, reducing errors and improving maintainability. His talk, grounded in practical examples, offered a compelling case for leveraging automation to strengthen PHP applications. This post examines four key themes: the evolution of PHP typing, benefits of static analysis, transforming arrays into objects, and practical implementation strategies.
The Evolution of PHP Typing
Damien Seguy opened by tracing the journey of PHP’s type system, from its loosely typed origins to the robust features introduced in recent versions. He highlighted how PHP’s gradual typing, with features like scalar type hints and return types, has improved code reliability. Damien emphasized that automatic type inference, supported by tools like PHPStan and Psalm, takes this further by detecting types without explicit declarations. This evolution, informed by his work at Exakat, enables developers to write safer, more predictable code.
Benefits of Static Analysis
A core focus of Damien’s talk was the power of static analysis in catching errors early. By analyzing code before execution, tools like PHPStan can identify type mismatches, undefined variables, and other issues that might only surface at runtime. Damien shared examples where static analysis prevented bugs in complex projects, enhancing code quality without requiring extensive manual type annotations. This approach, he argued, reduces debugging time and fosters confidence in large-scale PHP applications, aligning with modern development practices.
Transforming Arrays into Objects
Damien advocated for converting arrays into objects to enhance semantic clarity and type safety. He explained that arrays, often used for lists, lack the structural guarantees of objects. By defining classes with named properties, developers can leverage static analysis to catch errors like misspelled keys early. Drawing from his experience, Damien demonstrated how this transformation adds value to codebases, making them more maintainable and less prone to runtime errors, particularly in projects with complex data structures.
Practical Implementation Strategies
Concluding his presentation, Damien shared practical strategies for integrating type inference into PHP workflows. He recommended starting with simple static analysis checks and gradually adopting stricter rules as teams gain confidence. By using tools like Exakat, developers can automate type inference across legacy and new codebases. Damien’s insights emphasized incremental adoption, ensuring that teams can improve code quality without overwhelming refactoring efforts, making type inference accessible to all PHP developers.
Links:
[PHPForumParis2021] Front-End Quality: Why It’s Also the Backend Developer’s Job – Martin Supiot & Élie Sloïm
Martin Supiot and Élie Sloïm, experts in web quality, delivered a compelling joint presentation at Forum PHP 2021, arguing that backend developers play a critical role in ensuring front-end quality. Representing Opquast, Élie, a pioneer in web quality standards, and Martin, a former AFUP treasurer, emphasized the interconnectedness of front-end and backend development. Their talk provided practical strategies for improving user experience through collaboration. This post explores four themes: shared responsibility, enhancing user empathy, optimizing error handling, and avoiding third-party dependencies.
Shared Responsibility
Martin Supiot and Élie Sloïm opened by challenging the siloed mindset of front-end versus backend development. They argued that backend developers, through their work on APIs and data processing, directly impact front-end performance and accessibility. Drawing on Opquast’s quality checklist, Élie and Martin highlighted how backend choices, like efficient API responses, influence user experience. Their collaborative approach at Opquast underscores the need for cross-functional teamwork to deliver high-quality web applications.
Enhancing User Empathy
A central theme was fostering empathy for users, particularly those with limited technical capabilities. Martin and Élie stressed that backend developers must consider how their code affects user interactions, such as ensuring clear error messages or accessible data formats. By prioritizing user needs, developers can create inclusive applications. Élie’s work with Opquast’s guidelines provides a framework for backend developers to align their work with user-centric front-end outcomes, enhancing overall usability.
Optimizing Error Handling
The duo emphasized the importance of thoughtful error handling, such as personalized 404 and 403 pages, to guide users effectively. Martin explained that a generic error page might lead users to blame their connection, whereas a well-crafted response provides clarity. While 500 errors are harder to test, Élie and Martin advocated for backend systems that deliver meaningful feedback, ensuring users remain engaged rather than frustrated, a principle rooted in Opquast’s focus on quality assurance.
Avoiding Third-Party Dependencies
Concluding their talk, Martin and Élie cautioned against relying solely on third-party authentication systems like Google or Facebook. They noted that such dependencies can exclude users without accounts, potentially losing 30% of a site’s audience. By designing backend systems that support independent authentication, developers can enhance accessibility and inclusivity. This approach, informed by Opquast’s best practices, ensures that backend decisions prioritize user access and engagement.
Links:
[NodeCongress2021] Safely Handling Dynamic Data with TypeScript – Ethan Arrowood
In the realm of full-stack development, where APIs shuttle payloads across boundaries, ensuring type fidelity amid flux poses a perennial puzzle. Ethan Arrowood, a software engineer at Microsoft, navigates this terrain adeptly, advocating schemas as sentinels against runtime surprises. His discourse spotlights TypeScript’s prowess in taming erratic inputs—from form submissions to auth tokens—via symbiotic validation frameworks.
Ethan posits data as the lifeblood of modern apps: JSON’s ubiquity powers endpoints, yet its pliancy invites mismatches. Consider an employee dossier: id, name, employed boolean, company, age, projects array. Static typings guard assignments, but external fetches evade compile-time checks, risking undefined accesses or coerced primitives. Ethan’s remedy? Leverage JSON Schema for declarative constraints, transmuting fluid objects into rigid molds.
Bridging Schemas and Static Guarantees
Enter @sinclair/typebox, a runtime validator that births schemas from TypeScript generics, yielding dual benefits: enforcement and inference. Ethan illustrates with Fastify routes: define bodySchema as TypeBox’s TObject, embedding TString for id/name, TOptional(TBoolean) for employed, mirroring anticipated shapes. This artifact doubles as validator—Fastify’s schema prop ingests it for payload scrutiny—and type oracle, infusing handlers with precise annotations.
In practice, a POST endpoint parses body as TInfer, affording intellisense: body.name yields string, body.age number|undefined. Ethan live-codes this synergy, hovering reveals nested generics—TArray(TString) for projects—ensuring downstream ops like array iterations sidestep guards. Should validation falter, Fastify aborts with 400s, averting tainted flows.
This fusion extends to broader ecosystems: io-ts for branded types, Zod for ergonomic chaining. Ethan cautions reliance on validation logic; a flawed schema propagates peril, echoing JavaScript’s untyped underbelly. Yet, when aligned, it forges ironclad pipelines, where dynamic ingress aligns seamlessly with static egress.
Real-World Integrations and Ecosystem Synergies
Ethan’s Fastify demo crystallizes the workflow: register plugins, await readiness, log addresses— all scaffolded atop schema-derived types. VS Code’s hover unveils the schema’s blueprint, from optional fields to array innards, streamlining refactoring. For authentication, schemas vet JWT claims; forms, user inputs—universal applicability.
Gratitude flows to undraw for visuals, highlight.js for syntax, and tmcw/big for slides, underscoring open-source’s scaffolding role. Ethan’s ethos—connect via GitHub/Twitter—invites dialogue, amplifying Node.js and TypeScript’s communal momentum. By entwining validation with typing, developers reclaim assurance, rendering volatile data a predictable ally in resilient architectures.
Links:
[PHPForumParis2021] Design Patterns Explained to Children – Vincent Laurier
Vincent Laurier, a former mathematics teacher turned PHP developer, enchanted the Forum PHP 2021 audience with a unique approach to explaining design patterns through storytelling. Drawing on his recently published book, Vincent used a narrative style to make complex concepts accessible, likening design patterns to characters in a children’s tale. His talk blended pedagogy with practical advice, offering insights into both coding and project realization. This post covers three themes: simplifying design patterns, the circle of realization, and sharing knowledge through storytelling.
Simplifying Design Patterns
Vincent Laurier began by demystifying design patterns, presenting them as intuitive solutions to common coding challenges. Using analogies from his book, he likened patterns like Singleton and Factory to characters with distinct roles, making them relatable for developers of all levels. Vincent’s pedagogical background shone through as he avoided technical jargon, ensuring that even complex concepts like dependency injection were accessible. His approach, rooted in his passion for teaching, encouraged developers to view patterns as tools for clearer, more maintainable code.
The Circle of Realization
A key concept in Vincent’s talk was the “circle of realization,” a framework for turning ideas into completed projects. He described this as a cyclical process with five entry points—representing stages like ideation, planning, and execution—that developers must navigate to bring projects to fruition. Vincent emphasized maintaining momentum by addressing challenges at each stage, ensuring projects don’t stall in the “world of ideas.” This metaphor, drawn from his book, resonated with attendees seeking to complete personal or professional projects.
Sharing Knowledge Through Storytelling
Vincent concluded by highlighting the power of storytelling in knowledge transfer. His book, available in both French and English on platforms like Amazon, uses narratives to teach design patterns, reflecting his belief that stories engage and educate effectively. By sharing his journey of writing and publishing, Vincent inspired developers to communicate complex ideas simply, fostering collaboration and learning within the PHP community. His talk underscored the value of creative pedagogy in technical education.
Links:
[DevoxxFR 2022] Père Castor 🐻, raconte-nous une histoire (d’OPS)
Lors de Devoxx France 2022, David Aparicio, Data Ops chez OVHcloud, a partagé une conférence de 44 minutes sur l’apprentissage à partir des échecs en opérations informatiques. David a analysé les post-mortems d’incidents majeurs survenus chez des géants comme GitHub, Amazon, Google, OVHcloud, Apple, Fastly, Microsoft, GitLab et Facebook. En explorant les causes racines, les remédiations et les bonnes pratiques, il a montré comment tirer des leçons des erreurs pour renforcer la résilience des systèmes. Suivez OVHcloud sur ovhcloud.com et twitter.com/OVHcloud.
Comprendre les post-mortems
David a commencé par expliquer ce qu’est un post-mortem : un document rédigé après un incident pour comprendre ce qui s’est passé, identifier les causes et prévenir les récurrences. Il inclut l’historique de l’incident, les flux d’information, l’organisation (qui a agi, avec quelle équipe), les canaux de communication avec les clients, l’utilisation des ressources et les processus suivis. David a souligné l’importance de la transparence, citant des initiatives comme les meetups de développeurs où les échecs sont partagés pour démystifier les incidents.
Il a illustré son propos avec une histoire fictive d’Elliot, un junior qui, par erreur, supprime une base de données de production en suivant une documentation mal structurée. Cet incident, inspiré de cas réels chez AWS (2017), GitLab et DigitalOcean, montre les dangers d’un accès non contrôlé à la production. David a recommandé des garde-fous comme des approbations manuelles pour les commandes critiques (par exemple, DROP TABLE), des rôles RBAC stricts, et des tests réguliers des backups pour garantir leur fiabilité.
Les incidents personnels : le legacy à l’épreuve
David a partagé une expérience personnelle chez OVHcloud, où il gère le data lake pour répliquer les données internes. Lors d’une astreinte, un week-end d’été, il a été alerté d’un problème sur une infrastructure legacy sans documentation claire. Un service saturait sa file de connexions (1024 clients maximum), provoquant des refus. Sans réponse des développeurs, David a opté pour une solution KISS (Keep It Simple, Stupid) : une sonde vérifiant la connectivité toutes les cinq minutes, redémarrant le service si nécessaire. Ce script, en place depuis un an et demi, a redémarré le service 70 fois, évitant de nouveaux incidents.
Un autre incident concernait une application Java legacy, tombant après 20 à 40 minutes malgré des redémarrages. Les logs montraient des déconnexions ZooKeeper et des crashs JVM. Plutôt que d’ajuster la mémoire (heap tuning), David a découvert un script de nettoyage propriétaire dans l’historique. Appliqué deux fois par semaine, ce script a résolu le problème durablement. Ces cas illustrent l’importance de comprendre le legacy, d’éviter les solutions complexes et de documenter les correctifs.
Les pannes majeures : CDN et réseaux
David a analysé l’incident Fastly de juin 2021, où une erreur 503 a touché des sites comme The Guardian, The New York Times, Amazon, Twitter et la Maison Blanche. La cause : une configuration client déployée sans test le 8 juin, activée par une demande du 12 mai, révélant un point de défaillance unique (SPoF) dans le CDN. Résolu en 45 minutes, cet incident souligne l’importance de tester les changements en pré-production (par exemple, via blue-green deployments ou shadow traffic) et de personnaliser les messages d’erreur pour améliorer l’expérience utilisateur.
Un autre cas marquant est la panne Facebook de septembre 2021, causée par une mise à jour du protocole BGP (Border Gateway Protocol). Les serveurs DNS, incapables d’accéder aux datacenters, se sont mis en mode protection, coupant l’accès à Facebook, Instagram, WhatsApp et même aux outils internes (Messenger, LDAP). Les employés ne pouvaient ni badger ni consulter la documentation, obligeant une intervention physique avec des disqueuses pour accéder aux racks. David a recommandé des TTL (Time To Live) plus longs pour les DNS, des canaux de communication séparés et des routes de secours via d’autres cloud providers.
Bonnes pratiques et culture de l’échec
David a conclu en insistant sur la nécessité de ne pas blâmer les individus, comme dans le cas d’Elliot, mais de renforcer les processus. Il a proposé des tests réguliers de backups, des exercices de chaos engineering (par exemple, simuler une erreur 500 un vendredi après-midi), et l’adoption de pratiques DevSecOps pour intégrer la sécurité dès les tests unitaires. Il a également suggéré de consulter les post-mortems publics (comme ceux de GitLab ou ElasticSearch) pour s’inspirer et d’utiliser des outils comme Terraform pour automatiser les déploiements sécurisés. Enfin, il a encouragé à rejoindre OVHcloud pour expérimenter et apprendre des incidents dans un environnement transparent.
[NodeCongress2021] Instrumenting Node.js Internals – Alejandro Oviedo
Delving into the intricacies of runtime diagnostics reveals a persistent challenge for Node.js developers: unraveling opaque behaviors in live applications without invasive alterations. Alejandro Oviedo, a backend specialist from Buenos Aires, confronts this head-on by unveiling “instrument,” an open-source utility he crafted to illuminate network flows, filesystem interactions, and module loadings. This innovation stems from his encounters with elusive glitches, where conventional logging falls short, compelling a quest for non-disruptive observability.
Alejandro’s journey underscores a universal frustration—debugging sans exceptions or traces leaves one adrift, akin to navigating fog-shrouded waters. Even in controlled dev setups, grasping async invocations or dependency chains demands more than intuition. His tool intervenes subtly, wrapping native modules like HTTP, HTTPS, or FS to log invocations without reshaping source code, thus preserving original outputs while appending diagnostic summaries.
Enhancing Visibility Through Modular Wrappers
At the heart of instrument lies a configuration-driven approach, where users specify modules in an instrument.config.js file—HTTP for endpoint reconnaissance, REQUIRE for dynamic imports. Alejandro demonstrates with npm’s version query: invoking via instrument yields the anticipated 8.2.1 for Mocha, trailed by a concise report on GET requests to registry.npmjs.org, complete with user-agent headers and CI flags. This granularity exposes externalities, from URL patterns to payload details, sans performance penalties in non-prod realms.
Extending to refactoring scenarios, imagine auditing dynamic loads in an HTTP server; static analyzers falter against runtime evaluations, but instrument excels, flagging module_a.js imports across probes. Alejandro stresses its dev-centric ethos: add as a devDependency, execute with npx instrument node app.js, and harvest insights effortlessly. Caveats abound—overhead precludes prod use, and nascent bugs invite community scrutiny via GitHub.
Yet, this simplicity belies profound utility. By demystifying internals, developers sidestep trial-and-error marathons, accelerating triage from hours to moments. Alejandro’s creation not only empowers solo coders but fosters collaborative ecosystems, where shared configs standardize diagnostics across teams. In an era of sprawling Node.js deployments, such tools bridge the observability chasm, ensuring applications hum reliably under scrutiny.
Fostering Community-Driven Refinements
Alejandro invites scrutiny, urging PRs and issues on the repository, while teasing a Q&A for deeper dives. His Buenos Aires roots and international contributions—local meetups to global forums—infuse the project with grassroots vigor, mirroring Node.js’s collaborative spirit. As environments evolve, instrument’s extensibility promises adaptations, perhaps integrating with APMs for holistic tracing.
Through this lens, troubleshooting morphs from art to science, equipping practitioners to dissect and mend with precision. Alejandro’s endeavor reminds us: true resilience blooms from visibility, not obscurity.
Links:
Kafka Streams @ Carrefour : Traitement big data à la vitesse de l’éclair
Lors de Devoxx France 2022, François Sarradin et Jérémy Sebayhi, membres des équipes data de Carrefour, ont partagé un retour d’expérience de 45 minutes sur l’utilisation de Kafka Streams pour des pipelines big data en temps réel. François, technical lead chez Moshi, et Jérémy, ingénieur senior chez Carrefour, ont détaillé leur transition des systèmes batch Spark et Hadoop vers un traitement stream réactif sur Google Cloud Platform (GCP). Leur talk a couvert l’adoption de Kafka Streams pour le calcul des stocks et des prix, les défis rencontrés et les solutions créatives mises en œuvre. Découvrez Carrefour sur carrefour.com et Moshi sur moshi.fr.
Du batch au stream processing
François et Jérémy ont débuté en comparant le traitement batch et stream. La plateforme legacy de Carrefour, datant de 2014, reposait sur Spark et Hadoop pour des jobs batch, traitant les données comme des fichiers avec des entrées et sorties claires. Les erreurs étaient gérables en corrigeant les fichiers d’entrée et en relançant les pipelines. Le streaming, en revanche, implique des flux d’événements continus via des topics Kafka, où les erreurs nécessitent une gestion en temps réel sans perturber le pipeline. Un événement corrompu ne peut être simplement supprimé, car les données historiques peuvent couvrir des années, rendant le reprocessing impraticable.
Kafka Streams, un framework réactif basé sur Apache Kafka, a permis à Carrefour de passer au stream processing. Il exploite Kafka pour un transit de données scalable et RocksDB pour un stockage d’état colocalisé à faible latence. François a expliqué que les développeurs définissent des topologies—graphes acycliques dirigés (DAG) similaires à ceux de Spark—avec des opérations comme map, flatMap, reduce et join. Kafka Streams gère automatiquement la création des topics, les stores d’état et la résilience, simplifiant le développement. L’intégration avec les services GCP (GCS, GKE, BigTable) et les systèmes internes de Carrefour a permis des calculs de stocks et de prix en temps réel à l’échelle nationale.
Surmonter les défis d’adoption
Adopter Kafka Streams chez Carrefour n’a pas été sans obstacles. Jérémy a noté que beaucoup d’équipes manquaient d’expérience avec Kafka, mais la familiarité avec Spark a facilité la transition, les deux utilisant des paradigmes de transformation similaires. Les équipes ont développé indépendamment des pratiques pour le monitoring, la configuration et le déploiement, consolidées ensuite en best practices partagées. Cette approche pragmatique a créé une base commune pour les nouveaux projets, accélérant l’adoption.
Le changement nécessitait une adaptation culturelle au-delà des compétences techniques. La plateforme data de Carrefour, gérant des volumes massifs et des données à haute vélocité (stocks, prix, commandes), exigeait un changement de mindset du batch vers le réactif. Le stream processing implique des jointures continues avec des bases externes, contrairement aux datasets statiques des batchs. François et Jérémy ont souligné l’importance d’une documentation précoce et d’un accompagnement expert pour naviguer dans les complexités de Kafka Streams, surtout lors des déploiements en production.
Bonnes pratiques et architectures
François et Jérémy ont partagé les pratiques clés émergées sur deux ans. Pour les schémas des topics, ils utilisaient Schema Registry pour typer les données, préférant des clés obligatoires pour assurer la stabilité des partitions et évitant les champs optionnels pour prévenir les ruptures de contrat. Les valeurs des messages incluaient des champs optionnels pour la flexibilité, avec des champs obligatoires comme les IDs et timestamps pour le débogage et l’ordonnancement des événements.
Maintenir des topologies stateful posait des défis. Ajouter de nouvelles transformations (par exemple, une nouvelle source de données) nécessitait de retraiter les données historiques, risquant des émissions dupliquées. Ils ont proposé des solutions comme les déploiements blue-green, où la nouvelle version construit son état sans produire de sortie jusqu’à ce qu’elle soit prête, ou l’utilisation de topics compactés comme snapshots pour stocker uniquement le dernier état par clé. Ces approches minimisaient les perturbations mais exigeaient une planification rigoureuse, les déploiements blue-green doublant temporairement les besoins en ressources.
Métriques et monitoring
Le monitoring des applications Kafka Streams était crucial. François a mis en avant des métriques clés : lag (messages en attente par topic/consumer group), indiquant les points de contention ; end-to-end latency, mesurant le temps de traitement par nœud de topologie ; et rebalance events, déclenchés par des changements de consumer group, pouvant perturber les performances. Carrefour utilisait Prometheus pour collecter les métriques et Grafana pour des dashboards, assurant une détection proactive des problèmes. Jérémy a insisté sur l’importance des métriques custom via une couche web pour les health checks, les métriques JMX de Kafka Streams n’étant pas toujours suffisantes.
Ils ont aussi abordé les défis de déploiement, utilisant Kubernetes (GKE) avec des readiness probes pour surveiller les états des applications. Une surallocation de CPU pouvait retarder les réponses aux health checks, causant des évictions de consumer groups, d’où l’importance d’un tuning précis des ressources. François et Jérémy ont conclu en vantant l’écosystème robuste de Kafka Streams—connecteurs, bibliothèques de test, documentation—tout en notant que sa nature événementielle exige un mindset distinct du batch. Leur expérience chez Carrefour a démontré la puissance de Kafka Streams pour des données en temps réel à grande échelle, incitant le public à partager ses propres retours.
[PHPForumParis2021] Saving the Planet by Doing Less – Hélène Maitre-Marchois
Hélène Maitre-Marchois, a Scrum Master and co-founder of Fairness, delivered a thought-provoking presentation at Forum PHP 2021, urging developers to embrace digital sobriety to reduce the environmental impact of technology. Drawing on her work at Fairness, a cooperative focused on responsible digital solutions, Hélène challenged the audience to rethink feature development and prioritize sustainability. Her talk, blending ecological awareness with practical strategies, inspired developers to make impactful choices. This post explores four key themes: the environmental cost of digital technology, questioning feature necessity, optimizing user experience, and fostering sustainable practices.
The Environmental Cost of Digital Technology
Hélène Maitre-Marchois opened by highlighting the significant environmental footprint of digital technology, noting that it accounts for 3–4% of global greenhouse gas emissions, a figure growing by 8% annually. She emphasized that the internet is not intangible—data centers, networks, and user devices consume vast resources. Hélène referenced studies from GreenIT and The Shift Project, underscoring that user devices, with low recycling rates, contribute heavily to this impact. By framing coding as an energy-intensive activity, she urged developers to consider the ecological consequences of their work, from CPU usage to disk operations.
Questioning Feature Necessity
A core message of Hélène’s talk was the importance of challenging the need for every feature. She advocated for a “why” mindset, questioning whether functionalities truly serve users or merely satisfy client assumptions. Hélène shared examples where client-driven features, like flashy designs, were less valuable than anticipated when tested with users. By prioritizing minimal, purposeful development, developers can reduce resource consumption, aligning with the principles of eco-design that Fairness champions, ensuring applications are both functional and environmentally responsible.
Optimizing User Experience
Hélène stressed that sustainable design enhances user experience without sacrificing aesthetics. She suggested practical measures, such as using dark backgrounds to reduce screen energy consumption, as black pixels require less power than white ones on many displays. By optimizing user journeys and focusing on essential information, developers can create efficient, user-friendly applications. Hélène’s approach, rooted in her Scrum Master experience, emphasizes collaboration with designers and stakeholders to balance usability and sustainability, ensuring applications meet real user needs.
Fostering Sustainable Practices
Concluding her presentation, Hélène encouraged developers to adopt sustainable coding practices, such as optimizing database queries and choosing energy-efficient data formats. She highlighted the role of ethical designers and community initiatives like La Fresque du Numérique in promoting digital sobriety. By integrating these practices, developers can contribute to a cleaner internet, aligning with Fairness’ mission to build a responsible digital ecosystem. Hélène’s call to action inspired attendees to rethink their workflows and prioritize ecological responsibility in their projects.