Jonathan Lalou's Blog

Posts Tagged ‘Observability’

Understanding `elastic.apm.instrument_ancient_bytecode=true` in Elastic APM

Elastic APM (Application Performance Monitoring) is a powerful tool designed to provide visibility into your application’s performance by instrumenting code at runtime. Most of the time, Elastic APM dynamically attaches itself to Java applications, weaving in the necessary instrumentation logic to capture transactions, spans, and errors. However, some applications, especially legacy systems or those running on older bytecode, may require additional configuration. This is where the parameter elastic.apm.instrument_ancient_bytecode=true becomes relevant.

What Does This Parameter Do?

By default, the Elastic APM agent is optimized for modern JVM bytecode, typically generated by more recent versions of Java compilers. However, in certain environments, applications may rely on very old Java bytecode compiled with legacy compilers, or on classes transformed in ways that deviate from expected patterns. In such cases, the default instrumentation mechanisms may fail.

Setting elastic.apm.instrument_ancient_bytecode=true explicitly tells the agent to attempt instrumentation on bytecode that does not fully conform to current JVM standards. It essentially relaxes some of the agent’s safeguards and fallback logic, allowing it to process “ancient” or non-standard bytecode.

When Is This Necessary?

Most modern Java applications do not require this parameter. However, it becomes useful in scenarios such as:

Legacy Applications: Systems still running on bytecode generated by Java 5, 6, or even earlier.
Bytecode Manipulation: Applications that make heavy use of frameworks or tools that dynamically generate or transform bytecode in unusual ways.
Incompatible Class Structures: Some libraries written long ago may use patterns that modern instrumentation cannot safely parse.

Examples of Differences

Without the Parameter

The Elastic APM agent may skip certain classes entirely, resulting in gaps in transaction traces.
Errors such as “class not instrumented” may appear in logs when working with older or unusual bytecode.
Performance metrics may look incomplete, missing critical spans in legacy code paths.

With the Parameter Enabled

The agent attempts a broader set of instrumentation strategies, even for outdated or malformed bytecode.
Legacy classes and libraries are more likely to be traced successfully, providing a fuller view of application performance.
Developers gain visibility into workflows that would otherwise remain opaque, such as old JDBC calls or
proprietary frameworks compiled years ago.

Trade-offs and Risks

While enabling this parameter may seem like a straightforward fix, it should be approached with caution:

Stability Risks: Forcing instrumentation of very old bytecode could lead to runtime issues if the agent misinterprets structures.
Performance Overhead: Instrumenting non-standard classes may come with higher CPU or memory costs.
Support Limitations: Elastic primarily supports mainstream JVM versions, so using this
parameter places the application in less-tested territory.

Best Practices

Enable elastic.apm.instrument_ancient_bytecode only if you detect missing traces or errors in the agent logs related to class instrumentation.
Test thoroughly in a staging environment before applying it to production.
Document which modules require this setting and track their eventual migration to modern Java versions.

Conclusion

The elastic.apm.instrument_ancient_bytecode=true parameter is a niche but valuable option for teams maintaining legacy Java systems. By enabling it, organizations can bridge the gap between outdated bytecode and modern observability needs, ensuring that even older applications benefit from the insights provided by Elastic APM. However, this should be viewed as a temporary measure on the journey toward modernizing application stacks, not as a permanent fix.

Hashtags:
#ElasticAPM #JavaMonitoring #ApplicationPerformance #LegacySystems #DevOps #Observability #JavaDevelopment #PerformanceMonitoring #ElasticStack #SoftwareMaintenance

Posted in Uncategorized | Tags: ElasticAPM, JavaMonitoring, Observability | No Comments »

[DotJs2025] Live Coding with Ryan Dahl: Deno and OpenTelemetry

Author: Jonathan Lalou

Observability’s quest in serverless seas once drowned in bespoke boilerplate; enter Deno’s alchemy, fusing runtime rigor with OpenTelemetry’s ubiquity. Ryan Dahl, Node’s progenitor and Deno Land’s CEO, live-wove this tapestry at dotJS 2025, showcasing zero-config traces illuminating JS’s shadowy underbelly. From UCSD’s mathematical meadows to ML’s machine wilds, Ryan’s odyssey—secure-by-default Deno, TypeScript’s native nest—now embraces OTEL’s triad: logs, metrics, spans.

Ryan’s canvas: a toy API, user IDs fetched, errors eventual—mundane yet ripe for revelation. Deno’s infusion: Deno.metrics(), console.log piped to OTEL, spans auto-spanning HTTP arcs. Exporter’s elegance: Jaeger console, traces unfurling waterfalls—entrypoints to exits, attributes annotating arcs. Live, Ryan scaffolded: import OTEL, configure exporter, instrument fetches—redeploy to Deno Deploy, logs and traces blooming in tandem. Nanoseconds’ narrative: cold starts eclipsed, polyglot peace via WIT interfaces—Rust greeters yielding to TS handlers.

This symbiosis scales: production probes sans proxies, Deno Deploy’s horizon harboring holistic views. Ryan’s aside: Oracle’s JS trademark tyranny—a cease-and-desist specter stifling “JSConf”—spurred javascript.tm’s petition, amassing allies against anachronistic anchors.

Deno’s decree: OTEL’s open arms, JS’s joyful instrumentation—debugging demystified, deployments discerned.

Instrumentation’s Instantaneity

Ryan rendered realms: metrics via Deno.metrics(), logs laced with OTEL, spans spanning sans strife. Jaeger’s vista: waterfalls whispering workflows, attributes authoring arcs—live, a fetch’s fate traced, errors etched eternally.

Horizons and Heresies

Deno Deploy’s dawn: traces native, nanoseconds’ narrative. Ryan rallied: sign javascript.tm, reclaim JS’s soul—petition’s progress, public’s power against Oracle’s overhang.

Links:

Posted in en-US | Tags: Deno, DenoDeploy, dotJS2025, JavaScript, Metrics, Observability, OpenTelemetry, RyanDahl, Serverless, Tracing | No Comments »

[PHPForumParis2023] You Build It, You Run It: Observability for Developers – Smaïne Milianni

Author: Jonathan Lalou

Smaïne Milianni, a former taxi driver turned PHP developer, delivered an engaging talk at Forum PHP 2023, exploring the “You Build It, You Run It” philosophy and the critical role of observability in modern development. Now an Engineering Manager at Yousign, Smaïne shared insights from his decade-long journey in PHP, emphasizing how observability tools like logs, metrics, traces, and alerts empower developers to maintain robust applications. His practical approach and humorous delivery offered actionable strategies for PHP developers to enhance system reliability and foster a culture of continuous improvement.

The Essence of Observability

Smaïne introduced observability as the cornerstone of the “You Build It, You Run It” model, where developers are responsible for both building and maintaining their applications. He explained how observability encompasses logs, metrics, traces, and custom alerts to monitor system health. Using real-world examples, Smaïne illustrated how these tools help identify issues, such as application errors or system outages, before they escalate. His emphasis on proactive monitoring resonated with developers seeking to ensure their PHP applications remain stable and performant.

Implementing Observability in PHP

Diving into practical applications, Smaïne outlined how to integrate observability into PHP projects. He highlighted tools like Datadog for collecting metrics and traces, and demonstrated how to set up alerts for critical incidents, such as P1 outages that trigger SMS and email notifications. Smaïne stressed the importance of prioritizing alerts based on severity to avoid notification fatigue. His examples, drawn from his experience at Yousign, provided a clear roadmap for developers to implement observability, ensuring rapid issue detection and resolution.

The Power of Post-Mortems

Smaïne concluded by emphasizing the role of post-mortems in fostering a virtuous cycle of improvement. Responding to an audience question, he explained how his team conducts weekly manager reviews to track post-mortem actions, ensuring they are prioritized and addressed. By treating errors as learning opportunities rather than failures, Smaïne’s approach encourages developers to refine their code and systems iteratively. His talk inspired attendees to adopt observability practices that enhance both technical reliability and team collaboration.

Links:

Yousign website

Posted in en-US | Tags: devops, Observability, PHP, PHPForumParis2023, SmaïneMilianni, Yousign | No Comments »

[NodeCongress2021] Panel Discussion – Node.js in the Cloud

Author: Jonathan Lalou

Cloud paradigms reshape Node.js landscapes, blending serverless ephemera with containerized constancy, as dissected in this convocation. Moderated discourse features Ali Spittel, AWS Amplify advocate and digital nomad; Eran Hammer, Sideway founder weaving narrative webs; Ruben Casas, American Express engineer pioneering micro-frontends; and Slobodan Stojanovic, Cloud Horizon CTO scaling Vacation Tracker’s serverless saga.

Ali champions Amplify’s frictionless ingress: Git-based deploys, CI/CD alchemy transmute code to globals—Lambda for backends, AppSync for GraphQL. Eran probes costs: fixed fleets versus invocation metering, cold starts’ latency tax. Ruben extols IaC: CDK’s constructs blueprint stacks, Terraform’s declarative drifts ensure idempotence.

Slobodan chronicles evolution: singleton Lambda to hexagonal CQRS ensembles, LocalStack mocks integrations. Consensus: serverless abstracts ops, yet demands async mastery—promises over callbacks, hexagonal ports insulate.

Deployment Dynamics and Cost Conundrums

Deploys diverge: Amplify’s wizardry suits solos, Claudia.js blueprints APIs. Containers—Docker/K8s—orchestrate statefuls, Fargate abstracts. Costs confound: Slobodan’s $250/month belies bugs’ $300 spikes; alarms mitigate.

Ali lauds functions’ scalability sans provisioning; Eran tempers with vendor lock perils. Ruben integrates OneApp’s runtime swaps.

Observability and IoT Intersections

Tracing threads via X-Ray/OpenTelemetry; Datadog dashboards divine. IoT? Node’s WebSockets shine—process streams via Amplify, hexagonal fits serverless.

Panel’s tapestry—diverse voices—illuminates Node.js’s cloud ascent, from fledgling functions to enterprise echelons.

Links:

Posted in en-US | Tags: AliSpittel, EranHammer, IaC, Node, NodeCongress, NodeCongress2021, NodeJS, NodeJSCloud, Observability, RubenCasas, Serverless, SlobodanStojanovic | No Comments »

[NodeCongress2021] Logging, Metrics, and Tracing with Node.js – Thomas Hunter II

Author: Jonathan Lalou

Observability pillars—logs, gauges, spans—form the triad illuminating Node.js constellations, where opacity breeds outages. Thomas Hunter II, a Node.js luminary and author of “Distributed Systems with Node.js,” dissects these sentinels, adapting book chapters to unveil their synergies in service scrutiny.

Thomas frames logging as cloud-elevated console.logs: structured JSON extrudes states, severity tiers—error to silly—filter verbosity. Winston orchestrates: transports serialize to stdout/files, Pino accelerates with async flushes. Conventions prescribe correlation IDs, timestamps; aggregators like ELK ingest for faceted searches.

Metrics quantify aggregates: counters tally invocations, histograms bin latencies. Prometheus scrapes via prom-client, Grafana visualizes trends—spikes foretell fractures. Thomas codes a registry: gauge tracks heap, histogram times handlers, alerting deviations.

Tracing reconstructs causal chains: spans encapsulate ops, propagators thread contexts. OpenTelemetry standardizes; Jaeger self-hosts hierarchies, timelines dissect 131ms journeys—Memcache to Yelp. Datadog APM auto-instruments, flame graphs zoom Postgres/AWS latencies.

Instrumentation Patterns and Visualization Nuances

Thomas prototypes: async_hooks namespaces contexts, cls-r tracers bridge async gulfs. Zipkin’s dependency DAGs, Datadog’s y-axis strata—live Lob.com postcard fetches—demystify depths.

Thomas’s blueprint—Winston for persistence, Prometheus for pulses, Jaeger for journeys—equips Node.js artisans to navigate nebulous networks with crystalline clarity.

Links:

Posted in en-US | Tags: logging, Metrics, Node, NodeCongress, NodeCongress2021, NodeJS, Observability, OpenTelemetry, ThomasHunterII, Tracing | No Comments »

[NodeCongress2021] Push Notifications: Can’t Live With Em, Can’t Live Without Em – Avital Tzubeli

Author: Jonathan Lalou

In an era where digital alerts permeate daily rhythms, the orchestration of push notifications embodies a delicate equilibrium between immediacy and reliability. Avital Tzubeli, a backend engineer at Vonage, unravels this dynamic through her recounting of the message bus at the heart of their communications platform—a conduit dispatching 16 million dispatches daily, contending with temporal pressures and infrastructural strains. Drawing from Hebrew folklore, where a louse embarks on a globetrotting odyssey, Avital likens notifications to intrepid voyagers navigating service boundaries.

Avital’s tale unfolds across Vonage’s ecosystem: inbound triggers from Frizzle ingress via RabbitMQ queues, auto-scaling consumers in HTTP services validate payloads, appending trace IDs for audit trails. Continuation Local Storage (CLS-Hooked) embeds identifiers in request scopes, facilitating log enrichment without prop modifications. As payloads traverse to PushMe—Vonage’s dispatch hub—interceptors affix traces to Axios headers, ensuring end-to-end visibility.

This choreography yields sub-15ms latencies: Frizzle to HTTP in milliseconds, thence to PushMe, culminating in device delivery via APNS or FCM. Avital spotlights middleware elegance—CLS-Hooked instances persist contexts, auto-injecting IDs into logs or headers, oblivious to underlying transports.

Architectural Resilience and Observability

Resilience pivots on RabbitMQ’s durability: dead-letter exchanges quarantine failures, retries exponential backoffs temper bursts. Monitoring via Grafana dashboards tracks queue depths, consumer lags; alerts preempt pileups. Avital shares code vignettes—middleware instantiation, trace retrieval, log augmentation—revealing CLS-Hooked’s prowess in decoupling concerns.

For broader applicability, Avital posits analogous buses for event sourcing or microservice fan-outs: RabbitMQ’s ACK semantics guarantee at-least-once semantics, complemented by idempotent handlers. Blaming externalities like Apple for undelivered alerts underscores the perils of third-party dependencies, yet Vonage’s stack—Node.js scripts fueling the frenzy—exemplifies robust engineering.

Avital’s odyssey, though sans parasitic flair, affirms notifications’ global sprint, propelled by vigilant teams and scalable sinews.

Links:

Posted in en-US | Tags: AvitalTzubeli, MessageBus, Node, NodeCongress, NodeCongress2021, NodeJS, Observability, PushNotifications, RabbitMQ, Vonage | No Comments »

[NodeCongress2021] Examining Observability in Node.js – Liz Parody

Author: Jonathan Lalou

Observability transcends mere logging, emerging as a vital lens for dissecting Node.js applications amid escalating complexity. Liz Parody, Head of Developer Relations at NodeSource, unpacks this concept, drawing parallels to control theory where external signals unveil internal machinations. Her examination equips developers with strategies to illuminate asynchronous behaviors, preempting failures in production.

Liz delineates observability’s essence: inferring system states sans code perturbations, contrasting it with monitoring’s retrospective aggregation. In Node.js’s event-loop-driven world, this proves indispensable, as microservices and containers fragment visibility, amplifying “unknown unknowns” like latent memory leaks.

Leveraging Node.js Internals for Performance Insights

Node.js furnishes potent primitives for introspection. Performance hooks, via observers and timers, timestamp operations—marking search latencies across engines like DuckDuckGo—yielding millisecond granularities without external agents. Heap snapshots, triggered by –heapsnapshot-signal, capture V8 allocations for leak hunting, while trace-events chronicle GC cycles and loop idles.

Liz demonstrates profiling: –prof flags generate CPU logs, convertible to flame charts via tools like 0x, pinpointing hotspots in async chains. The V8 inspector, invoked remotely, mirrors Chrome DevTools for live edits and async stack traces, though she warns against production exposure due to event-loop halts.

External Augmentations and Benchmark Realities

Complementing internals, libraries like blocking-monitor flag loop stalls exceeding thresholds, while APMs—New Relic, Datadog—offer dashboards for error rates and latencies. Liz critiques their overhead: agents wrap runtimes, inflating memory by megabytes and startups by seconds, per benchmarks where vanilla Node.js outpaces instrumented variants by 600%.

Enter N|Solid, NodeSource’s runtime: embedding observability at V8 levels adds negligible latency—2MB footprint, sub-10ms resolutions—delivering cluster views of heap, CPU, and GC without intermediaries. Liz’s metrics affirm its edge: 10,000 RPS versus competitors’ 1,500, underscoring low-impact alternatives for mission-critical deployments.

Liz’s synthesis urges proactive instrumentation, blending internals with judicious externals to cultivate robust, performant Node.js landscapes.

Links:

Posted in en-US | Tags: LizParody, Node, NodeCongress, NodeCongress2021, NodeJS, NodeSource, Nsolid, Observability, PerformanceHooks, V8Inspector | No Comments »

[NodeCongress2021] Comprehensive Observability via Distributed Tracing on Node.js – Chinmay Gaikwad

Author: Jonathan Lalou

As Node.js architectures swell in complexity, particularly within microservices paradigms, maintaining visibility into system dynamics becomes paramount. Chinmay Gaikwad addresses this imperative, advocating distributed tracing as a cornerstone for holistic observability. His discourse illuminates the hurdles of scaling real-time applications and positions tracing tools as enablers of confident expansion.

Microservices, while promoting modularity, often obscure transaction flows across disparate services, complicating root-cause analysis. Chinmay articulates common pitfalls: elusive errors in nested calls, latency spikes from inter-service dependencies, and the opacity of containerized deployments. Without granular insights, teams grapple with “unknown unknowns,” where failures cascade undetected, eroding reliability and user trust.

Tackling Visualization Challenges in Distributed Environments

Effective observability demands mapping service interactions alongside performance metrics, a task distributed tracing excels at. By propagating context—such as trace IDs—across requests, tools like Jaeger or Zipkin reconstruct end-to-end journeys, highlighting bottlenecks from ingress to egress. Chinmay emphasizes Node.js-specific integrations, where middleware instruments HTTP, gRPC, or database queries, capturing spans that aggregate into flame graphs for intuitive bottleneck identification.

In practice, this manifests as dashboards revealing service health: error rates, throughput variances, and latency histograms. For Node.js, libraries like OpenTelemetry provide vendor-agnostic instrumentation, embedding traces in event loops without substantial overhead. Chinmay’s examples underscore exporting traces to backends for querying, enabling alerts on anomalies like sudden p99 latency surges, thus preempting outages.

Forging Sustainable Strategies for Resilient Systems

Beyond detection, Chinmay advocates embedding tracing in CI/CD pipelines, ensuring observability evolves with code. This proactive stance—coupled with service meshes for automated propagation—cultivates a feedback loop, where insights inform architectural refinements. Ultimately, distributed tracing transcends monitoring, empowering Node.js developers to architect fault-tolerant, scalable realms where complexity yields to clarity.

Links:

Posted in en-US | Tags: ChinmayGaikwad, DistributedTracing, Microservices, Node, NodeCongress, NodeCongress2021, NodeJS, Observability, OpenTelemetry | No Comments »

[NodeCongress2021] Can You Change the Behavior of a Running Node.js Process From the Outside? – Vladimir de Turckheim

Author: Jonathan Lalou

Runtime modifications to live applications represent a fascinating frontier in Node.js engineering, where traditional redeployments yield to dynamic interventions. Vladimir de Turckheim, a seasoned Node.js collaborator, delves into this realm, demonstrating techniques to alter process conduct without code alterations or restarts. His session reveals the debugger’s untapped potential, transforming it from a mere inspection tool into a conduit for real-time behavioral shifts.

Vladimir begins with a relatable scenario: a bare-bones HTTP server lacking logs, emblematic of hasty development oversights. Rather than refactoring and redeploying, he advocates injecting logging logic externally, leveraging Node.js’s signal-handling capabilities. By emitting SIGUSR1, the process enters debug mode, exposing a WebSocket endpoint for remote connections— a feature ripe for production diagnostics, as Vladimir notes in his Screen blog contributions on memory leak hunting.

Harnessing the DevTools Protocol for Introspection

Central to Vladimir’s methodology is the Chrome DevTools Protocol, the backbone of Node.js debugging. Forgoing graphical interfaces, he employs programmatic access via the inspector module, querying V8’s heap for object introspection. This low-level API enables pinpointing instances—like an HTTP server’s singleton—through prototype traversal and property enumeration, yielding object IDs as memory pointers.

Vladimir’s live demo exemplifies this: post-debug activation, a secondary Node.js injector script evaluates expressions remotely, first globalizing a patching function on the process object for universal accessibility. Subsequent calls invoke this function on the server instance, swapping ‘request’ event listeners with wrappers that prepend console logs—capturing methods and URLs—before delegating to originals. This monkey-patching preserves event emission order, ensuring seamless augmentation.

Such precision stems from protocol commands like Runtime.evaluate and Runtime.callFunctionOn, which execute snippets in the target’s context. Vladimir cautions on cleanup—releasing object IDs and closing sessions via inspector.close—to avert leaks, underscoring the approach’s suitability for controlled environments with SSH access, where administrative privileges mitigate security risks.

Practical Implications and Beyond Debugging

While ostensibly a code injection showcase, Vladimir reframes the talk as a tribute to DevTools’ extensibility. Beyond logging, it facilitates bespoke profilers or heap dumps for elusive issues, bypassing UI limitations in IDEs like VS Code. For production, this enables non-intrusive observability, aligning with Screen’s mission of code-agnostic server hardening.

Vladimir concludes by encouraging custom tool-building, from granular CPU sampling to event tampering, all grounded in the protocol’s robustness. His narrative not only equips attendees with actionable dark arts but also elevates debugging from reactive firefighting to proactive mastery, fostering resilient Node.js ecosystems.

Links:

Posted in en-US | Tags: Debugging, DevToolsProtocol, Node, NodeCongress, NodeCongress2021, NodeJS, Observability, RuntimeInjection, Screen, VladimirDeTurckheim | No Comments »