Jonathan Lalou's Blog

Posts Tagged ‘Kibana’

Have you ever had a perfectly working Docker Compose stack that mysteriously stopped working after a routine software update? It’s a frustrating experience that can consume hours of debugging. This post is a chronicle of just such a problem, involving a local Elastic Stack, Docker’s recent versions, and a simple, yet critical, configuration oversight.

The stack in question was a straightforward setup for local development, enabling a quick start for Elasticsearch, Kibana, and the APM Server. The key to its simplicity was the environment variable xpack.security.enabled=false, which effectively disabled security for a seamless, local-only experience.

The configuration looked like this:

version: "3.9"

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.16.1
    container_name: elasticsearch
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
    ports:
      - "9200:9200"
      - "9600:9600"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    restart: always

  kibana:
    image: docker.elastic.co/kibana/kibana:8.16.1
    container_name: kibana
    depends_on:
      - elasticsearch
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
      - xpack.apm.enabled=true
    ports:
      - "5601:5601"
    restart: always

  apm-server:
    image: docker.elastic.co/apm/apm-server:8.16.1
    container_name: apm-server
    depends_on:
      - elasticsearch
    environment:
      - APM_SERVER_LICENSE=trial
      - X_PACK_SECURITY_USER=elastic
      - X_PACK_SECURITY_PASSWORD=changeme
    ports:
      - "8200:8200"
    restart: always

This setup worked flawlessly for months. But after a hiatus and a few Docker updates, the stack refused to start. Countless hours were spent trying different versions, troubleshooting network issues, and even experimenting with new configurations like Fleet and health checks—all without success. The solution, it turned out, was to roll back to a four-year-old version of Docker (20.10.x), which immediately got the stack running again.

The question was: what had changed?

The Root Cause: A Subtle Security Misalignment

The culprit wasn’t a major Docker bug but a subtle incompatibility in the configuration that was handled differently by newer Docker versions. The issue lies with the apm-server configuration.

Even though security was explicitly disabled in the elasticsearch service with xpack.security.enabled=false, the apm-server was still configured to use authentication with X_PACK_SECURITY_USER=elastic and X_PACK_SECURITY_PASSWORD=changeme.

In older Docker versions, the APM server’s attempt to authenticate against an unsecured Elasticsearch instance might have failed silently or been handled gracefully, allowing the stack to proceed. However, recent versions of Docker and the Elastic stack are more stringent and robust in their security protocols. The APM server’s inability to authenticate against the non-secured Elasticsearch instance led to a fatal startup error, halting the entire stack.

The Solution: A Simple YAML Fix

The solution is to simply align the security settings across all services. Since Elasticsearch is running without security, the APM server should also be configured to connect without authentication.

By removing the authentication environment variables from the apm-server service, the stack starts correctly on the latest Docker versions.

Here is the corrected docker-compose.yml:

version: "3.9"

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.16.1
    container_name: elasticsearch
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
    ports:
      - "9200:9200"
      - "9600:9600"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    restart: always

  kibana:
    image: docker.elastic.co/kibana/kibana:8.16.1
    container_name: kibana
    depends_on:
      - elasticsearch
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
      - xpack.apm.enabled=true
    ports:
      - "5601:5601"
    restart: always

  apm-server:
    image: docker.elastic.co/apm/apm-server:8.16.1
    container_name: apm-server
    depends_on:
      - elasticsearch
    # The fix is here: remove security environment variables
    environment:
      - APM_SERVER_LICENSE=trial
    ports:
      - "8200:8200"
    restart: always

This experience highlights an important lesson in development: what works today may not work tomorrow due to underlying changes in a platform’s behavior. While a quick downgrade can get you back on track, a deeper investigation into the root cause often leads to a more robust and forward-compatible solution.

If you’re working with Elastic APM in a Java application, you might wonder when to use `@CaptureSpan` versus `@CaptureTransaction`. Both are powerful tools for observability, but they serve different purposes.
🔹 `@CaptureTransaction`:
Use this at the entry point of a request, typically at a controller, service method, or a background job. It defines the start of a transaction and allows you to trace how a request propagates through your system.
🔹 `@CaptureSpan`:
Use this to track sub-operations within a transaction, such as database queries, HTTP calls, or specific business logic. It helps break down execution time and pinpoint performance bottlenecks inside a transaction.

📌 Best Practices:

✅ Apply @CaptureTransaction at the highest-level method handling a request.
✅ Use @CaptureSpan for key internal operations you want to monitor.
✅ Avoid excessive spans—instrument only critical code paths to reduce overhead.

By balancing these annotations effectively, you can get detailed insights into your app’s performance while keeping APM overhead minimal.

At Devoxx France 2015, David Pilato and Colin Surprenant, both deeply involved with Elasticsearch, shared a compelling narrative on leveraging the ELK stack (Elasticsearch, Logstash, Kibana) to transform raw data into actionable insights. David, a French customs service developer, and Colin, an Elasticsearch engineer, demonstrated how they met a marketing team’s data analysis needs in a fraction of the expected time.

Rapid Data Insights with ELK

David recounted a scenario where his marketing team needed to understand customer behavior and sentiment on Twitter. Using Logstash’s Twitter input plugin, he ingested data with simple configuration, leveraging API keys from Twitter’s developer portal. Elasticsearch indexed this data, enabling rapid searches, while Kibana visualized patterns, revealing customer trends in under 30 minutes.

This efficiency, David noted, showcases ELK’s power for real-time analytics.

Handling Dynamic Data Challenges

Colin addressed technical nuances, such as Logstash’s file input managing log rotations intelligently. Their setup handled dynamic file changes, ensuring continuous data flow. Q&A clarified plugin architectures resembling Ruby-like syntax, simplifying configuration. The duo’s approach turned complex data into clear visualizations, freeing time for open-source contributions.

This agility, Colin emphasized, accelerates business decision-making.

Community and Future Engagement

David and Colin invited attendees to a workshop and BOF session, promoting hands-on ELK exploration. Their open-source advocacy, evidenced by David’s RSS River plugin and Elasticsearch community leadership, underscored the stack’s accessibility. They encouraged leveraging GitHub resources for further learning.

This session inspires developers to harness ELK for data-driven solutions.

Links:

• David Pilato on Twitter/X
• David Pilato on GitHub
• Elasticsearch company website