Jonathan Lalou's Blog

1) Coaching Team Members Through Debugging

When junior engineers struggle with incidents, I walk them through the scientific method of debugging:

1. Reproduce the problem.
2. Collect evidence (logs, metrics, traces).
3. Form a hypothesis.
4. Test, measure, refine.

For example, in a memory leak case, I let a junior take the heap dump and explain findings, stepping in only to validate conclusions.

2) Introducing SRE Practices to New Teams

In teams without SRE culture, I start small:

• Define a single SLO for a critical endpoint.
• Introduce a burn-rate alert tied to that SLO.
• Run a blameless postmortem after the first incident.

This creates buy-in without overwhelming the team with jargon.

3) Prioritizing and Delegating in High-Pressure Situations

During outages, prioritization is key:

• Delegate evidence gathering (thread dumps, logs) to one engineer.
• Keep communication flowing with stakeholders (status every 15 minutes).
• Focus leadership on mitigation and rollback decisions.

After stabilization, I lead the postmortem, ensuring learnings feed back into automation, monitoring, and runbooks.

1) White-Box vs Black-Box Monitoring

White-box: metrics from inside the system (CPU, memory, app metrics). Example: http_server_requests_seconds from Spring Actuator.

Black-box: synthetic probes simulating user behavior (ping APIs, load test flows). Example: periodic “buy flow” test in production.

2) Tracing Distributed Transactions

Use OpenTelemetry to propagate context across microservices:

// Spring Boot setup
implementation "io.opentelemetry:opentelemetry-exporter-otlp:1.30.0"

// Annotate spans
Span span = tracer.spanBuilder("checkout").startSpan();
try (Scope scope = span.makeCurrent()) {
    paymentService.charge(card);
    inventoryService.reserve(item);
} finally {
    span.end();
}

These traces flow into Jaeger or Grafana Tempo to visualize bottlenecks across services.

3) Example Dashboard for a High-Value Service

• Availability: % successful requests (SLO vs actual).
• Latency: p95/p99 end-to-end response times.
• Error Rate: 4xx vs 5xx breakdown.
• Dependency Health: DB latency, cache hit ratio, downstream service SLOs.
• User metrics: active sessions, checkout success rate.

1) Approaching Memory Leaks

Memory leaks in Java often manifest as OutOfMemoryError exceptions or rising heap usage visible in monitoring dashboards. My approach:

1. Reproduce in staging: Apply the same traffic profile (e.g., JMeter load test).
2. Collect a heap dump:

   jmap -dump:format=b,file=heap.hprof <PID>

3. Analyze with tools: Eclipse MAT, VisualVM, or YourKit to detect uncollected references.
4. Fix common causes:
   • Unclosed streams or ResultSets.
   • Static collections holding references.
   • Caches without eviction policies (e.g., replace HashMap with Caffeine).

2) Profiling and Fixing High CPU Usage

High CPU can stem from tight loops, inefficient queries, or excessive logging.

• Step 1: Sample threads

  jstack <PID> > thread-dump.txt

  Identify “hot” threads consuming CPU.

• Step 2: Profile with async profilers like async-profiler or Java Flight Recorder.

  java -XX:StartFlightRecording=duration=60s,filename=recording.jfr -jar app.jar

• Step 3: Refactor:
  • Replace String concatenation in loops with StringBuilder.
  • Optimize regex (use Pattern reuse instead of String.matches()).
  • Review logging level (DEBUG inside loops is expensive).

3) Tuning GC for Low-Latency Services

Garbage collection (GC) can cause pauses. For trading, gaming, or API services, tuning matters:

• Choose the right collector:
  • G1GC for balanced throughput and latency (default in recent JDKs).
  • ZGC or Shenandoah for ultra-low latency workloads (<10ms pauses).
• Sample configs:

  -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:+ParallelRefProcEnabled

• Monitor GC logs with GC Toolkit or Grafana dashboards.

4) Handling Database Bottlenecks

Spring apps often hit bottlenecks in DB queries rather than CPU.

1. Enable SQL logging: in application.properties

   spring.jpa.show-sql=true

2. Profile queries: Use p6spy or database AWR reports.
3. Fixes:
   • Add missing indexes (EXPLAIN ANALYZE is your friend).
   • Batch inserts (saveAll() in Spring Data with hibernate.jdbc.batch_size).
   • Introduce caching (Spring Cache, Redis) for hot reads.
   • Use connection pools like HikariCP with tuned settings:

     spring.datasource.hikari.maximum-pool-size=30