1) White-Box vs Black-Box Monitoring
White-box: metrics from inside the system (CPU, memory, app metrics). Example: http_server_requests_seconds from Spring Actuator.
Black-box: synthetic probes simulating user behavior (ping APIs, load test flows). Example: periodic “buy flow” test in production.
2) Tracing Distributed Transactions
Use OpenTelemetry to propagate context across microservices:
// Spring Boot setup
implementation "io.opentelemetry:opentelemetry-exporter-otlp:1.30.0"
// Annotate spans
Span span = tracer.spanBuilder("checkout").startSpan();
try (Scope scope = span.makeCurrent()) {
paymentService.charge(card);
inventoryService.reserve(item);
} finally {
span.end();
}These traces flow into Jaeger or Grafana Tempo to visualize bottlenecks across services.
3) Example Dashboard for a High-Value Service
- Availability: % successful requests (SLO vs actual).
- Latency: p95/p99 end-to-end response times.
- Error Rate: 4xx vs 5xx breakdown.
- Dependency Health: DB latency, cache hit ratio, downstream service SLOs.
- User metrics: active sessions, checkout success rate.