Distributed Tracing

Status: Production Ready Last Updated: 2026-02-24

Description

Distributed tracing tracks requests as they flow through multiple services, providing end-to-end visibility into latency, errors, and dependencies. It is essential for debugging and performance optimization in microservice architectures.

When to Use

• Setting up trace propagation (W3C TraceContext, B3)
• Configuring Jaeger for trace collection and visualization
• Configuring Zipkin for trace collection and visualization
• Understanding span relationships (parent/child, follows-from)
• Choosing between head-based and tail-based sampling
• Planning trace storage and retention policies
• Correlating traces with logs and metrics

Key Concepts

Trace Propagation Formats

Span Relationships

Sampling Strategies

Best Practices

1. Use W3C TraceContext — it is the industry standard and supported by all major tracing backends
2. Propagate context through all transport layers — HTTP headers, message queue metadata, gRPC metadata
3. Set span names to describe the operation — POST /api/orders not handler
4. Record errors on spans — set span status to ERROR and call recordException
5. Use tail-based sampling for production — capture all error and high-latency traces while sampling normal traffic
6. Correlate traces with logs — inject trace_id and span_id into log records for cross-signal navigation
7. Set retention policies — keep error traces longer than normal traces to support incident investigation
8. Monitor trace pipeline health — alert on dropped spans, exporter failures, and collector queue depth

Common Patterns

Jaeger Setup (Docker Compose)

services:
  jaeger:
    image: jaegertracing/all-in-one:latest
    ports:
      - "16686:16686"   # Jaeger UI
      - "4317:4317"     # OTLP gRPC
      - "4318:4318"     # OTLP HTTP
    environment:
      COLLECTOR_OTLP_ENABLED: "true"

Zipkin Setup (Docker Compose)

services:
  zipkin:
    image: openzipkin/zipkin:latest
    ports:
      - "9411:9411"     # Zipkin UI + API

W3C TraceContext Header

traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01

Format: version-traceId-spanId-traceFlags

Correlating Traces with Logs

import { trace, context } from "@opentelemetry/api";

function getTraceContext() {
  const span = trace.getActiveSpan();
  if (!span) return {};
  const ctx = span.spanContext();
  return {
    trace_id: ctx.traceId,
    span_id: ctx.spanId,
    trace_flags: ctx.traceFlags,
  };
}

// Inject into structured logs
logger.info({ ...getTraceContext(), message: "Order processed", orderId });

Anti-Patterns

Last verified: 2026-02-24 | Skill version: 1.0.0