Datadog APM
Distributed tracing, service maps, and performance analysis.
Requirements
Datadog Labs Pup should be installed. See Setup Pup if not.
Quick Start
pup auth login
pup apm services list
pup apm traces list --service api-gateway --duration 1h
Services
List Services
pup apm services list
pup apm services list --env production
Service Details
pup apm services get api-gateway --json
Service Map
# View dependencies
pup apm service-map --service api-gateway --json
Traces
Search Traces
# By service
pup apm traces list --service api-gateway --duration 1h
# Errors only
pup apm traces list --service api-gateway --status error
# Slow traces (>1s)
pup apm traces list --service api-gateway --min-duration 1000ms
# With specific tag
pup apm traces list --query "@http.url:/api/users"
Get Trace Detail
pup apm traces get <trace_id> --json
Key Metrics
| Metric | What It Measures |
|---|---|
trace.http.request.hits | Request count |
trace.http.request.duration | Latency |
trace.http.request.errors | Error count |
trace.http.request.apdex | User satisfaction |
⚠️ Trace Sampling
Not all traces are kept. Understand sampling:
| Mode | What's Kept |
|---|---|
| Head-based | Random % at start |
| Error/Slow | All errors, slow traces |
| Retention | What's indexed (billed) |
# Check retention filters
pup apm retention-filters list
Trace Retention Costs
| Retention | Cost |
|---|---|
| Indexed spans | $$$ per million |
| Ingested spans | $ per million |
Best practice: Only index what you need for search.
Service Level Objectives
Link APM to SLOs:
pup slos create \
--name "API Latency p99 < 200ms" \
--type metric \
--numerator "sum:trace.http.request.hits{service:api,@duration:<200000000}" \
--denominator "sum:trace.http.request.hits{service:api}" \
--target 99.0
Common Queries
| Goal | Query |
|---|---|
| Slowest endpoints | avg:trace.http.request.duration{*} by {resource_name} |
| Error rate | sum:trace.http.request.errors{*} / sum:trace.http.request.hits{*} |
| Throughput | sum:trace.http.request.hits{*}.as_rate() |
Troubleshooting
| Problem | Fix |
|---|---|
| No traces | Check ddtrace installed, DD_TRACE_ENABLED=true |
| Missing service | Verify DD_SERVICE env var |
| Traces not linked | Check trace headers propagated |
| High cardinality | Don't tag with user_id/request_id |