go-observabilitySafety 90Repository ShareFavorite skill

When to Use

• Adding structured logging to a service
• Implementing distributed tracing across microservices
• Collecting and exporting metrics
• Exporting observability data to object storage for long-term retention
• Setting up Gin/gRPC middleware for automatic instrumentation

Critical Patterns

Three Pillars

┌──────────────┐    ┌──────────────┐    ┌──────────────┐
│   Logging    │    │   Tracing    │    │   Metrics    │
│   (slog)     │    │   (OTel)     │    │   (OTel)     │
└──────┬───────┘    └──────┬───────┘    └──────┬───────┘
       │                   │                   │
       └───────────┬───────┴───────────────────┘
                   │
         ┌─────────▼──────────┐
         │  OTel Collector    │     ← Deployed as sidecar or standalone
         └─────────┬──────────┘
                   │
       ┌───────────┼───────────┐
       ▼           ▼           ▼
   Object Store  Jaeger    Prometheus
   (long-term)  (traces)   (metrics)

Structured Logging (slog)

Reference: assets/logger.go

Logging Convention

Reference: assets/logging_convention.go

Distributed Tracing (OpenTelemetry)

Reference: assets/tracing.go

Metrics (OpenTelemetry)

Reference: assets/metrics.go

Custom Business Metrics

Reference: assets/business_metrics.go

Gin Middleware

Reference: assets/gin_middleware.go

gRPC Interceptors

Reference: assets/grpc_interceptors.go

Export to Object Storage

Use the OpenTelemetry Collector to batch and export to object storage:

Reference: assets/otel-collector-config.yaml

Local Dev Stack (docker-compose)

Reference: assets/docker-compose-observability.yml

Commands

# OpenTelemetry SDK
go get go.opentelemetry.io/otel
go get go.opentelemetry.io/otel/sdk
go get go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc
go get go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc

# Gin instrumentation
go get go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin

# gRPC instrumentation
go get go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc

# View traces locally
open http://localhost:16686  # Jaeger UI

# View metrics
open http://localhost:9090   # Prometheus UI

Anti-Patterns