Worker Tuning

Guidance for configuring and optimizing Temporal worker performance.

Worker Options Overview

Key configuration parameters:

Basic Configuration

import (
    "go.temporal.io/sdk/worker"
)

func createWorker(c client.Client) worker.Worker {
    options := worker.Options{
        // Concurrency limits
        MaxConcurrentActivityExecutionSize:         100,
        MaxConcurrentWorkflowTaskExecutionSize:     100,
        MaxConcurrentLocalActivityExecutionSize:    100,

        // Rate limiting
        WorkerActivitiesPerSecond: 1000,

        // Pollers
        MaxConcurrentActivityTaskPollers:  5,
        MaxConcurrentWorkflowTaskPollers:  5,
    }

    return worker.New(c, "my-task-queue", options)
}

Tuning Guidelines

Activity Concurrency

Factors to consider:

• Resource usage per activity (CPU, memory, network)
• External service rate limits
• Database connection pools
• Available system resources

Calculation:

MaxConcurrentActivityExecutionSize = min(
    Available CPU cores × activities per core,
    External service rate limit,
    Database connection pool size,
    Available memory / memory per activity
)

Example configurations:

Workflow Task Concurrency

Workflow tasks are typically fast (< 1s). Higher concurrency is usually safe:

MaxConcurrentWorkflowTaskExecutionSize: 1000,  // Default is good

Reduce if workflows have expensive queries or local activities.

Poller Configuration

Pollers per task queue:

// For high-throughput queues
MaxConcurrentActivityTaskPollers:  10,
MaxConcurrentWorkflowTaskPollers:  10,

// For low-throughput queues
MaxConcurrentActivityTaskPollers:  2,
MaxConcurrentWorkflowTaskPollers:  2,

Rule of thumb: More pollers help with latency-sensitive workloads but consume more connections.

Task Queue Strategies

Dedicated Task Queues

Separate task queues for different activity types:

// CPU-intensive activities
cpuWorker := worker.New(c, "cpu-intensive", worker.Options{
    MaxConcurrentActivityExecutionSize: 4,  // Limited to CPU cores
})
cpuWorker.RegisterActivity(RenderVideo)
cpuWorker.RegisterActivity(ProcessImage)

// I/O-bound activities
ioWorker := worker.New(c, "io-bound", worker.Options{
    MaxConcurrentActivityExecutionSize: 100,  // Higher concurrency
})
ioWorker.RegisterActivity(FetchData)
ioWorker.RegisterActivity(SendEmail)

Priority Task Queues

// High priority - more workers
highPriorityWorker := worker.New(c, "orders-high-priority", worker.Options{
    MaxConcurrentActivityExecutionSize: 50,
})

// Normal priority
normalWorker := worker.New(c, "orders-normal", worker.Options{
    MaxConcurrentActivityExecutionSize: 20,
})

Sticky Execution

Workflows prefer returning to the same worker (sticky execution). Configure cache:

worker.Options{
    StickyScheduleToStartTimeout: 5 * time.Second,  // How long to wait for sticky worker
}

Resource Management

Memory Considerations

// Limit concurrent executions to control memory
worker.Options{
    MaxConcurrentActivityExecutionSize:         50,   // Each activity uses memory
    MaxConcurrentWorkflowTaskExecutionSize:     100,  // Workflow state in memory
}

Estimate memory:

• Workflow state: 1-10 KB per workflow
• Activity execution: Depends on payload
• Pollers: ~1 MB per poller

CPU Considerations

// For CPU-bound activities
runtime.GOMAXPROCS(4)  // Limit Go runtime to 4 cores

worker.Options{
    MaxConcurrentActivityExecutionSize: 4,  // Match GOMAXPROCS
}

Connection Pooling

Each poller maintains connections. Balance pollers vs. connections:

// Total connections ≈ pollers × 2
worker.Options{
    MaxConcurrentActivityTaskPollers:  5,   // ~10 connections
    MaxConcurrentWorkflowTaskPollers:  5,   // ~10 connections
}

Scaling Patterns

Horizontal Scaling

Scale workers across nodes:

# Kubernetes deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: temporal-worker
spec:
  replicas: 10  # Scale horizontally
  template:
    spec:
      containers:
      - name: worker
        resources:
          requests:
            cpu: "2"
            memory: "2Gi"
          limits:
            cpu: "4"
            memory: "4Gi"

Auto-Scaling

Scale based on task queue backlog:

# HPA based on custom metrics
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: worker-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: temporal-worker
  minReplicas: 2
  maxReplicas: 20
  metrics:
  - type: External
    external:
      metric:
        name: temporal_task_queue_depth
        selector:
          matchLabels:
            task_queue: "my-queue"
      target:
        type: Value
        value: "100"

Vertical Scaling

Increase resources and concurrency:

// Larger instance, higher concurrency
worker.Options{
    MaxConcurrentActivityExecutionSize:        200,
    MaxConcurrentWorkflowTaskExecutionSize:    500,
    MaxConcurrentActivityTaskPollers:          20,
    MaxConcurrentWorkflowTaskPollers:          20,
}

Monitoring Worker Performance

Key Metrics

# Activity execution rate
rate(temporal_worker_activity_task_execution_total[5m])

# Schedule-to-start latency (waiting for worker)
histogram_quantile(0.99, rate(temporal_activity_schedule_to_start_latency_bucket[5m]))

# Execution latency
histogram_quantile(0.99, rate(temporal_activity_execution_latency_bucket[5m]))

# Task queue backlog
temporal_task_queue_depth{task_queue="my-queue"}

Performance Indicators

Troubleshooting

High Schedule-to-Start Latency

Causes:

• Not enough workers
• Not enough pollers
• Task queue name mismatch

Solutions:

• Scale up worker replicas
• Increase MaxConcurrentActivityTaskPollers
• Verify task queue names match

Activities Timing Out

Causes:

• Concurrency too high for resources
• External service slow
• Activity too large

Solutions:

• Reduce MaxConcurrentActivityExecutionSize
• Add activity heartbeats
• Break into smaller activities

Worker Memory Issues

Causes:

• Too many concurrent executions
• Large payloads in memory
• Memory leaks

Solutions:

• Reduce concurrency limits
• Use smaller payloads
• Profile memory usage

Configuration Examples

Low-Latency Worker

worker.Options{
    MaxConcurrentActivityExecutionSize:    200,
    MaxConcurrentWorkflowTaskExecutionSize: 500,
    MaxConcurrentActivityTaskPollers:      20,
    MaxConcurrentWorkflowTaskPollers:      20,
}

Resource-Constrained Worker

worker.Options{
    MaxConcurrentActivityExecutionSize:    10,
    MaxConcurrentWorkflowTaskExecutionSize: 50,
    MaxConcurrentActivityTaskPollers:      2,
    MaxConcurrentWorkflowTaskPollers:      2,
}

Batch Processing Worker

worker.Options{
    MaxConcurrentActivityExecutionSize:    50,
    WorkerActivitiesPerSecond:             100,  // Rate limit
    MaxConcurrentActivityTaskPollers:      5,
}

Additional Resources

Reference Files

For detailed tuning patterns, consult:

• references/scaling-patterns.md - Advanced scaling strategies
• references/performance-testing.md - Load testing approaches