Observability and monitoring

Observability is knowing what's happening inside your system without adding instrumentation. Logs, metrics, and traces are the three pillars. Without observability, production issues are invisible — you find out from user complaints. With it, you catch problems before users do.

Advanced

5 min read

The problem: production blindness

Your application is running in production. A Lambda function starts failing silently (processing messages but crashing before deleting them, so messages get retried infinitely). Users don't complain immediately because retries happen in the background. Days later, your SQS bill is 10× normal because millions of messages are being retried.

Without observability, you never know this happened until the bill shocked you. With observability, alarms fire the instant messages start backing up in the DLQ or the Lambda error rate spikes.

Three pillars: logs, metrics, traces

Logs: individual events in text form.

2025-08-23T14:32:15.123Z INFO   Processing message id=msg-123 from queue
2025-08-23T14:32:16.456Z ERROR  Database connection timeout: attempted 3 retries

Good for debugging ("why did this specific request fail?") but hard to aggregate at scale (millions of logs per second).

Metrics: quantitative data aggregated over time.

Metric	Value
Lambda invocations	1,000 / min
Lambda errors	50 / min (5% error rate)
DLQ depth	10,000 messages
Database connections	250 / 500 max

Good for dashboards and alerting. You can't say "customer X's request failed" from metrics, but you can say "error rate is 5% and rising."

Traces: request flow through distributed systems.

Good for understanding latency ("where did the time go?") and finding bottlenecks in microservices.

Logs: structured JSON over unstructured text

Unstructured logs are hard to parse at scale:

"Payment processed for customer alice in 234ms"

A human can read this. A log aggregation system can't reliably extract the customer name or latency.

Structured JSON is queryable:

{
  "timestamp": "2025-08-23T14:32:15.123Z",
  "level": "INFO",
  "message": "Payment processed",
  "customer_id": "alice",
  "duration_ms": 234,
  "status": "success"
}

Now you can query: "show me all payments for alice" or "show me payments slower than 1000ms."

Every log line should be valid JSON with key-value pairs. CloudWatch Logs Insights can then parse and query them.

import json
 
def handler(event, context):
    try:
        result = process_payment(event)
        print(json.dumps({
            "timestamp": datetime.utcnow().isoformat(),
            "level": "INFO",
            "message": "Payment processed",
            "customer_id": event['customer_id'],
            "amount": event['amount'],
            "duration_ms": (datetime.utcnow() - start_time).total_seconds() * 1000,
            "status": "success"
        }))
    except Exception as e:
        print(json.dumps({
            "level": "ERROR",
            "message": str(e),
            "customer_id": event['customer_id'],
            "status": "failed"
        }))

Metrics: CloudWatch dashboards and alarms

CloudWatch Metrics are time-series data points. For Lambda:

Invocations: total number of invocations
Errors: number of invocations that threw an exception
Duration: how long invocations took (in milliseconds)
Throttles: invocations rejected due to concurrency limits

Create a dashboard:

resource "aws_cloudwatch_dashboard" "api" {
  dashboard_name = "api-dashboard"
 
  dashboard_body = jsonencode({
    widgets = [
      {
        type = "metric"
        properties = {
          metrics = [
            ["AWS/Lambda", "Invocations", { stat = "Sum" }],
            [".", "Errors", { stat = "Sum" }],
            [".", "Duration", { stat = "Average" }]
          ]
          period = 300
          stat   = "Average"
          region = "us-east-1"
          title  = "Lambda Performance"
        }
      }
    ]
  })
}

Create alarms on metrics:

resource "aws_cloudwatch_metric_alarm" "lambda_errors" {
  alarm_name          = "lambda-high-error-rate"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 2
  metric_name         = "Errors"
  namespace           = "AWS/Lambda"
  period              = 60
  statistic           = "Sum"
  threshold           = 10  # more than 10 errors per minute
  alarm_actions       = [aws_sns_topic.alerts.arn]
}
 
resource "aws_cloudwatch_metric_alarm" "sqs_dlq_depth" {
  alarm_name          = "sqs-dlq-has-messages"
  comparison_operator = "GreaterThanOrEqualToThreshold"
  evaluation_periods  = 1
  metric_name         = "ApproximateNumberOfMessagesVisible"
  namespace           = "AWS/SQS"
  period              = 60
  statistic           = "Average"
  threshold           = 1  # alert if any message in DLQ
  dimensions = {
    QueueName = aws_sqs_queue.jobs_dlq.name
  }
  alarm_actions = [aws_sns_topic.critical_alerts.arn]
}

When thresholds are breached, SNS publishes to a topic (which can trigger PagerDuty, email, Slack, etc.), and you're paged.

Traces: X-Ray for distributed tracing

In a microservices architecture, a single user request touches many services. X-Ray traces the full path:

from aws_xray_sdk.core import xray_recorder
from aws_xray_sdk.core import patch_all
 
patch_all()  # Auto-instrument AWS SDK calls
 
@xray_recorder.capture('process_payment')
def process_payment(event):
    # Every AWS SDK call (S3, DynamoDB, etc.) is automatically traced
    customer = get_customer_data(event['customer_id'])  # traced
    charge_card(customer, event['amount'])  # traced
    send_confirmation(customer)  # traced
    return {'status': 'success'}

X-Ray records the full request tree with timing, so you see "API call took 500ms; 300ms was database, 150ms was S3, 50ms was processing." Bottlenecks become obvious.

Real observability: connecting logs, metrics, traces

Best practice: correlation IDs. Every request gets a unique ID that flows through all services:

import uuid
 
def handler(event, context):
    correlation_id = str(uuid.uuid4())
    
    # Pass to all downstream services
    log_and_record({
        "correlation_id": correlation_id,
        "message": "Processing request",
        "customer_id": event['customer_id']
    })
    
    result = call_downstream_service(
        event,
        headers={'X-Correlation-ID': correlation_id}
    )
    
    return result

Now you can query logs for a specific correlation_id, trace the request through all services, and correlate metrics to that exact request flow.

The observability readiness checklist

Before deploying to production:

✅ Structured JSON logs for all important events (request start, errors, external API calls, state changes)
✅ Metrics on key business functions (payments processed, users created, errors)
✅ DLQ alarms (depth > 0 should trigger an alert)
✅ Error rate alarms (Lambda errors, API 5xx responses)
✅ Latency alarms (if avg response time > normal, alert)
✅ Infrastructure limits (database connections, Lambda throttles, SQS visibility timeout / Lambda timeout ratio)
✅ Dashboards to visualize all of the above
✅ Correlation IDs to trace requests across services
✅ Log retention (how long to keep logs; balance costs vs. audit needs)

Without these, production is a black box and you'll discover problems from angry users.