Real production gotchas and lessons learned

Infrastructure as code and cloud platforms have edge cases and traps. These are real bugs and gotchas from production systems: the SQS visibility timeout mismatch, the Lambda singleton connection pool, the presigned URL checksum headers, the OIDC trust policy that was too broad.

Advanced

11 min read

Gotcha 1: SQS visibility timeout mismatched with consumer timeout

A Lambda function consumes messages from SQS. The Lambda timeout is set to 30 seconds (max time to process a message). The SQS queue visibility timeout is the default 30 seconds.

A message takes 25 seconds to process. The Lambda takes 5 more seconds to return and clean up. While cleanup is running, SQS assumes the message failed (visibility timeout has elapsed), makes it visible again, and another Lambda starts processing the same message. Result: duplicate work, two charges instead of one.

Fix: visibility timeout should be at least 6× the Lambda timeout, not equal to it. If Lambda timeout is 30 seconds, set SQS visibility timeout to 180+ seconds.

resource "aws_sqs_queue" "jobs" {
  visibility_timeout_seconds = 180  # 6 × 30s lambda timeout
}
 
resource "aws_lambda_event_source_mapping" "jobs" {
  function_name    = aws_lambda_function.process.arn
  event_source_arn = aws_sqs_queue.jobs.arn
}

Lesson: the consumer's timeout (Lambda) and the queue's visibility timeout (SQS) are not the same timeout — they have different meanings and need different values.

Gotcha 2: Lambda database connection exhaustion

A Lambda function opens a database connection at the start and closes it at the end. With 10 concurrent Lambdas and each opening a connection, the database can handle it. But at 1,000 concurrent Lambdas, the database hits its connection limit (default 100 for many databases) and subsequent Lambdas fail immediately with "too many connections."

Root cause: Lambda creates a fresh container (and thus a fresh connection) for each concurrent invocation, not a shared pool. At very high concurrency, this naturally hits the database's limit.

Fix: Lambda instances reuse connections between invocations (if code is structured correctly), so keep the pool small and lazily-initialize:

db_pool = None
 
def get_pool():
    global db_pool
    if not db_pool:
        db_pool = psycopg2.pool.SimpleConnectionPool(1, 2, ...)  # max 2 per Lambda instance
    return db_pool
 
def handler(event, context):
    pool = get_pool()
    conn = pool.getconn()
    try:
        # use conn
        pass
    finally:
        pool.putconn(conn)  # return to pool, don't close

Even better: RDS Proxy (AWS's connection pooler) sits between Lambda and the database, accepting connections from unlimited Lambdas and pooling them internally.

Lesson: serverless compute scales concurrency easily; stateful resources (databases) do not. Plan for extreme concurrency.

Gotcha 3: Presigned URL checksum validation breaking uploads

After upgrading AWS SDK v3, presigned PUT URLs for S3 uploads start failing with "SignatureDoesNotMatch" errors. The SDK now automatically adds checksum headers to presigned requests; browsers can't compute checksums client-side, so the signature becomes invalid.

# Old code — works fine
url = s3.generate_presigned_url('put_object', Params={...})
 
# Browser upload:
# PUT url with file body → AWS validates signature → 403 SignatureDoesNotMatch

Fix: explicitly disable automatic checksum validation for presigned URLs:

s3 = boto3.client('s3', config=Config(
    s3={'payload_signing_enabled': False}
))
url = s3.generate_presigned_url('put_object', Params={...})

Lesson: SDK behavior changes across versions; test major upgrades against real use cases (direct browser uploads, not just backend-to-S3 operations).

Gotcha 4: GitHub Actions OIDC trust policy scoped too broadly

A GitHub Actions workflow assumes an AWS IAM role to deploy. The trust policy says:

Condition = {
  StringLike = {
    "token.actions.githubusercontent.com:sub" = "repo:alice/my-app:*"
  }
}

This allows any branch in the repository to assume the role. Later, a security researcher finds a bug, files a PR with a fix, and also adds their own CI step that deploys malicious code to production using the role. The trust policy is too permissive.

Fix: scope to specific branches or environments:

Condition = {
  StringLike = {
    "token.actions.githubusercontent.com:sub" = "repo:alice/my-app:ref:refs/heads/main"
  }
}

Now only the main branch can assume the role; PRs and other branches cannot.

Lesson: "works" and "secure" are different; start with the broadest trust policy that works for development, then tighten it after validating the setup.

Gotcha 5: Infrastructure code and docs describing different systems

A repository's README.md describes an AWS ECS Fargate + RDS + Lambda architecture. The Terraform code exists (ecs.tf, rds.tf, lambda.tf). But the actual deployment went to a different platform (Vercel, Heroku) for cost/simplicity reasons. Months later, a new engineer reads the docs, assumes the infrastructure described is real, and makes changes based on the wrong architecture.

Root cause: docs and code were written, then the actual deployment changed, but the docs were never updated.

Fix: either keep docs and code synchronized (difficult) or minimize aspiration docs. Better: have a single source of truth (e.g., terraform state list output, cloud console screenshots) and reference it in docs with a timestamp. Mark aspirational architecture clearly: "We plan to use ECS in Q3; current deployment is Vercel."

Lesson: treat infrastructure documentation as code — if it diverges from reality, it misleads. Either automate sync or mark docs with publication dates/validity caveats.

Gotcha 6: Partial batch failure without explicit handling

A Lambda receives 10 messages from SQS. It processes them in a loop, deleting each after success. On message #7, an exception occurs. The Lambda crashes without returning; SQS sees the batch as "incomplete," keeps all 10 messages (resetting visibility), and schedules them for retry.

Result: messages #1-6 are processed twice (or more), causing duplicates.

Fix: handle failures explicitly and return only failed message IDs so SQS knows which ones to retry:

response = sqs.receive_message(MaxNumberOfMessages=10, ...)
failed = []
 
for msg in response['Messages']:
    try:
        process(msg)
        sqs.delete_message(...)
    except Exception as e:
        failed.append(msg['ReceiptHandle'])
        logger.error(f"Failed {msg['MessageId']}: {e}")
 
# Let failed ones auto-retry via visibility timeout

Lesson: "at-least-once" delivery means you must be prepared for duplicates and make processing idempotent.

Gotcha 7: Load balancer idle timeout cutting long SSE streams

A Lambda or backend service streams responses via Server-Sent Events (SSE). A client connects and receives updates. After 60 seconds of inactivity on the connection (no data sent), the load balancer in front of the service kills the connection with a reset, thinking it's dead.

The application code is fine; the infrastructure is the culprit.

Fix: raise the load balancer's idle timeout to accommodate long-lived connections:

resource "aws_lb_target_group" "api" {
  deregistration_delay = 300  # graceful drain on shutdown
 
  stickiness {
    type = "lb_cookie"
  }
}
 
# For ALB specifically, set target group attribute
resource "aws_lb_target_group_attachment" "..." {
  # ...
}
 
resource "aws_lb_listener" "api" {
  default_action {
    target_group_arn = aws_lb_target_group.api.arn
  }
}
 
# ALB idle timeout is in the load balancer itself
resource "aws_lb" "api" {
  idle_timeout = 120  # 120 seconds, can be raised to ~3600 for SSE
}

Lesson: infrastructure configurations affect application behavior in non-obvious ways; monitor connection resets and idleness metrics.

Gotcha 8: a serverless platform's plan tier silently rejects your cron schedule

A background job (send appointment reminders) was scheduled to run every 5 minutes:

{
  "crons": [{ "path": "/api/cron/reminders", "schedule": "*/5 * * * *" }]
}

The deploy succeeded. No error, no warning, no failed build. The job simply never ran — not once, from the day the file was introduced. The actual cause took real digging to find, because the first, more familiar suspects (a broken webhook, a misconfigured route, a bad git merge) were all ruled out one by one before landing on the true cause: the hosting platform's free/hobby tier silently caps cron jobs at once per day. A schedule requesting anything more frequent isn't rejected at deploy time — it just never fires, as if the feature quietly doesn't exist at that tier.

Fix: match the schedule to what the tier actually allows —

{
  "crons": [{ "path": "/api/cron/reminders", "schedule": "0 7 * * *" }]
}

— and design the job itself to not depend on tight, frequent ticks (see the next gotcha).

Lesson: a paid-tier feature can be silently unavailable on a free tier, with zero error surfaced anywhere in the deploy or runtime logs. Read the specific platform's plan-tier limits for the exact feature being used — "it deployed cleanly" is not evidence that it actually runs.

Gotcha 9: a "fire it and move on" job stamps success before checking whether anything actually happened

A reminder job finds appointments happening soon, sends a WhatsApp reminder for each, and marks them done:

# Wrong: marks the reminder "sent" regardless of whether the send worked
for appt in due_appointments:
    send_whatsapp_reminder(appt)          # returns a result, but it's ignored
    appt.reminder_sent_at = now()          # stamped unconditionally
    appt.save()

send_whatsapp_reminder was written, correctly, to never raise — a third-party messaging integration being briefly unconfigured or down should never take down a booking system. But that same "never throw" design has a sharp edge here: the calling code has no exception to catch, so it has to actually check the returned result — and this call site didn't. The moment the messaging integration was unconfigured or temporarily failing, every due reminder got permanently stamped reminder_sent_at, and that reminder was gone for good — the next sweep skips it, because as far as the data is concerned, it already went out.

# Right: only stamp success on an actual success signal
for appt in due_appointments:
    result = send_whatsapp_reminder(appt)
    if result.success:
        appt.reminder_sent_at = now()
        appt.save()
    # else: leave it unstamped — the next sweep will pick it up and retry

This pairs with a second, complementary design choice worth calling out: instead of a fixed catch-up window ("check appointments due in the next 5 minutes," which a missed cron tick or a temporary platform outage can cause to skip entirely), the sweep re-evaluates "which appointments are due and still unstamped" on every run. A missed tick, a Hobby-tier cron limited to once a day (the previous gotcha), or a brief outage doesn't lose anything — the next run just picks up whatever's still outstanding. Combine that with isolating each row's failure (one bad appointment's exception doesn't abort the rest of the batch — the same principle as Gotcha 6's partial-batch handling) and the whole job becomes self-healing: it tolerates being run late, run rarely, or interrupted mid-batch, and still eventually converges on "every due reminder actually sent."

Lesson: "never throw" and "always succeeds" are not the same claim. A function designed to fail safely still needs its caller to check whether it actually worked — and a scheduled job's correctness should be evaluated by asking "what happens if a run is missed, delayed, or partially fails," not just "does it work when everything goes right."

Gotcha 10: a trailing slash in one config value broke links across the whole app

A base URL was configured with a trailing slash — https://app.example.com/ instead of https://app.example.com — and every place in the codebase that built a link did the obvious thing:

link = f"{settings.SITE_BASE_URL}/booking/{token}"
# with the trailing-slash config value, this produces:
# https://app.example.com//booking/abc123   <- double slash

The double slash surfaced inconsistently — some downstream consumers (browsers, most link previews) silently normalized it away, others (strict URL parsers, some webhook validators) didn't, so the bug looked intermittent and consumer-specific rather than what it actually was: one bad value at the source, multiplied across every call site that used it.

Fix: sanitize once, where the value enters the system, not at each place it's used:

SITE_BASE_URL = os.environ["SITE_BASE_URL"].rstrip("/")

Lesson: when a value is consumed in many places, validating or sanitizing it at every call site means every call site can independently get it wrong (and did, here, until the pattern was noticed). Fixing it once at the boundary — where the value first enters the application — makes it correct everywhere downstream by construction, instead of by discipline.

Summary: the pattern

All these gotchas have the same shape: stated behavior (docs, defaults, initial assumptions) diverges from actual behavior under stress/edge cases.

SQS visibility timeout: thought it was a single timeout; actually two independent timings.
Lambda concurrency: thought horizontal scaling was free; actually bounded by dependent resources.
SDK upgrades: thought new version was backward compatible; actually changed request signing.
Trust policies: thought "works" and "secure" were the same; actually requires deliberate narrowing.
Docs vs code: thought docs described the system; actually described aspirations.
Error handling: thought "process a batch" was a single atomic unit; actually needs explicit error handling per item.
Infrastructure: thought the service code was isolated; actually depends on load balancer settings.
Plan-tier limits: thought a cron schedule either works or fails loudly; actually can be silently capped by the hosting tier.
Fire-and-forget jobs: thought "never throws" meant "always succeeds"; actually still needs the caller to check the result.
Shared config values: thought sanitizing at each use site was enough; actually one bad value at the source beats validation scattered everywhere it's consumed.

Prevention: test at realistic scale, read release notes, measure actual behavior (not assumed), keep docs and code synchronized, assume defaults are wrong, and design scheduled/background jobs to be self-healing rather than dependent on every tick firing exactly on time.