Queues and async job processing

Why synchronous backends fail under load: a request handler that does everything (database write, send email, generate PDF) blocks the response. Queues decouple work — write to a queue, respond immediately, process the work later. SQS, reliability guarantees, and the visibility-timeout trap.

Intermediate

4 min read

The problem: synchronous work blocks responses

A user uploads a document. Your backend:

Validates the file
Stores it in S3
Extracts text (slow)
Writes metadata to the database
Sends a confirmation email
Returns a response

All of this happens inside the HTTP request handler. If extracting text takes 5 seconds and 100 users upload simultaneously, you need to handle 100 concurrent requests waiting 5 seconds each — a massive resource drain and a terrible user experience (the response doesn't come back until all work is done).

Queues fix this: validate the upload, store it, add a message to a queue saying "process this file," and respond immediately. A background worker picks up messages from the queue later and does the slow work asynchronously.

How queues work: producers and consumers

A queue (SQS, RabbitMQ, Kafka) is a staging area for work. A producer (your HTTP handler) pushes messages into the queue; a consumer (a background worker) pulls messages and processes them.

The producer doesn't wait for the consumer — it just enqueues and moves on. The consumer processes at its own pace. If the consumer crashes mid-process, the message re-appears in the queue (via a mechanism called visibility timeout, covered below) and another consumer picks it up. Work gets done reliably even if individual workers fail.

SQS: AWS's managed queue service

SQS (Simple Queue Service) is AWS's queue service — no infrastructure to manage, scales automatically, pay only for messages.

Produce a message:

import boto3
 
sqs = boto3.client('sqs')
sqs.send_message(
    QueueUrl='https://sqs.us-east-1.amazonaws.com/123456789/my-jobs',
    MessageBody=json.dumps({
        'file_id': 'abc123',
        'action': 'process_document'
    })
)

Consume messages:

while True:
    response = sqs.receive_message(
        QueueUrl='...',
        MaxNumberOfMessages=10,  # receive up to 10 at once
        WaitTimeSeconds=20       # long-poll: wait up to 20s for messages
    )
    
    for message in response.get('Messages', []):
        job = json.loads(message['Body'])
        try:
            process_document(job['file_id'])
            # Delete after successful processing
            sqs.delete_message(QueueUrl='...', ReceiptHandle=message['ReceiptHandle'])
        except Exception as e:
            # Don't delete — message reappears after visibility timeout
            logger.error(f"Failed: {e}")

Visibility timeout: the reliability trick

When a consumer receives a message, SQS hides it for a configurable period (the visibility timeout, default 30 seconds). If the consumer crashes before deleting the message, it reappears after the timeout and another consumer picks it up. This is how SQS guarantees at-least-once delivery even if workers fail.

But there's a trap: if your processing takes 2 minutes and visibility timeout is 30 seconds, the message becomes visible again while the original worker is still processing it. Two workers process the same message, creating duplicate work.

The fix: set visibility timeout to at least 6× your expected processing time. If processing takes 10 seconds, set visibility timeout to 60 seconds.

sqs.receive_message(
    QueueUrl='...',
    VisibilityTimeout=60  # message stays hidden for 60 seconds
)

Dead-letter queues: the failure escape hatch

If a message fails repeatedly (consumer keeps crashing on it), it shouldn't loop forever. Instead, move it to a dead-letter queue (DLQ) after N retries.

resource "aws_sqs_queue" "jobs" {
  name                      = "my-jobs"
  visibility_timeout_seconds = 60
  message_retention_seconds = 86400
 
  redrive_policy = jsonencode({
    deadLetterTargetArn = aws_sqs_queue.jobs_dlq.arn
    maxReceiveCount     = 3  # after 3 failed attempts, send to DLQ
  })
}
 
resource "aws_sqs_queue" "jobs_dlq" {
  name                      = "my-jobs-dlq"
  message_retention_seconds = 1209600  # keep DLQ messages longer for forensics
}

When a message is received 3 times without being deleted, SQS automatically moves it to the DLQ. A separate monitoring process alerts on DLQ depth (if depth > 0, something is broken).

Partial batch failure: a modern pattern

When a consumer receives a batch of 10 messages and processes 9 successfully but 1 fails, what happens?

Old way: if you fail on any message, the whole batch is retried, causing duplicate processing of the 9 successful messages.

New way (recommended): return exactly which message IDs failed:

response = sqs.receive_message(MaxNumberOfMessages=10, ...)
 
failed_ids = []
for message in response['Messages']:
    try:
        process_document(json.loads(message['Body']))
        sqs.delete_message(...)  # success
    except Exception:
        failed_ids.append(message['ReceiptHandle'])  # track failure
 
# The failed message stays in queue (visibility timeout resets)
# The successful messages are gone

This ensures that only the actually-failed message retries, not the whole batch.

Trade-offs and when queues aren't the answer

Queues add latency — the user's action doesn't complete immediately, it completes "eventually." For uploads that need to be processed and verified before the user can proceed, a queue works great (user sees "processing" status). For something that needs to complete before returning a response (a payment charge, a database lookup), use a queue for side effects only (sending a confirmation email) but do the critical work synchronously.

Queues also add complexity — an additional service to monitor, messages to track, failure modes to understand. For small projects with low traffic, the extra infrastructure might not be worth it.