In web scraping and data ingestion, change is constant. External distributor APIs frequently change their JSON response schemas without warning. If your crawler code expects a certain dictionary structure and hits a schema change, it will throw an exception, causing the SQS message to fail and return to the queue.

Without proper handling, these messages will loop indefinitely, consuming compute cycles and spamming your error log. This is known as a poison pill message. The solution is to configure a Dead-Letter Queue (DLQ).

1. What is a Dead-Letter Queue?

A DLQ is a secondary SQS queue that automatically receives messages that failed processing a specified number of times (the maxReceiveCount). Isolating these poison pills prevents them from blocking the processing of valid messages in the main queue.

2. Configuring DLQ in CloudFormation

We declare the main queue, the DLQ, and the redrive policy specifying how many times a message should be retried before being sent to the DLQ:

{
  "Resources": {
    "MyDeadLetterQueue": {
      "Type": "AWS::SQS::Queue"
    },
    "MySourceQueue": {
      "Type": "AWS::SQS::Queue",
      "Properties": {
        "RedrivePolicy": {
          "deadLetterTargetArn": { "Fn::GetAtt": ["MyDeadLetterQueue", "Arn"] },
          "maxReceiveCount": 3
        }
      }
    }
  }
}

Once a message lands in the DLQ, SRE teams can set up CloudWatch alarms, download the malformed payloads, adjust the parser code, and redrive the messages back to the source queue.

Automated DLQ Redrive & Schema Quarantine Inspection

Attach CloudWatch alarms to ApproximateNumberOfMessagesVisible > 0 on every Dead-Letter Queue so malformed vendor payloads are inspected and replayed via SQS DLQ Redrive once parser schemas are updated.