Data pipelines often depend on third-party APIs for data ingestion. However, these external APIs are prone to rate limits, network timeouts, and temporary service outages. If your pipeline crashes at the first failure, your downstream dashboards will display stale data.

To build a resilient ETL pipeline, you must implement Exponential Backoff Jitter and a Dead-Letter Queue (DLQ) for failed records.

1. Retrying with Exponential Backoff and Jitter

Retrying immediately when an API returns an error can overload the provider. Instead, we wait longer between each retry, adding a random jitter to prevent synchronized retries:

import time
import random

def call_with_retry(api_func, max_retries=5):
    for attempt in range(max_retries):
        try:
            return api_func()
        except Exception as e:
            if attempt == max_retries - 1:
                raise e
            # Calculate backoff with random jitter
            sleep_time = (2 ** attempt) + random.uniform(0, 1)
            time.sleep(sleep_time)

2. The Dead-Letter Queue (DLQ) Pattern

If a record fails to parse or download after all retry attempts, we write it to a Dead-Letter Queue (e.g., a S3 bucket or a database table failed_records) along with the error log. The main pipeline can then continue processing the remaining records, and developers can audit the DLQ later.

Implementing these patterns keeps your data flowing even during external service disruptions.

Decorated Exponential Backoff with Jitter in Python

Always combine exponential backoff with full random jitter so concurrent worker processes do not synchronize their retry waves ("thundering herd") when an upstream vendor API recovers from a brief outage. Read more in Freelance Python Programmer: Django, FastAPI & Asyncio.