Handling LLM API 429 Rate-Limit Errors: Backoff, Headers, and Fallback Chains That Work

Why a 429 Is Different From Every Other Error

A 429 is the provider telling you your credentials are fine and your request was refused on volume grounds — not a malformed request, not a bad key. That matters because the fix is time and pacing, not a code change. OpenAI’s documentation is explicit that a Retry-After header “does not mean that quota, billing, or other errors that require user action can be resolved by retrying.” In other words, there are several distinct 429s hiding behind one status code, and treating them all the same is how you burn your retry budget on nothing.

The error splits cleanly into two families:

• 4xx client errors (other than 429 and 408) — your fault. Bad JSON, a too-long context window, an expired key. Retrying is pointless; fix the request.
• 429 and 5xx errors — the provider’s constraint. These are retryable, but how you retry depends on which flavor of 429 you actually hit.

Within the 429 family itself there are at least four situations worth distinguishing, because the correct reaction differs:

• Per-minute request or token cap — you sent too fast. Waiting for the documented window fixes it.
• Spend or credit cap — the account is out of money or hit an org/project spend limit. Waiting does not fix it; retrying burns quota on nothing.
• Acceleration / ramp limit — your traffic spiked faster than the provider allows, even under your nominal caps. The fix is a gradual ramp, not a retry storm.
• Upstream gateway throttle — an aggregator relaying a provider’s own limit. Retrying the same route makes it worse; fail over instead.

OpenAI distinguishes the spend-cap case with error codes like credit_balance_exhausted or organization_spend_limit_exceeded, while Anthropic surfaces an enforced_spend_limit_reached detail code with no retry-after header. Learn to read the error body, not just the status line.

Read the Response Headers Before You Retry

The single most useful thing a 429 response gives you is in its headers, and most people skip it. When a retry-after header is present, it tells you exactly how many seconds to wait before the next attempt is likely to succeed. Retrying sooner just gets you another 429.

Each provider structures this differently, which is worth internalizing if you use more than one:

Anthropic’s token-bucket model is a subtle but important detail: capacity replenishes continuously rather than resetting on a fixed clock boundary. That is why a burst can trip a 429 seconds after a previous burst succeeded, and why the reset value is a timestamp, not “wait until the top of the minute.” Anthropic is also explicit that “a rate of 60 requests per minute might be enforced as 1 request per second” — meaning short bursts can exceed a limit that a naive per-minute counter thinks you’re under.

Here is the key resilience rule: a header set belongs to whichever model actually served the request, not the endpoint you called. If you route through an OpenAI-compatible aggregator, some routes return a full x-ratelimit-* set and others return none at all. Your parser must degrade gracefully to plain backoff when a header is missing, rather than throwing a KeyError.

Implement Exponential Backoff With Jitter (Not Fixed Sleeps)

The reason fixed delays fail is the thundering herd problem. Imagine a thousand clients all hit a limit at the same instant. If every one of them waits exactly one second and retries, they all collide again, get throttled again, and repeat the cycle in lockstep forever. Exponential backoff spaces those retries over a growing interval; jitter adds randomness so clients stop synchronizing.

Here is a minimal, correct retry loop that honors retry-after when present and falls back to jittered backoff when it is not:

import random
import time

import httpx

RETRYABLE = (429, 500, 502, 503, 504)

def call_with_backoff(url: str, payload: dict, headers: dict,
                      max_retries: int = 5) -> httpx.Response:
    for attempt in range(max_retries):
        resp = httpx.post(url, json=payload, headers=headers)
        if resp.status_code not in RETRYABLE:
            return resp

        # Prefer the server's explicit instruction.
        retry_after = resp.headers.get("retry-after")
        if retry_after:
            try:
                wait = float(retry_after)
                time.sleep(min(wait, 120.0))
                continue
            except ValueError:
                pass  # fall through to backoff

        # No usable header -> jittered exponential backoff.
        base = min(2 ** attempt, 16.0)   # 1, 2, 4, 8, 16s cap
        wait = random.uniform(0, base)   # full jitter
        time.sleep(wait)

    raise RuntimeError("max retries exhausted")

Three details in that loop matter more than they look:

• Cap the sleep. Most providers will not make you wait minutes for a per-minute cap, but a rogue or malformed retry-after value should never stall your worker for an hour. Clamp it.
• Full jitter beats “equal jitter.” random.uniform(0, base) scatters clients across the entire window, which breaks synchronization better than a small random bump around a fixed value.
• Do not stack a second retry layer on top of the SDK’s. The official OpenAI and Anthropic clients already retry transient failures a couple of times and already honor retry-after. If you wrap every call in your own loop and leave the SDK retries on, your “3 attempts” silently become nine, and the waits compound. Either own the loop yourself, or raise the SDK’s max_retries and let it manage the pacing — not both.

Distinguish What “Waiting” Will and Won’t Fix

The most common production bug around 429s is a retry loop hammering an exhausted credit balance. The request fails, you wait ten seconds, it fails again, you wait twenty seconds — and you loop until your alerting notices, never actually recovering. The signal is in the body, not the status line.

A quick decision tree to run on every 429:

• Is there a retry-after header? If yes, honor it, capped. If the error body names a spend/credit cap, skip straight to “fix billing.”
• Does the error body name billing? Look for credit_balance_exhausted, organization_spend_limit_exceeded, project_spend_limit_exceeded, or Anthropic’s enforced_spend_limit_reached. Do not retry — log it, alert a human, and stop.
• Did traffic just spike? If you ramped requests faster than your baseline, back off to your previous rate and grow it gradually (OpenAI suggests no more than ~50% increase every 15 minutes once near the TPM ceiling).
• Is the limit the provider’s, not yours? Anthropic’s 529 overloaded_error and a provider-side 503 mean their servers are at capacity, not that you misbehaved. Back off calmly, or fail over — retrying the same route harder makes it worse.

Keeping those four cases separate is what separates a graceful degradation from a cascading failure.

Build a Fallback Chain for When Retries Aren’t Enough

Retrying the same provider only helps for transient limits. For extended outages or a hard spike, you need somewhere else to send the request. A fallback chain routes to a secondary model or provider once the local retry budget is spent, and it is the single highest-leverage reliability change you can make to an LLM-backed service.

The elegant part: if you route through an OpenAI-compatible endpoint, failing over is often just a model string change, not a second integration. Here is a sketch that tries a primary model, then a cheap fallback, then gives up cleanly:

ROUTES = [
    {"url": "https://api.openai.com/v1/chat/completions",
     "model": "gpt-4o", "key": "PRIMARY"},
    {"url": "https://api.anthropic.com/v1/messages",
     "model": "claude-sonnet-4-5", "key": "FALLBACK"},
]

def call_with_fallback(payload: dict) -> dict:
    last_err = None
    for route in ROUTES:
        try:
            resp = call_with_backoff(route["url"], payload,
                                     auth_headers(route["key"]),
                                     max_retries=3)
            if resp.status_code == 200:
                return resp.json()
            last_err = f"{route['model']} -> {resp.status_code}"
        except RuntimeError as e:
            last_err = str(e)
            continue  # retries exhausted on this route; move on
    raise RuntimeError(f"all routes failed: {last_err}")

Two design notes worth calling out:

• Bound each route’s retries. Don’t let a single provider eat the entire global retry budget while a healthy fallback sits idle. Give each route a small max_retries and move on.
• A circuit breaker beats retrying harder. Once a route’s error rate crosses a threshold (say 50% over 60 seconds), open the circuit and route around it entirely for a cooldown period, letting a few probe requests through afterward. This stops you from piling latency onto a provider that is already down and gives it room to recover.

Verify Your Fix Under Load

The mistake people make after wiring this up is never testing it under the failure mode it was built for. A retry loop that looks correct in review can still synchronize every worker, exhaust a thread pool, or loop forever on a spend-cap 429. A few concrete checks before you call it done:

• Simulate a 429. Point your client at a stub that returns 429 with a retry-after: 2 header and confirm it waits (roughly) two seconds and succeeds on retry, rather than spinning.
• Simulate a headerless 429. Confirm it falls back to jittered exponential backoff and stops after max_retries, rather than looping forever.
• Simulate a billing 429. Return a 429 whose body names credit_balance_exhausted and confirm the client does not retry — it logs and aborts.
• Watch the logs. Track status codes, latency, token counts, and retry counts per route. A 429 rate above ~5 per minute, or fallback usage above ~50%, is your early warning that a primary provider is in trouble and you should investigate before it becomes an outage.

A small but revealing final detail: spend five minutes checking whether your SDK already retries for you. The installed OpenAI client defaults to two retries, honors retry-after-ms first and then retry-after, caps a server-directed delay around two minutes, and refuses to retry when the server asks for longer than that. If a 429 is surfacing in your own code, it has often already failed three times inside the SDK — which is a hint that the limit itself, not your retry logic, is the problem worth fixing.

Conclusion

429 rate-limit errors are not anomalies; they are the environment telling you how fast you are allowed to go. The reliable response is a layered one: read and honor retry-after, fall back to jittered exponential backoff when it is absent, refuse to retry billing-cap 429s, and route around a struggling provider with a fallback chain. None of this is glamorous, but it is the difference between a service that degrades gracefully under load and one that takes a short throttle and amplifies it into a full outage.

For more on the operations side of running local and hosted AI reliably, see our guide on ComfyUI automation via WebSocket events and history polling and our post on building a Python harness for the ComfyUI API.

Primary References

• OpenAI API error codes — rate limits and retry guidance
• Anthropic rate limits — headers and spend caps