Graceful Degradation for AI-Enhanced Applications: When the Model Is Down

An AI feature that worked perfectly in development fails in production the first time the model provider’s API returns a 503. Then it fails when the rate limit kicks in at peak traffic. Then it fails when the model responds with “I cannot answer that question” instead of the expected output. Each failure mode is different, but the result is the same: your user sees an error, a spinner that never resolves, or worse — stale nonsense presented as an AI-generated insight.

Graceful degradation is the practice of reducing functionality instead of breaking the page when an AI dependency fails. It is not optional for production applications. This post covers the failure landscape, the circuit-breaker pattern adapted for AI calls, model fallback chains, caching strategies, and the UX patterns that tell the user something is reduced without leaving them stranded.

The Failure Landscape Is Not Binary

Model API outages are rarely a clean “up or down.” Production incidents fall into a spectrum of failure modes:

Hard outages (HTTP 503/504). The provider is overloaded or undergoing maintenance. Your call fails immediately. These are the easiest to detect and handle — the error code is unambiguous.

Rate limits (HTTP 429). The provider is telling you to slow down. Depending on your plan and burst capacity, rate limits may kick in at 100 requests per minute or 10,000. The response includes a Retry-After header, but in a real-time application, waiting five seconds to retry may already be too long.

Latency spikes. The model responds eventually, but a 300 ms p50 becomes a 12-second p99 during an incident. Your application times out its HTTP client before the response arrives, leaving an orphaned request consuming provider credits and returning nothing.

Content-filter rejections. The provider’s safety classifier flags the input or output as violating policy. The model returns “I am unable to generate a response.” For applications processing user-generated content, this happens regularly and unpredictably.

Silent degradation (nonsense output). The model returns a response that parses as valid JSON or text but is factually wrong, incoherent, or hallucinated. There is no error code. Detection requires a separate quality-assessment step.

A resilient AI application must handle all six modes, not just the hard outages.

Circuit-Breaker Pattern for AI Calls

Standard circuit breakers (closed → open → half-open → closed) translate well to AI API calls with a few modifications specific to model behaviour.

Implementation outline. Wrap each model API call in a circuit breaker that tracks consecutive failures. A failure includes HTTP 5xx, HTTP 429 beyond retry budget, timeouts, and content-filter rejections. Set the threshold at 5 consecutive failures for a 60-second open window. During the open window, the breaker short-circuits to the fallback immediately without making the API call.

Half-open probes. After the open window expires, allow one test request through. Use a synthetic probe — a simple deterministic query like “Return the word ‘okay’” — rather than a real user request. If the probe succeeds, close the circuit and resume normal traffic. If it fails, reset the open window.

Per-endpoint breakers. Each model endpoint (gpt-4o-mini, claude-3-haiku, your embedding service) should have its own circuit breaker. A rate limit on the cheap model should not degrade the expensive model’s circuit.

from pybreaker import CircuitBreaker

llm_breaker = CircuitBreaker(
    fail_max=5,
    reset_timeout=60,
    exclude=[HTTPError(429)]  # Don't count rate limits toward open threshold
)

@llm_breaker
def call_primary_model(prompt):
    return provider.chat(prompt)

The exclude clause is important: rate limits should not trigger circuit-open state because they are transient, usually resolve within seconds, and counting them toward the breaker threshold would keep the circuit open longer than necessary after a short burst.

Model Fallback Chains in Practice

When the primary model fails, the fallback chain determines how much functionality your user retains. A complete chain looks like this:

Tier 1: Primary model — your best model for the task. If the circuit breaker is open or the call fails, proceed to Tier 2.

Tier 2: Secondary model — a cheaper or different provider’s model. If the primary is OpenAI and the failure is a provider-wide outage, a secondary from Anthropic or Google may still be available. Target latency: within 2× the primary’s p95.

Tier 3: Last-known-good cache — return the most recent successful response for a semantically similar query. Only appropriate for factual or template-based outputs (documentation queries, code snippet generation). Not appropriate for personalised or real-time data.

Tier 4: Degraded UX — inform the user that the AI feature is unavailable and offer a manual alternative. No model call is made at this level.

The fallback should never skip tiers — if Tier 1 fails and Tier 2 also fails, the chain must go to Tier 3 or 4, not loop back to Tier 1.

Caching Last-Known-Good Responses

For factual queries where correctness does not depend on recency — API documentation lookups, company policy questions, product specification retrieval — caching the last successful model response prevents repeated failed calls and keeps the feature alive during an outage.

Key insight: you are not caching to save money (that is prompt caching). You are caching to survive an outage. The cache-hit path must return in under 50 ms with no external API calls.

When it works. “What does our return policy say about opened products?” — the answer changes rarely, and serving a 24-hour-old response is better than showing “Feature unavailable.”

When it fails. “What is the current stock level of SKU-1234?” — the answer changes every few minutes. A cached response is worse than showing “Real-time data unavailable” because it actively misleads the user.

Use TTL-based invalidation with conservative defaults. A 5-minute TTL for dynamic data, 24-hour TTL for static knowledge-base content. Track cache-hit ratios per endpoint and reduce TTLs when hit ratios drop without a corresponding outage — it means the cache is serving stale data that users are overriding.

Degraded UX Patterns

The most important rule of degraded AI UX is: never let the user think the AI responded when the fallback fired. A cached response or lower-quality model output presented without disclosure erodes trust faster than showing “Feature unavailable.”

Pattern 1: Feature-level degradation. Instead of hiding the AI assistant or showing a generic error, replace the AI output area with an explanatory message and a manual alternative. “Smart reply is unavailable — type your response below.” The rest of the page works normally.

Pattern 2: Deferred processing. For asynchronous AI features (summarisation, analysis, content generation), accept the request and queue it for processing when the model recovers. Show a “Processing — results will appear when ready” banner with an estimated wait time.

Pattern 3: Partial fallback. If the model is available but degraded (high latency, reduced context window), shift from real-time streaming to a “Generate” button pattern. The user clicks, waits a few seconds, and receives a result. This trades interactivity for availability without breaking the workflow.

Pattern 4: Transparent fallback notification. When a fallback chain step activates, log it and optionally surface a subtle indicator to the user: “Results may be abbreviated (model cache)” or “Analysis running on backup system.” Power users appreciate knowing the system is degraded; casual users ignore it until they need to know.

Monitoring and Alerting for Degradation

Reactive monitoring tells you a provider was down after the incident resolves. Proactive monitoring tracks the signals that predict degradation before the page breaks.

Track per endpoint:

• p50/p95/p99 latency — a latency trend crossing your HTTP client timeout threshold means silent failures are accumulating
• Error-rate-by-class — separate 5xx, 429, timeout, and content-filter errors. A spike in content-filter rejections often signals a prompt change or provider policy update, not an infrastructure issue
• Fallback-chain-hit rate — the fraction of requests resolved at each tier. A rising Tier 3 or Tier 4 hit rate means the primary and secondary models are both failing
• Circuit-breaker state transitions — how often each breaker opens, how long it stays open, and how many probe requests fail before recovery
• Cache-hit ratio for degraded mode — when the cache is serving live traffic during an outage, a dropping hit ratio means the outage is longer than your TTL

Set alert thresholds on trends, not absolute numbers. A 2× latency increase sustained for 5 minutes is actionable. A single breaker opening is not — unless it stays open after the reset window expires.

Conclusion

Graceful degradation is not a design afterthought for AI-enhanced applications. It is a core reliability requirement, equivalent to database connection pooling or load balancing. Without it, every model API incident becomes a user-facing outage. With it, your application absorbs failures at the infrastructure layer and continues serving reduced but functional features.

The circuit-breaker stub above gives you a working starting point — drop it into your AI call wrapper and configure the thresholds for your primary model provider. Then add one fallback tier per week until your chain covers hard outages, rate limits, and content-filter rejections. Test the full chain during your next intentional outage drill: kill the primary API key and verify the application still renders a usable page with a clear explanation.

When the next provider incident happens — and it will — your users will not see a broken page. They will see a slightly less capable page with an honest explanation. That is the difference between an application built on AI and one that depends on it.

For observability patterns that pair with graceful degradation, see Observability for AI Agents: What to Log, Trace, and Alert On.