The AI App Degradation Drill: What Your Product Should Do When the Model API Fails

Your onboarding flow asks an AI model to turn a new customer's notes into a project plan. The customer spends 12 minutes entering details, clicks Create plan, and waits. The model provider is overloaded. Your server retries several times, the browser retries again, and the customer sees a spinner until it becomes a generic error.

What should happen next?

“Call another model” sounds like an answer, but it hides the actual product decisions. Does the second model support the same structured output? May the customer's data be sent to that provider? Could two delayed requests both create a plan? Will the fallback quietly produce lower-quality work? Does anyone know whether the first request finished?

An AI app is not reliable because every model call succeeds. It is reliable when a dependency can fail without losing the user's intent, duplicating a consequential action, or pretending that an unverified result is complete.

This guide gives a small team a practical degradation drill. It is not a promise of high availability, and it does not replace a production reliability review. It is a way to expose failure decisions while they are still cheap to change.

Model Failure Is a Product State, Not Just an Exception

Provider APIs document several kinds of failure. OpenAI distinguishes rate-limit errors from quota and request errors. Anthropic documents temporary overload as a 529 response and attaches a request ID to API responses. Gemini documents 503 UNAVAILABLE for overload or lack of capacity and recommends exponential backoff for retriable 429 and 503 conditions.

Those codes are useful, but your user does not experience a status code. They experience one of five product states:

1. The request is invalid. Retrying the same payload will not fix an unsupported parameter, oversized input, bad credential, or exhausted account budget.
2. The dependency is temporarily unavailable. A bounded retry may work, but synchronized or layered retries can add load during an outage.
3. The outcome is unknown. Your timeout expired, but the remote system may still have completed work. Repeating a side effect can create duplicates.
4. The response arrived but is unusable. It may violate a schema, omit evidence, fail a policy check, or be too uncertain for the promised task.
5. A downstream tool failed. The model responded, but search, storage, email, payment, or another tool did not. A fluent final message can conceal an incomplete workflow.

Treating every case as “AI error, retry” collapses different risks into one loop. The first design task is to translate technical failures into explicit product states.

Decide What Must Survive Before Choosing a Fallback

For one important user journey, write a survival contract. It should be short enough that a founder, designer, and engineer make the same decision during an incident.

For the onboarding example:

• The customer's original notes must be saved before generation begins.
• One click must create at most one project-plan job.
• A plan cannot be labeled complete until its required sections pass validation.
• If generation is delayed, the customer can leave and return without losing progress.
• If a different model or provider would receive the data, the route must comply with the product's data promises.
• The interface must distinguish queued, processing, degraded, failed, and complete states.

This contract matters more than an uptime slogan. It tells you what to preserve when full functionality is unavailable: usually user input, correct state, reversibility, and honest communication.

Build a Degradation Ladder

A fallback should remove optional work before it changes the meaning of the product. Use a ladder rather than one emergency branch.

The order is deliberate. A founder should first ask, “Can we do less?” rather than “Which other model can we call?”

For a writing assistant, degraded mode might preserve the draft and offer a deterministic outline. For document extraction, it might let the user download the original file and show that extraction is pending. For an agent that sends refunds, modifies permissions, or submits legal records, the safe degraded mode may be no automated action at all.

Microsoft's Well-Architected guidance recommends that a degraded experience notify users about what remains available and what changed. That is especially important for AI products because a plausible-looking fallback can be mistaken for the promised result.

Why “Just Add a Backup Model” Is Not a Complete Plan

Model failover can help, but it creates a second system to validate and operate.

A smaller model is not automatically equivalent

A cheaper or faster model may follow a schema differently, use tools less reliably, handle long context poorly, or change the tone and refusal behavior users expect. Define a separate acceptance threshold for the fallback. If it cannot meet the core promise, label it as a draft or do not show it.

A second provider changes the data path

Review retention, region, subprocessors, credentials, logging, and contractual commitments before routing real user data elsewhere. Do not make an incident the first time the team decides whether cross-provider processing is allowed.

Failover can duplicate side effects

If the first request times out after sending an email or creating a record, a second attempt may repeat the action. Use an idempotency key tied to the user's logical action, store operation state, and make tools return stable identifiers. Where a tool cannot guarantee idempotency, reconcile the existing state before retrying.

A fallback that is never exercised is only a diagram

Prompts, tool schemas, SDKs, model names, and provider behavior change. Run representative cases through the fallback on a schedule. Compare required fields and policy outcomes, not whether the prose looks good.

The 90-Minute AI Dependency Failure Drill

Run this against a staging environment with synthetic data. Do not create a real provider incident or send destructive tool calls.

Minutes 0–15: choose one user promise

Pick one journey that creates real value: generating a proposal, classifying an inbound lead, extracting an invoice, answering from a knowledge base, or completing onboarding. Do not test “the AI feature” as one giant surface.

Write down:

• The user's input and the value they expect.
• The maximum useful wait time.
• Which state must be durable before the model call.
• Which actions are reversible and which have external side effects.
• The minimum acceptable result.
• The result that must never be shown as complete.

Minutes 15–30: map the dependency path

Draw the actual path from click to outcome:

browser → app API → queue → model → tool → validator → database → notification

Mark the owner, timeout, retry behavior, request identifier, and fallback for each hop. Look for hidden multiplicative retries. Three attempts in the browser, three in the app server, and three in an SDK can turn one user action into 27 downstream attempts.

Amazon's Builders' Library warns that retries add load and can delay recovery when a service is already overloaded. It recommends timeouts, capped backoff, jitter, retry limits, and idempotency for operations with side effects. Google SRE similarly recommends a process-wide retry budget and warns against retrying independently at multiple layers.

Choose one layer to own retries. Set an end-to-end deadline so a downstream call cannot continue beyond the point where its result is useful. Preserve provider request IDs in logs, but never show internal credentials or sensitive prompts in user-facing errors.

Minutes 30–55: inject six safe failures

Use a stub, proxy, feature flag, or test double to produce controlled outcomes.

Also test two failures together. A common false confidence comes from testing provider overload while the queue, cache, database, and notification service are healthy. Try an overloaded model plus a full queue, or a valid model response plus an unavailable tool.

Minutes 55–70: inspect the user experience

For each failure, answer these questions from the interface—not from logs:

• Is the original input still present?
• Does the user know whether anything was created or sent?
• Is the next action safe to click more than once?
• Does the message explain what is delayed or unavailable?
• Can the user leave and recover the task later?
• Is a degraded result visibly different from a verified result?
• Is there a non-AI path when one is genuinely useful?

Avoid messages such as “Something went wrong” when you know the useful state. Prefer: “Your notes are saved. Plan generation is delayed, and nothing has been sent. You can close this page; we will keep the job for 30 minutes.” Only make that promise if the queue and notification path actually support it.

Minutes 70–85: rehearse recovery

Restore the dependency and observe what happens to queued, timed-out, and partially completed work.

Recovery is where duplicate jobs often appear. Limit the rate at which queued work resumes so a healthy provider is not hit by a synchronized wave. Revalidate work before marking it complete. Expire jobs that are no longer useful. Preserve a record of why a job was retried, rerouted, canceled, or completed.

Test the operator controls as well: can one person pause new AI work, disable a fallback route, drain a queue gradually, and find affected users without deploying new code? A kill switch that nobody can locate under pressure is not an operational control.

Minutes 85–90: record a go/no-go decision

The drill passes only if the team can explain:

• Which mode the product entered and why.
• Which user promises remained true.
• Which work was lost, delayed, or reduced.
• Whether any side effect could be duplicated.
• How recovery was controlled.
• Which residual risk the team accepts.

Turn every ambiguous answer into an owner and a test, not an assumption.

Minimum Implementation Controls

You do not need a multi-region platform to make an early AI product more honest. Start with controls that reduce ambiguity.

Durable intent before generation

Save the user's input and create a stable job ID before making the remote call. A browser tab should not be the only copy of valuable work.

Explicit job states

Use states such as accepted, running, degraded, needs_review, complete, failed, and canceled. Define allowed transitions. Do not infer completion merely because a worker process exited without an exception.

Output validation outside the model

Validate schemas, required citations, permissions, totals, and business invariants in code where possible. A second model can assist with subjective review, but it should not be the only judge of whether a consequential action occurred correctly.

One retry owner and one budget

Classify errors as retriable, non-retriable, or outcome-unknown. Set a small retry budget with exponential backoff and jitter. Measure attempts per logical user action, not only total API calls.

Idempotent tools and reconciliation

Pass a stable key to side-effecting operations. Store remote IDs. When an outcome is unknown, query existing state before trying again. If neither idempotency nor reconciliation is possible, require human review.

Failure isolation

Separate critical flows from optional enrichment. A failing recommendation call should not consume all workers needed for account access or saved drafts. Concurrency caps, separate queues, and bulkheads can keep a nonessential feature from exhausting the whole product.

Truthful status and observability

Log the logical job ID, provider request ID, selected route, model, attempt number, latency, terminal state, validator result, and fallback reason. Minimize or redact sensitive content. Give users status that maps to the same underlying state machine operators see.

Metrics That Reveal Real Degradation

Overall success rate can hide a bad user experience. Track:

• First-attempt success and eventual success separately.
• Percentage of jobs entering each degraded mode.
• Time to an honest terminal or queued state.
• Attempts per logical action and retry-budget exhaustion.
• Duplicate or reconciled side effects.
• Queue age, abandonment, and completion after recovery.
• Validation failures by primary and fallback route.
• User retries caused by unclear interface state.

Set an alert on the transition into degradation, not just a total outage. A fallback that silently serves worse results for hours may be more damaging to trust than a short, clear interruption.

Where Graceful Degradation Should Stop

This framework is appropriate for ordinary early-stage AI apps where the team can define a smaller safe outcome. It is not sufficient for medical, legal, financial, safety-critical, or regulated decisions. Those systems may require formal hazard analysis, contractual availability targets, validated human procedures, audit controls, and qualified reliability engineering.

Do not automatically route sensitive data to another provider, lower a safety threshold, omit required evidence, or turn a write action into “best effort” just to preserve an availability number. Sometimes the correct degraded mode is to save the request and stop.

Also remember that a drill proves only the cases you exercised. It cannot establish that a provider, network, queue, or fallback will behave the same way in every real incident. Repeat the drill after changing models, SDKs, tool contracts, queues, data policies, or critical prompts.

The Founder Degradation Checklist

Before launch, require clear answers:

• Is valuable user input saved before any model call?
• Do we distinguish invalid, transient, unknown-outcome, invalid-output, and tool failures?
• Is there one owner for retries and a bounded retry budget?
• Do retries use backoff and jitter?
• Are side effects idempotent or reconcilable?
• Can the product operate in named normal, constrained, degraded, queued, and unavailable modes?
• Is every fallback tested against the minimum product promise?
• Have data and contract implications of cross-provider routing been reviewed?
• Does the interface preserve input and state what did and did not happen?
• Can users safely return to queued work?
• Can an operator pause, reroute, and gradually resume work?
• Do logs connect one user action to its attempts without exposing unnecessary sensitive data?
• Have we injected slow, rate-limited, overloaded, permanent, invalid-output, and unknown-side-effect failures?
• Have we tested recovery, not only failure entry?
• Do we know which cases must stop instead of degrade?

A polished demo proves that an AI app can work. A degradation drill shows whether the product can fail without lying, duplicating action, or discarding the user's effort. For a founder, that is a more durable quality signal than another model upgrade: the product remains understandable even when its most impressive dependency is unavailable.

References

• OpenAI, API Error Codes, including distinctions among request, rate-limit, quota, and server errors.
• Anthropic, Claude API Errors, including overload, timeout, and request-ID behavior.
• Google AI for Developers, Gemini API Troubleshooting Guide, including 503 capacity failures and retry guidance.
• Amazon Builders' Library, Timeouts, Retries, and Backoff with Jitter, on bounded retries, overload amplification, timeouts, and idempotency.
• Google SRE Book, Addressing Cascading Failures, on retry budgets, load shedding, overload tests, and degraded results.
• Microsoft Azure Well-Architected Framework, Self-Healing and Self-Preservation Strategies, including explicit degraded modes and user notification.
• Allen Institute for AI, What Building Shippy Taught Us About Building Agents, an engineering case study on deterministic tools, isolation, guardrails, and workflow-grounded evaluation.