The AI-Built Interface State Audit: Test What the Demo Never Shows

An AI app builder can produce a convincing dashboard in an afternoon. The sample account has data, the network is fast, every permission is generous, and the founder knows exactly where to click. The demo looks finished.

Then a real user arrives with an empty account. They double-click Create because the first click appears to do nothing. Their session expires while an AI job is running. They open a saved link from another workspace. The model returns a valid response with one required field missing. The interface still has to tell the truth.

This is where many generated products stop feeling like products. The normal screenshot exists, but the states between “before” and “success” were never designed as part of the promise.

This guide provides a 75-minute interface state audit for founders and small teams. It does not certify accessibility, security, or reliability. It helps you find observable contradictions: moments when the interface says nothing, says the wrong thing, exposes an action the user cannot safely take, or loses the path back to useful work.

Use this as the interface layer of Y Build's AI app launch quality review. If the failure originates in a model or tool dependency, continue with the AI app degradation drill. If an action can send, spend, publish, delete, or change access, define human approval gates before polishing the screen.

A Screen Is Not One State

Take a simple “generate client report” page. A design file may show a text area, a generate button, and a finished report. The running product needs more states:

• First visit with no reports.
• Existing reports but no report selected.
• Input that is missing, too long, unsupported, or no longer available.
• Submission accepted but no response yet.
• Slow work that continues after the user navigates away.
• Partial output that has not passed validation.
• A recoverable failure with input preserved.
• An unknown outcome where the server may have completed the action.
• A user who can view the report but cannot regenerate, share, or delete it.
• A report deleted, archived, or changed in another tab.
• A successful result that a keyboard or screen-reader user was never notified about.

These are not decorative variants. Each one answers a product question: What exists? What is happening? What may this person do? What happened to their input? What should they do next?

The audit therefore evaluates a state transition, not a screenshot:

starting state → user action → waiting state → outcome → safe next action

If the team cannot name all five parts for an important journey, the interface is probably relying on assumptions the user does not share.

Build a State Matrix Before Testing

Choose one high-value journey, such as creating a proposal, inviting a teammate, generating an analysis, uploading a document, or changing a subscription. Do not inventory the whole app on the first pass.

Create a matrix with one row per meaningful state:

Writing this table often reveals a missing backend capability. If the proposed loading message says “You can close this page,” but work exists only in browser memory, that message is false. If the error promises “Nothing was sent,” but the remote outcome is unknown, the product needs reconciliation—not better copy.

The Six Hidden-State Gates

1. Empty must teach, not imitate success

An empty table with column headings proves only that the component rendered. A useful empty state explains why the area is empty, what the user will gain by adding something, and the smallest valid next action.

Test at least three kinds of empty:

• New empty: the user has never created an item.
• Filtered empty: items exist, but the current search or filter hides them.
• Unavailable empty: items should exist, but access, deletion, or a failed load prevents display.

Do not use the same illustration and “No results” message for all three. Resetting a filter is appropriate for filtered empty; creating duplicate data is not. A permission or loading failure must not masquerade as an empty account.

Also remove demo data before this test. Generated dashboards sometimes use invented charts or sample records to make the first screen feel complete. Label samples unmistakably or keep them out of the real workspace. A founder should be able to tell which values came from the product, the user, and a placeholder.

2. Waiting must show acknowledgement and ownership

When a click produces no fast visual response, users click again. Chrome's Interaction to Next Paint guidance explains why immediate visual feedback matters: a slow response can make a page appear unresponsive even when background work has started.

The waiting state should answer:

• Was the action received?
• Can the user safely click again?
• May they leave this page?
• Is work happening in the browser or on the server?
• How will they find the result later?
• Can cancellation actually stop the work or only hide it?

A spinner answers none of those questions by itself. Use a short acknowledgement first, then a more durable state if work continues. “Generating” is acceptable for a brief operation. A longer job needs a saved status, an identifier, and an honest return path.

Do not disable the entire page as a reflex. The HTML disabled attribute removes controls from focus and form submission, so careless disabling can hide values or leave keyboard users without context. Disable only the action that truly cannot be repeated, keep the status available, and make the server enforce the rule too.

3. Invalid must be specific and preserve effort

W3C's WCAG guidance says that automatically detected input errors should identify the item and describe the error in text. GOV.UK's design system adds a practical copy rule: say what happened and how to fix it; avoid generic messages such as “An error occurred.”

For the selected journey, test:

• Every required field omitted individually.
• All required fields omitted together.
• Wrong format, excessive length, unsupported file, and expired resource.
• A value that passes browser validation but violates a business rule.
• A server-side rejection after the UI initially accepts the input.

The user should not have to reconstruct valid fields after one field fails. Put the error next to the affected input, provide a summary for a long form, move focus intentionally when appropriate, and keep the correction message consistent. Color alone is not an explanation.

Remember that fetch() resolves even when the server returns an HTTP error status; client code must inspect the response status. An interface that treats every resolved promise as success can show a green confirmation for a 403, 422, or 500 response.

4. Permission must match the action, resource, and tenant

Hiding an Edit button is not authorization. The server must still reject the write. But an interface that shows unusable or dangerous controls also creates false expectations and makes testing harder.

Build a small authorization matrix for the journey:

OWASP recommends expressing authorization as actor, resource, and action, then regression-testing the matrix as the product changes. Test both the interface and the request. Change an object ID, open a copied deep link, switch workspaces in another tab, and repeat the action after a role is removed. A denial should reveal enough to recover without confirming that a private resource exists.

Do not turn every denial into “Upgrade.” Some are plan limits, but others are ownership, role, tenant, policy, or expired-session boundaries. The next step must match the actual reason.

5. Repeated action must not create repeated consequences

Test double-clicks, impatient retries, refresh during submission, back-button resubmission, two tabs, and a network response that arrives after the UI times out.

The visual fix—changing the button to “Working”—reduces accidental repeats but does not guarantee one outcome. Use a stable idempotency key or logical operation ID on consequential create/update requests. Stripe's API documentation illustrates the core contract: retries with the same key can return the first result instead of creating a second object.

Your product needs its own version of that contract for operations such as:

• Creating a paid order.
• Sending an invitation or campaign.
• Publishing a document.
• Starting an expensive generation job.
• Applying an AI-recommended change.

Define whether “Cancel” prevents execution, requests best-effort cancellation, or only stops waiting in the browser. AbortController can abort a browser request, but that does not prove the server or downstream tool reversed work already accepted. Copy must reflect the system's actual boundary.

6. Completion must be observable and recoverable

A toast that disappears after two seconds is weak evidence that work completed. The final state should be tied to a durable result: a saved record, a validated artifact, a receipt, a version, or a remote identifier.

W3C's status-message criterion requires important dynamic messages to be programmatically determinable so assistive technology can present them without moving focus. Test success, progress, and error announcements with keyboard navigation and at least one screen reader; do not infer accessibility from visible text alone.

Then test recovery:

• Refresh after success.
• Open the result from its durable URL.
• Return after signing out and back in.
• Retry from a failed state.
• Change the underlying resource in another tab.
• Reopen the app after the session expires mid-job.

If the refreshed page returns to a spinner, the toast was not completion. If a failed job cannot be distinguished from a never-started job, the product has no reliable recovery path.

The 75-Minute Founder Audit

Minutes 0–10: define the promise

Write one sentence: “When this journey succeeds, the user has ______, and we can prove it by ______.” Choose a durable proof, not “the success message appeared.”

List the highest-consequence action and the data that must survive a failure.

Minutes 10–25: complete the matrix

Add empty, invalid, waiting, permission-denied, repeated-action, failed, and complete rows. For each row, state what the interface may claim and what the server must prove. Mark any promise the current architecture cannot support.

Minutes 25–50: force the states

Use test accounts and synthetic data. Slow the network, block one API request, return representative 4xx and 5xx responses, remove a role, switch tenants, submit twice, refresh, and expire the session. Never run destructive tests against customer or production data.

Record a screenshot or short video plus the observed request/result for every failed gate. A screenshot alone cannot prove that no duplicate write occurred.

Minutes 50–65: test access and recovery

Complete the journey with keyboard-only navigation. Check that focus remains understandable, errors are connected to inputs, status changes are announced, and disabled controls do not trap the user. Refresh each terminal state and verify that it remains truthful.

Minutes 65–75: set the release gate

Classify findings:

• Block launch: data loss, cross-tenant access, false success, duplicate consequential action, irreversible action without review, or no safe recovery.
• Fix before broad promotion: generic errors, ambiguous waiting, broken empty paths, missing status announcements, or mobile/keyboard blockers.
• Backlog with owner: cosmetic inconsistency that does not change meaning or prevent completion.

The audit passes only when every launch-blocking finding has a reproducible fix and a regression check.

Turn Observations Into Stable Checks

Automate the transitions that must not regress. Playwright's web-first assertions retry until an asynchronous condition reaches the expected state, which is more representative than checking immediately after a click. Assert meaning, not implementation details:

• A specific status becomes visible after submission.
• The primary action cannot create a second logical request.
• Valid input remains after a server rejection.
• A member from another tenant sees no protected data.
• A completed item remains after reload.
• Keyboard focus reaches the error summary or relevant field as designed.

Screenshot comparisons can catch visual drift, but Playwright warns that rendering varies across operating systems and browser environments. Keep baselines in a consistent environment and pair visual checks with behavioral assertions. A perfect screenshot can still contain a dead button, an unauthorized response, or a status message no assistive technology can detect.

Start with one automated test per block-launch risk. Do not chase exhaustive UI coverage while the product's state model is still changing.

Common Failure Modes

Designing copy for capabilities that do not exist. “We will notify you” requires a durable job and notification path. “Nothing was sent” requires a known outcome. Treating client state as system truth. A disabled button, local spinner, or canceled request does not prove that the server prevented, completed, or reversed an action. Using one error component for every cause. Invalid input, expired authentication, denied authorization, provider failure, and unknown outcome require different next steps. Testing only with an admin account. Generous permissions conceal missing tenant and role boundaries. Using snapshots as the whole test strategy. Visual similarity does not prove business rules, accessibility, persistence, or absence of duplicate side effects. Making every edge case a launch blocker. Prioritize by consequence and frequency. A rare spacing issue is not equivalent to a repeat charge or leaked workspace.

Where This Audit Stops

This workflow is useful for ordinary web products and early AI apps. It is not a substitute for a WCAG conformance audit, penetration test, threat model, load test, privacy review, or regulated validation. High-risk medical, financial, legal, safety-critical, or employment systems need qualified review and controls beyond a founder exercise.

The audit also proves only the states you forced. It cannot show that every network sequence, browser, assistive technology, race condition, or external dependency is safe. Repeat it when roles, billing, state storage, job execution, model providers, or consequential actions change.

The important shift is simple: stop asking whether the generated screen looks finished. Ask whether every important transition remains truthful when the user is new, wrong, slow, unauthorized, impatient, disconnected, or returning later. That is where an AI-built demo starts becoming a dependable product.

References

• W3C Web Accessibility Initiative, Web Content Accessibility Guidelines (WCAG) 2.2, especially error prevention, component states, and programmatically determinable status messages.
• W3C Web Accessibility Initiative, Understanding Error Identification, on identifying inputs and describing detected errors in text.
• GOV.UK Design System, Error Message, practical guidance for specific, consistent, actionable error copy.
• Playwright, Best Practices and Visual Comparisons, on web-first assertions and the limits of screenshot baselines.
• Chrome team, Interaction to Next Paint, on interaction responsiveness and visible feedback after user input.
• OWASP Cheat Sheet Series, Authorization Regression Testing, on actor-resource-action matrices and continuous access-control tests.
• Stripe API Reference, Idempotent Requests, an implementation example for preventing duplicate outcomes during safe retries.
• MDN Web Docs, Using the Fetch API and AbortController, on HTTP response handling and browser-side request cancellation boundaries.