Before You Merge AI-Generated Code: A Founder's Evidence Pack

An AI coding agent fixes a checkout bug, updates three dependencies, changes an environment variable, and opens a pull request. The summary is polished. The tests are green. Another AI reviewer leaves two minor comments, the agent resolves them, and the merge button becomes available.

Is the change ready?

Not necessarily. The tests may cover the function the agent edited while missing the refund path. The dependency update may introduce a new install script. The environment change may expose a secret to a browser bundle. The AI review may have evaluated the diff with assumptions similar to those used to create it. A coherent explanation can make an incomplete change feel safer than it is.

For a non-technical founder, the answer is not to become a senior engineer overnight. It is to require a small merge evidence pack: a structured record showing what the change is meant to do, what it can affect, which checks ran on the exact revision, who or what reviewed the risky parts, and how the team will recover if production behaves differently.

The pack does not prove that code is bug-free. No review process can. It creates a defensible decision boundary between “the agent says it is done” and “we have enough independent evidence to release this change.”

This is the change-control layer of Y Build's AI app launch quality review. After the merge, verify the user-visible result with the AI-built interface state audit, especially when the change affects waiting, permissions, repeated actions, or recovery.

A Pull Request Is a Proposal, Not Proof

Modern coding agents can write a change, explain it, respond to review comments, and revise the same branch. That workflow is useful, but it can collapse several roles into one system.

The author may claim that the change is narrow. The generated tests may confirm the author's interpretation. An AI reviewer may inspect the same visible diff and accept the same incomplete requirement. All three outputs can agree while the real user journey still fails.

GitHub's documentation is unusually direct about this boundary: Copilot code review is not guaranteed to find every problem, can make mistakes, and should be supplemented with human review. It also notes that some file types, including dependency-management files, are excluded from Copilot code review. That does not make AI review useless. It means an AI review comment is input to a decision, not the decision itself.

The same principle applies to human-written code. Humans also misunderstand requirements, write weak tests, and approve familiar-looking changes too quickly. AI changes the operating conditions: it can produce larger diffs faster, generate convincing explanations, and make follow-up edits with very little friction. Your acceptance process must therefore anchor on evidence tied to the exact change, not confidence in the author.

Start With a Change Contract

Before reviewing files, write a five-part change contract in plain language. A founder should be able to understand it without reading code.

1. User outcome: What should a user be able to do after this change?
2. Explicit non-goals: What adjacent behavior must remain unchanged?
3. Affected surfaces: Which screens, APIs, data stores, permissions, scheduled jobs, third-party services, and configuration values could be touched?
4. Failure conditions: What would make the change unsafe or incomplete even if the happy path works?
5. Acceptance evidence: Which observable results would justify merging?

For a hypothetical checkout fix, the contract might say:

A customer can retry one failed card payment without creating a second order. This change does not alter pricing, taxes, refunds, or subscription renewals. Acceptance requires a failed-payment test, a retry test, a duplicate-submission test, confirmation that no secret enters client code, and a verified rollback path.

This paragraph is more useful than “fix checkout.” It gives reviewers a boundary. It also exposes requirement drift: if the diff changes refund code or adds a payment dependency, the change is no longer as narrow as described.

The contract should be written before the final implementation is accepted. If the agent writes both the code and the contract afterward, it can unconsciously describe what it built rather than what the product needed.

Build the Merge Evidence Pack

The evidence pack can live in the pull request description. It should be short for routine work and deeper for high-impact changes. The following six sections cover the minimum useful record.

1. Intent and scope

Include the change contract, the user journey being changed, and links to the original issue, support report, design, or decision note. If there is no source requirement, say who approved the intended behavior.

List non-goals explicitly. “No database migration,” “no new data sent to third parties,” and “no change to authorization rules” are testable boundaries. “Small cleanup” is not.

2. A risk map of the diff

Do not treat every changed line equally. Ask the agent to group changed files by product risk, then have a reviewer verify the grouping.

This map is deliberately product-oriented. A founder may not understand an abstract class hierarchy, but they can ask whether a retry might create two charges or whether an error state preserves user input.

If the agent cannot explain why a changed file exists, stop. If generated artifacts, lockfiles, migrations, or infrastructure files are hidden in a large diff, review them separately.

3. Executable evidence tied to the exact revision

“Tests passed” is incomplete. Record:

• the commit identifier that was tested;
• the exact checks that ran;
• whether they passed, failed, or were skipped;
• the environment used;
• the important scenario each check covers;
• any expected check that did not run.

GitHub status checks can report whether external processes such as continuous integration passed for a commit. Protected branches can require those checks before merge. Use those controls, but configure trusted check sources where possible: GitHub notes that users or integrations with write permission can set check or commit status, so a green label is meaningful only when you know who produced it.

For user-facing changes, include scenario evidence as well as unit tests. That might be a staging URL, a short screen recording, an accessibility result, an API transcript with sensitive values removed, or a before-and-after screenshot. Evidence should show the changed behavior and at least one failure or boundary state, not only the ideal demo.

Generated tests deserve review. Check whether they assert a meaningful outcome or merely reproduce the implementation. A test that confirms “the function returned success” may miss whether two orders were created, the wrong account was updated, or the user saw a false completion message.

4. Supply-chain and data-path evidence

Dependency changes need their own line item. Record every direct dependency added, removed, or upgraded; why it is necessary; whether its lockfile and install behavior changed; and whether known vulnerability or license checks ran.

GitHub's dependency review is designed for this boundary. It can show added, removed, and updated dependencies in a pull request and can enforce a check that fails when new dependencies have known vulnerabilities. Availability depends on repository and plan, so a small team may need a different scanner or a manual lockfile review. The control matters more than the vendor.

Then trace data movement. Did the change add analytics, an AI provider, error reporting, remote storage, or a new API call? What user data leaves your system? Does it include prompts, uploaded files, identifiers, or secrets? Is the destination consistent with your privacy notice and product promise?

A code change can pass functional tests and still be unacceptable because it silently changes who receives customer data.

5. Independent review and ownership

Assign a human owner for the merge. That person does not need to inspect every syntax detail, but they must own the decision that the evidence matches the product risk.

Use independent technical review for changes involving authentication, payments, destructive actions, migrations, secrets, production infrastructure, or sensitive data. “Independent” means the reviewer did not merely ask the same agent that wrote the code whether it is safe. Another model can broaden the search for issues, but it is still automated assistance, not accountable approval.

GitHub protected branches can require pull request reviews, dismiss approvals when new commits are pushed, require review from code owners, and prevent the most recent pusher from approving their own change. Small teams do not need every rule for every repository. They do need a rule that prevents a risky AI-generated revision from bypassing the person responsible for it.

After review comments are resolved, verify the final commit again. An agent's “small fix” to address a comment is still a new change. Stale test results or approval on an earlier revision should not authorize it.

6. Release and recovery evidence

Merge approval and production release are separate decisions. Before merging, record:

• whether the change can be enabled gradually or behind a feature flag;
• the production signal that will show it is working;
• the signal that will trigger rollback or disablement;
• who will watch those signals and for how long;
• the exact rollback, revert, or forward-fix path;
• any irreversible data operation.

GitHub deployment environments can require approval, restrict deployment branches, delay a job, and withhold environment secrets until protection rules pass. Even if your platform uses different controls, preserve the separation: passing code review should not automatically grant an unreviewed workflow access to production credentials.

Do not write “rollback if needed.” Test the mechanism when risk justifies it. A revert may fail after a database migration, background job, webhook, or external notification has already changed state. In those cases, the recovery plan may be a forward migration, a kill switch, a reconciliation job, or manual repair.

Match Evidence Depth to Consequence

Requiring the same ceremony for a typo and a payment rewrite encourages people to bypass the process. Use simple risk tiers.

Risk is about consequence, not line count. A one-line permission change can be high risk. A large generated test fixture can be low risk. Ask what could happen to a real user, real data, or real money if the assumption is wrong.

If classification is uncertain, choose the higher tier until someone can explain why it is safe to lower it.

A Founder-Friendly Review Sequence

For a medium-risk change, use this sequence:

1. Read the change contract without opening the diff. Can you explain the promised user outcome and non-goals?
2. Check the file list. Do any payment, auth, data, config, dependency, or deployment files appear unexpectedly?
3. Open the checks on the final commit. Did the expected checks actually run, and are results from trusted systems?
4. Exercise the changed journey in staging. Test the normal path, one failure path, and a repeat or retry.
5. Read unresolved and resolved review conversations. Did a new commit invalidate an earlier approval or test result?
6. Confirm the release owner, production signal, stop condition, and recovery action.
7. Approve only the identified commit. If the branch changes, repeat the relevant evidence.

This sequence does not require the founder to judge every implementation choice. It lets them detect mismatches between the story, the files, the checks, the observed product, and the release plan.

Failure Modes That Make Weak Reviews Look Strong

The agent reviews its own interpretation

The code, tests, summary, and review all derive from one incomplete requirement. Add an independent product scenario or reviewer who can challenge the premise.

Green checks ran on the wrong revision

A review fix lands after CI or staging evidence was collected. Tie every result to the final commit and dismiss stale approvals for important branches.

The diff is small but the blast radius is large

A configuration, permission, routing, or migration line can affect every user. Classify by consequence, not diff size.

Happy-path evidence hides state errors

The demo works once, but double-clicking, retrying, refreshing, using an expired session, or returning after a timeout breaks the workflow. Include boundary and recovery states.

AI comments create the appearance of scrutiny

A long list of stylistic suggestions can distract from missing authorization, data, or rollback questions. Review coverage matters more than comment volume.

The rollback exists only on paper

Reverting code does not undo an email, charge, deleted record, changed schema, or disclosed secret. Name the irreversible effects and recovery owner before release.

What This Framework Does Not Do

An evidence pack is not a security audit, penetration test, compliance assessment, or substitute for experienced engineering judgment. NIST's Secure Software Development Framework treats code review and analysis as practices within a broader secure development process. OWASP likewise presents manual security code review as important alongside automated scanners, not as a universal guarantee.

For regulated data, financial movement, healthcare, safety-critical systems, cryptography, identity infrastructure, or a major migration, bring in qualified specialists. If no one on the team can understand the relevant code or failure modes, do not use a checklist to manufacture confidence. Reduce the scope, delay the release, or obtain technical review.

Nor should every small team build an elaborate approval bureaucracy before it has users. Start with a protected main branch, final-commit checks, a plain-language change contract, a named merge owner, and a recovery note. Add deeper controls where real risk and repeated failures justify them.

The Merge Checklist

Before merging an AI-generated change, confirm:

• The user outcome and non-goals are written in plain language.
• The changed files match the stated scope.
• High-risk surfaces are identified explicitly.
• Tests and checks are attached to the final commit.
• Skipped, missing, or flaky checks are disclosed.
• At least one failure or boundary state was exercised.
• Dependency and data-path changes were reviewed separately.
• AI review findings were validated rather than accepted automatically.
• A human owns the merge decision.
• High-risk code received independent technical review.
• New commits invalidate stale evidence where appropriate.
• Production success and stop signals are named.
• Rollback or recovery is specific and feasible.
• Irreversible effects are understood before deployment.

The goal is not to make AI-generated code harder to ship. It is to make the reason for shipping legible.

An agent can produce a persuasive pull request in minutes. Trust should take one step longer. Require a contract, inspect the risk, attach evidence to the final revision, separate review from release, and know how to recover. Then the merge button represents a decision your team can defend—not merely the end of the agent's task.

References

• GitHub Docs: About GitHub Copilot code review
• GitHub Docs: About protected branches
• GitHub Docs: About status checks
• GitHub Docs: Reviewing dependency changes in a pull request
• GitHub Docs: Deployments and environments
• NIST SP 800-218: Secure Software Development Framework
• OWASP Code Review Guide