Testing and Verification While Vibe Coding

The short answer: The core risk in vibe-coding tests is that the agent writes both the code and the tests, so the tests rubber-stamp whatever the code already does. They pass, you feel safe, and real bugs ship anyway. The fix is to stop treating "all tests passing" as verification: write tests from the spec and intended behaviour, not the implementation; cover the edge cases and error paths the agent skipped; never let the agent "fix" a red test by weakening the assertion; and run the suite yourself. Green does not mean correct.

Of all the vibe-coding failure modes, this is the most seductive, because it comes wrapped in a reassuring green checkmark. The agent builds a feature, writes a test suite, runs it, and reports "all 40 tests passing." It feels verified. It usually isn't. The agent tested that the code does what the code does — which is always true — and told you nothing about whether the code does what you meant.

Why agent-written tests give false confidence

Ask one agent to write a function and its tests in the same breath and you get a closed loop. It has one mental model of the behaviour, and it encodes that model twice: once as the implementation, once as the assertion. If the model is wrong, both are wrong in the same direction, and the test passes anyway. The test isn't checking the code against reality — it's checking the code against a copy of itself.

This is how a feature ships with a confident-looking suite and a bug that any human spec-reader would have caught in a minute. I've seen "227 tests pass" sitting on top of a method that had zero real coverage — its collaborators were mocked out so thoroughly that no test ever exercised the actual path where the bug lived. The number was real. The verification was not.

Write tests from the spec, not the implementation

The only test worth writing is one that could fail. That means the expected value has to come from somewhere other than the code under test — from the requirement, the spec, a worked example you computed by hand.

The first test will pass forever, even if applyDiscount starts adding tax or dropping a decimal. The second nails a concrete fact from the requirement, so it breaks the moment the behaviour drifts. Hard-code the expected answers. If you find yourself writing a formula in the assertion, you're probably writing the implementation a second time.

Test the edges and the error paths

Agents test the happy path because that's what they just built and what the demo exercises. The bugs live everywhere else: the empty list, the zero, the negative number, the duplicate, the expired token, the row that was deleted mid-request, the API call that times out. For anything that can fail, assert that it fails the way you intended — the right error, the right status, no partial write left behind. A payment that half-charges on a network blip is the kind of thing a happy-path suite waves straight through.

Don't let the agent weaken a failing test

This is the trap that quietly destroys a suite. A test goes red; you tell the agent to fix it; the agent "fixes" it by editing the assertion to match the current (wrong) output. The suite goes green and the bug is now enshrined as expected behaviour. A red test is a question: is the code wrong, or is the test wrong? Answer it yourself. If the behaviour is genuinely correct and the test was wrong, fine — but you decide that, reading the spec, not the agent reconciling the two numbers in front of it.

Prioritize by blast radius, not coverage percentage

Coverage percentage is the vanity metric of vibe-coded testing — trivially gamed by tautological tests that touch every line and verify nothing. Spend your testing budget where a bug actually hurts:

• Critical paths — the two or three flows that, if broken, mean the product is down.
• Money — pricing, charges, refunds, totals. Off-by-one here is a real-world loss.
• Auth — authentication and authorization. "Can this specific user do this specific thing" is where agents cut corners.
• Data integrity — anything that writes to a database, especially multi-step writes that must be all-or-nothing.

A dozen real tests on those four areas beat 90% coverage made of green-but-empty assertions.

Run the tests yourself

The whole chapter reduces to one habit: don't trust "all passing" as a reported string. Run the suite in your own terminal. Read what it actually asserts. Better still, break the code on purpose — comment out the discount, flip a comparison — and confirm a test goes red. A suite that stays green when you sabotage the code isn't testing anything, and that five-second experiment tells you more than any coverage report. The same mindset lives in a healthy code-review culture: a green check is the start of the conversation, not the end of it.

Use agents and skills deliberately

The right move isn't to stop using agents for tests — it's to split the roles so one isn't grading its own homework. Have the agent write the implementation, then bring a separate reviewer with a specialist lens:

• A coding-standards skill to enforce that tests assert against fixed, spec-derived values rather than re-deriving them from the code.
• An api-and-interface-design skill to pin down the error contract — exact status codes and shapes — so your error-path tests have something concrete to assert against.
• A product-manager / PRD skill to turn fuzzy intent into explicit acceptance criteria, which are the raw material for tests that can actually fail.

A generalist agent produced the code and a plausible suite; a reviewer agent reading the spec instead of the implementation is what catches the tautology.

This is one of the vibe-coding deep-dives — start from the vibe-coding precautions overview. When a spec-driven test does go red, how to debug anything is the next step, and verification is the safety net that lets you keep shipping fast without breaking things.