Reviewing AI-Generated Test Cases: Where Time Goes

Two years ago the question was whether a language model could write a usable test case. That question closed. The World Quality Report 2025-26, which surveyed 2,000 executives across 23 countries, found 89% of organizations piloting or deploying generative AI in quality engineering, and 15% running it at enterprise scale.

The same report puts the average testing productivity gain at 19%, with a third of respondents seeing between 1% and 9%. That spread is the interesting part. Nearly everyone adopted the tools. Most got a fraction of what they expected, and the reason shows up as soon as you watch where the hours go: the bottleneck moved into review, and most teams have not restructured around that.

What work replaced the writing?

A model hands you forty cases in four minutes. Someone still reads all forty. That reader decides which ones test the thing that matters, which duplicate each other under different wording, and which describe behaviour the product never promised.

Reviewing forty generated cases takes longer than reviewing forty hand-written ones, for a reason that surprises people the first time: hand-written cases carry visible signals about the author's confidence. A tester who was unsure writes a short case, or leaves a note, or asks in Slack. Generated cases arrive uniformly polished. The case built on a misread requirement looks exactly like the case built on a correct one.

So the reviewer loses the shortcuts they used to have. They read every line at full attention, or they ship someone else's misunderstanding into the regression suite.

Free and open sourceCasely writes these cases for youAttach a spec and one file of your team's existing test cases in Claude. Casely copies your columns, names the gaps it found in the spec, and exports a single Excel file your tracker imports in one pass.Read the install docs→

Generate from the spec, not from the implementation

This is the practical finding worth changing your process over.

Point a model at your code and ask for tests, and it writes tests that assert what the code does. If the code has a bug, the generated test locks the bug in and turns green.

A 2026 study named this the misguidance effect: buggy code steers a model toward tests that assert the erroneous behaviour instead of exposing it. The authors' fix was to replace the code in the prompt with a generated specification docstring, which raised the average number of effective bug-finding tests from 104 to 187 across their benchmark.

Tests generated from requirements do not have that failure mode. The spec describes what the product owes the user, so a test built from it disagrees with a wrong implementation, which is the entire point of the exercise.

The rule that follows: feed the model the requirements document. Feed it the code only when you want tests that describe current behaviour, which is a different job with a different name, usually called characterization testing.

Why did coverage numbers stop meaning what they meant?

Teams reach for line coverage to prove the generated suite works. A replicability study across 11 models tested whether coverage and mutation scores on LLM-written suites predict real-bug detection, and the answer came back conditional. When the code handed to the model is sound and you are generating regression tests, the metrics carry real signal. When the code already contains the bug you are trying to expose, they stop being reliable indicators, for the same reason the misguidance effect exists: the suite agrees with the implementation, so it scores well against code it never challenges.

That second case is the one most teams are in when it matters. The mechanism is easy to see once you look at a generated suite. A model will happily produce fifteen cases that each execute the same happy path with different variable names. Coverage climbs. Defect detection does not move. The suite exercises the code without ever disagreeing with it.

Judge a generated suite by what it would catch, not by what it touches. Take a defect you shipped last quarter, reintroduce it in a branch, and run the suite. If nothing turns red, the coverage number was decoration.

When does the maintenance bill arrive?

Every generated test is a test somebody keeps green. Teams who generated a thousand cases in a sprint met that arithmetic a quarter later, when a refactor broke four hundred of them at once and nobody could tell which failures mattered.

Two habits keep the bill down. Delete aggressively during review, because a case that duplicates another one costs maintenance forever and buys nothing. And keep generated cases traceable to the requirement they came from, so when the requirement changes you know which cases to regenerate instead of reading all four hundred.

What stays with you

Risk. A model has no idea that the payments module broke production in March, that the client escalated about export formatting twice, or that the team rewrote session handling last week. It weights every requirement in the document the same. You do not.

Exploratory work. Scripted cases find the bugs somebody predicted. The expensive bugs are the ones nobody wrote a case for, and finding those is still a human sitting with the product and a suspicious feeling.

Your team's conventions. ID schemes, the columns your tracker expects, which edge cases the team agreed to skip and why. None of that lives in the spec. Hand the model an example file if you want it honoured.

Where the honest gain sits

AI removes the mechanical decomposition: splitting an input range into classes, walking each boundary, laying out a decision table. That work took real hours and it followed rules, which is what these models do well.

It does not remove the judgment, and the teams reporting the biggest gains are the ones who moved their senior testers off drafting and onto reviewing. The teams reporting disappointment usually generated a large suite, skipped the review, and discovered in the next release which cases were fiction.