Flaky Tests: Root Causes and How to Fix Them
A flaky test passes on one run and fails on the next against the exact same code. Nothing in the application changed between runs, only the test's own reliability did, and that is what separates flakiness from a real regression: a regression fails consistently once it starts, a flaky test fails inconsistently for reasons that have nothing to do with whether the feature works.
The cost isn't the failed run itself, it's what a team does after seeing enough of them. A suite where two or three tests fail at random on an otherwise healthy build trains people to re-run the pipeline before reading why it went red, and once "just run it again" becomes the normal response, a genuine regression buried in that same run gets the same shrug. A test suite exists to make red mean broken. Flaky tests are the fastest way to make red mean nothing.
What actually causes a flaky test
Most flaky tests trace back to one of a small number of causes, and each one has a fix that addresses the cause rather than papering over the symptom.
| Cause | What it looks like | Fix |
|---|---|---|
| A fixed wait instead of a real one | Passes locally, fails in CI when the app responds a beat slower | Wait for a specific state (an element visible, a response received, a record written) instead of sleeping a fixed number of seconds |
| Shared state between tests | Passes alone, fails as part of the full suite; order changes the result | Give each test its own data, and reset anything shared before the run instead of trusting the last test to clean up after itself |
| An uncontrolled external dependency | Fails when a third-party API is slow, rate-limited, or briefly down | Mock or stub anything the system under test doesn't own |
| The system clock or a random seed | Fails once a day near a date boundary, or intermittently with no visible pattern | Fix the clock and the random seed for the duration of the test run |
| A fragile locator | Fails after an unrelated markup or copy change, not a behavior change | Select by a stable test ID, not by visible text or position in the DOM |
| Leftover state from a previous run | Fails only after a run that didn't clean up, never on a fresh environment | Tear down what the test created every time, including when the test itself fails |
Most of these share a pattern: the test is checking a moment in time it doesn't actually control. A fixed wait assumes the app always responds within that window. A shared fixture assumes no other test touches it first. A real locator, a real clock, a real network call: each one hands control of the test's outcome to something outside the test.
Free and open sourceCasely writes these cases for youAttach a spec and one file of your team's existing test cases in Claude. Casely copies your columns, names the gaps it found in the spec, and exports a single Excel file your tracker imports in one pass.Read the install docsRetrying isn't a fix
A CI job that reruns failed tests up to three times before marking the build red treats every flaky test as a coin flip and hopes it lands heads eventually. It works, in the narrow sense that the pipeline turns green. It also means a test that fails one run in five is now indistinguishable, in the dashboard, from a test that never fails, and the actual cause never gets investigated because nothing forces it to.
There's a narrow, legitimate use for a rerun: a known, external, unowned dependency (a shared staging environment someone else deploys to, a third-party sandbox with its own uptime) that genuinely blips independent of anything the test does. Even there, the rerun buys time, not a fix, and it should come with a ticket, not a shrug.
Finding the actual cause
Start by reproducing the failure outside the full suite. Run the test alone, in a loop, twenty or fifty times. If it fails alone at the same rate it fails in the suite, the cause is internal to the test: a wait, a locator, a dependency it doesn't control. If it only fails as part of the full run, the cause is almost always shared state or an order dependency, and the fix is isolating what the test reads and writes, not touching the test's own logic at all.
From there, the timestamp in the failure log usually points at the mechanism directly: a timeout that fires a fixed number of milliseconds after the test starts is a wait problem, an assertion on a value that belongs to a different test's fixture is a state problem, a connection error to a URL outside your own infrastructure is a dependency problem. Track failure rate per test over time rather than treating a single red run as evidence either way. One failure could be anything. A test that fails one run in ten, consistently, over two weeks, has a real cause worth finding.
Where it fails
Not every intermittent failure is the test's fault. A test that fails once in twenty runs might be exposing a real race condition in the product, one that only shows up under specific timing and would eventually hit a real user the same way. Labeling every intermittent failure "flaky" and moving on is how a genuine concurrency bug ships to production wearing a false diagnosis.
Some categories carry more inherent timing risk than others. A unit test with no I/O has almost no excuse to flake. An end-to-end test driving a real browser against a real backend has more moving parts outside its control, network latency, render timing, animation frames, and a zero-flake bar for that category is not realistic even with every fix in the table above applied correctly.
A quarantine list without an owner and a deadline becomes a graveyard. Pulling a flaky test out of the required suite so it stops blocking merges is a reasonable short-term move. Leaving it there indefinitely, with nobody assigned to fix it, means the coverage that test represented is gone and nobody notices until the bug it would have caught reaches production.
What's worth automating
Detecting flakiness, flagging a test whose pass rate on unchanged code drops below a threshold across enough runs, is mechanical, and most CI platforms already do it without asking anyone to watch a dashboard. Diagnosing why a specific test flakes, and telling a real intermittent product bug apart from a test-only timing problem, stays a judgment call that needs someone to actually read the failure and the code around it, not just count how often it happens.
That's a different problem from deciding whether the right test cases exist in the first place. Casely generates test cases from requirements, the titles, steps, and expected results a suite should cover, and has no opinion on how reliably a given case executes once it's automated. A correctly generated case can still be flaky in execution, and fixing execution reliability doesn't tell you whether the suite is testing the right things to begin with. Both problems are real, and neither substitutes for the other.