A flaky test sometimes passes and sometimes fails without a clear change in the code under test. The result appears random. In practice, the cause usually involves uncontrolled state, a race, timing, data, or a dependency the test does not account for.
The team's problem extends beyond wasted execution time. When red statuses repeatedly turn out to be noise, people stop trusting them and miss real defects. Managing instability is therefore part of test-system quality rather than optional post-release cleanup.
Do not assume a failure is false
An intermittent failure may indicate a real product race condition: for example, one parallel response overwrites current state. That matters to users even if the next test run passes. The flaky label describes observed instability but does not prove the test is at fault.
Classify causes after investigating: application, test code, data, infrastructure, or external dependency. Keep an unknown category until evidence emerges. Prematurely labeling something an environment issue often closes a question the team has not examined.
Preserve the first failure
Before rerunning, save the build, environment, seed, inputs, order, and parallelism. For UI tests, retain a trace or screenshot, actual requests, and page state. For APIs, keep a sanitized request, response, and correlation ID. An event sequence helps with asynchronous processes.
Repeating without preserving context may destroy the failure condition. If a shared user has already changed, the second run is a different experiment. Compare the original failure and successful retry as two observations rather than replacing one result with the other.
System state and isolation
A shared database, account, file, or cart creates dependencies between tests. One case changes the locale while another expects English text; one deletes a record that another assumes exists. These issues often appear only in parallel or after a particular predecessor.
Run the scenario alone first, then with suspected neighbors and the relevant parallelism. Use unique data and explicit cleanup. A failure disappearing in sequential mode is evidence, but is insufficient reason to disable parallelism permanently.
Asynchronous behavior and incorrect waits
A one-second pause assumes the operation always completes within that time. A busy CI agent can violate the assumption. A proper wait observes a business state or required technical signal within a justified timeout.
However, the wait itself can be wrong. A button appearing does not mean its handler is attached, and 202 does not mean an operation has finished. Choose evidence matching the promised result. Polling should read state rather than repeat a side-effecting action.
Practical example: an order disappears from the list
Two tests share one account. The first creates an order; the second deletes all the user's drafts. Both pass sequentially, but in parallel the first sometimes cannot find its record. A random delay changes the failure frequency without removing the cause.
Record the created order ID and deletion log, then rerun the tests in parallel with separate users. Expect no cross-test deletion. Fix this with scenario-specific data and targeted cleanup of owned IDs. Verification should retain the original parallelism and include enough repetitions.
If the problem persists with separate users, continue investigating: the application may incorrectly scope deletion by owner. Test isolation must not conceal a real access defect. Confirming the cause requires observing which records were changed and by whom.
Time, randomness, and order
Date tests may depend on time zones, crossing midnight, and server clocks. Control time at an appropriate level and test boundaries separately. Comparing “created now” without an allowed interval is often unstable even when the product is correct.
Random data is useful only when reproducible. Save the seed and relevant inputs. Do not sort results in a test when order itself is a requirement. Otherwise, fixing instability removes a check of actual behavior.
Measuring the problem
Count initial failures, successful retries, final failures, and investigation time separately. Define each metric's denominator and period. For example, the proportion of cases passing on retry differs from the proportion of job runs requiring a restart.
Ten repetitions without failure do not prove a rare cause was fixed. Match verification to the original frequency and reproduction conditions. Confirming the fix mechanism, such as independent IDs and nonoverlapping operations, matters more than simply accumulating passing runs.
Retries and quarantine
Limited retries can handle a known temporary failure and gather additional evidence. They must retain the first result and an explicit instability metric. Unlimited retries or automatic reruns until green destroy the check's meaning.
Quarantine is acceptable temporarily with an owner, reason, deadline, and compensating control for important risk. Keep running the excluded case separately to observe the issue. Do not silently remove a critical check and treat the remaining passing suite as equivalent.
The repair process
Form a hypothesis and experiment. Use different IDs for shared records, fixed clocks for time, and event logs plus controlled response ordering for races. Change one significant condition at a time so success can be connected to a cause.
After fixing the issue, add a narrow regression check for the mechanism if it can be reproduced at a lower level. Then repeat the original broader journey. This preserves defect protection and reduces future detection costs.
Common mistakes and conclusions
Dangerous approaches include increasing every timeout, ignoring exceptions, clicking again without checking state, and disabling cases without owners. They reduce visibility without establishing product correctness.
Systematic flaky-test management requires evidence, cause classification, and a verifiable fix. Stability comes from controlling data, time, dependencies, and waits, while preserving the initial failure as useful evidence even after a successful retry.