A flaky test produces different verdicts without a relevant change to the system being evaluated. The immediate response is often to retry it.

A retry can reveal that the result is not reproducible. It cannot explain why.

If the next run passes, the underlying cause may still be:

  • state left by another test;
  • an incorrect wait;
  • a race condition in the product;
  • an unstable external service;
  • a saturated test host;
  • a damaged cable;
  • a measurement near a physical threshold.

Calling all of those “test flakiness” places different engineering problems into one bucket and gives them the same treatment.

Preserve the first failure

The first rule is simple: a passing retry must not erase the original result.

Retain:

  • the original verdict and evidence;
  • each retry verdict;
  • whether the environment was reset between attempts;
  • timing and state differences;
  • product, framework, dependency, host, and rig versions.

A retry without a reset tests persistence. A retry after a full reset tests reproducibility from a known condition. Those are different experiments and should be labelled accordingly.

Classify by the component that introduced variation

A practical taxonomy includes:

Test logic and data

Examples:

  • incomplete setup or cleanup;
  • order dependency;
  • shared mutable data;
  • unstable selector or observation;
  • arbitrary sleeps;
  • assertion near an undefined threshold;
  • use of wall-clock time;
  • accidental dependence on locale or environment.

Test framework and orchestration

Examples:

  • resource collisions between parallel jobs;
  • worker scheduling;
  • incorrect timeout propagation;
  • lost artifacts;
  • framework-level state;
  • incompatible plugin or library versions.

System under test

Examples:

  • race conditions;
  • uninitialised state;
  • nondeterministic startup;
  • resource leak;
  • real intermittent hardware fault;
  • timing behaviour sensitive to load.

A flaky result can reveal a real product defect. Automatically quarantining it as a test problem can discard valuable evidence.

External dependencies

Examples:

  • service availability;
  • rate limiting;
  • eventual consistency;
  • changing test data;
  • network delay;
  • authentication or certificate renewal.

Host and test infrastructure

Examples:

  • CPU, memory, storage, or network contention;
  • instrument or driver fault;
  • fixture wear;
  • temperature or power instability;
  • camera or sensor freshness;
  • operating-system update.

The classification can begin as “unknown,” but unknown should create an investigation path rather than permanent tolerance.

Collect discriminating evidence

Evidence should help distinguish one cause family from another.

Useful signals include:

  • initial state and setup result;
  • event timestamps and wait history;
  • resource utilisation;
  • dependency request and response identifiers;
  • product logs and traces;
  • rig-health signals;
  • execution order and parallel neighbours;
  • exact software and configuration versions.

If a test fails because an element or state did not appear, record what was observed during the wait. “Timeout” alone does not tell whether the product was slow, the observer was stale, or the expected event never happened.

Reproduce by changing one dimension

After preserving the failure, use targeted experiments:

  • run the test alone;
  • run it after its usual predecessor;
  • reverse suite order;
  • hold the product version constant and change the rig;
  • hold the rig constant and change the product unit;
  • reduce host load;
  • replace a dependency with a controlled fake;
  • increase logging without changing the timeout;
  • repeat from a verified initial state;
  • inject a known delay or fault.

Each experiment should test a hypothesis. Repeating the same opaque workflow twenty times may estimate failure frequency while revealing little about the cause.

Fix synchronization at the state boundary

Arbitrary sleeps are a common source of instability. Replace them with an explicit condition and a bounded wait:

  • wait for the application state;
  • wait for a new frame, not any frame;
  • wait for a device response with matching correlation;
  • wait for measured voltage to enter and remain inside a range;
  • wait for a job to reach a terminal state.

The condition should describe the state needed for the next operation. The timeout should produce evidence of what prevented that state.

Treat thresholds as engineering decisions

Noise around a measurement limit can cause alternating verdicts. Before adding tolerance, examine:

  • sensor and instrument accuracy;
  • resolution and sampling;
  • calibration;
  • environmental variation;
  • product requirement;
  • decision consequence.

Use hysteresis or an uncertainty policy only when justified. A wider limit chosen because the test sometimes fails is not a stability fix.

Quarantine with ownership and an exit rule

Quarantine may protect the main signal while an unstable test is investigated. It should contain:

  • named owner;
  • first observed date;
  • failure examples;
  • current classification;
  • operational impact;
  • investigation or repair action;
  • review date and exit criteria.

A permanently ignored test is no longer a test. If the behaviour does not justify repair, remove or redesign the test explicitly.

Track rates by cause, not only by test

Useful programme-level views include:

  • first-run failure rate;
  • pass-on-retry rate;
  • failure family;
  • affected rig, host, dependency, or product version;
  • time to classification;
  • time to repair;
  • recurrence after repair.

Do not use the retry pass rate as proof of test health. A test that passes on its second attempt every day is producing a consistently poor first signal.

Make reliability part of test ownership

New tests should define:

  • initial-state contract;
  • synchronisation strategy;
  • evidence on timeout;
  • external dependencies;
  • parallelisation constraints;
  • expected execution envelope;
  • owner and maintenance path.

Review those properties with the test logic.

The purpose of automation is a fast, reliable, precise feedback loop. Retries are one diagnostic tool inside that loop. Classification turns inconsistent results into repairable engineering work.

Source notes for review