Automation is often evaluated by how many tests it runs and how quickly it reports green or red. Those numbers matter, but they miss the quality that determines whether engineers will trust the system: what evidence remains after the run?

A pass should demonstrate that the expected behaviour was observed under a known configuration. A failure should narrow the investigation rather than open a debate about whether the test itself was confused. Evidence-first automation designs the run record before scaling the test count.

A verdict is a conclusion, not evidence

“Failed: expected READY” is a conclusion. It does not tell the reader:

  • which software and configuration were used;
  • what state the product started in;
  • which command was sent;
  • whether the interface acknowledged it;
  • what values were observed during the wait;
  • when the timeout occurred;
  • whether the product log, bus trace, measurement, and screenshot describe the same moment;
  • whether recovery changed the device before artifacts were collected.

When those details are missing, engineers rerun tests to obtain information that should have been captured the first time. Intermittent failures may disappear, and the most useful evidence is lost.

Give every run a correlation identity

Create one identifier for the execution and propagate it through the automation layers where possible.

The identifier can connect:

  • suite and test case;
  • device and bench allocation;
  • software, firmware, model, and configuration versions;
  • orchestration logs;
  • driver and device-control events;
  • traces from supporting services;
  • measurements and bus captures;
  • screenshots, images, or short video;
  • CI job and source revision;
  • recovery actions and final verdict.

Not every external device can store the identifier, but the automation system can preserve a mapping between the run, device timestamps, filenames, and capture windows.

Correlation is more valuable than simply collecting more data. Ten unrelated logs may create less evidence than three well-correlated signals.

Capture intent, observation, and decision

A useful test record separates three things:

  • Intent: what action the test requested and why.
  • Observation: what the product and test equipment actually reported.
  • Decision: which rule converted those observations into a verdict.

Consider a startup-time requirement. The run should preserve the start event, the definition of “ready,” the observed readiness signal, the measured interval, the allowed limit, and the resulting comparison.

If the readiness definition changes later, the team can see which rule produced the historical result. If only “pass” was stored, that context is gone.

Use a layered evidence model

Keep the main report readable while allowing deeper diagnosis.

Level 1: campaign summary

The campaign view should answer:

  • what was tested;
  • which build and environment were used;
  • how many scenarios completed, failed, were blocked, or were not executed;
  • whether the bench remained healthy;
  • where the evidence is stored.

Level 2: test narrative

The test-level view should show:

  • important actions and state transitions;
  • relevant observations and limits;
  • failure category;
  • recovery attempts;
  • links to selected artifacts.

Level 3: raw diagnostic artifacts

Retain the deeper material needed by specialists:

  • complete logs and traces;
  • bus or protocol captures;
  • measurement files;
  • images or clips;
  • crash dumps;
  • device diagnostics;
  • calibration and rig-health records.

The report should not paste every raw artifact into one enormous page. It should make the evidence discoverable and preserve its relationship to the test.

Capture before recovery changes the scene

Recovery is often destructive to evidence. A reboot clears volatile diagnostics. Reconnecting may reset interface counters. Returning a robot to its safe pose removes the exact physical state that caused a problem.

Define a failure-capture order:

  1. Freeze the high-level verdict context.
  2. Read volatile diagnostics.
  3. capture the relevant measurement and communication window.
  4. Take the required image or short clip.
  5. Record equipment and interlock status.
  6. Begin recovery.
  7. Record every recovery action and its result.

Safety remains the priority; an emergency safe-state action should never wait for perfect evidence. For ordinary failures, however, capture order dramatically improves diagnostic value.

Record test-system health

A product failure and a test-system failure must not look identical.

Include health evidence for components that can invalidate the verdict:

  • instrument connection and self-test;
  • calibration identity or validity;
  • robot and actuator state;
  • camera frame freshness and exposure checks;
  • communication error counters;
  • power supply mode and protection state;
  • disk space and artifact-write result;
  • clock or synchronisation status when timing matters.

If required evidence is missing, the correct verdict may be “inconclusive” or “blocked,” not product fail.

Make evidence retention intentional

Different artifacts have different value and size.

A practical policy might retain:

  • compact summaries and structured results for every run;
  • detailed logs for a defined period;
  • complete evidence for failures and release candidates;
  • short video windows around relevant events instead of endless recordings;
  • selected successful baselines for comparison;
  • checksums or digests where artifact integrity matters.

CI systems commonly treat test outputs, screenshots, logs, and crash data as workflow artifacts. Retention should be long enough for the team’s review and release cycle, while respecting storage cost, confidentiality, and applicable data rules.

Design the evidence contract before scaling

Before adding hundreds of tests, define a minimum evidence contract:

  • unique run identity;
  • product, software, configuration, and equipment identity;
  • action and state-transition timeline;
  • observed values and acceptance rules;
  • explicit failure category;
  • test-system health;
  • recovery history;
  • stable links to diagnostic artifacts.

Then review real failures with the engineers who diagnose them. Remove noise, add missing context, and automate the collection that repeatedly requires manual work.

The strongest automation does not merely execute without people. It returns enough structured evidence that people can make the next engineering decision quickly and confidently.

Source notes for review