Many unstable automation suites are blamed on test cases when the real problem lives one layer lower.

A test says “power on the device,” “connect,” “start measurement,” or “return to home position.” The underlying library sends a command and assumes the requested state now exists. When hardware is slow, already active, disconnected, partially initialised, or recovering from a previous failure, the test produces a timeout or a misleading assertion.

Adding sleeps may reduce the visible failure rate, but it does not make the system deterministic. A reliable device-control layer must model state, verify postconditions, bound every wait, and expose recovery as part of the result.

Treat commands as state transitions

“Start measurement” is not merely a message sent to an instrument. It is a requested transition:

Ready → Measuring

That transition has:

  • allowed source states;
  • a command or sequence of commands;
  • an expected completion condition;
  • a maximum duration;
  • observable intermediate behaviour;
  • defined failure outcomes;
  • a recovery or safe-state path.

This model immediately prevents several common errors. A start command can be rejected when the device is disconnected. A repeated start can be treated as idempotent when the device is already measuring. A timeout can report the last observed state instead of only saying that a keyword failed.

Make readiness explicit

Connection is not readiness.

A serial port can be open while a device is booting. A network socket can exist while the service is loading configuration. A robot controller can respond while its axes are not enabled. A camera can stream frames before exposure and focus have stabilised.

Define readiness through product-specific evidence:

  • a status register reaches an allowed value;
  • required services or communication channels respond;
  • configuration and firmware identity match expectations;
  • health checks pass;
  • the device reports no blocking diagnostic state;
  • required instruments have valid calibration or self-test status.

The control layer should offer ensure_ready() rather than forcing every test to repeat these rules.

Prefer idempotent operations

An idempotent operation can be requested repeatedly without causing additional unintended change.

For device automation, useful examples include:

  • ensure_power_on() instead of toggle_power();
  • ensure_output_off() instead of press_output_button();
  • move_to_safe_pose() instead of a relative movement;
  • ensure_application_closed() instead of sending an unconditional close sequence.

Physical systems are not perfectly idempotent, but the interface can move in that direction by observing current state before acting and verifying state afterward.

Avoid hiding every inconsistency behind a retry. A retry is appropriate only when the failure is understood, transient, bounded, and visible. Repeating a physical action without knowing whether the first action occurred can damage equipment or invalidate the scenario.

Replace sleeps with bounded waits

A fixed sleep encodes neither the expected condition nor the actual performance of the device.

Use a bounded wait that records:

  • the condition being awaited;
  • the polling or event mechanism;
  • start and completion timestamps;
  • the last observed value;
  • the timeout limit;
  • evidence collected during the wait.

Fast devices continue immediately. Slow but valid devices remain supported. Failed devices produce a diagnostic message that points to the missing postcondition.

Where polling is required, choose an interval that respects both response time and interface load. More frequent polling is not always better, especially for slow buses or instruments.

Separate the driver from the workflow

A maintainable architecture normally has at least three layers:

  • Driver: protocol-level operations such as serial frames, REST calls, GPIO, bus messages, or vendor API calls.
  • Device service: state, readiness, postconditions, timeouts, recovery, and normalised errors.
  • Test keywords or workflows: domain language such as “prepare product for wake-up validation.”

Robot Framework supports this separation well: Python libraries can expose stable domain keywords while the lower-level implementation remains ordinary Python. Tests stay readable, and protocol details do not leak into every suite.

The same separation also makes migration easier. Replacing an instrument model should mostly affect the driver and configuration, not the business-level test intent.

Make recovery a first-class operation

Recovery is not “try again.”

A recovery flow should answer:

  • What state might the device currently be in?
  • Which operations are safe from that state?
  • Can communication be re-established without changing the test condition?
  • Does the product require a soft reset, hard reset, power cycle, or operator intervention?
  • Which evidence must be preserved before recovery changes the state?
  • When should the campaign stop rather than continue?

Use a recovery budget. One known communication reconnect may be acceptable. Repeated power cycles across many tests are evidence of a system problem and should fail the campaign or quarantine the equipment.

Normalise failures without erasing detail

Different drivers produce different exceptions and status codes. The device service should map them into a small set of meaningful categories while keeping the original detail:

  • configuration error;
  • unavailable device;
  • communication failure;
  • invalid source state;
  • transition timeout;
  • failed postcondition;
  • safety or interlock stop;
  • recovery failed.

This lets the orchestration and reporting layer respond consistently. It also makes failure trends useful: a rise in communication failures tells a different story from a rise in product postcondition failures.

A practical implementation sequence

Start with one unstable device and one valuable operation:

  1. List the states that matter to that operation.
  2. Define allowed transitions and observable postconditions.
  3. Replace fixed sleeps with bounded waits.
  4. Add explicit safe-state and recovery operations.
  5. Capture transition timing and last observed state.
  6. Expose one high-level keyword to the tests.
  7. Inject communication and timeout failures to verify recovery.

The result should make the test case simpler because complexity has moved into the correct layer.

Reliable automation does not mean hardware never fails. It means the system knows what it was trying to do, what state it observed, how long it waited, whether recovery was attempted, and why execution continued or stopped.

Source notes for review