“The device shall respond within 200 milliseconds” looks like an easy requirement to automate. Record a start time, wait for the response, subtract, and compare.

The difficult part is deciding what starts the interval, what ends it, which clock measures it, which operating conditions apply, how much the observer contributes, and what one slow result means.

Without those definitions, a timing test can alternate between green and red while the product remains unchanged.

Define the two observable events

A timing requirement is an interval between two events. Name both events in terms the test system can observe.

The start might be:

  • the physical edge at a product input;
  • the transmission of the final byte of a command;
  • the acknowledgement from an interface;
  • the moment an actuator contacts a control.

The end might be:

  • the first valid output edge;
  • a state transition on a bus;
  • a rendered display frame;
  • a diagnostic state;
  • completion of an application transaction.

These choices are not equivalent. Measuring from host API call to host callback includes driver, transport, scheduling, and observer latency. That may be correct for an end-to-end user requirement, but it should not be mistaken for internal product execution time.

Write the measurement boundary into the test result.

Use the right clock

Elapsed time should be measured with a monotonic clock. A wall clock can be corrected by time synchronisation or manual adjustment and may jump.

For events captured by one process, a high-resolution monotonic counter is usually appropriate. For events captured by different instruments or hosts, establish a shared trigger, synchronised clock, or measured offset.

Record clock resolution and the timestamp source. More decimal places do not create more measurement accuracy.

Where hardware timestamps are available, understand where in the interface stack they are created. A timestamp generated after buffering or decoding may exclude or include different delays than expected.

Account for the observer

Every observation path adds uncertainty:

  • polling interval;
  • camera frame period and exposure;
  • bus adapter buffering;
  • serial-driver buffering;
  • operating-system scheduling;
  • network transit;
  • instrument trigger and acquisition delay;
  • application logging delay.

Estimate or measure the observation error. If a camera runs at a low frame rate, it cannot support a narrow timing claim merely because the software stores timestamps with microsecond precision.

For critical requirements, compare the automation path with an independent instrument or reference setup.

Control the operating conditions

Timing behaviour may change with:

  • processor and bus load;
  • temperature and voltage;
  • cache and storage state;
  • network quality;
  • device mode;
  • first execution versus warmed execution;
  • background services;
  • software and configuration versions.

Define the intended operating envelope. If the requirement applies under maximum load, create and verify that load. If the requirement applies after wake-up, do not measure only an already warmed path.

Attach the conditions to the evidence so a reviewer can compare runs.

Replace sleeps with state-aware waits

A fixed sleep is not a timing measurement and is not reliable synchronisation.

Use a bounded wait that observes the end condition repeatedly or subscribes to the relevant event. Preserve the first observation time, not the time at which a later assertion happened to run.

Separate two ideas:

  • measurement deadline: the maximum interval permitted by the requirement;
  • test timeout: the maximum period after which the automation stops waiting and collects failure evidence.

The timeout may be longer than the requirement deadline so the test can distinguish “late response” from “no response.”

Collect a distribution

One passing observation proves very little about a variable system. Repeat the scenario under controlled conditions and examine the distribution.

Useful summaries can include:

  • minimum and maximum;
  • median and selected percentiles;
  • number of observations;
  • failures beyond the requirement boundary;
  • warm versus cold groups;
  • condition-specific slices.

Do not invent an acceptable percentile after seeing the data. The decision rule should follow the requirement and the consequence of missing it.

A hard real-time requirement may treat any observed deadline miss as a failure. A user-experience target may be expressed through a percentile and an explicit outlier policy. The test should reflect the product contract.

Preserve every miss

Aggregates can hide the event that matters. For every missed deadline, retain:

  • raw start and end timestamps;
  • the complete event sequence around the miss;
  • device load and mode;
  • rig and host health;
  • relevant logs, traces, and measurements;
  • software, hardware, and configuration versions.

The slowest observation is often where contention, priority inversion, resource exhaustion, or an unexpected retry becomes visible.

Distinguish product variation from test variation

Run diagnostic experiments:

  • measure the observation path against a known signal;
  • vary polling interval and confirm the measured result converges;
  • repeat on an idle and loaded test host;
  • capture the same event through two independent channels;
  • inject a controlled delay and verify that the automation detects it;
  • compare multiple rigs or instruments.

If the measured distribution changes when only the host load changes, the observer is contributing materially.

Avoid the retry trap

Retrying a failed timing test and accepting the next pass changes the requirement to “eventually responds within the limit on one attempt.” That may not be the intended product behaviour.

A retry can be useful as a diagnostic experiment, but preserve the original miss and classify the repeated result. Do not erase it from the report.

Make the result reviewable

A strong timing report states:

  • event definitions;
  • measurement boundary and clock;
  • operating conditions;
  • instrument and observer characteristics;
  • requirement and decision rule;
  • distribution and sample count;
  • evidence for every miss;
  • known uncertainty.

This turns a fragile stopwatch check into a defensible validation method.

Timing tests do not become stable by widening the limit until they pass. They become stable when the system defines what it measures, controls the conditions, understands the observer, and treats variation as evidence.

Source notes for review