Separating Product Defects from Test-Rig Failures

Editorial disclosure: This is a representative engagement scenario based on common engineering constraints. It does not describe a named client, claim completed work for a specific organization, or present invented performance figures. Replace intended outcomes with approved, measured results only when they are available.

Context

An embedded regression campaign produces failures that require manual reruns and bench inspection. Some are genuine product defects; others come from fixture wear, unstable communication, stale test data, camera drift, or orchestration errors.

The engagement would add an evidence-driven classification layer so a failed assertion does not automatically become a product verdict.

The problem

Blind reruns can hide intermittent product defects and normalize weak benches. Sending every failure to a product engineer creates noise, while treating every retry success as infrastructure trouble can suppress real issues.

The system needs an explicit taxonomy and observations that discriminate among product, bench, environment, automation, and unknown causes.

Engineering constraints

• Classification confidence varies by available evidence.
• Unknown must remain a valid outcome.
• Reruns can change state and contaminate diagnosis.
• The classifier itself must be reviewable and correctable.

The approach

We would begin with a failure taxonomy grounded in historical incidents. Each category maps to observable evidence and permitted actions. For example, loss of fixture feedback before a product response suggests a different path from a valid physical action followed by a diagnostic fault.

The runner preserves first-failure evidence, executes only category-specific health checks, and records any rerun as a separate attempt. Initial classification can be deterministic rules; statistical assistance is considered only after enough reviewed examples exist.

A review interface shows the evidence behind the classification and allows engineers to correct it. Corrections feed taxonomy and rule improvement rather than disappearing into comments.

Proposed system architecture

• Versioned failure taxonomy with unknown and mixed-cause states.
• Independent product, rig, environment, and orchestration health signals.
• Rule engine with explainable evidence references.
• Attempt graph that preserves first failure and all recovery actions.
• Human review and correction history.

Validation strategy

The pilot replays known incidents and injects fixture, communication, observer, and product faults. Success means the system presents the right evidence and escalation path, not that every case is forced into a confident label.

Reviewers track disagreement and unknown cases to identify missing sensors or ambiguous policies.

Intended operational outcome

The intended outcome is cleaner product feedback and more honest infrastructure health. Engineers can prioritize genuine defects, quarantine unreliable rigs, and see uncertainty instead of relying on hidden reruns.

Published claims should use reviewed incident data and state the taxonomy coverage and unresolved categories.

What a real engagement would require

• Historical failure examples and reviewer decisions.
• Access to product, fixture, environment, and runner telemetry.
• Agreement on escalation and rerun policy.
• Named owners for taxonomy changes and bench quarantine.

Source notes

• OpenTelemetry signals — A primary reference for correlating traces, metrics, and logs across a distributed workflow.
• Robot Framework User Guide — Execution, library, logging, and reporting concepts used in the automation architecture.
• Scikit-learn model evaluation — Reference for choosing metrics that reflect the actual cost of prediction errors.