A computer-vision model does not encounter an average image in production. It encounters a specific camera, lens, distance, illumination, motion, product variant, background, and moment in the process.

An overall evaluation score compresses those conditions into one number. That number can be useful for comparison, but it can also hide a failure concentrated in the exact slice that matters operationally.

Validate the operating envelope, not only the dataset average.

Define the operating design domain

Write down the conditions under which the vision system is expected to work. Borrowing the language of engineered systems, this is its operational design domain.

Depending on the application, include:

  • lighting direction, intensity, colour, and flicker;
  • camera and lens versions;
  • focus, exposure, contamination, and vibration;
  • object distance, scale, orientation, and position;
  • motion speed and blur;
  • occlusion and partial visibility;
  • product variants and acceptable appearance variation;
  • backgrounds, fixtures, and reflections;
  • environmental temperature or moisture where they affect imaging;
  • upstream process states that change what the camera sees.

For each factor, define a normal range, boundary conditions, and conditions in which the system should reject the input rather than make a confident decision.

This transforms “works in the lab” into a testable claim.

Build a condition matrix

A dataset count by class is not enough. Cross the target classes with the conditions that can change performance.

The matrix will not always be complete; some combinations may be impossible or rare. Its value is making coverage visible. Empty cells become explicit data gaps rather than invisible assumptions.

Track provenance for every sample:

  • capture site and session;
  • camera, lens, and firmware;
  • product or batch identity;
  • relevant environmental settings;
  • annotation version and reviewer;
  • transformations applied after capture.

Provenance supports both representative splitting and diagnosis.

Prevent related samples from leaking across splits

Video frames, burst captures, and repeated images of the same object are strongly related. If adjacent frames appear in training and test sets, the evaluation may reward memorisation of a scene rather than generalisation.

Group related samples before splitting. Depending on the intended deployment, hold out entire:

  • capture sessions;
  • product units or batches;
  • cameras;
  • sites;
  • users;
  • time periods.

Choose the split boundary that resembles the novelty the system will encounter after deployment.

Keep a final test set isolated from routine model selection. Use separate validation data for threshold and architecture decisions.

Report slices and error types

Overall precision, recall, or detection quality should be accompanied by results for the operational slices.

Ask:

  • Which conditions create false positives?
  • Which conditions create false negatives?
  • Does confidence become unreliable in a weak slice?
  • Are small objects or boundary positions affected?
  • Does one hardware revision behave differently?
  • What happens at the edge of the permitted lighting range?

Use a confusion matrix or class-specific metrics to show which errors occur. Precision–recall analysis is useful when the operating threshold must balance missed detections against false alarms.

Do not choose the threshold only from a global curve. A threshold that looks reasonable overall may be unsuitable for a high-cost condition.

Test image quality as an input contract

The system should distinguish “target absent” from “image not suitable for a decision.”

Add input-quality checks where practical:

  • exposure and contrast;
  • sharpness or motion blur;
  • camera availability;
  • obstruction or contamination;
  • expected field of view;
  • frame freshness;
  • plausible colour or intensity distribution.

When the input violates the contract, the safe output may be “invalid image,” followed by recapture, cleaning, recalibration, or human review. Forcing every image into a business class creates confident-looking errors.

Stress the boundaries deliberately

Representative data shows typical operation. Stress tests show where the system stops being dependable.

Vary one condition at a time when possible:

  • reduce illumination toward the boundary;
  • introduce increasing blur;
  • move the target toward the field edge;
  • add controlled occlusion;
  • change background texture or reflection;
  • contaminate or defocus the lens in measured steps.

The goal is not to create arbitrary synthetic difficulty. It is to map the transition from acceptable to unacceptable behaviour and decide how the product should respond near that boundary.

Robotic and automated validation rigs can make these tests repeatable by controlling position, lighting, motion, and product state while synchronising images with independent evidence.

Investigate failures as condition clusters

When a field error arrives, avoid treating it as an isolated bad image. Attach the relevant condition metadata and look for clusters.

A set of failures may reveal:

  • one camera calibration;
  • one production station;
  • a new material finish;
  • seasonal lighting;
  • a changed preprocessing version;
  • an upstream mechanical shift.

This can lead to a system fix rather than another undirected training run.

Monitor inputs and outcomes in operation

Production monitoring should cover the imaging system and the model:

  • input-quality failures;
  • condition and source distributions;
  • prediction and confidence distributions;
  • processing latency and dropped frames;
  • model, preprocessing, camera, and configuration versions;
  • reviewed errors and delayed ground truth;
  • rates of abstention, recapture, or human escalation.

Alert thresholds should reflect the process. A sudden change in exposure or camera identity may be actionable before outcome labels are available.

Retain samples only under an explicit privacy, security, and data-governance policy. Often, compact metadata and selected reviewed cases provide more value than indiscriminate image retention.

Replace one reassuring number with an operating map

The purpose of a condition matrix is not to create more charts. It is to connect model evaluation to the environment the product must survive.

A release decision should state where the system works, where it requires safeguards, and where it is not authorised to decide. It should name weak slices and the evidence used to improve or contain them.

Computer vision fails in concrete conditions. The evaluation should be concrete enough to reveal them.

Source notes for review