The expensive part of a custom machine-learning project is rarely the first training run. It is discovering late that the target was ambiguous, the labels were unreliable, the available data did not represent deployment, or the required latency made the chosen approach impractical.

A good feasibility study is designed to discover those constraints early. Its result is not necessarily a model. It is an evidence-based decision to proceed, change the formulation, collect different data, use a simpler method, or stop.

Start with the decision

“Use machine learning to inspect products” is not yet a problem definition. Ask what decision the system should support and what happens after its prediction.

For example:

  • What object, event, or condition is being classified, detected, estimated, or forecast?
  • Who or what consumes the result?
  • Is the output advisory, automatically actionable, or safety relevant?
  • What is the cost of a false positive, false negative, delayed result, or abstention?
  • What simpler rule or measurement is used today?

The target should be observable and linked to a useful action. If qualified people cannot agree on the correct label from the available evidence, the model will inherit that ambiguity.

Audit labels before architectures

Labels are a measurement process. Document who creates them, from which evidence, with which instructions, and how disagreements are resolved.

Inspect:

  • class definitions and boundary cases;
  • missing and default labels;
  • label balance;
  • duplicate or related samples;
  • disagreement between annotators;
  • changes in policy over time;
  • labels inferred from a proxy rather than the real outcome.

A large dataset with inconsistent labels can be less useful than a smaller, carefully defined set. Sample the data manually and turn ambiguous cases into explicit policy decisions.

For novel problems, a short annotation exercise is often more informative than an early architecture search. It reveals whether the task can be specified consistently and how much evidence a human needs to make the decision.

Map the data to deployment

Training data should represent the conditions in which the system will operate. That includes not just typical cases but sources of variation:

  • hardware and sensor versions;
  • sites, users, and product variants;
  • lighting, environment, and background;
  • seasonal or process changes;
  • rare but important classes;
  • failure and recovery conditions.

Record provenance. Samples created by the same session, device, person, or product unit may be highly related. Randomly splitting those samples can place near-duplicates in training and test sets, producing an optimistic evaluation.

Split along the boundary that represents future use. Depending on the application, that may mean holding out time periods, devices, sites, users, or production batches.

Establish the simplest useful baseline

The baseline is not an administrative formality. It tells the team whether machine learning adds value and how much complexity may be justified.

Useful baselines include:

  • the current manual or rules-based process;
  • a simple threshold;
  • a majority or frequency-based predictor;
  • linear or tree-based models;
  • a pretrained model used without extensive customisation.

If a simple method meets the operational requirement, it may be the correct solution. It will usually be easier to explain, deploy, monitor, and maintain.

If the baseline fails, inspect why. The result may reveal poor signal, noisy labels, leakage, or a target that needs to be reformulated.

Choose metrics from error costs

Accuracy is rarely enough. In an imbalanced problem, a model can achieve a high accuracy while missing the class that matters.

Select metrics that reflect the decision:

  • precision when false alarms are expensive;
  • recall when missed cases are expensive;
  • precision–recall behaviour across thresholds;
  • class-specific confusion matrices;
  • calibration when probabilities influence a downstream decision;
  • latency, memory, or energy when deployment is constrained;
  • abstention and review rate when humans handle uncertain cases.

Evaluate by meaningful slices, not only the overall average. A model that appears acceptable overall may fail on one device revision or operating condition.

Define the minimum acceptable result before running a broad experiment. Otherwise, the team can be tempted to reinterpret whatever number appears.

Test a thin end-to-end path

Model quality is only one feasibility dimension. Build the smallest path that resembles deployment:

  1. acquire a representative input;
  2. apply the intended preprocessing;
  3. run the candidate model or baseline;
  4. apply threshold and business rules;
  5. deliver the result to the consuming workflow;
  6. preserve evidence and timing.

This exposes integration, latency, formatting, and observability constraints. A promising notebook result may depend on information or compute that will not exist in the target environment.

Include operational constraints in the decision

The feasibility report should address:

  • target hardware and runtime;
  • throughput and latency envelope;
  • memory, power, and thermal limits;
  • privacy and data-residency requirements;
  • update and rollback process;
  • monitoring signals and ground-truth delay;
  • fallback behaviour when confidence or input quality is poor;
  • ownership of data, model, and service maintenance.

These are part of the model’s usefulness, not post-project deployment details.

End with Go, Change, or Stop

A good feasibility study can reach three valid outcomes:

  • Go: the signal, data, baseline, metrics, and operating constraints support a focused implementation.
  • Change: the use case may be viable after relabelling, new data collection, a different target, additional sensors, or a simpler workflow.
  • Stop: the evidence does not justify further model development under the current constraints.

For a Go decision, define the next experiment and acceptance criteria. For Change, define what evidence would resolve the uncertainty. For Stop, record why so the same unsupported assumption is not revived later.

This discipline keeps custom ML connected to an operational result. Training becomes one instrument in the investigation, not the project’s purpose.

Source notes for review