The deployment is 70 % of the project
The model is the part of the project everyone wants to talk about. The deployment is the part that determines whether the project survives. Below is the runbook we've converged on after a few dozen anomaly stations.Latency budgets that actually exist on a line
- Continuous flow, 200 mm/s, 100 mm part pitch → 500 ms part-to-part → model + IO budget < 200 ms
- Indexing station, 1 part/sec → ~700 ms budget after camera read and operator UI update
- Sample inspection, manual loading → 2-5 s budget, comfortable
The budget includes camera read, network transfer, preprocessing, model inference, post-processing, threshold check, and result publish. Most projects we've inherited had spent 100 % of the budget on the model and were surprised by the rest.
Practical numbers: PatchCore on a CPU is 200-500 ms per image at 512x512. EfficientAD on an RTX A2000 is 8-15 ms. Hailo-8 at INT8 with a converted FastFlow model is 5-12 ms. Pick your model around the budget, not the reverse.
Hot model reload, without pausing the line
The system has to swap models without dropping frames. The pattern:- Model loader runs in a separate process / thread
- New model loads + warms up on shadow data while old model serves traffic
- Atomic pointer swap once new model is ready
- Old model unload in next idle window
This is a 30-line change in code, and it's the difference between "we updated the model overnight" and "we stopped the line for an hour to update the model".
Drift detection, three signals
- Score distribution drift — the cheapest. Rolling 7-day quantiles of the anomaly score on accepted parts. Alert if the 95th percentile crosses 2σ from baseline.
- Embedding-space drift — more sensitive. Sample 10k recent good-part embeddings, compute MMD or KL vs the calibration set. Alert at threshold.
- Operator override rate — the most expensive but ground-truth. Track per-shift, per-camera. Anything trending up is an investigation.
We require all three. Score drift catches lighting changes. Embedding drift catches subtle camera or product changes. Override rate catches the cases where the first two missed.
The operator UI nobody plans for
The model's job is to flag. The operator's job is to decide. The UI between them determines whether they trust the system.What works:
- Single anomaly score, color-coded (green / yellow / red) — not a probability, not a heatmap by default
- Heatmap on click — the operator opens it to investigate, not by default. Defaults to noise.
- "Why did this score?" button — shows the most-anomalous patch with a similar good-patch reference. Closes the credibility gap.
- Override + reason code — every override captures a reason. Reasons feed retraining.
- No "confidence" displayed — operators interpret confidence inconsistently. Just show the score and the threshold.
Fallback to rule-based — keep it
Every anomaly station we ship has a rule-based reject path running in parallel. Out-of-bounds dimension, wrong color, missing component — anything that can be checked deterministically. The ML model catches what rules miss; the rules catch what the ML model gets confidently wrong.Belt and braces. It's not elegant but it ships.
What's your deployment stack? Curious about anyone running anomaly detection at the edge with no operator in the loop.
