Computer Vision Consulting: The Mistakes That Kill Projects
0
0
Computer vision projects fail more often than most organizations expect.
Not spectacularly. Quietly. The model performs well in development and poorly in production. The accuracy that looked promising in testing doesn’t hold up on real operational data. The system works for six months and then degrades as conditions change. The deployment succeeds technically but the organization doesn’t trust the outputs enough to act on them.
These failures share common roots. Understanding them before a project starts is worth more than any technical capability comparison between consulting firms.
Mistake 1: Starting With the Model Instead of the Problem
This is the most common and most consequential mistake in computer vision consulting engagements.
The sequence that produces failures: identify a problem that seems like a computer vision problem, select a model architecture, collect some training data, train the model, discover it doesn’t perform well enough, try to fix it with more data or a different architecture, eventually conclude that “the technology isn’t ready.”
The sequence that produces success: precisely define what the system needs to do and what performance is required, assess whether the available data and imaging conditions can support that performance level, design the training data collection strategy before collecting a single image, then select the architecture.
The difference isn’t technical sophistication. It’s the order of operations. Problem definition before model selection isn’t a process preference — it’s the thing that determines whether the model is solving the right problem.
What good computer vision consulting looks like here: A structured feasibility assessment before any development commitment. The output is a clear answer: yes, this is solvable with CV under these conditions, or no, here’s what would need to be different.
Mistake 2: Underestimating the Training Data Problem
Training data is the work. Everything else is downstream of it.
This sounds obvious. It’s consistently underestimated. Here’s why.
The amount of data required isn’t fixed — it depends on the problem complexity, the number of classes, the visual variability within each class, and the accuracy requirements. For some problems, a few hundred well-labeled examples per class are sufficient. For others, tens of thousands are needed and accuracy is still limited by data diversity rather than model architecture.
The diversity of data matters as much as the volume. A model trained on defect images from day shifts in summer lighting will underperform on night shifts in winter. A model trained on products from one production line may struggle on products from a slightly different line. The training data needs to represent the full range of conditions the model will encounter in production — not just the common cases.
The labeling quality determines the ceiling on model performance. Inconsistent labeling — where different annotators classify the same image differently — introduces noise that limits accuracy regardless of how sophisticated the model is. Label quality needs to be managed, not assumed.
What good computer vision consulting looks like here: Data strategy as a first-class deliverable. Before the first model is trained, there should be a documented plan for how much data is needed, what diversity it needs to cover, how it will be collected, who will label it, what the annotation guidelines are, and how labeling quality will be validated.
Mistake 3: Evaluating on the Wrong Data
Test performance that doesn’t predict production performance is not test performance — it’s false confidence.
The most common form of this mistake: the test set is drawn from the same distribution as the training data. The model learns the training distribution well, performs well on the test set, and fails on production data that has different characteristics.
Common sources of distribution mismatch:
- Seasonal or time-of-day variation in lighting that wasn’t represented in training
- Equipment variation — different cameras, different lens wear — that wasn’t captured
- Product variation over time — new formulations, new suppliers — that postdates the training data
- Operational variation — different operators, different handling — that wasn’t in the controlled training environment
Good evaluation for computer vision uses test sets that are specifically designed to capture production distribution rather than drawn conveniently from training data. This requires deliberate effort and, for the first deployment in a new environment, often requires collecting test data from the production environment itself.
What good computer vision consulting looks like here: Separate evaluation strategy designed before model development. The test set should be built to challenge the model, not to confirm it works.
Mistake 4: Treating Deployment as the Finish Line
The model is in production. The project is done. This is where many computer vision engagements end — and where many production systems begin to fail.
Computer vision models in production face conditions that don’t stay constant:
- Equipment changes — new cameras, lens replacement, camera repositioning
- Environmental changes — seasonal lighting variation, facility modifications
- Product changes — new SKUs, new formulations, new suppliers
- Process changes — new operators, new handling procedures, modified production line
Any of these can shift the input distribution away from what the model was trained on. Without monitoring, the shift is invisible until someone notices the outputs are wrong. By then, the gap between what the model sees and what it’s trained on may be large.
What good computer vision consulting looks like here: Monitoring designed alongside the model, not after deployment. Specifically: input distribution monitoring that detects when the images coming in look different from training data, output distribution monitoring that detects when prediction confidence patterns change, and a retraining process that’s documented and tested before it’s needed.
Mistake 5: Accuracy Requirements Set After Development
This one is particularly expensive because it turns successful model development into failed project delivery.
The pattern: development proceeds without defined accuracy requirements. The model achieves, say, 92% accuracy. The client, seeing the number for the first time, decides 92% isn’t good enough — they need 99%. The consulting firm either tries to improve the model (often hitting diminishing returns) or disputes whether 99% was ever the requirement.
Accuracy requirements should be defined in the problem definition phase, before any development begins. They should be:
- Specific to the task: overall accuracy is rarely the right metric. Precision and recall on specific defect classes, or accuracy on specific object types, are usually more meaningful.
- Business-grounded: what accuracy level is required for the system to deliver the business value that justified the project? This is a business question, not a technical one.
- Achievable: some accuracy requirements are not achievable with the available data. Knowing this before development is the most valuable output of the feasibility assessment.
What good computer vision consulting looks like here: Performance thresholds documented in the problem definition phase, agreed by both sides, and used as the explicit success criteria for evaluation.
Mistake 6: Skipping the Human Oversight Design
Computer vision systems make mistakes. All of them, always. The question is what happens when they do.
For many applications — high-volume sortation, for example — automated decisions at 98% accuracy are valuable because the cost of 2% errors is acceptable and the cost of manual inspection at volume is not. For other applications — safety-critical inspection, medical imaging, any system affecting individuals’ legal rights or significant financial decisions — 2% errors at high volume is not acceptable and human review for low-confidence outputs is essential.
The oversight model — which outputs get automated action, which get flagged for human review, what the escalation threshold is, who the human reviewers are and how they’re trained — needs to be designed before deployment, not discovered after an incident.
What good computer vision consulting looks like here: Explicit oversight design as a deliverable. Documented confidence thresholds for automated action. Human review workflow for flagged outputs. Performance metrics for the human review layer.
The Common Thread
Every mistake above has the same underlying cause: insufficient upfront work before development begins.
Problem definition that’s vague allows architecture decisions to be made without understanding the problem. Training data strategy that’s skipped produces data that doesn’t represent production. Evaluation design that’s deferred produces metrics that don’t predict production performance. Monitoring that’s planned post-deployment catches drift only after it’s caused harm. Accuracy requirements set during development rather than before become moving targets. Oversight design skipped becomes an incident waiting to happen.
The computer vision consulting engagements that succeed invest heavily in the work before development. The ones that fail rush to development and discover the gaps in production.
The mistakes above are preventable. They’re also consistent enough across projects and industries that a consulting firm with real production experience will have a process designed to avoid them.
The questions to ask before engaging any computer vision consulting firm: how do they approach feasibility assessment, training data strategy, evaluation design, monitoring, accuracy requirements, and oversight? Specific answers indicate experience. Vague answers indicate the experience that comes from having made these mistakes on client projects.
Artificial Intelligence – The Data Scientist