- We asked five frontier vision-language models to label 237 items from real Metal America jobs (build stage, next step, whether a slab is finished, which photo is later) and to estimate ground slope on 106 slab sites.
- Categorical labels are ready for automation. A majority vote of the five models was right 98% of the time, against 96% for the best single model. All five agreed on 86% of labels, at 98% accuracy.
- Model agreement cannot catch shared blind spots. The only labels all five models got wrong were cases where, on review, our own answer key was wrong. Labels need checking against real outcomes, including human ones.
- Measurement labels are not ready. All five vision-language models read the ground flatter than it is. None of them labeled a single steep or severe lot correctly, and their median was still 3.4 points of grade off.
Definition. AI data labeling is using models to automatically tag training data, such as which action is happening in a video and whether it succeeded.
Why is data labeling the bottleneck in robot learning?
A robot learns a task from examples that say what happened, when each step started and ended, and whether it worked. Those annotations are the labels. In most robotics datasets a person writes them by hand, which costs time for every hour of footage and gets harder as tasks get longer. A construction job has dozens of steps across several trades. If every hour of crew footage needed a person to label it, the data would not scale.
This study asks a narrow, practical question. Which kinds of construction labels can AI write today without a person, how accurate are they, and how would we know when to trust them?
What did we test?
We reused the raw outputs from two pilots on our own job data and analyzed them together. In the first, five models (GPT-5.5, GPT-5.4 mini, Claude Opus 5.5, Claude Sonnet 5.5, and Claude Haiku 5.5) labeled 90 photos from 13 jobs on four tasks. In the second, the same five models estimated the slope of 106 slab sites from crew photos, scored against 1-meter USGS lidar elevation. We asked three questions. How accurate is each model on each label type? Does combining models help? Does model agreement tell us which labels to trust?
How accurate is each model on each label type?
| Model | Build stage | Next step | Slab finished | Which photo is later |
|---|---|---|---|---|
| GPT-5.5 | 95% | 95% | 94% | 98% |
| GPT-5.4 mini | 95% | 95% | 97% | 98% |
| Claude Opus 5.5 | 94% | 95% | 91% | 98% |
| Claude Sonnet 5.5 | 95% | 92% | 91% | 98% |
| Claude Haiku 5.5 | 96% | 88% | 94% | 100% |
| Majority vote of all 5 | 99% | 99% | 97% | 98% |
| Guess the most common answer | 31% | 34% | 65% | 50% |
Every model is far above the naive baseline on every categorical task. Combining them helps. The majority answer of all five beat every single model, 98% against 96% for the best one (GPT-5.4 mini).
Can model agreement decide which labels to auto-accept?
| Auto-accept a label when | Share of labels auto-accepted | Accuracy of auto-accepted labels |
|---|---|---|
| All 5 agree | 86% | 98% |
| At least 4 of 5 agree | 95% | 99% |
| At least 3 of 5 agree | 100% | 98% |
On this data, mostly. When all five models agreed, they were right 98% of the time, and that covered 86% of all labels. Accepting labels where at least four of five agree covered 95% at 99%. In a real pipeline, a person would only need to look at a small minority of labels.
Agreement has a hard limit, though. When models disagreed they were still right 97% of the time, so disagreement did not reliably flag errors. And the errors that mattered most were unanimous.
What happens when every model disagrees with the answer key?
We looked at every label where all five models disagreed with our answer key. In one, the photo shows vapor barrier and rebar before the pour. Our label said the slab was already poured, and all five models were right. In the other, the slab carries a finished tag in our files, but a power trowel is still sitting on it. The models said it was not finished yet, which is a reasonable reading.
This is the most useful finding in the study. When five independent models confidently disagree with a label, check the label first. Human labels have errors too, and this answer key was written without field review. The fix is the same for both: check labels against what actually happened on the job.
Can AI measure site slope well enough to label it?
| Method | Mean error (pts) | Median error (pts) | Exact price band | Within one band | Steep and severe lots right | Paid-grading lots caught | Bias (pts) |
|---|---|---|---|---|---|---|---|
| Claude Sonnet 5.5 | 3.4 | 1.8 | 38% | 82% | 0% | 58% | -2.1 |
| GPT-5.5 | 3.4 | 2.4 | 34% | 88% | 0% | 91% | -0.5 |
| Claude Opus 5.5 | 3.4 | 1.8 | 36% | 81% | 0% | 63% | -1.9 |
| GPT-5.4 mini | 3.7 | 3.1 | 22% | 86% | 0% | 98% | -0.0 |
| Claude Haiku 5.5 | 3.8 | 2.8 | 33% | 76% | 0% | 81% | -1.4 |
| Median of the 5 models | 3.4 | 39% | |||||
| Depth Anything V2 (monocular depth) | 9.1 | 4.7 | 28% | 59% | 12% | 72% | +4.5 |
| Baseline: call every lot flat | 4.1 | 1.3 | 46% | 63% | 0% | 0% | -3.9 |
| Reference noise check | 2.3 | 1.1 | 65% | 94% | 44% | 93% | +0.1 |
Not yet. Claude Sonnet 5.5 had the lowest error (3.38 points) and GPT-5.5 was a close second (3.39), but no model landed the exact price band more often than simply calling every lot flat. The median of all five models did not help either (3.4 points). GPT-5.5 was the most useful for triage, catching 91% of lots that need paid grading.
The five vision-language models fail in a consistent direction. Every one of them reads the ground flatter than it is, while the monocular depth model errs the other way and much more widely. The worst site was a 26% grade that the models put at 8%. The next two were 22% read as 5% and 20% read as 4%.
| Method | Flat (0 to 2%) | Slight (3 to 5%) | Moderate (6 to 10%) | Steep (11 to 20%) | Severe (21%+) |
|---|---|---|---|---|---|
| Sites in band | 49 | 18 | 23 | 12 | 4 |
| Claude Sonnet 5.5 | 71% | 11% | 13% | 0% | 0% |
| GPT-5.5 | 31% | 78% | 30% | 0% | 0% |
| GPT-5.4 mini | 4% | 78% | 30% | 0% | 0% |
| Baseline: call every lot flat | 100% | 0% | 0% | 0% | 0% |
Broken out by band, the pattern is stark. The vision-language models do reasonably on flat and slightly sloped lots and get none of the 16 steep or severe lots right. Those are the lots where dirt work costs the most, which is exactly where a wrong label hurts.
Does model agreement help for measurements?
| Keep the sites where models agree most | Mean error (pts) | Exact price band |
|---|---|---|
| Most-agreeing 25% | 2.7 | 42% |
| Most-agreeing 50% | 2.8 | 42% |
| Most-agreeing 75% | 2.8 | 40% |
| Most-agreeing 100% | 3.4 | 39% |
A little. Keeping only the sites where the five models agree most cut the mean error from 3.4 to 2.7 points, but it did not fix band accuracy, because all five models share the same bias. When every model is wrong in the same direction, agreement looks like confidence.
What does this mean for scaling data labeling?
- Categorical labels can be automated now. Build stage, next step, completion, and order can be written by a vote of several models, with a person reviewing a small minority.
- Agreement is a filter, not a check. It reduces review work, but shared blind spots pass straight through. Every label still needs testing against an outside source of truth.
- Outcomes are that source of truth. Job records, step order, inspection results, and callbacks give us something to check labels against. That is what lets AI labeling scale without quietly getting worse.
- Measurements need calibration data. For quantities like slope, a small amount of field ground truth, such as a crew's tape-measure reading when setting forms, is needed to correct the bias before AI labels can be trusted.
What are the limitations?
The categorical set is small (13 jobs) and easy by design, because ambiguous photos were left out. The answer key was written by one annotator before any model ran and has not had full field review, and this study found at least one error in it. The slope reference is public elevation data, not a site survey, and the crew photos were taken during or after the pour rather than before. Model results are a single run at default settings.
What comes next?
Our field team will review the answer key, we will add the ambiguous photos back, and we will run the same agreement analysis on first-person crew video, where labels are action segments rather than single photos. For slope, crews will log one form-height reading per job to give us real ground truth. Detailed results for each pilot are in construction progress monitoring and site slope from photos.
Analysis of raw model outputs from two pilot studies on Metal America job data. Accuracy intervals are Wilson 95%. Majority vote uses the most common answer across five models. The slope ensemble is the median of five estimates.
Frequently asked questions
What is AI data labeling?
Using models to automatically tag training data, such as which action is happening in a video, when each step starts and ends, and whether it succeeded.
Why is data labeling a bottleneck for robotics?
Robots learn from labeled actions and outcomes. Labeling by hand costs time for every hour of footage, so it does not scale to the amount of data robots need.
Can AI label construction footage?
For categorical labels like build stage, next step, and completion, yes. A majority vote of five frontier vision models was right 98% of the time on our job photos. Measurements like ground slope are not reliable yet.
Does model agreement catch labeling errors?
It filters most of the work, but not shared blind spots. In our data, the only labels all five models got wrong were cases where our own answer key was wrong, so every label still needs checking against real job outcomes.
Sources
- Ratner et al., Snorkel: Rapid Training Data Creation with Weak Supervision
- Grauman et al., Ego4D: Around the World in 3,000 Hours of Egocentric Video
Related research
- Physical AI: The Seven Pillars Self-Driving Proved
- Construction AI: An LLM Benchmark on 387 Questions
- Construction Robotics: A Task Graph from Real Jobs
- Construction Progress Monitoring with VLMs
- How we capture construction data
- The Metal America Index map