- We asked five frontier vision-language models to read 90 photos from 13 of our real jobs: what stage the work is at, what comes next, whether a slab is finished, and which of two photos was taken later.
- Every model scored between 88% and 100% on every task, against baselines of 31 to 65%.
- Coarse progress recognition from a single phone photo is close to solved for our kinds of jobs.
- Judging whether work was done correctly is a different, harder problem. It needs footage tied to real outcomes, which is what we are now collecting.
Definition. Construction progress monitoring is tracking which stage of work a job site has reached, such as forms set, slab poured, or frame erected.
Part of our work on AI data labeling for construction.
Why does progress monitoring matter for construction robots?
A machine working on a job site needs to know where the job stands before it can act. Is the slab poured? Has it cured? Is the frame up? Can sheeting start? On a real site, that state changes daily and nobody writes it down consistently.
If general-purpose vision models can already read job state from a photo, the world-model problem for construction starts from a much higher floor than it did a few years ago. We tested that on our own jobs.
What did we ask the models?
Stage. Which of nine build stages does the photo show? The stages run from site preparation, forms, steel reinforcement, pour and finished slab, through steel frame, sheeting, interior framing, and a finished building.
Next step. Given the photo, what should the crew do next?
Slab finished. Is the concrete work done?
Which came later. Given two photos of the same job taken on different days, which is later? This tests whether a model understands build order, not just appearance.
What data did we use?
90 unique images from 13 jobs: ten residential slabs, two steel buildings, and one barndominium (a steel-frame home) interior. The answer keys for "slab finished" and "which came later" come directly from company records: our field team's own "finished" tags, camera sequence numbers, and capture dates. Stage labels were assigned on a fixed rubric before any model was run, and ambiguous photos were excluded. Faces, names, and addresses are kept out of anything published.
What did we find?
| Model | Stage | Next step | Slab finished | Which came later |
|---|---|---|---|---|
| Claude Haiku 5.5 | 96% | 88% | 94% | 100% |
| Claude Opus 5.5 | 94% | 95% | 91% | 98% |
| Claude Sonnet 5.5 | 95% | 92% | 91% | 98% |
| GPT-5.4 mini | 95% | 95% | 97% | 98% |
| GPT-5.5 | 95% | 95% | 94% | 98% |
| Majority-class baseline | 31% | 34% | 65% | 50% |
Even the smallest, cheapest model reached about 95% on stage recognition. At this sample size the models are indistinguishable from each other. All are far above baseline.
Where did vision-language models fail?
The errors were few but consistent across every model, which suggests shared gaps in how these models understand construction rather than random noise.
A labeling error in our own answer key. One photo every model "missed" turned out, on review, to show vapor barrier and rebar before the pour. Our label said the slab was already poured. All five models were right and the label was wrong. See our AI data labeling analysis for what this means for checking labels at scale.
Fresh versus finished. Slabs still being machine-smoothed were called unfinished even when our records mark them complete. With a power trowel still on the slab, the models are arguably right, which again points to noise in our own labels.
Frame versus sheeting. A steel frame with a few panels attached sits on a boundary even people argue about.
Can AI judge construction quality yet?
This pilot tests whether models can recognize what has been done, not whether it was done right. We set out to test quality judgment too: whether a model can tell a slab that will pass inspection from one that will crack, or a frame that is out of square. We could not, because we found no photo-level records of pass, fail, or callback (a return visit to fix a problem). That gap is the point of our crew camera program: footage of the work, tied to the inspection result and to any callbacks that follow.
What are the limitations?
Thirteen jobs is small, and two steel builds supply most of the building-stage photos. Removing ambiguous photos made the task easier than real check-in photos will be. Stage labels are being reviewed by our field operations team before we treat these numbers as final.
What comes next?
Next, a quality set: about fifty jobs with photos and video plus their real outcomes, scored on whether models can predict the outcome from the footage. Ambiguous photos will be added back and reported separately.
Pilot study. One image per question at 768 px. Models queried at temperature 0. Total model cost about $3.
Frequently asked questions
Can AI monitor construction progress from photos?
Yes, at the level of build stage. Five frontier vision-language models identified stage, next step, and slab completion on our job photos 88 to 100% of the time.
Where do vision-language models fail on job sites?
Every model mistook a plastic curing blanket on a fresh slab for an unpoured vapor barrier.
Can AI judge construction quality?
Not yet tested. Judging whether work was done right needs footage tied to inspection results and callbacks, which we are now collecting.
How big was the test?
A pilot: 90 photos from 13 real jobs, four tasks, five models.
Sources
Related research
- AI Data Labeling for Construction Robotics
- Physical AI: The Seven Pillars Self-Driving Proved
- Construction AI: An LLM Benchmark on 387 Questions
- Construction Robotics: A Task Graph from Real Jobs
- How we capture construction data
- The Metal America Index map