- We built 387 questions from our own price books, engineering specs, lead times, and policies: the questions real customers and estimators ask every day, each with a known correct answer.
- We asked six frontier and open models to answer them twice: once on their own, and once with the relevant pages of our data in the prompt.
- On their own, the models averaged 19% correct. With our data they averaged 90%.
- With our data, a small open model outscored the strongest model without it by almost four to one. The data mattered more than the model.
- Most of what our data did not fix came from places where our own documents disagree with each other.
Definition. A construction LLM benchmark is a fixed set of real construction questions with verified answers, used to measure how accurately language models answer them.
How well do LLMs know construction?
General-purpose models know a lot about construction in the abstract. They can explain what a vapor barrier is or why snow load matters. But a business runs on specifics: what this building costs in this county, how long this manufacturer takes, whether this product can be sold in this state. That knowledge is local, it changes monthly, and almost none of it is on the public internet.
For physical AI this is the same gap one level up. A robot that knows how to drive a screw in general still has to know which screw, how many, in what pattern, for this building. We wanted to measure how big the gap between general knowledge and operating knowledge is, and how much a company's own data closes it.
What questions are in the benchmark?
Every question has a single verifiable answer drawn from our own systems. They cover five broad areas: prices (small structures like carports, commercial buildings, homes, and foundations), specifications (how much wind and snow a structure must withstand, and foundation requirements), timelines (how long delivery and installation take), rules (permits, financing, and company policies), and process (the steps to install a building). They are phrased the way a customer or an estimator would actually ask. Numeric answers are scored against a tolerance. Written answers are graded by a separate model that sees the reference answer and is blind to which model wrote the response.
| Category | Questions | No data | With our data |
|---|---|---|---|
| Delivery timelines | 15 | 3% | 94% |
| Foundation prices | 46 | 5% | 93% |
| Permits and engineering | 30 | 8% | 93% |
| Financing rules | 22 | 12% | 78% |
| Home build prices | 41 | 15% | 99% |
| Company policies | 19 | 18% | 91% |
| Home build products | 8 | 19% | 91% |
| Structural specs (wind and snow) | 44 | 21% | 97% |
| Commercial building prices | 73 | 24% | 91% |
| Small structure prices | 33 | 24% | 75% |
| Installation steps | 14 | 25% | 88% |
| Small structure products | 20 | 26% | 90% |
| Foundation specs | 22 | 38% | 96% |
How did we test retrieval?
No data. The model answers from what it learned in training, the way it would if a customer asked it directly.
With our data. We retrieve the relevant pages from our price books, specs, and policies and place them in the prompt before the question. This is simple retrieval, not fine-tuning.
What did we find?
Without our data, every model failed most questions. The gap between the best and worst model was about 14 points. With our data, every model passed most questions, and the smallest, cheapest model in the test scored within a few points of the best. Once the right information is in front of it, model size stops being the deciding factor.
Do LLMs underestimate construction prices?
| Model | Median price error, no data | Share of prices too low | Median price error, with data |
|---|---|---|---|
| GPT-6.1 Sol | 24% | 57% | 0% |
| Claude Opus 5.5 | 26% | 60% | 0% |
| Gemini 2.5 Flash | 29% | 50% | 0% |
| Llama 4 Maverick | 55% | 76% | 0% |
| DeepSeek V4 Flash | 39% | 80% | 0% |
| Mistral Large 3 | 59% | 76% | 0% |
When a model doesn't know a construction price, it tends to guess low. Concrete slabs came in roughly 40% under our real prices, and stamped engineering drawings more than 80% under. A customer quoted by a general-purpose assistant would be told a project costs far less than it does.
Why do general models fail on company-specific rules?
When a model doesn't know a company's rule, it fills the gap with what a typical company would do. Every real business has rules that differ from the typical case, and the model gives no sign that it is guessing. The clearest failures were questions where all six models agreed and were wrong. Asked whether we can add change orders after a customer signs a concrete quote, all six said yes, because change orders are normal in construction. Our signed price is the most the customer pays unless they change the scope. Asked whether our sales agents can quote monthly payments by text, four of six said yes. Our rules forbid it.
What did retrieval not fix?
Even with retrieval, a few categories stayed weak: financing rules and some carport pricing. When we traced those misses, most came from our own documents disagreeing. One document listed one set of payment terms and another listed a different set. Two sources gave two different setup fees. The model faithfully reproduced whichever page it was shown. Retrieval cannot be more consistent than the data behind it.
What are the limitations?
Two models ran the full 387 questions. Four ran a fixed 199-question subset with the same category mix, because of rate limits. Some newer models were not available to us and were not tested. Written answers were graded by a model, which can be wrong. Questions with specific prices are kept private.
What comes next?
The obvious next step is to fix our own contradictions, then rerun. The larger lesson carries straight into robotics. General models bring broad competence, but operating in a real business takes local, current, verified knowledge, and that only comes from the business. We are building the same thing for physical work: footage of how our crews actually do each task, tied to what it cost and whether it passed.
387 questions, 13 categories. Retrieval-augmented condition uses the same source pages for every model. Grader: a separate model, blinded to the answering model. Total model cost about $16.
Frequently asked questions
How accurate are LLMs on construction questions?
On 199 questions every model answered, six frontier and open models averaged 19% correct on their own and 91% with retrieved company data in the prompt.
Which model did best?
With company data, Claude Opus 5.5, DeepSeek, and Llama 4 all scored above 93%. Without it, the best model scored 25%.
Do LLMs underestimate construction prices?
Yes. Without company data, models priced construction work 25% to 55% low on the median answer. Concrete slabs came in about 40% under.
Does retrieval fix everything?
No. Most remaining errors came from company documents that contradicted each other. Retrieval is only as consistent as the data behind it.
Sources
- Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP
- Stanford HELM: Holistic Evaluation of Language Models
Related research
- AI Data Labeling for Construction Robotics
- Physical AI: The Seven Pillars Self-Driving Proved
- Construction Robotics: A Task Graph from Real Jobs
- Construction Progress Monitoring with VLMs
- How we capture construction data
- The Metal America Index map