Why do general AI tools get engineering answers wrong?
General AI assistants get engineering answers wrong because they generate plausible text rather than checked facts. NIST AI 600-1 calls the result confabulation. Published tests from 2023 to 2025 show invented references, calculation slips, misread code tables, outdated editions, gaps in long documents, misplaced confidence and answers that change when asked again.
By Jun Zhong, Editor·Researched and drafted by Beacon's AI newsroom·Checked by Jun Zhong, Editor·Last checked ·Next review
Why does it matter?
In April 2023, researchers asked two versions of ChatGPT to write short literature reviews with references. Of the references GPT-3.5 produced, 55% did not exist; for GPT-4 the share was 18% 7. The US National Institute of Standards and Technology (NIST) calls confidently stated false output confabulation, and notes that the confident tone is what leads users to believe and act on it 1. Below are the documented failure modes and the checks that frameworks recommend.
Why does it happen?
A large language model (LLM) produces text by predicting likely next words from the statistical patterns of its training data. NIST explains that the same process yields correct answers and false or self-contradicting ones, and that the risk is highest in long answers and in fields that need specialist context 1.
Researchers at OpenAI and Georgia Tech add two causes. Training and benchmark scoring give no credit for "I don't know", so a guess scores better than admitting doubt; and facts that appear only once in the training data put a floor under the error rate, while often repeated facts are rarely wrong 13.
What are the failure modes?
Each row gives the mechanism, an invented example, a check, and evidence with the models and test dates. Results age fast, so each row names the models and the date they were tested 7.
| Failure mode | Mechanism | Engineering example (illustrative) | How to catch it | Evidence (models, date tested) |
|---|---|---|---|---|
| Fabricated references or clauses | Citation-shaped text is generated, not retrieved 1 | A paragraph number in an inspection code that does not exist | Open the cited document and find the clause 1 | 55% of GPT-3.5's and 18% of GPT-4's references were invented; 43% and 24% of the real ones had substantive errors (April 2023) 7 |
| Arithmetic and unit errors | Numbers come out as predicted text; calculation tools cut these errors 1,9 | A rate in mils per year read as millimetres per year, about 39 times too high | Redo the calculation in a checked tool, units carried through | Calculation caused 29.0% of GPT-3.5's errors on college science problems, 6.2% when it wrote Python; the best score, GPT-4 with tools, was 43.22% (2023 models) 9 |
| Misread code tables | Tables in codes are interpreted worse than prose clauses 10 | An allowable value read from the wrong material or temperature row | Read the table in the code itself | On four-option questions from Chinese building codes, GPT-4o scored 60.05% on text clauses and 44.44% on tables; the best model, DeepSeek-R1, 84.02% and 63.49% (2025) 10 |
| Right fact, wrong use | A fact recalled alone is not reliably carried into a multistep task 10 | Correct design temperature when asked, another one used in the calculation | Check every input the answer uses, not only the result | A model gave the correct seismic grade when asked directly but not when drafting a full design proposal (2025) 10 |
| Outdated or superseded edition | Effective knowledge cutoffs can differ sharply from reported ones, partly because web crawls keep old copies 12 | Requirements quoted from an edition of API 570 older than the 5th, of February 2024 15 | Ask which edition is used; check the publisher's current catalog | Effective and reported cutoffs differed across the open models studied (2024) 12 |
| Lost in a long document | Information in the middle of a long input is used worse than at its start or end 11 | A finding on page 140 of a 300-page inspection history missing from a summary | Ask for the page and the passage, then read it | With the answer mid-context among 20 or 30 documents, gpt-3.5-turbo-0613 scored below its 56.1% with no documents (2023) 11 |
| Confidence that does not track accuracy | Scoring rewards guessing 13; users over-trust automated output 1 | A sour service limit stated flatly, with no caveat | Judge the evidence, not the tone | For some models, GPT-4 among them, stated confidence did not track correctness: on compound safety questions GPT-4 gave 4.0 of 5 on its six wrong answers and 1.0 on its one right answer (2025) 8 |
| Different answers to the same question | Rewording or a fresh try can change the answer 14,13 | Two corrosion allowances for one line on two tries | Ask twice, reworded; disagreement is a warning | DeepSeek-V3 gave three different wrong dates in three tries (11 May 2025) 13; ChatGPT contradicted itself on over 10% of its own paraphrases (2023) 14 |
How are these errors caught?
NIST AI 600-1 (July 2024), the generative AI profile of NIST's AI Risk Management Framework, suggests comparing output with known ground truth by several methods including human review, fact-checking generated information, verifying sources and citations, and not judging a system from narrow or anecdotal tests 1. Its parent, AI RMF 1.0 (NIST AI 100-1, January 2023), asks for accuracy measured on test sets that represent real use, and human intervention where a system cannot detect or correct its errors 2.
ISO/IEC 42001:2023, published on 18 December 2023, sets requirements for establishing, running and improving an AI management system, and applies to any organisation that provides or uses AI systems 3.
In Great Britain, the Health and Safety Executive (HSE) said on 12 June 2026 that existing goal-setting law covers AI: where AI affects workplace safety, employers assess the risk and reduce it so far as is reasonably practicable 4. An HSE Science Division survey of 247 AI use cases in 14 sectors found language models widely used to support risk assessment, listed inaccuracies in AI-assisted safety assessments as a risk, and recorded output checking and mandatory human approval of safety-critical outputs as controls 5.
The American Society of Civil Engineers (ASCE), in Policy Statement 573 of July 2024, holds that AI cannot replace a licensed professional engineer's judgement and that responsibility stays with the engineer 6.
Is a hallucination the same as an outdated answer or a calculation error?
No. NIST's term covers false content stated confidently, including invented citations and self-contradiction 1. An outdated answer may have been correct for an earlier edition 12, and a calculation error can follow a correct formula: SciBench records cases where the model wrote the right formula and then failed in the arithmetic steps 9. Each needs a different check: the source, the edition, or the arithmetic.
Where integrity problems show up
| Integrity task | Where a wrong AI answer would enter (illustrative, not a reported incident) | Failure mode | Evidence |
|---|---|---|---|
| Minimum required thickness | Allowable stress or other values taken from a code table | Misread table, right fact used wrongly | Code table questions scored well below text clause questions 10 |
| Corrosion rate | Conversion between mm/y and mils per year | Arithmetic and unit errors | Calculation was a leading error source 9 |
| Remaining life and inspection intervals | Margin divided by rate; interval rules of the governing code | Arithmetic, outdated edition | 9,12 |
| Standard editions | API 510 is in its 11th edition (October 2022) and API 570 in its 5th (February 2024) | Outdated or superseded edition | 15 |
| Review of inspection history | Hundreds of pages of past reports | Lost in a long document | 11 |
Common questions
- Is a hallucination the same as an outdated answer?
- No. NIST defines confabulation as false content stated with confidence, while an outdated answer may have been right for an earlier edition of a code. Both fail in use, but one is checked against the source and the other against the current edition.
- Do newer models still make these errors?
- Less often in some tests, but not never. GPT-4 fabricated 18% of its references in April 2023 against 55% for GPT-3.5, and a 2025 analysis argues that scoring which rewards guessing keeps errors in place.
- Does pasting the document into the chat solve it?
- It helps with facts the model never saw, but long inputs bring their own failure. In a 2023 test, with the answer buried in the middle of 20 or 30 documents, a model was right less often than with no documents at all, so ask for the page and read it.
Beacon explains these concepts in its own words for education. It does not reproduce or replace the standards cited, and Winda is not affiliated with or endorsed by the organizations that publish them. Engineering decisions need the current edition of the standard and a qualified engineer.
Sources
- 1.NIST AI 600-1, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (July 2024), section 2.2 and suggested actions MP-2.3-001, MP-2.3-003, MS-2.5-001 and MS-2.5-003 · US National Institute of Standards and Technology (NIST) ·
- 2.NIST AI 100-1, Artificial Intelligence Risk Management Framework (AI RMF 1.0) (January 2023), section 3.1, Valid and Reliable · US National Institute of Standards and Technology (NIST) ·
- 3.ISO/IEC 42001:2023, Information technology, Artificial intelligence, Management system (published 18 December 2023) · NSAI Standards (listing of ISO/IEC 42001) ·
- 4.HSE's regulatory approach to Artificial Intelligence (AI) · UK Health and Safety Executive (HSE) ·
- 5.Understanding How Industry Uses AI in HSE-Regulated Sectors (C. Chambers, F. Anderson, C. Williamson, F. McGarry), Hazards 35, Symposium Series No. 172, Crown Copyright 2025 · UK Health and Safety Executive (HSE) Science Division, published by IChemE
- 6.Policy Statement 573, Artificial intelligence and engineering responsibility (adopted by the Board of Direction 18 July 2024) · American Society of Civil Engineers (ASCE) ·
- 7.Fabrication and errors in the bibliographic citations generated by ChatGPT (W. H. Walters, E. I. Wilder), Scientific Reports 13, 14045 · Manhattan College and Lehman College, City University of New York, in Scientific Reports ·
- 8.A framework for evaluating the chemical knowledge and reasoning abilities of large language models against the expertise of chemists (A. Mirza, N. Alampara, K. M. Jablonka et al.), Nature Chemistry, 2025; preprint arXiv:2404.01475 · Friedrich Schiller University Jena, EPFL and others, in Nature Chemistry (open access) ·
- 9.SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models (X. Wang, Z. Hu, P. Lu, Y. Zhu et al.) · University of California, Los Angeles, and others, in Proceedings of ICML 2024 (PMLR 235) ·
- 10.AECBench: A Hierarchical Benchmark for Knowledge Evaluation of Large Language Models in the AEC Field (C. Liang, Z. Huang, X. Zhao et al.), arXiv:2509.18776v3; published in Advanced Engineering Informatics 71 (2026) · Tongji University and East China Architectural Design and Research Institute (arXiv preprint) ·
- 11.Lost in the Middle: How Language Models Use Long Contexts (N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, P. Liang), arXiv:2307.03172 · Stanford University, University of California, Berkeley, and Samaya AI (arXiv preprint, accepted by Transactions of the ACL) ·
- 12.Dated Data: Tracing Knowledge Cutoffs in Large Language Models (J. Cheng, M. Marone, O. Weller, D. Lawrie, D. Khashabi, B. Van Durme) · Johns Hopkins University, at COLM 2024
- 13.Why Language Models Hallucinate (A. T. Kalai, O. Nachum, S. S. Vempala, E. Zhang), arXiv:2509.04664 · OpenAI and Georgia Institute of Technology (arXiv preprint) ·
- 14.Consistency Analysis of ChatGPT (M. E. Jang, T. Lukasiewicz), arXiv:2303.06273 · University of Oxford and TU Wien (arXiv preprint) ·
- 15.2025 Publications Catalog (entries for API 510, 11th edition, October 2022, and API 570, 5th edition, February 2024, with prices effective 1 January 2025) · American Petroleum Institute (API)