Skip to content
Beaconby Winda
FactData, Digital and AI · Trustworthy AI

Why do general AI tools get engineering answers wrong?

·15 sources

Short answer

General AI assistants get engineering answers wrong because they generate plausible text rather than checked facts. NIST AI 600-1 calls the result confabulation. Published tests from 2023 to 2025 show invented references, calculation slips, misread code tables, outdated editions, gaps in long documents, misplaced confidence and answers that change when asked again.

By , Editor·Researched and drafted by ·Checked by , Editor·Last checked ·Next review

Why does it matter?

In April 2023, researchers asked two versions of ChatGPT to write short literature reviews with references. Of the references GPT-3.5 produced, 55% did not exist; for GPT-4 the share was 18% 7. The US National Institute of Standards and Technology (NIST) calls confidently stated false output confabulation, and notes that the confident tone is what leads users to believe and act on it 1. Below are the documented failure modes and the checks that frameworks recommend.

Why does it happen?

A large language model (LLM) produces text by predicting likely next words from the statistical patterns of its training data. NIST explains that the same process yields correct answers and false or self-contradicting ones, and that the risk is highest in long answers and in fields that need specialist context 1.

Researchers at OpenAI and Georgia Tech add two causes. Training and benchmark scoring give no credit for "I don't know", so a guess scores better than admitting doubt; and facts that appear only once in the training data put a floor under the error rate, while often repeated facts are rarely wrong 13.

What are the failure modes?

Each row gives the mechanism, an invented example, a check, and evidence with the models and test dates. Results age fast, so each row names the models and the date they were tested 7.

Failure modeMechanismEngineering example (illustrative)How to catch itEvidence (models, date tested)
Fabricated references or clausesCitation-shaped text is generated, not retrieved 1A paragraph number in an inspection code that does not existOpen the cited document and find the clause 155% of GPT-3.5's and 18% of GPT-4's references were invented; 43% and 24% of the real ones had substantive errors (April 2023) 7
Arithmetic and unit errorsNumbers come out as predicted text; calculation tools cut these errors 1,9A rate in mils per year read as millimetres per year, about 39 times too highRedo the calculation in a checked tool, units carried throughCalculation caused 29.0% of GPT-3.5's errors on college science problems, 6.2% when it wrote Python; the best score, GPT-4 with tools, was 43.22% (2023 models) 9
Misread code tablesTables in codes are interpreted worse than prose clauses 10An allowable value read from the wrong material or temperature rowRead the table in the code itselfOn four-option questions from Chinese building codes, GPT-4o scored 60.05% on text clauses and 44.44% on tables; the best model, DeepSeek-R1, 84.02% and 63.49% (2025) 10
Right fact, wrong useA fact recalled alone is not reliably carried into a multistep task 10Correct design temperature when asked, another one used in the calculationCheck every input the answer uses, not only the resultA model gave the correct seismic grade when asked directly but not when drafting a full design proposal (2025) 10
Outdated or superseded editionEffective knowledge cutoffs can differ sharply from reported ones, partly because web crawls keep old copies 12Requirements quoted from an edition of API 570 older than the 5th, of February 2024 15Ask which edition is used; check the publisher's current catalogEffective and reported cutoffs differed across the open models studied (2024) 12
Lost in a long documentInformation in the middle of a long input is used worse than at its start or end 11A finding on page 140 of a 300-page inspection history missing from a summaryAsk for the page and the passage, then read itWith the answer mid-context among 20 or 30 documents, gpt-3.5-turbo-0613 scored below its 56.1% with no documents (2023) 11
Confidence that does not track accuracyScoring rewards guessing 13; users over-trust automated output 1A sour service limit stated flatly, with no caveatJudge the evidence, not the toneFor some models, GPT-4 among them, stated confidence did not track correctness: on compound safety questions GPT-4 gave 4.0 of 5 on its six wrong answers and 1.0 on its one right answer (2025) 8
Different answers to the same questionRewording or a fresh try can change the answer 14,13Two corrosion allowances for one line on two triesAsk twice, reworded; disagreement is a warningDeepSeek-V3 gave three different wrong dates in three tries (11 May 2025) 13; ChatGPT contradicted itself on over 10% of its own paraphrases (2023) 14

How are these errors caught?

NIST AI 600-1 (July 2024), the generative AI profile of NIST's AI Risk Management Framework, suggests comparing output with known ground truth by several methods including human review, fact-checking generated information, verifying sources and citations, and not judging a system from narrow or anecdotal tests 1. Its parent, AI RMF 1.0 (NIST AI 100-1, January 2023), asks for accuracy measured on test sets that represent real use, and human intervention where a system cannot detect or correct its errors 2.

ISO/IEC 42001:2023, published on 18 December 2023, sets requirements for establishing, running and improving an AI management system, and applies to any organisation that provides or uses AI systems 3.

In Great Britain, the Health and Safety Executive (HSE) said on 12 June 2026 that existing goal-setting law covers AI: where AI affects workplace safety, employers assess the risk and reduce it so far as is reasonably practicable 4. An HSE Science Division survey of 247 AI use cases in 14 sectors found language models widely used to support risk assessment, listed inaccuracies in AI-assisted safety assessments as a risk, and recorded output checking and mandatory human approval of safety-critical outputs as controls 5.

The American Society of Civil Engineers (ASCE), in Policy Statement 573 of July 2024, holds that AI cannot replace a licensed professional engineer's judgement and that responsibility stays with the engineer 6.

Is a hallucination the same as an outdated answer or a calculation error?

No. NIST's term covers false content stated confidently, including invented citations and self-contradiction 1. An outdated answer may have been correct for an earlier edition 12, and a calculation error can follow a correct formula: SciBench records cases where the model wrote the right formula and then failed in the arithmetic steps 9. Each needs a different check: the source, the edition, or the arithmetic.

Where integrity problems show up

Integrity taskWhere a wrong AI answer would enter (illustrative, not a reported incident)Failure modeEvidence
Minimum required thicknessAllowable stress or other values taken from a code tableMisread table, right fact used wronglyCode table questions scored well below text clause questions 10
Corrosion rateConversion between mm/y and mils per yearArithmetic and unit errorsCalculation was a leading error source 9
Remaining life and inspection intervalsMargin divided by rate; interval rules of the governing codeArithmetic, outdated edition9,12
Standard editionsAPI 510 is in its 11th edition (October 2022) and API 570 in its 5th (February 2024)Outdated or superseded edition15
Review of inspection historyHundreds of pages of past reportsLost in a long document11

Common questions

Is a hallucination the same as an outdated answer?
No. NIST defines confabulation as false content stated with confidence, while an outdated answer may have been right for an earlier edition of a code. Both fail in use, but one is checked against the source and the other against the current edition.
Do newer models still make these errors?
Less often in some tests, but not never. GPT-4 fabricated 18% of its references in April 2023 against 55% for GPT-3.5, and a 2025 analysis argues that scoring which rewards guessing keeps errors in place.
Does pasting the document into the chat solve it?
It helps with facts the model never saw, but long inputs bring their own failure. In a 2023 test, with the answer buried in the middle of 20 or 30 documents, a model was right less often than with no documents at all, so ask for the page and read it.

Beacon explains these concepts in its own words for education. It does not reproduce or replace the standards cited, and Winda is not affiliated with or endorsed by the organizations that publish them. Engineering decisions need the current edition of the standard and a qualified engineer.

Sources

  1. 1.NIST AI 600-1, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (July 2024), section 2.2 and suggested actions MP-2.3-001, MP-2.3-003, MS-2.5-001 and MS-2.5-003 · US National Institute of Standards and Technology (NIST) ·
  2. 2.NIST AI 100-1, Artificial Intelligence Risk Management Framework (AI RMF 1.0) (January 2023), section 3.1, Valid and Reliable · US National Institute of Standards and Technology (NIST) ·
  3. 3.ISO/IEC 42001:2023, Information technology, Artificial intelligence, Management system (published 18 December 2023) · NSAI Standards (listing of ISO/IEC 42001) ·
  4. 4.HSE's regulatory approach to Artificial Intelligence (AI) · UK Health and Safety Executive (HSE) ·
  5. 5.Understanding How Industry Uses AI in HSE-Regulated Sectors (C. Chambers, F. Anderson, C. Williamson, F. McGarry), Hazards 35, Symposium Series No. 172, Crown Copyright 2025 · UK Health and Safety Executive (HSE) Science Division, published by IChemE
  6. 6.Policy Statement 573, Artificial intelligence and engineering responsibility (adopted by the Board of Direction 18 July 2024) · American Society of Civil Engineers (ASCE) ·
  7. 7.Fabrication and errors in the bibliographic citations generated by ChatGPT (W. H. Walters, E. I. Wilder), Scientific Reports 13, 14045 · Manhattan College and Lehman College, City University of New York, in Scientific Reports ·
  8. 8.A framework for evaluating the chemical knowledge and reasoning abilities of large language models against the expertise of chemists (A. Mirza, N. Alampara, K. M. Jablonka et al.), Nature Chemistry, 2025; preprint arXiv:2404.01475 · Friedrich Schiller University Jena, EPFL and others, in Nature Chemistry (open access) ·
  9. 9.SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models (X. Wang, Z. Hu, P. Lu, Y. Zhu et al.) · University of California, Los Angeles, and others, in Proceedings of ICML 2024 (PMLR 235) ·
  10. 10.AECBench: A Hierarchical Benchmark for Knowledge Evaluation of Large Language Models in the AEC Field (C. Liang, Z. Huang, X. Zhao et al.), arXiv:2509.18776v3; published in Advanced Engineering Informatics 71 (2026) · Tongji University and East China Architectural Design and Research Institute (arXiv preprint) ·
  11. 11.Lost in the Middle: How Language Models Use Long Contexts (N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, P. Liang), arXiv:2307.03172 · Stanford University, University of California, Berkeley, and Samaya AI (arXiv preprint, accepted by Transactions of the ACL) ·
  12. 12.Dated Data: Tracing Knowledge Cutoffs in Large Language Models (J. Cheng, M. Marone, O. Weller, D. Lawrie, D. Khashabi, B. Van Durme) · Johns Hopkins University, at COLM 2024
  13. 13.Why Language Models Hallucinate (A. T. Kalai, O. Nachum, S. S. Vempala, E. Zhang), arXiv:2509.04664 · OpenAI and Georgia Institute of Technology (arXiv preprint) ·
  14. 14.Consistency Analysis of ChatGPT (M. E. Jang, T. Lukasiewicz), arXiv:2303.06273 · University of Oxford and TU Wien (arXiv preprint) ·
  15. 15.2025 Publications Catalog (entries for API 510, 11th edition, October 2022, and API 570, 5th edition, February 2024, with prices effective 1 January 2025) · American Petroleum Institute (API)