An AI model can draft a quarterly review in minutes. Checking it can take longer than writing it by hand, because nothing in the finished document tells you which figures are right. The problem is not that models are careless. It is that a figure in prose carries no evidence of where it came from.
This article is for the people who sign off on reports: analytics leads, quality and compliance managers, finance controllers and auditors. It describes where numbers in generated reports go wrong, what makes a figure checkable, and what a reviewer should still look at when the tooling has done its part.
Where the wrong numbers come from
In our work, wrong figures in AI-drafted reports fall into a small number of patterns. Knowing them helps target review effort.
- Invented figures. The model writes a plausible number that no query produced, often a percentage or a growth rate “filled in” to complete a sentence.
- The right number, the wrong scope. The figure is real but belongs to a different period, region or filter than the sentence claims, for example year-to-date presented as a quarter.
- Drift between formats. The text rounds one way, the table another and the chart a third. In spreadsheets and slide decks, a stale cached formula or a template placeholder can show a number the analysis never computed.
- A query that answered a different question. Join traps, nulls, duplicates and ties make two reasonable queries disagree. Only one of them matches what the reader thinks the question was.
- Conflicts smoothed over. Two systems report different values for the same thing, and the report quietly picks one, or averages them.
- Small numbers that should not be shown. A count of three patients at one site is accurate and still a disclosure risk.
The first three are the easiest to prevent mechanically. The last three need explicit rules.
Five properties of a checkable figure
A figure is checkable when a reviewer, or a program, can answer “where did this come from?” and “is it still what was computed?” without redoing the analysis. That takes five properties.
1. It has an identity
The text should not contain the number at all. It should refer to a value that was computed and recorded separately, and the number is filled in when the document is produced. If the writing step, human or model, cannot type digits, it cannot invent a figure or round one differently from the table. This single change removes most invented and drifting numbers.
2. It has lineage
Each recorded value should carry what produced it: the exact query text and the time of the data it read, the version of the calculation it used, or, for a figure taken from a document, the quoted passage and the page it was found on. Lineage turns review from “redo the work” into “open the source”.
3. It was computed twice, independently
Running the same query twice proves little. An independent check reaches the same number by a different route: a yearly total recombined from monthly figures, a distinct count recomputed by grouping on the key, a statistic computed by a second implementation, or a SQL result matched against a differently written query that agrees. A mismatch should stop the report, not produce a warning someone may skip.
Some values have no independent second route, for example a figure produced by exploratory code. Those should be marked, and released only when a named person accepts them.
4. Its state is honest
Not every requested figure can be produced. A good report says which figures are insufficient (not enough data, with how much was found and needed), conflicting (two sources disagree, with both readings shown), or unchecked (computed, but without an independent check). These states should be printed the same way in every format. A report that always has a number for everything is a warning sign.
5. The delivered file says the same thing
Checking the analysis is not the same as checking the file the board receives. After a workbook, deck or PDF is produced, it should be opened again by a program and every shown figure compared with the recorded value, including its formatting and rounding. Chart axes for bars and columns should start at zero. A file that cannot be read back should not be sent.
Sentences, not only numbers
A sentence can contain only correct numbers and still mislead. Three patterns are common in generated text.
- A level presented as a movement. “Readmissions rose to 14%” needs a prior value. If only the current period was computed, the word “rose” is unsupported, however plausible.
- A cause the data does not measure. “Margins fell because of freight costs” needs evidence about freight costs. A correlation in the same paragraph is not that evidence.
- A superlative without a comparison. “The highest in the region” needs every other member of the region in the data.
Statistical words deserve the same care. “Significant”, “trend” and “outlier” have technical meanings that readers rely on. They should appear only where a test, a regression or a control-chart rule supports them, and when a report makes many comparisons, the tests should be adjusted together so that something does not look significant by chance.
A practical way to review sentences is to break each one into its claims (a level, a change, a cause, a comparison, a superlative) and ask, for each claim, whether the cited figures state it, contradict it or are silent. Automated checks can do the same decomposition, but a model should only be allowed to object to a sentence, never to approve one that a rule has refused.
What a reviewer should still ask
Even with every mechanical check in place, review is not finished. These questions remain the reviewer’s:
- Was the question asked the right one, and did the plan that ran actually answer it?
- Do the definitions match the organization’s own: what counts as an admission, a customer, a closed case?
- Which figures could not be checked independently, and who accepted them?
- Where two systems disagreed, which one was chosen, and by what rule?
- Were small counts suppressed, and could a hidden one be worked out from the totals?
- Does the file state whether a person reviewed it, and who signed it?
When evaluating a tool or a vendor, the same ideas become direct questions: can the model write a number that code did not compute; is each figure checked a second, independent way; are the delivered files read back; and can you open the source of any number from the file itself?
What no tool can check
Mechanical checks prove that a figure is what the query computed and that the file prints it faithfully. They cannot prove that the source data was right, that a definition was the appropriate one, or that a conclusion follows. Those remain matters of judgment, and a report should make it easy to see where that judgment was exercised and by whom.