Check the numbers
- research
- verification
Numbers in AI drafts deserve a different trust level from the prose around them. The words are usually fine; the figures are where quiet failures live — a percentage that does not recompute, a total that double-counts, a “12%” that is really 9.5%. The rule is simple: numbers get recomputed, not read.
Three checks, every time
- Recompute. Do the arithmetic yourself — even a rough pass. 4.2% churn down to 3.8% is a drop of 0.4 points, which is a ~9.5% relative improvement — not the “12%” a draft claimed. Derived figures (growth rates, averages, totals) are where models slip most often.
- Sanity-check the magnitude. Does the size pass the smell test? A three-store bakery with €2M weekly revenue does not. Order-of-magnitude checks catch most invented or garbled numbers in two seconds.
- Check units and scope. Percent or percentage points. Per month or per year. Country or region. Responders or the whole population. A number is only as precise as its unit — and scope words are where shrinkage and inflation happen quietly.
The extract-and-check habit
How it works, step by step
- Extract every figure into a list with its label and the sentence it came from.
- Mark the load-bearing ones — the figures the decision depends on — versus decorative numbers.
- Recompute the load-bearing figures by hand, or with a calculator, and write the working down.
- Sanity-check magnitude and units for the rest; fix or delete what fails.
- Keep the working with the document. A number whose arithmetic you can show survives any review.
Weak prompt
Make these numbers sound more impressive for the deck
Stronger prompt
List every figure in this draft with its value, unit, period and source sentence. Then recalculate each derived figure — growth rates, totals, averages — and show the working for each. Do not add any numbers I have not given you.
Why the stronger one works
- The weak prompt asks the model to optimise the impression — the exact direction numbers should never be pushed.
- The strong prompt uses the model for what it is genuinely good at: extraction, transcription and bookkeeping.
- “Do not add any numbers I have not given you” removes the largest failure mode in one clause.
A bad example
Ship: “Revenue grew 30% year on year, reaching €152k from €120k.”
Two real numbers and one invented relationship: €152k from €120k is a 26.7% increase. Nobody recomputed, and the error is now a quotable “30%”.
A better example
Ship: “Revenue grew from €120k to €152k — up €32k, or 26.7% (calculation in the appendix).”
The raw numbers carry the weight; the derived figure shows its work; the appendix makes the check trivial for the next reader.
Why it works
Language models produce numbers the way they produce words — as likely continuations, not as arithmetic. Simple calculations are often right; multi-step or derived ones fail in plausible-looking ways. That asymmetry is exactly why the habit matters more than any single result: an AI assistant is an excellent extractor and transcriber, and a verifier only of last resort. Checks you delegated to it are checks you did not do — and numbers, being the most quotable thing in any document, carry the failure furthest. Anything unsupported hidden in prose stays arguable; an unsupported number ships as fact.
Practice
Check the number claim
A draft analytics summary, mostly written by an AI assistant, contains this line: “Customer churn improved from 4.2% to 3.8% — a 12% reduction in churn.” The number is going into a board slide.
What is the right first move?
Hint
The raw numbers can be right while the percentage derived from them is wrong — check the derivation, not just the source.
Absolute change and relative change are two different figures: 4.2 → 3.8 is a 0.4-point move, which is a certain percentage of the starting value.
Why this is the answer
“4.2% to 3.8%” is an absolute drop of 0.4 percentage points; as a relative change it is 0.4 ÷ 4.2 ≈ 9.5% — not the claimed 12%. Nothing looks wrong: the format is right, the direction is right, and it is off by a quarter. Derived figures (rates, totals, growth) are where AI drafts fail most plausibly, so they get recomputed before they ship — especially the one destined for a board slide, where the number will be quoted without its context.
Transfer
- Decks: the biggest number on the slide is the one to recompute first — it is the one everyone will quote.
- Budgets and forecasts: check totals against their parts; double-counting hides in long tables.
- Percentages: write both forms — “up 0.4 points (+16.7%)” — and readers stop confusing them.
- Any derived figure without visible working: mark it, recompute it, or cut it.
Key takeaways
- Numbers get recomputed, not read — especially derived ones (rates, totals, averages).
- Magnitude and units catch what arithmetic misses: per month or per year? points or percent?
- Use AI as extractor and transcriber; the arithmetic check is yours to do.
Next
More research lessons arrive in a later wave — the track hub keeps the path in view. Meanwhile the work track continues with AI for presentations — and the practice floor keeps every muscle warm. The caveat entry pairs well with this lesson’s habit of writing down uncertainty.