A required field made my AI fabricate statistics

I run a pipeline that generates explainer articles. LLM in the middle, structured output, published in several languages. It had been running for a while and the articles looked good: clean layout, a chart, and near the top of each one a confident little box with a statistic. Something in the shape of “68% of people never change the default.” A number, a source, an authoritative ring to it.

Not one of those numbers had been researched. The pipeline had never looked up a single statistic in its life. It asked the model for a number and printed whatever came back.

No clever eval caught this. I was cleaning up something unrelated and read the prompt properly for the first time in months.

The field that forced a lie

The output schema had a required field. statistic.text and statistic.source, described in the prompt as an “eye-catching stat” for the top of the article. Required. Every article had to have one.

The prompt then went further and told the model what to do when it had no real number. Round to a safe order of magnitude. And strip the year off the source, so the article would look evergreen instead of dated.

I had written instructions that said: always produce a statistic, invent a plausible magnitude if you have to, and remove the one piece of metadata anyone could check it against. That does not occasionally permit a hallucination. It demands one, every time the model happens not to know a real figure.

So it produced them. Confidently, in every language, each wearing a real-sounding source: a named institute, an industry association, a government statistics office. This was content people actually act on, which is the category where being wrong is not a rounding error.

The chart code was doing the same thing one floor down. That block asked the model for “actual statistics from official sources” and, trying to be helpful, offered “Bureau of Labor Statistics” as an example. So the chart invented its own numbers too, separately from the stat box, and the two never had to agree with each other.

Why “just tell it to be honest” does not work

My first instinct was the wrong one. Add a line to the prompt: only use real statistics, cite a verifiable source, do not make anything up.

That does nothing. A language model cannot tell a hallucinated number from a remembered one, because from the inside they feel identical. Ask it to only cite real sources and it agrees, sincerely, and then hands you a real-sounding source with exactly the same sincerity whether or not the thing exists. You cannot turn honesty on from inside something that has no way to check itself.

Which means the bug was structural, not linguistic. The field was still required and the model still had no number, so it filled the blank. Politeness does not enter into it.

Once you see it that way the fix is obvious and slightly boring.

The statistic field became optional. No real number, no section, nothing to invent. A number is only allowed now if a real source URL for it sits in the article’s own source list, otherwise the model says it in words (“in most cases”, “within a short window”) without a fake decimal attached. The chart block lost its “use official statistics” line and its Bureau of Labor Statistics example, and now draws only values that already appear in the article text.

The quality checker needed fixing too. It used to deduct points for a missing statistic, which was rewarding the exact behaviour I was trying to kill.

The difference showed up immediately. Where the model used to reach for an official-sounding report that did not exist, it now points at a primary source that does, or it says nothing.

Then came the unpleasant part. I went through the 28 live articles by hand. Between them they carried 35 of these statistic boxes, and I removed 27 of them, because they were either fabricated or could not be traced back to anything. Eight survived. Four of the 35 looked plausible enough that I checked each against a real search: three were simply wrong (a claimed 16.8 million was really 6.2 million, another had no source I could find anywhere, a third was overstated) and one was close enough that I corrected it to the exact value and kept it. Twenty-seven out, eight kept. Not a flattering ratio. An invented source is worse than no number at all, so the rule I settled on is that untraceable numbers do not ship.

The same bug wearing a different hat

A week later the identical mistake turned up somewhere I did not expect it, which is the reason this post exists at all.

To cut cost I had moved the cheap mechanical calls to a smaller model. Not the article text, that stays on the strong one, just the scaffolding: layout code, the SVG, the chart data. Roughly halved the per-article cost. I was pleased with myself for about a week.

One of those mechanical calls rewrites the entire page component and needs a lot of output tokens. The small model caps out at 8192. The call asked for more. And the API did not complain. It returned HTTP 200, a perfectly successful response, with a quiet finish_reason: length and the code cut off in the middle of a JSX tag. The build died several minutes later with “Unexpected token ArticleLayout”, an error pointing nowhere near the actual cause. I spent an afternoon on that one.

Same shape as the statistics bug, wearing a different hat. Whether the output fits inside the token limit is something I could have checked in two lines of code. Instead I assumed it held, because the prompt implied it should.

The fix was the same kind of boring. Asking for more tokens than a model can produce now throws a config error before the call goes out, a truncated response fails the run instead of quietly moving downstream, and that call no longer goes to a model too small for the job.

The actual lesson

Both bugs came from one habit: asking the model, through the prompt, to guarantee something the code around it could have enforced.

A prompt is a request, not a guarantee. If a property can be checked mechanically, check it mechanically. The output fits in the token limit. A cited number traces to a real source. A required section has real content, or there is no section. The moment one of those lives only in the prompt, you have not prevented the failure, you have agreed not to look at it.

Here is the part that should worry anyone shipping LLM output to real users. Your model will fill any blank you make mandatory. Require a statistic and you get a statistic, real or not. Reward a number and you get numbers. What you are looking at is not dishonesty, it is your own structure reflected back at you. So fix the structure.

Topics: ai-contentllmprompt-engineeringbuilding-in-publiclessons