How to Check an AI Policy Claim · Hanh D. Brown

Three companies put a promise about invented citations in writing.

One marketing page used the word eliminate. Another used avoid. A third offered legal citations that were hallucination free. Then researchers measured two of them.

Those two invent content between 17 and 33 percent of the time. Not none. Zero was the promise you were sold. That range is the measurement.

Sit with that gap. These are the tools built specifically to solve this problem. They are sold to people whose work depends on getting it right, and whose clients carry the cost when it is wrong.

That gap is why this page exists. Not the general worry that Artificial Intelligence (AI) makes things up, which everyone has heard by now. The narrower and more useful fact underneath it.

Can you trust an AI answer about the law?#

Not without opening the source. The reason is measured rather than theoretical, and the measurement is worse than most people expect. Researchers put specific, verifiable questions to general models about random federal court cases. Then they counted what came back wrong.

Between 58 percent with one model and 88 percent with another (Dahl and colleagues). Not edge cases. Not trick questions. Ordinary questions with checkable answers, about real cases on real court records.

Two further findings matter more than the headline number. The models often failed to correct a person whose question contained a wrong assumption. Ask about a case that does not exist and the answer arrives anyway, built on your error and adding to it.

The second is worse. They could not reliably tell when they were making something up. There is no inner signal to read.

That closes off the shortcut most people reach for. You cannot use the tone as a guide. Confidence is present either way, in the right answer and the invented one, in the same font on the same page.

None of this makes the tools useless. I use them daily. It means the output is a draft to be checked, never an answer to be repeated. The checking has to be done by a person with the document open in front of them.

Why does an AI invent a citation that looks right?#

A model is not looking anything up. It produces the next likely piece of text, given everything that came before it, and a legal citation has a very regular shape. That shape is learnable. Producing a convincing one takes no knowledge of whether the case behind it was ever decided.

Volume, reporter, page, year. Party names in a familiar rhythm. Numbers in the ranges numbers usually fall in.

Regularity of that kind makes the shape easy to produce and hard to doubt. The model has seen thousands of real citations. It has learned what one looks like, the way you have learned what a phone number looks like, and it can produce a convincing one on demand.

A prediction that looks right is the dangerous kind. A garbled reference gets caught by anyone awake. A clean one, correctly formatted, attached to a real sounding case, sails past a tired reader at the end of a long day.

It is like a menu written by someone who has never been in the kitchen. Every dish is spelled correctly. Every price is sensible. The document is perfectly formed and tells you nothing at all about what that kitchen can actually cook.

Calling this lying gives it too much credit. Lying would mean the machine knows better and chooses otherwise. This is a shape appearing where a fact was wanted.

Which is why the fix is not a better prompt. It is not a sterner instruction to be accurate, and it is not a more careful tone. It is a person opening the document.

Partly, and far less than advertised. Legal research companies answered the problem by building tools that retrieve a real source first and write the answer from it. That approach genuinely helps and it does not do what the marketing promised, which is a distinction the buyer has to make rather than the seller.

Researchers ran the first preregistered evaluation of these products (Magesh and colleagues). Preregistered matters here. It means the test was published before the results came in, so nobody could quietly move the target afterwards.

The tools from two of the largest legal publishers each invent content between 17 and 33 percent of the time. Put that in a working week and it stops being abstract. Check twenty answers and somewhere between three and seven of them carry something that is not there.

That same paper quotes the vendors. Eliminating hallucinations. Avoiding them. Hallucination free citations.

Both things are true together, and the pairing is the whole point. These tools beat a general chatbot by a wide margin. They are also wrong often enough that trusting the label without checking is still a mistake.

It is like a smoke alarm advertised as fireproof. The alarm is a real improvement on no alarm at all. Believing the label is still how a house burns down.

So the useful question is never whether a tool makes things up. Ask how often. Ask who measured it, and whether they had money riding on the answer. A vendor with no number has answered you.

What has to be true before a citation counts?#

Three things have to be true, and none of them implies the others, which is the part people get wrong. A reference can be real and still support nothing. It can support the sentence perfectly and describe a rule that stopped being law a year ago. Each test is separate.

First, does the document exist. Second, does the passage actually say what the sentence claims. Third, is the thing still in force.

A citation can pass the first and fail the second. The case is real. It simply stands for something else entirely, which is the failure that survives every automated check and arrives on the reader’s screen looking finished. It can pass the first two and fail the third, describing a rule repealed last year in words that read exactly like current law.

Most people stop after the first test. The citation checkers I have used stop there too, which leaves four distinct ways for a finished looking citation to be wrong:

• The document does not exist. The reference is well formed and points at nothing.
• The document exists and says something else. Your quote was never in it.
• The quote is real and reversed. The sentence after it undoes the claim.
• The rule was repealed. The old words read exactly like current law.

Only the first of those is caught by a citation checker.

Skipping the third is the quiet failure. A model has read the repealed version and the live version, and nothing in the language marks which is which. Only dates do that. Check the date every time.

There is a habit worth building here, and it costs about a minute. Before a citation goes into anything you send, write the three answers down beside it. Exists, says it, in force. Three words and a date.

Written answers are harder to fake to yourself than remembered ones. A vague sense that you checked something is the state most bad citations travel in. The note turns that feeling into a record. Believing you were careful and being able to show it are not the same thing. Write it down.

How do you check a quote against the original?#

Open the document and search it for the words. No Artificial Intelligence (AI) tool does this step for you. That is the entire method, and it is duller than people expect, which is exactly why it gets skipped for something that feels more like real research.

There is no clever version of this step. There is only the version where somebody does it.

Get the actual document rather than a page describing it. A summary is not the source. Neither is a press release about the source, and neither is a second article citing the first.

Then search inside it. Pick a short distinctive phrase from the quote rather than the whole sentence, because one changed word breaks an exact match and sends you looking for a problem that is not there. Five or six words is usually right.

If the phrase is absent, the quote is not from that document. The source being real changes nothing. That is the moment to stop.

Then read what sits around the passage with your own eye on the screen. A sentence can appear word for word and still be reversed by the line after it. Summaries drop that line constantly, and it is the line that carries the qualification. This is the failure behind how AI systems get described.

A quote can carry a named report, a working link and an entirely plausible claim, and still appear nowhere in that report. Every step of it was checkable the whole time. Nobody had opened the page and searched it.

What do you do when two sources disagree?#

Report both sources and say which one carries more weight. Never pick quietly and never split the difference, because a disagreement between sources is information and averaging it away destroys the most useful thing on the page. A reader who is told the question is contested is better served than one handed false calm.

Weight comes from the kind of document, never from how clearly it is written. The full ranking is set out in Check a Policy Claim, and the short version is that a law outranks a regulation, a regulation outranks a proposal, and a proposal binds nobody at all.

What matters here is the procedure rather than the ranking. When a lower ranked source contradicts a higher one, the higher one governs. Mention the disagreement anyway. When two of equal rank disagree, the later one usually governs, and the reader still deserves to know the question is contested.

Saying that out loud is not weakness. It is the most honest line on the page, and it is close to the point of what these systems say.

Machines are poor at this particular job and the reason is structural. A model is trained to produce one fluent answer, so a genuine split in the record gets smoothed into a single confident paragraph. The disagreement does not survive the summary. It is the first thing lost and the last thing missed.

Which means a contested question is exactly where a person has to step in. Not because the tool is careless, but because resolving a conflict requires knowing what kind of document each side is, and that is a judgement about authority rather than a pattern in text.

All of this takes minutes, and the cost of skipping it falls unevenly. Someone caring for an ageing parent, checking at midnight whether a plan pays for a home visit, has no research department behind them and no second opinion to call. They have one sentence on a screen and a decision to make by morning.