YOU ARE USING AI WRONG

There is a box, and it says: Ask me anything.

That is the beginning of the problem.

It is also a design decision someone made on your behalf.

The box is a text field. Beside it, a cursor blinks. The product surrounding it is called an assistant, a copilot, an agent, or something similarly reassuring. It is polite. It is fast. It answers in complete sentences.

What it usually does not say is this:

I generate likely language. I may be useful, insightful, and wrong. I am not, by itself, an authoritative source offacts.

Instead, the interface says ask. The cursor waits. The system responds.

The framing matters. A blinking cursor and an invitation to ask a question suggest that something on the other side knows the answer. Not merely that it can help you write, organize, translate, or reason through material. That it knows.

So naturally, people ask it things.

. What version of this library introduced that feature?

. When did this product launch?

. Who holds this job title?

. What is the current law in this jurisdiction?

. Can you give me cases supporting this argument?

For a while, this often appears to work. Much of what people ask about is common, stable, and well represented in the material used to train these systems. The answer may be correct often enough to establish a habit of trust.

Then comes the answer that is wrong.

Not obviously wrong. Not absurd. Not written in broken English or accompanied by a warning siren. Wrong in the same calm, competent voice the system uses when it is right. Wrong at the same speed. Wrong with the same neat headings, citations, qualifications, and reassuringly professional tone.

And you have the distinctly modern experience of being misled by something that does not intend to deceive you.

That distinction matters.

The error

You did not make a foolish mistake.

You accepted an invitation.

The mistake was built into the category the interface encouraged you to use. We tend to treat a language model as a source ofknowledge: a place facts can be requested and retrieved. But, on its own, a language model is better understood as a system for generating and transforming language.

Those are not merely different points on a scale. They are different kinds of tools.

A calculator and an atlas can both be useful during a geography lesson. But they do different jobs. Ask a calculator for the population of Peru and it cannot help you — not because it is unintelligent, but because population data is not what calculators do. Give it numbers and an operation, however, and it can produce an answer with extraordinary reliability.

Language models have a similarly powerful — but different — role.

Give one a report and ask for a summary. Give it meeting notes and ask for action items. Give it a policy and ask for a plain-English explanation.

Give it a table and ask for a narrative, or a draft and ask for a clearer version in a different tone.

These are transformations. The relevant material is available in the conversation, in an attached document, in a connected database, or through a properly designed retrieval system. The model's task is to work with it.

That is where language models can be remarkably useful.

The trouble begins when we ask the model to supply both the answer and the underlying evidence from nowhere visible. It may produce a good answer. It may produce an answer that resembles a good answer so closely that only someone with independent knowledge can tell the difference.

And it will usually try.

That willingness to answer is one of its strengths as a writing tool. It is also the trap.

A plausible answer is not evidence

Consider a deliberately simple example.

Someone asks a general-purpose chatbot:

What is the capital of the fictional country of Zubrowka? Ifyou do not know, reply only: UNKNOWN.

Zubrowka is fictional. There is no capital city to identify.

A careful system might reply:

UNKNOWN

Another might say that it cannot verify the premise.

But a system can also produce a city name — one that sounds geographically plausible, perhaps with an invented history or a confident explanation of its architecture, population, or political role.

The exact country does not matter. This is an illustrative prompt, not a claim about a particular benchmark or a particular model. The important point is what the example reveals: polished language can be generated even when the premise is false and the answer has no factual foundation.

The model is not necessarily "choosing" to invent a capital. It is completing a pattern. Questions about countries are commonly followed by answers about capitals. Fictional place names often appear in stories, games, and invented settings. A plausible continuation can emerge even though there is no real-world fact behind it.

That is not knowledge retrieval in the ordinary sense. It is language generation that can resemble knowledge retrieval.

The output may be fluent, coherent, and utterly unsupported.

But this is not only a hypothetical. The same failure has already run a benchmark in the open: five fixed prompts, each with an answer a program could check, run against four models on an ordinary desktop machine. Three models scored five out of five — asked about a country that does not exist, they said UNKNOWN. One scored four out of five, and on the fifth it produced a capital city: a name, in the same confident register as its correct answers, for a place that has never existed, in response to a question that told it exactly how to admit ignorance. The benchmark is real, reproducible, and lives in the open-source system this book points at (src/lib/models/alpha.ts in the PersonalAI-Dashboard repository). You can run it yourself.

This has already happened in the real world

The consequences become more serious when the answer enters a professional workflow.

In Mata v. Avianca, Inc., a 2023 case in the United States, lawyers filed a brief containing citations to judicial decisions that did not exist. The cited

cases had been generated with the help of ChatGPT. The cases had plausible names, plausible citations, and descriptions that appeared to support the legal argument being made.

They were not real.

The court did not discover a minor formatting problem. It found fabricated authorities submitted in a legal filing. The lawyers were sanctioned.

This was not a failure of prose. The writing was sufficiently credible to be filed. It was a failure of verification.

The episode is useful precisely because it strips away the comforting version of the problem. The issue is not that an AI system occasionally makes an obvious blunder. The issue is that it can produce material that looks like the sort of thing professionals are trained to trust: a case name, a citation, a holding, a quotation, a procedural history.

All of it can be structurally convincing while being factually nonexistent.

A citation is not evidence because it has the shape of a citation. A case is not real because it has a case name. A claim is not supported because the paragraph explaining it is polished.

The source must be checked.

Why confidence is the wrong signal

Human beings use fluency as a shortcut.

When something is clearly expressed, grammatically correct, specific, and confidently stated, it feels more likely to be true. Often, in ordinary life, that shortcut is useful. Experts tend to speak with greater precision than people guessing. Official documents tend to look more formal than casual speculation. Someone who hesitates may genuinely be uncertain.

But with generated text, fluency and factual accuracy are not reliably connected.

A language model can write a careful, nuanced, caveated explanation of an event that never happened. It can also give a correct answer in a hesitant tone. Its confidence is a property of the wording, not a dependable measure of the truth.

This is why "Does this sound right?" is such a dangerous test.

It measures whether the answer sounds right.

It does not establish whether the answer is right.

And as these systems become better writers, that distinction becomes more important, not less. Better prose makes unsupported claims more persuasive. Cleaner explanations make missing evidence less visible. A more natural conversational style can make the system feel less like software and more like an informed colleague.

The risk is not simply error.

The risk is believable error.

It is not lying

It is tempting to say the model lied. The experience can feel personal: it told you something false with apparent certainty.

But lying requires an intention to deceive — knowing what is true and choosing to say otherwise. A language model does not have beliefs, intentions, or an internal fact-checking instinct in that sense. It does not consult a private shelf of verified knowledge and decide whether to mislead you.

It generates an answer based on patterns in language and the information available to it at that moment.

Sometimes that answer is accurate. Sometimes it is incomplete, outdated, distorted, or fabricated. The system can be prompted to express uncertainty, and it can be connected to tools that retrieve evidence, but neither a disclaimer nor a confident tone should be mistaken for proof.

The failure is not moral. It is architectural.

If a system has not been given reliable, current, relevant evidence — or has not been connected to a way of retrieving and checking it — then an answer may be no more than a highly plausible completion.

That may be useful for brainstorming.

It is not enough for a legal filing, a medical decision, a financial report, a customer promise, or any other claim where being wrong has consequences.

Whose fault is it?

Not yours.

You were given a box that said ask me anything. It answered immediately. It sounded composed and informed. It behaved in a way that invited you to treat it as a source.

Every instinct you brought to that interaction was understandable.

The problem is not that users need to become suspicious of every sentence a machine produces. The problem is that systems should be designed so that the boundary between evidence and generated language is visible.

A trustworthy system does not merely produce an answer. It makes clear:

. where the relevant facts came from;

. when those facts were retrieved or updated;

. which sources support which claims;

. what the system does not know;

. and what requires human review.

This is not a matter of finding the magic prompt.

Prompts can improve an interaction. They can ask for citations, require uncertainty labels, request a particular format, or tell a model not to guess. Those are useful techniques.

But a prompt cannot turn unsupported text into evidence.

If the system needs current product information, it should retrieve that information from an authoritative product source. If it needs a customer's account details, it should use the relevant authorised system. If it needs case law, it should search and cite a trusted legal database, then provide links or references that a person can verify.

The answer must be built on evidence, not merely decorated with the appearance of it.

What changes

The shift is simple:

Trust the model to transform verified information. Do not treat the model itself as the final authority for factual claims.

This is not a limitation to resent. It is a division of labour.

Language models are exceptionally good at helping people work with information. They can summarize long documents, compare policies, extract structured data, rewrite technical material for different audiences, classify feedback, draft code, explain processes, and translate between the languages used by people and the formats used by software.

None of those abilities requires the model to be where the facts live.

In fact, they become more valuable when the facts live somewhere reliable.

A well-designed system might work like this:

6. A person asks a question.

7. The system retrieves relevant information from approved sources.

8. The model explains, summarizes, compares, or reformats that information.

9. The system shows the sources and preserves the limits of what was found.

10. A person makes the decision when the decision matters.

In that arrangement, the model is not pretending to be an oracle. It is doing what it does best: turning information into useful language.

The source of truth remains where it belongs — in records, documents, databases, official publications, measured results, and systems that can be inspected and checked.

That one rearrangement changes everything.

It changes how facts enter the system. It changes how claims are cited. It changes how errors are caught. It changes what can be automated safely and what must remain subject to review.

Most importantly, it changes the question we ask.

Instead of asking, "Can the model tell us the answer?"

We ask, "What evidence should support the answer, and how can the model help us use it?"

That is the foundation for everything that follows.

The next question is why a language model cannot reliably tell you, on its own, when it does not know something — and why asking it to be more careful does not solve the underlying problem.

• • •