What an AI Model Actually Is

When you type a question and text comes back, it is easy to picture something on the other side reading it. What is actually there is a file, and a program that runs that file. This page is about what the file does, so you can tell which behaviour comes from the model and which comes from the software someone built around it.

What is in the file

A trained model is a large collection of numbers, called parameters or weights. Training set those numbers. After training they are fixed, and every run uses the same ones.

Nothing in the file is a sentence you could open and read. There is no row labelled France that holds Paris. The parameters encode relationships that were useful for predicting text during training, spread across billions of values, and no single one of them corresponds to a fact.

What happens during one run

Take a language model of the kind behind most assistants today. Four things happen.

First, your text is cut into pieces called tokens and each piece becomes a number. That step has its own page: from text to tokens.

Second, the parameters transform that sequence of numbers into a score for every token the model knows.

Third, the program picks one token from that ranking. Not always the top one, because most systems sample rather than always taking the maximum, which is why the same question can produce different wording twice.

Fourth, the chosen token is added to the sequence and the whole thing runs again.

That loop is what people mean by generation. It describes autoregressive language models specifically. Image models, speech models and classifiers are also models, and they do not work this way, so be careful about carrying this description across to them.

Why plausible is not the same as true

Ask a database for the capital of France and it looks for a row. If the row is missing, you get an error.

Ask a language model and there is no lookup. It produces the continuation its parameters score highest. For the capital of France that continuation is Paris, and it is correct, because the pattern in the training text matches the world.

Now ask it who the first person to walk on Mars was. A name still scores highest. Nothing in those four steps checks whether anyone has walked on Mars. The run produces a likely continuation either way, and the machinery is identical in both cases.

Three separate things get confused here, and it is worth keeping them apart:

• How likely the model scored that continuation.
• How confident it sounds, which is a property of the wording.
• Whether it is true.

The second one misleads people most. Most text the model learned from is written in a confident register, so a confident register is the likely continuation. The tone is a pattern, not a report on the model's own reliability.

A system can estimate the first of those three with a separate calibration or scoring mechanism. Ordinary generation does not measure the third at all. If you want a truth check, something else has to perform it.

One consequence is worth seeing directly: a question with a false premise still has a likely continuation. Ask which year the Treaty of Ipswich ended the Anglo-Danish War and you will probably get a year, because the shape of the question makes a year the likely next thing. Being asked a malformed question does not, by itself, produce an objection.

What the model does not do on its own

When someone says the AI looked up their order, or refused to send the email, or remembered last week's conversation, they are almost always describing the application, not the model.

These are separate mechanisms that someone builds:

• fetching documents and putting them in the input
• calling tools and running actions
• keeping state between sessions
• validating the output against a schema
• applying policy about what is allowed
• evaluating whether the result was any good

The model contributes one thing: given the input it was handed, a ranking over what comes next. Everything else in the list is ordinary software, and each part can be inspected, tested and fixed on its own. That is the useful consequence of this whole page. When the output is wrong, the model is one suspect among several, and often not the right one.

What this changes

Give it the facts rather than asking it to hold them. Transformation of text you supply is a different reliability problem from recall of text you did not.

Expect variation between runs, and do not read a confident tone as a signal about accuracy.

When something goes wrong, ask which part failed before assuming it was the model.

For what changes the file itself, see training, fine-tuning, and inference.