From Text to Tokens

A language model never receives your sentence. It receives a list of integers. The component that converts one into the other is the tokenizer, and knowing what it does explains a few things that otherwise look like the model being strange.

Why not just use letters, or just use words

You could give every character a number. Then the vocabulary is tiny, but every sentence becomes a long sequence, and long sequences cost more to process.

You could give every word a number instead. Then sequences are short, but the vocabulary has to be enormous, and it still fails the first time someone types a word that was not in it.

Tokenizers sit between the two. Common words get their own token. Rarer words get split into pieces that are themselves common. The vocabulary stays a manageable size, sequences stay reasonably short, and nothing is unrepresentable, because worst case a word falls back to smaller and smaller fragments.

What a token is

A token is one entry in that vocabulary. It might be a whole word, part of a word, a punctuation mark, or a space attached to the word that follows it.

So a made-up word like "Smartification" might arrive as three tokens rather than one, something like Smart, ifi, cation. The model sees three integers in sequence, not a word it recognises.

This is why one visible word does not reliably equal one token, and why character count is a poor predictor of token count. A short sentence of unusual words can cost more tokens than a longer sentence of ordinary ones.

A toy version

Real tokenizers are built by algorithms such as byte pair encoding, which start from characters and repeatedly merge the most frequent adjacent pair until the vocabulary reaches a target size. The merge table is learned from data.

The snippet below is not that. It splits on a fixed rule and learns nothing. It is here only to make the splitting step concrete:

import re

text = "Smartification!"
tokens = re.findall(r"[A-Za-z]+|[^A-Za-z\s]", text)
print(tokens)
# ['Smartification', '!']

Notice that it keeps "Smartification" whole, which a real subword tokenizer would not. That gap is the point. A toy splitter shows you that splitting happens; it does not tell you how any particular model will split your text. For that, use the tokenizer that ships with the model you are actually calling.

Where token counts show up in practice

Two places, both operational.

Context limits. Models accept a bounded number of tokens per request. The bound is counted in tokens, not characters or words, so the only reliable way to know whether something fits is to tokenize it.

Cost and latency. Where a provider charges by volume, it charges per token, and processing time generally grows with sequence length. Estimating either from character count will drift, in a direction that depends on what you are sending.

Both of these are reasons to measure rather than estimate. Exact limits and prices change by model and provider, so check the ones you are using rather than trusting a number in an article.

A caution about what tokenization explains

Tokenization is often used to explain why models miscount letters in a word, or stumble over comparing numbers. The intuition is reasonable: if a word arrives as one integer, the individual letters are not separately present in the input.

Treat that as a plausible contributing factor rather than a settled cause. This page has not measured it, and the failures people point to usually have more than one thing going on. The safe statement is the narrow one: tokenization changes how input is segmented and represented before the model sees it. Where it goes beyond that, say less, or go and measure it.

For what happens after the integers reach the model, see what an AI model actually is. For how the token budget for a single run is filled, see context windows as working memory.