What Are Embeddings?
An embedding turns text into a list of numbers, so that related text lands close together. What the numbers, dimensions and similarity scores mean.
Search for "small dog" and a good system finds the passage that only says "puppy". No shared word made that happen. The system compared two lists of numbers and found them close. Those lists are embeddings.
OpenAI's documentation describes an embedding as a vector, a list of floating point numbers. An embedding model produces the list for a piece of text, positioned so that text with related meaning gets similar numbers. Everything else on this page follows from that.
From a sentence to a point
A sentence
"my small dog"
The embedding model
Numbers for each piece.
Pool into one
Often by averaging.
A point in space
1536 numbers, say.
The model reads the text as tokens and produces numbers for each one. One sentence needs one list, so the per-token numbers are combined. Sentence-BERT produces one embedding per sentence by pooling the model's per-token outputs, and its default is the mean. That averaged list is what people mean by a mean embedding.
What the dimensions are
Each number in the list is one dimension. OpenAI's text-embedding-3-small returns 1536 numbers by default and text-embedding-3-large returns 3072, and a dimensions parameter shortens the list. More dimensions can hold finer distinctions; fewer are cheaper to store and faster to compare. The right length is the shortest one that still finds the right material on your own data.
The individual numbers have no names. Dimension 412 does not mean "animal" or "formal". Meaning lives in where the whole list points, relative to other lists, which is why people call the collection an embedding space.
How similarity is measured
Two embeddings are compared by the angle between them, called cosine similarity. Pointing the same way scores 1; unrelated scores near 0. OpenAI's embeddings are normalised to length 1, so cosine similarity can be computed as a dot product and gives the same ranking as Euclidean distance. For these embeddings, the two common distance measures put results in the same order.
A similarity score is a measure of closeness. It is not a probability, and 0.82 does not mean 82% sure.
What embeddings are used for
OpenAI lists search, clustering, recommendations, anomaly detection, diversity measurement and classification as uses. The best-known one is the retrieval step of retrieval-augmented generation: embed the question, find the nearest passages, and give those to the model. The same comparison groups support tickets by topic, finds near-duplicates, and flags the document that looks like nothing else.
What nearness does and does not tell you
Nearness in an embedding space measures what the model learned to treat as related. That is often what you want, and sometimes exactly what goes wrong.
Nearness tells you
- Similar meaning, even in different words
- The same subject
- Related, as the model learned it
Nearness does not tell you
- Which one is correct
- Which one is current
- Which one answers the question
A score near 1 is closeness, which is different from confidence.
A policy replaced last month sits right next to the policy that replaced it, because they are about the same subject. An embedding search can return either. Which one is current, and which one answers the question, has to be decided by something other than distance: filters, dates, a reranker, or a person. Different embedding models also arrange the space differently, so a model that works well on a public benchmark can still put your material in the wrong neighbourhoods. Test it on your own queries.
What this changes
Read a similarity score as closeness, never as confidence. Pick dimensions by testing on your own data. And when an embedding search returns something plausible and wrong, the search worked as designed: the fix belongs in what happens after the nearest neighbours are found, which is where the context for the model gets assembled.
Where this helps, and where it stops
- What it is
- A plain explanation of text embeddings: the vector, its dimensions, similarity scores, how one sentence gets one embedding, and what nearness means.
- What it does not guarantee
- A comparison of embedding models or vector databases, or a treatment of how embedding models are trained.
- When the distinction matters
- Someone is building search or retrieval, choosing an embedding model or dimension, or reading a similarity score as if it were a confidence.