Machine learning · intermediate
Embeddings
An embedding is a list of numbers (a vector) that a model assigns to a piece of text so that texts with similar meaning get vectors that lie close together. The model learns the numbers from data; no single number has a meaning on its own, only the position relative to other vectors does.
Why it matters
Embeddings are how software compares meaning instead of spelling. Semantic search, retrieval for chat assistants, recommendations, clustering and duplicate detection all reduce to the same operation: embed the items, embed the query, and return the nearest vectors. Knowing that explains why a search can match text that shares no words with the query, and why it can also return something merely similar rather than correct.
Mental model
How to reason about embeddings
Picture every text as a point on a map with hundreds of axes. Embedding places the point; comparing two texts means measuring between their points, usually by the angle between the vectors (cosine similarity). Search is then geometry: find the points nearest the query's point. Nothing is looked up by keyword, and nothing is understood at query time beyond that measurement.
Analogy
A library that shelves books by subject rather than by title. You find related books by walking to the right shelf and looking at the neighbours, even when their titles share no words, and a book shelved in the wrong place stays hard to find however good it is.
Examples
See the boundary, not just the happy path
Worked example · Embed a sentence
embed("How do I undo my last commit?") → [0.021, -0.413, 0.087, …] (1,536 numbers)The model returns a fixed-length vector whatever the length of the text. The individual numbers are not readable; the vector is only useful when compared with other vectors from the same model.
Worked example · Search by meaning
query: "undo my last commit" → nearest: "git revert", "git reset --soft", "git commit --amend"None of the top results contains the word undo. They rank first because their vectors lie closest to the query's vector, which is what similar meaning looks like in the embedding space.
Worked example · Compare with cosine similarity
cos(a, b) = (a · b) / (|a| |b|) → 1 same direction, 0 unrelated, −1 oppositeCosine similarity measures the angle between two vectors and ignores their lengths, so a one-line note and a long article on the same subject can still score close to 1.
Useful contrast · Keyword search
grep -ri "undo" docs/Keyword search matches characters, so it finds every line containing undo and misses a page about git revert that never uses the word. It is exact and predictable where embeddings are approximate and fuzzy.
Avoid · Mixing models
index built with model A, queries embedded with model BEach model has its own space, with different axes and often a different number of dimensions. Distances between vectors from different models are meaningless, so changing the model means embedding everything again.
Common mistakes
Misconceptions to remove early
Reading meaning into single numbers
The axes of a learned embedding are not labelled features such as size or sentiment. Meaning lives in distances and directions between whole vectors, so inspecting one coordinate tells you nothing.
Treating nearest as correct
Nearest-neighbour search always returns something, even when nothing relevant exists. Close vectors mean similar wording or topic, not a true or matching answer, so results need a similarity threshold or a later check.
Comparing vectors from different models
Vectors are only comparable inside the space of the model that produced them. A new model or a new version changes the space, and stored vectors must be regenerated before they can be searched with it.
Assuming distance and cosine always agree
Straight-line distance is affected by vector length and cosine similarity is not. They rank neighbours identically only when vectors are normalized to the same length, which many embedding models do but not all.
Quick check
Can you predict the result?
1. A search for "undo my last commit" returns a page about git revert that never uses the word undo. Why?
- • Its embedding lies close to the query's embedding
- • The search engine expanded undo into a list of synonyms
- • The page was tagged with the keyword undo by hand
2. Two texts on the same subject differ greatly in length. Which measure is designed to ignore that difference?
- • Cosine similarity, which compares direction only
- • Straight-line distance between the vectors
- • The number of dimensions in each vector
3. You replace your embedding model with a newer one. What must happen to the vectors already stored?
- • They must be regenerated with the new model
- • Nothing: all embedding models share one space
- • They only need to be sorted again
4. Why can a nearest-neighbour search return a confident-looking wrong result?
Keep building