A tool tells you your page scores 0.82 cosine similarity against the target query, and the competitor scores 0.86. What are you supposed to do with that? The implied answer is "add more semantically related content until the number goes up." That instruction assumes the number is on a meaningful scale, that the gap of 0.04 represents something, and that the model producing the score resembles the model doing the retrieval. None of those three assumptions survive contact with how embeddings actually work.
Cosine similarity is a useful tool. It is just not a score, in the sense of a grade. It is a coordinate in a space defined entirely by the model that produced it, and it only becomes interpretable when you compare it to other coordinates from the same model.
What a cosine score actually measures
An embedding model maps text to a fixed-length vector. Cosine similarity between two vectors is the cosine of the angle between them after normalizing for length — a number from -1 to 1, though in practice most text embedding models produce values crowded into a much narrower positive band.
What the model learned to put close together depends entirely on its training objective. A model trained with a contrastive objective on query–passage pairs (the typical retrieval setup) learns that a question and the passage answering it should be close. A model trained on sentence-pair similarity learns that paraphrases should be close. A model trained on next-token prediction and then pooled into a vector learns something fuzzier: topical and stylistic co-occurrence. Those are different notions of "similar," and they disagree constantly. A page about the risks of a procedure and a page about the benefits of that procedure are topically near-identical and pragmatically opposite. Most embedding models score them as very similar, because most embedding models were never trained to care about the distinction.
Why the absolute number isn't interpretable
Embedding spaces are anisotropic: the vectors do not spread evenly across the sphere, they occupy a narrow cone. This is a well-documented property of contextual embedding models in the NLP literature, and you can reproduce it in ten minutes. Embed a hundred deliberately unrelated sentences with a general-purpose model and look at the pairwise similarities. You will not get a distribution centered on zero. With many models you get a tight band of clearly positive values, because there is a dominant direction that every vector partly shares.
That has three consequences for anyone reading a similarity readout:
- The floor is not zero. If unrelated text scores in the 0.6s with your model, then 0.72 is a weak result, not a good one. Thresholds like "aim for 0.8" are meaningless without knowing the model's baseline.
- Scales are not portable. A 0.86 from one model and a 0.86 from another are unrelated quantities. If your tool changes its embedding model — which vendors do, quietly — your historical scores become incomparable. Any trendline that spans a model change is noise.
- Format and length shift scores. Pooled embeddings drift with input length, boilerplate, and even formatting. A page whose extracted text includes nav and footer will score differently from the same page with a clean extraction, with no content change at all.
Why the model scoring you isn't the model ranking you
Google has documented that it uses neural language understanding in ranking — BERT-based systems, neural matching, MUM — and its published research describes dense retrieval and reranking architectures. What it has never published is the production model, its training data, or its scoring scale. Your similarity score is produced by a public model that shares a general family resemblance with what large retrieval systems use, and nothing more.
The architectural gap matters more than the weights. Dense retrieval with a bi-encoder is a first stage. It exists to pull a few hundred candidates out of billions cheaply, because each document can be embedded once and indexed. Final ordering typically comes from rerankers that see the query and document together, plus lexical signals, plus everything else in the ranking stack.
| Method | What it sees | Where it's used | What a score means |
|---|---|---|---|
| Bi-encoder (cosine) | Query and doc encoded separately | Candidate retrieval, clustering | Relative proximity in one model's space |
| Cross-encoder / reranker | Query and doc together | Reordering a shortlist | Calibrated relevance, still model-specific |
| Lexical (BM25) | Term overlap, term rarity | Retrieval, hybrid blends | Term-match strength, not meaning |
Optimizing a page to raise a bi-encoder cosine score is optimizing for the recall stage of a system whose precision stage you cannot see.
Normalize against a reference set instead of a threshold
The fix is to stop reading the raw number and start reading its rank within a comparable population. Embed the query, embed the pages that currently rank for it, embed yours, and ask where you sit in that distribution.
import numpy as np
# all vectors L2-normalized, from ONE model
sims = ranking_pages @ q # shape (n,)
mine = float(my_page @ q)
pct = (sims < mine).mean() * 100
print(f"raw {mine:.3f} | baseline {sims.mean():.3f} "
f"| {pct:.0f}th pct of {len(sims)} ranking pages")Now the output means something: below the median of pages Google already chose. Add a control set of clearly off-topic pages to establish the floor, and the spread between floor and ranking-set median tells you how much signal the model has for this query at all. If your off-topic control scores 0.64 and the ranking set averages 0.69, this model is not discriminating for this query and no amount of editing will make its score informative.
Chunking decides the score before the model does
If you embed a 3,000-word page as one vector, you get an average of everything on it, and the specific paragraph that answers the query gets diluted by the other twenty. Run the same comparison chunk by chunk and the picture usually inverts: the page's best chunk beats a competitor whose whole-page score was higher.
Since retrieval systems generally operate on passages rather than whole documents, the max-over-chunks score is the more faithful measurement. Report both. A page with a strong max and a weak mean is focused; a page with a mediocre max and a high mean is uniformly vague, which is the more common failure.
What to do with this
- Pin the model. Record model name and version alongside every score. Never compare scores across models or across a vendor's silent upgrade.
- Establish the floor. Embed 50 deliberately unrelated documents once per model. That mean is your zero, not 0.0.
- Report percentile, not raw cosine. Position against pages that already rank for the query, and state the sample size.
- Score chunks, keep the max and the mean. Use the same chunk boundaries you would expect a retriever to use — headed sections of a few hundred words.
- Use cosine for what it's good at: finding pages that compete with each other, surfacing internal link candidates, grouping a large corpus before a human looks at it. These are relative-order tasks, which is exactly what the measure supports.
- Treat a low score as a question, not a verdict. It tells you the model finds your page distant from the query. Read the page and decide whether that reflects an actual gap in coverage or an artifact of length, boilerplate, or a model that doesn't discriminate here.