WriteMySEO / Blog / Cosine Similarity Thresholds Don't Transfer Between Models
SEO technology

Cosine Similarity Thresholds Don't Transfer Between Models

A 0.82 cutoff that worked with one embedding model is meaningless with the next. Here's why score scales differ, and how to calibrate thresholds against your own corpus.

Somewhere in your content tooling there is a hardcoded number. Maybe if similarity > 0.85: flag_as_cannibalization. Maybe a clustering script that merges pages above 0.9. It worked when you wrote it. Then you swapped the embedding model for a cheaper or newer one, re-ran the job, and suddenly every page on the site is a duplicate of every other page — or nothing is.

The threshold didn't break. It was never a property of the text in the first place. Cosine similarity is a number produced by a specific model's geometry, and that geometry changes completely between models, between versions of the same model, and sometimes between task modes of the same version.

Why the same pair of texts scores differently under two models

Cosine similarity measures the angle between two vectors. What varies is how the model distributes vectors across the available space.

Embedding spaces are anisotropic. Research on contextual representations has repeatedly found that embeddings from transformer models do not fill the sphere uniformly; they occupy a narrow cone. When every vector points roughly the same direction, even unrelated documents have a high baseline cosine. Two paragraphs about completely different topics can score comfortably above zero simply because the model's entire output lives in one region of the space.

Contrastive training objectives stretch or compress that cone. Modern retrieval embedding models are trained with contrastive losses and a temperature parameter that directly controls how aggressively positives are pulled together and negatives pushed apart. A low temperature produces a wide, well-spread score distribution; a higher one produces scores bunched near the top. Two models can rank an identical set of candidate documents in exactly the same order while assigning scores that differ by half a point.

Dimensionality tricks change the distribution too. Models trained with Matryoshka representation learning let you truncate a vector to fewer dimensions. Truncation preserves most of the ranking quality but it does not preserve the absolute scores. Same model, different dimension count, different threshold.

The practical consequence: a cosine of 0.78 is not "fairly similar." It is 0.78 for this model, and you cannot interpret it until you know what unrelated text scores.

Calibrate against a random baseline, not against zero

The fix is cheap. Before you pick any threshold, sample random pairs from your own corpus and look at the distribution of their scores. That gives you the model's floor — the score two documents get for merely being written in the same language, by the same brand, about the same general industry.

import numpy as np

# vecs: L2-normalised embeddings for your corpus, shape (n, d)
rng = np.random.default_rng(0)
i = rng.integers(0, len(vecs), 20000)
j = rng.integers(0, len(vecs), 20000)
mask = i != j
baseline = (vecs[i[mask]] * vecs[j[mask]]).sum(axis=1)

print(np.percentile(baseline, [50, 95, 99, 99.9]))

Now express every decision as a percentile of that baseline rather than a raw score. "These two pages are more similar than 99.9% of random page pairs on this site" is a statement that survives a model change. "These two pages score 0.91" is not.

Decision ruleSurvives a model swap?Survives a corpus change?
Raw cosine above fixed constantNoNo
Percentile of random-pair baselineYesMostly
Top-k nearest neighboursYesYes
Score margin over the k+1th resultYesYes

The baseline also tells you something about your own site. If your random-pair median is very high, your pages share a large amount of boilerplate — navigation, footers, repeated CTAs, templated intros — and the embedding is partly encoding your template rather than your content. That is a content problem the calibration exposes for free.

Prefixes and task modes silently change the geometry

Several widely used embedding families expect an instruction prefix. The E5 model family's own documentation requires inputs be prefixed with query: or passage: , and BGE-style models expect an instruction string on the query side. Google's embedding API exposes a task_type parameter with distinct values for retrieval documents, retrieval queries, semantic similarity, and classification.

These are not cosmetic. A symmetric similarity mode and an asymmetric retrieval mode produce different vectors for identical text, and scores from one are not comparable to scores from the other. The failure mode is quiet: if you embed documents with the retrieval-document mode and queries with the plain similarity mode, nothing errors, results just get worse in a way that looks like "the model isn't very good."

If you are embedding short queries against long passages, you want the asymmetric mode. If you are comparing two pages to each other for clustering or cannibalisation, you want the symmetric similarity mode. Mixing them inside one pipeline is the most common source of mysterious score drift I see in SEO tooling.

Length and boilerplate compress scores toward the middle

Long inputs get truncated at the model's token limit, and anything past that limit contributes nothing. Two 4,000-word guides may be compared entirely on their first few hundred tokens, which on most sites is the header, the intro and the table of contents. Short chunks have the opposite problem: fewer tokens means more variance, so a two-sentence chunk can land anywhere.

Both distortions push toward the same bad outcome. Pages look similar because their templates are similar, and distinct pages look identical because only shared scaffolding made it inside the context window. Strip boilerplate before embedding and check the token count of what you actually sent.

What to change in your pipeline

  1. Store the model identifier, version, dimension count and prefix or task mode alongside every vector. If that metadata isn't in your vector store, you cannot tell which comparisons are valid.
  2. Never compare vectors produced by different models or different task modes. Re-embed the whole corpus on an upgrade; a partial re-embed produces a split space that returns confident nonsense.
  3. Derive thresholds from a random-pair baseline on your own corpus, and recompute them every time the model or the corpus changes materially.
  4. Build a small labelled set — a few hundred pairs you have judged as duplicate, related, or unrelated — and pick the percentile cutoff that matches your judgments. This is the only way to know your threshold means what you think.
  5. Prefer rank-based decisions to score-based ones. Top-k neighbours and score margins are stable across models; absolute cutoffs are not.
  6. Log the baseline distribution with every run. When the distribution shifts, something upstream changed — a new template, a truncation bug, a silently updated API model.
embeddingsvector searchretrievalmeasurement

WriteMySEO produces marketing content, not legal, medical, financial, or compliance advice. Figures cited reflect publicly reported industry data at time of writing and shift over time.

Get started

We write this well about your industry, every month.

AI-drafted, human-reviewed SEO content on a flat subscription. Blog posts, metadata, schema, and internal links, shipped on a monthly rhythm.

See plans

More from the blog