Finding Near-Duplicate Pages Before Google Clusters Them

Most duplicate-content checks in SEO tools compare title tags, meta descriptions, or an exact hash of the HTML. Those checks find almost nothing useful. A timestamp, a CSRF token, or a rotating "related posts" block changes the hash of every page, so exact matching returns zero. Title comparison finds labeling problems, not content problems.

The pages that actually get clustered are the ones where 1,200 tokens of template surround 150 tokens of unique text. Google has documented that it groups pages it considers duplicates into a cluster and then selects one URL to represent it. You want to know which of your pages are sitting in the same cluster before Search Console tells you by dropping most of them out of the index.

Why exact-match checks miss the pages that matter

Near-duplication on real sites comes from a small number of generators:

• Templated page sets — location, product-variant, or "X in Y" pages where the differentiating content is a few swapped nouns.
• Faceted and parameterized URLs that render the same items in a different order.
• Syndicated or supplier-provided copy reused verbatim across your site and everyone else's.
• Pagination and archive views where the template dominates the item list.
• Rewritten pages that cover the same intent in different words — invisible to literal matching but a real cannibalization risk.

Each of these needs a different detection method, and two of them need you to fix your text extraction before you measure anything.

Strip boilerplate first, or you will just measure your own navigation

If your header, nav, sidebar, and footer contribute 700 tokens to every page and the unique body is 150 tokens, then every page on your site is roughly 80% identical to every other page. Run similarity on raw HTML text and you will get a beautiful matrix of 0.8 scores that tells you nothing.

Two ways to remove it:

1. Main-content extraction — run each page through a readability-style extractor (trafilatura, Mozilla Readability, or your own DOM heuristics) and keep only the article body.
2. Corpus-wide document frequency on blocks — split each page into text blocks, hash each block, count how many pages contain each hash, and drop blocks appearing on more than some share of the crawl. This is more robust on sites where the template is irregular, because it learns your boilerplate instead of guessing.

While you are there, record the boilerplate ratio per page: boilerplate tokens divided by total tokens. Pages above 0.8 are thin regardless of what the similarity numbers say later, and that single column is often more actionable than the pairwise analysis.

How do you compare a million pages without a million-squared comparisons?

Use the shingling approach Broder and colleagues described for AltaVista in the late 1990s: represent each document as the set of its overlapping word n-grams ("shingles"), then measure Jaccard similarity between sets. Five-word shingles are a reasonable default — short enough to survive small edits, long enough that common phrases don't collide.

Comparing every pair is quadratic. MinHash plus locality-sensitive hashing gets you to roughly linear: MinHash gives each document a short signature whose collision probability approximates Jaccard similarity, and LSH banding buckets documents so that only likely-similar pairs are ever compared.

from datasketch import MinHash, MinHashLSH

def shingles(text, w=5):
    t = text.lower().split()
    return {" ".join(t[i:i+w]) for i in range(len(t)-w+1)}

lsh = MinHashLSH(threshold=0.7, num_perm=128)
sigs = {}

for url, body in extracted_main_content.items():   # boilerplate already removed
    m = MinHash(num_perm=128)
    for s in shingles(body):
        m.update(s.encode("utf8"))
    sigs[url] = m
    lsh.insert(url, m)

for url, m in sigs.items():
    near = [u for u in lsh.query(m) if u != url]
    if near:
        print(url, near[:5])

This finds literal reuse: syndication, copy-paste variants, template-dominated page sets. It will not find a paraphrase.

Where embeddings tell you something shingles cannot

Embedding two pages and taking cosine similarity catches semantic overlap with no shared wording. That is the one thing shingling structurally cannot do.

The trap is that general-purpose text embeddings are topical proximity detectors, not duplicate detectors. On a single-topic site, unrelated pages routinely sit at cosine 0.8 or higher simply because they share vocabulary and domain register. Any threshold you borrowed from a blog post is meaningless on your corpus. Calibrate instead: embed a random sample, compute pairwise cosine, look at the distribution, and treat only the extreme upper tail as signal.

Used together, the two methods disambiguate each other:

What thresholds mean, and why you should validate by hand

Pick thresholds from your own distribution, not from a default. Plot the similarity histogram; real corpora usually show a dense low-similarity mass and a separated tail from the templated page sets. Set the cut where the tail begins, then manually inspect twenty pairs at that boundary. If more than a couple look like legitimate distinct pages, raise it.

Bias toward precision over recall. The actions this analysis triggers — consolidation, redirects, noindex — are destructive and slow to reverse. Missing a few near-duplicates costs you less than merging pages that were earning independent impressions.

How do you know whether Google already clustered them?

This is the part most similarity analyses skip, and it is the only external calibration available. Google's own clustering decision is partially visible:

• Search Console page indexing report — the statuses Duplicate without user-selected canonical, Duplicate, Google chose different canonical than user, and Alternate page with proper canonical tag are Google telling you it collapsed a cluster.
• URL Inspection API — compare userCanonical with googleCanonical for each URL in your candidate clusters. A mismatch means Google picked a different representative than you declared.
• Query–URL data — within a cluster, count how many members earn impressions. Clusters where one URL takes essentially everything have already been resolved.

Join those results back onto your similarity scores. The similarity level at which Google starts choosing its own canonical on your site is the threshold that matters, and it is empirical rather than theoretical. Google has documented that canonical selection considers signals including redirects, rel=canonical, sitemap inclusion, internal linking, and an HTTPS preference; the clustering step that happens before selection is not documented in detail, so treat any specific similarity cutoff as inference from your own data, not a published rule.

What to do with this

1. Crawl, extract main content, and log a boilerplate ratio per URL. Fix extraction before trusting any similarity number.
2. Run MinHash + LSH at a deliberately loose threshold to generate candidate pairs, then compute exact Jaccard on candidates only.
3. Embed the same cleaned text and add cosine similarity as a second column. Use the quadrant table to classify, not a single score.
4. Pull URL Inspection results for every cluster member and mark which clusters Google has already collapsed.
5. Act by category: consolidate literal duplicates, differentiate or merge same-intent pairs, and for large templated sets decide whether the unique content justifies a separate URL at all — if the answer is no for most of the set, the fix is a template change, not a redirect map.
6. Store the similarity scores and re-run after template releases. A deploy that adds 300 tokens of boilerplate to every page will move these numbers before it moves your traffic.