The keyword clustering question is really a page-count question: given five hundred keywords, how many pages should exist, and which keywords belong to which page? Get it wrong in one direction and you build near-duplicate pages competing with each other. Get it wrong in the other and one page chases four intents and satisfies none.
Most clustering tools answer this with semantic similarity — embeddings or shared words. That measures the wrong thing. Two keywords belong on the same page not when they mean similar things, but when Google treats them as the same intent. And Google publishes its opinion on that question every day, in the search results themselves.
What each method actually measures
Semantic clustering embeds each keyword as a vector and groups keywords whose vectors sit close together. It measures linguistic similarity: "crm software" and "crm platform" land near each other because the language is near-identical.
SERP-overlap clustering pulls the top results for each keyword and groups keywords whose result sets share URLs. It measures Google's behavior: if the same pages rank for both keywords, Google is satisfying both with one intent, and one page can plausibly cover both. If the result sets are disjoint, Google has decided these are different jobs — and a single page is fighting that decision.
The difference matters because language similarity and intent equivalence come apart constantly, in both directions.
Where semantic clustering gets the page count wrong
It merges what should be split. Keywords can be near-synonyms linguistically while carrying different intents. A phrase plus the word "best" often flips a SERP from vendor pages to comparison listicles; adding "for small business" can flip it again. An embedding model sees three phrasings of one concept. The SERP shows three different result-set compositions — which means three pages, or at least not one.
It splits what should be merged. Conversely, lexically distant queries — an abbreviation and its expansion, a symptom phrased two ways, jargon versus plain language — frequently return substantially the same results. Semantically they may sit in different neighborhoods; behaviorally they are one page's keyword set.
The reasonable inference from both failure modes is the same: embeddings model language in general, while ranking reflects what Google has learned from behavior on these specific queries. When the two disagree about intent, the SERP is the better evidence, because the SERP is the environment you are actually entering.
The worked method
The mechanics are simple enough to run in a spreadsheet at small scale or a script at large scale.
- Collect SERPs. For each keyword, pull the top 10 organic URLs from the same country and device, within as tight a time window as you can manage. This is the expensive step — you need a rank tracking or SERP API, and cost scales with keyword count.
- Normalize URLs. Strip tracking parameters, unify protocol and trailing slashes. Decide whether to compare at URL level (stricter, better for page-count decisions) or domain level (looser, more forgiving of pagination and near-duplicates).
- Score every pair. For each pair of keywords, count shared URLs between their top 10s. That count, 0–10, is your similarity score. Optionally weight shared URLs by position, since agreement at positions 1–3 is stronger evidence than agreement at 8–10.
- Set a threshold. Common practice is to treat roughly 3–4 shared URLs out of 10 as "same intent." There is no canonical value — it is a sensitivity dial, and you should inspect clusters at a couple of settings before committing.
- Choose a linkage rule. This choice changes results more than the threshold does:
- Chained (single-linkage): connect any pair above threshold and take connected components. Produces fewer, bigger clusters, but one bridge keyword can chain two unrelated groups together.
- Hub-and-spoke: pick the highest-volume keyword as the hub; every member must overlap with the hub. Tighter, more interpretable, biased toward the head term's intent.
- Complete-linkage: every member must overlap with every other member. Strictest, yields many small clusters, safest against accidental merges.
For page-count decisions, hub-and-spoke or complete-linkage is usually the right call: the cost of wrongly merging two intents into one page is higher than the cost of an extra small cluster.
Each resulting cluster is a page candidate: the hub is the primary target, the members are the secondary terms, and the cluster count is your answer to "how many pages."
The limitations you should price in
- SERPs move. An overlap snapshot is dated the day you pull it. Volatile or news-adjacent verticals need re-clustering more often than stable ones; for most sites, re-running quarterly or before major content planning is proportionate.
- Mixed SERPs are ambiguous. Some queries return deliberately diversified results spanning several intents. Low overlap between two such keywords does not prove different intent — it may mean both SERPs are hedging. Flag low-coherence clusters for human review rather than trusting the arithmetic.
- Localization matters. Overlap computed on US desktop SERPs says nothing about the German mobile clustering. Cluster per market you actually target.
- Cost scales with ambition. Pairwise comparison of 10,000 keywords is fifty million pairs. In practice you pre-block by shared terms or a cheap embedding pass, then compute SERP overlap only within blocks — semantics as the coarse filter, SERPs as the decision.
What to do
- Use both methods in sequence: semantic similarity to pre-group cheaply, SERP overlap to make the actual page-count decision inside each rough group.
- Start with hub-and-spoke linkage and a threshold around 3 of 10, then eyeball a sample of clusters and tune before trusting the output at scale.
- Weight by position if your tooling allows it; top-3 agreement is the signal worth paying for.
- Audit existing content against the clusters: two of your pages inside one cluster is a consolidation candidate; one page straddling two clusters is a split candidate.
- Date-stamp every clustering run and re-pull before the next planning cycle — the clusters are a snapshot of Google's opinion, and that opinion gets revised.