WriteMySEO / Blog / How Tokenizers Decide Which Queries Your Page Can Match
SEO technology

How Tokenizers Decide Which Queries Your Page Can Match

Lexical indexes and embedding models split text into different units. That mismatch explains missed SKUs, hyphenated brands, and why hybrid retrieval exists.

Search engines do not match words. They match tokens — the units left over after a text normalization pipeline has finished chopping, folding, and rewriting your copy. Every claim you make about "the page targets this keyword" is really a claim about the token sequence your content produces versus the token sequence the query produces.

Most of the time the two pipelines agree and you never think about it. The interesting failures happen at the edges: product codes, hyphenated brands, diacritics, symbols, and languages without spaces. And they fail differently in a lexical index than in a vector index, which is the single best argument for running both.

What happens to text before anything gets indexed

A classic inverted index stores postings lists keyed by token. Getting from raw HTML to those keys involves a sequence that varies by engine but almost always includes:

The critical property: the query goes through the same pipeline. If a token in the document and a token in the query normalize to the same key, they can match. If they do not, the lexical layer contributes exactly zero, no matter how obviously related the strings look to a human.

Why subword tokenizers disagree with lexical ones

Neural retrieval does not use word tokens. Transformer models use a fixed subword vocabulary — WordPiece for BERT-family models, BPE for the GPT family, SentencePiece for many multilingual models — typically tens of thousands of entries. Anything not in the vocabulary gets shattered into pieces that are.

That is a feature: it guarantees no out-of-vocabulary failures. It is also why a rare brand name becomes four meaningless fragments whose combined embedding sits nowhere useful in vector space.

Run your own strings through it rather than guessing:

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("bert-base-uncased")
for s in ["Zerolag", "WD-40", "SKU AB-1200X", "C++"]:
    print(s, "->", tok.tokenize(s))

In WordPiece output, pieces that continue a previous token are prefixed with ##. A common brand resolves to one or two pieces; an invented one typically explodes into four or five. The specific split depends entirely on the model's vocabulary, so verify with the model you actually deploy — do not assume the result transfers from one tokenizer to another.

The practical consequence for embeddings: a string that produces many low-information subwords gives the model very little to work with. Two unrelated SKUs that share a prefix can land closer together than the SKU and its own product page's prose description.

Which failures belong to which layer

InputLexical indexEmbedding model
AB-1200XUsually splits on the hyphen into two tokens; exact-string recall depends on whether the engine also keeps the joined formShatters into subwords; weak, unstable vector
C++, C#Punctuation often stripped, collapsing to cTokenizes, but the symbol carries little learned meaning in most general models
co-operation vs cooperationDifferent keys unless a joining rule existsUsually fine — semantically adjacent
Müller vs MuellerDifferent keys unless folding is configuredUsually fine
东京都Depends entirely on the segmenterHandled by multilingual subword vocab
Synonyms, paraphraseNo match at allThis is the layer's whole purpose

Read the columns. Lexical retrieval is strong exactly where embeddings are weak (rare literal strings) and weak exactly where embeddings are strong (meaning). That is the mechanism behind hybrid retrieval — not a vendor fashion.

What Google does, and what you can only infer

Google has documented using BERT and MUM in ranking, and its own developer material on tokenization for ML uses WordPiece. What Google's production index tokenization looks like — how it handles hyphens, symbol-bearing terms, or digit-letter boundaries — is not published, and it has clearly changed over time. Queries like c++ tutorial return sensible results today, which tells you symbol-bearing terms get special handling somewhere, but it does not tell you how.

So treat Google as a black box with observable behavior, and treat your own stack as the place to actually measure. The inference that carries over is structural rather than vendor-specific: any retrieval system has a token-matching layer and a semantic layer, and strings that tokenize badly are at risk in whichever layer you rely on most.

Where this shows up in data you already have

What to do about it

  1. Tokenize your top 200 brand, model, and SKU strings with the exact embedding model you ship and with your lexical analyzer. Flag anything producing more than three subword pieces — those are your fragile strings.
  2. Write the variants once, in prose. If customers type AB1200X, AB-1200X, and AB 1200 X, the page should contain at least the two most common forms naturally — in a spec table, for instance. This is not keyword stuffing; it is making the literal strings indexable.
  3. Use alternateName in your Organization or Product schema for spelling variants and transliterations. It costs nothing and gives parsers an explicit mapping.
  4. Run hybrid retrieval in anything you control — BM25 plus vectors — specifically so rare literals have a lexical path. Our post on reciprocal rank fusion covers how to combine them without pretending the scores are comparable.
  5. Configure your site search analyzer deliberately: decide on diacritic folding, hyphen joining, and digit-letter splitting, write the decision down, and re-run your zero-result export a month later to confirm it moved.

The diagnostic question is always the same: does the exact string a user types survive both pipelines intact? If you cannot answer it from a test run rather than intuition, that is the first thing to fix.

tokenizationretrievalembeddingssite search

WriteMySEO produces marketing content, not legal, medical, financial, or compliance advice. Figures cited reflect publicly reported industry data at time of writing and shift over time.

Get started

We write this well about your industry, every month.

AI-drafted, human-reviewed SEO content on a flat subscription. Blog posts, metadata, schema, and internal links, shipped on a monthly rhythm.

See plans

More from the blog