Search engines do not match words. They match tokens — the units left over after a text normalization pipeline has finished chopping, folding, and rewriting your copy. Every claim you make about "the page targets this keyword" is really a claim about the token sequence your content produces versus the token sequence the query produces.
Most of the time the two pipelines agree and you never think about it. The interesting failures happen at the edges: product codes, hyphenated brands, diacritics, symbols, and languages without spaces. And they fail differently in a lexical index than in a vector index, which is the single best argument for running both.
What happens to text before anything gets indexed
A classic inverted index stores postings lists keyed by token. Getting from raw HTML to those keys involves a sequence that varies by engine but almost always includes:
- Unicode normalization (NFC/NFKC), which collapses visually identical characters with different code points.
- Case folding, usually to lowercase.
- Punctuation handling — stripped, treated as a boundary, or preserved for specific patterns.
- Splitting rules for hyphens, slashes, underscores, and digit-letter boundaries.
- Diacritic folding, so café and cafe share a key (optional, and language-dependent).
- Stemming or lemmatization, so running and runs collapse toward run.
- Segmentation for Chinese, Japanese, Thai and other scripts that do not delimit words with spaces.
The critical property: the query goes through the same pipeline. If a token in the document and a token in the query normalize to the same key, they can match. If they do not, the lexical layer contributes exactly zero, no matter how obviously related the strings look to a human.
Why subword tokenizers disagree with lexical ones
Neural retrieval does not use word tokens. Transformer models use a fixed subword vocabulary — WordPiece for BERT-family models, BPE for the GPT family, SentencePiece for many multilingual models — typically tens of thousands of entries. Anything not in the vocabulary gets shattered into pieces that are.
That is a feature: it guarantees no out-of-vocabulary failures. It is also why a rare brand name becomes four meaningless fragments whose combined embedding sits nowhere useful in vector space.
Run your own strings through it rather than guessing:
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("bert-base-uncased")
for s in ["Zerolag", "WD-40", "SKU AB-1200X", "C++"]:
print(s, "->", tok.tokenize(s))
In WordPiece output, pieces that continue a previous token are prefixed with ##. A common brand resolves to one or two pieces; an invented one typically explodes into four or five. The specific split depends entirely on the model's vocabulary, so verify with the model you actually deploy — do not assume the result transfers from one tokenizer to another.
The practical consequence for embeddings: a string that produces many low-information subwords gives the model very little to work with. Two unrelated SKUs that share a prefix can land closer together than the SKU and its own product page's prose description.
Which failures belong to which layer
| Input | Lexical index | Embedding model |
|---|---|---|
AB-1200X | Usually splits on the hyphen into two tokens; exact-string recall depends on whether the engine also keeps the joined form | Shatters into subwords; weak, unstable vector |
C++, C# | Punctuation often stripped, collapsing to c | Tokenizes, but the symbol carries little learned meaning in most general models |
co-operation vs cooperation | Different keys unless a joining rule exists | Usually fine — semantically adjacent |
Müller vs Mueller | Different keys unless folding is configured | Usually fine |
东京都 | Depends entirely on the segmenter | Handled by multilingual subword vocab |
| Synonyms, paraphrase | No match at all | This is the layer's whole purpose |
Read the columns. Lexical retrieval is strong exactly where embeddings are weak (rare literal strings) and weak exactly where embeddings are strong (meaning). That is the mechanism behind hybrid retrieval — not a vendor fashion.
What Google does, and what you can only infer
Google has documented using BERT and MUM in ranking, and its own developer material on tokenization for ML uses WordPiece. What Google's production index tokenization looks like — how it handles hyphens, symbol-bearing terms, or digit-letter boundaries — is not published, and it has clearly changed over time. Queries like c++ tutorial return sensible results today, which tells you symbol-bearing terms get special handling somewhere, but it does not tell you how.
So treat Google as a black box with observable behavior, and treat your own stack as the place to actually measure. The inference that carries over is structural rather than vendor-specific: any retrieval system has a token-matching layer and a semantic layer, and strings that tokenize badly are at risk in whichever layer you rely on most.
Where this shows up in data you already have
- Internal site search zero-result logs. Hyphen and spacing variants of the same product are the single most common cause. Export the zero-result queries, normalize them yourself, and check how many have a non-zero-result sibling that differs only by punctuation.
- Search Console query rows.
wd 40andwd-40can appear as separate query strings with different impression counts. Grouping only on exact strings understates demand for anything punctuated. - RAG evaluation sets. If your retriever misses questions containing part numbers while handling conceptual questions well, that is a tokenization symptom, not a chunking one.
- Non-Latin markets. A segmentation mismatch between your on-site search engine and the way users type produces recall collapse that looks like "nobody searches for this."
What to do about it
- Tokenize your top 200 brand, model, and SKU strings with the exact embedding model you ship and with your lexical analyzer. Flag anything producing more than three subword pieces — those are your fragile strings.
- Write the variants once, in prose. If customers type
AB1200X,AB-1200X, andAB 1200 X, the page should contain at least the two most common forms naturally — in a spec table, for instance. This is not keyword stuffing; it is making the literal strings indexable. - Use
alternateNamein yourOrganizationorProductschema for spelling variants and transliterations. It costs nothing and gives parsers an explicit mapping. - Run hybrid retrieval in anything you control — BM25 plus vectors — specifically so rare literals have a lexical path. Our post on reciprocal rank fusion covers how to combine them without pretending the scores are comparable.
- Configure your site search analyzer deliberately: decide on diacritic folding, hyphen joining, and digit-letter splitting, write the decision down, and re-run your zero-result export a month later to confirm it moved.
The diagnostic question is always the same: does the exact string a user types survive both pipelines intact? If you cannot answer it from a test run rather than intuition, that is the first thing to fix.