Computing Internal PageRank From Your Own Crawl Data

Every desktop crawler ships a 0–100 "link score" or "internal PageRank" column, and people treat it like a readout from Google. It isn't. It's a power-iteration PageRank computed over the link graph your crawler happened to build, with a handful of modeling decisions baked in that the UI never exposes.

That doesn't make it useless. The rank ordering of your own URLs, compared against itself before and after a change, is one of the few internal-linking measurements you can actually compute. The absolute number is noise. Here's what the algorithm does, which choices move the result, and how to compute it yourself so you know what's in it.

What the algorithm actually computes

PageRank models a random surfer who follows a link with probability α (conventionally 0.85) and otherwise teleports to a random page. The score of a page is the stationary probability of that surfer being there:

PR(p) = (1-α)/N + α * Σ [ PR(q) / outdegree(q) ]  for all q linking to p

It's solved iteratively — start every page at 1/N, apply the update until scores stop moving. Two details matter more than the formula:

• Outdegree is a divisor. A page with 400 links passes 1/400th of its score down each edge. Bloated navigation doesn't just dilute "link equity" metaphorically; it's literally division.
• Dangling nodes need a rule. Pages with zero outbound internal links (PDFs, terminal pages, anything your crawler stopped at) are sinks. Standard implementations redistribute their mass uniformly across all pages. Some crawlers instead let the mass evaporate, which depresses every score. This is one reason two tools report different numbers for the same site.

The scores are a probability distribution — they sum to 1 across N pages. That means adding URLs lowers everyone's score even if nothing else changed. A 0–100 display is a normalization of that distribution, usually against the site's maximum, which is nearly always the homepage.

Why your crawler's graph isn't Google's graph

Google computes link-based scores over the whole web. Your crawler computes over a bounded set of URLs it could fetch. The gaps are structural, not cosmetic:

• External links are missing on both sides. A product page with hundreds of referring domains looks identical to one with none. Internal PageRank can't see earned links, which is the main source of real authority.
• The crawl boundary is arbitrary. Subdomains, alternate locales, and anything behind a parameter rule you excluded simply don't exist in the graph.
• JavaScript-injected links only exist if you rendered. If your crawl ran in HTML-only mode and your faceted nav or related-products module is client-side, half your edges are absent. Google renders; your default config might not.
• rel="nofollow" handling is a guess. Google announced in 2019 that link attributes including nofollow became hints rather than directives, effective for crawling and indexing in March 2020. How your crawler treats them — drop the edge, keep it, keep it but don't divide by it — is a modeling choice with no provably correct answer.

Redirects and canonicals change the answer more than α does

This is the decision most people never inspect. If 10,000 internal links point at http:// or trailing-slash variants that 301 to the canonical URL, you have two options, and they produce different graphs:

1. Treat the redirecting URL as its own node. It accumulates score and passes it down one edge to the target. The target still ends up high, but the redirecting URLs show up as high-scoring phantoms in your report.
2. Collapse edges onto the final target. The redirecting URLs vanish from the graph and the target absorbs all of it directly.

Option 2 is closer to how Google consolidates signals, and it's what you want for analysis. Most crawlers default to option 1 unless you tell them otherwise. Same question applies to rel=canonical: do non-canonical duplicates get folded into their canonical target, or counted separately? Fold them, for the same reason.

Boilerplate links will dominate unless you separate them

Raw PageRank weights every edge on a page equally. Your header, footer, and sidebar appear on every URL, so the handful of targets they point at collect an enormous share of the mass. The top of your "internal link score" report is usually just a list of your navigation — which you already knew.

Google almost certainly doesn't weight edges uniformly. The reasonable-surfer patent describes weighting links by the probability a user clicks them, and Google representatives have said for years that position and prominence affect how much a link counts. We can't replicate their weighting, but we can isolate the effect.

Compute the graph twice: once with all links, once with boilerplate modules excluded (most crawlers can restrict link extraction to a content container via CSS selector). The content-only graph tells you where your editorial and template-contextual linking actually sends authority. The delta between a page's rank in the two graphs is the interesting number: a page that's top-decile with nav and bottom-decile without is being propped up by a menu item, not by anything a reader would follow.

Computing it yourself in about fifteen lines

Export the crawl's inlinks as source,target pairs, resolve redirects and canonicals first, then:

import csv, networkx as nx

G = nx.DiGraph()
with open("edges.csv") as f:
    for src, dst in csv.reader(f):
        if src != dst:                 # drop self-links
            G.add_edge(src, dst)       # DiGraph dedupes repeats

pr = nx.pagerank(G, alpha=0.85)

# optional: teleport only to pages that earn external links or GSC clicks
# seeds = {url: weight for ...}
# pr = nx.pagerank(G, alpha=0.85, personalization=seeds)

The personalization vector is the upgrade worth making. Uniform teleport pretends every URL is equally likely to be an entry point. Seeding the vector with pages that have referring domains, or with Search Console click counts, gives you a graph where authority originates where it actually originates. The ranking shifts noticeably on most sites.

What to do with the output

• Join it to Search Console data, not to nothing. Pages in the top internal-PageRank decile with near-zero impressions are distribution you're spending on nothing. Pages with strong impressions in the bottom decile are the actionable list.
• Report percentiles and rank changes, never the raw score. The score depends on N. Add a thousand blog posts and every number drops; nothing was lost.
• Re-run with identical settings before and after a linking change and compare rank movement for a defined set of target URLs. That's a controlled measurement. Comparing to last quarter's crawl with different exclusion rules is not.
• Check the two-graph delta (all links vs. content-only) for your priority pages. If they only rank well with boilerplate included, the linking is structural, not editorial.
• Never present it as a Google metric. It's a model of your own site's topology. That's genuinely worth knowing, and it's all it is.