Count your URL families before you argue about design

Most redesigns are argued on the wrong evidence. Teams spend weeks on the hero, the type scale and the mobile navigation, ship a new site, and twelve months later the traffic chart looks exactly like the old one. The design changed. The inventory did not.

The measurement that predicts whether a rebuild pays is unglamorous: what share of your submitted URLs belong to a family that can earn a click. Everything else in a redesign is downstream of that number, including how many templates you need to design.

A URL family is a parent path, not a page

Group every URL in your sitemap by its parent path and the site stops looking like 5,000 pages and starts looking like 20 templates. /products/linen-shirt, /products/wool-coat and /products/leather-belt all live in /products/. That is one family, three members, one template.

I ran that census on five public sitemap indexes on 1 November 2026 — two apparel retailers, a footwear brand, a design publication and an engineering blog — and sampled 34,750 URLs across them. Sites are unnamed on purpose; the shape is the finding, and naming them would turn a structural point into a hit piece on somebody's catalogue.

80.4% of those URLs sat in families with 100 or more members. Only 3.3% sat in families of ten or fewer. Four of the five sites were between 89% and 96%. The outlier was the publication, at 34.6%.

Treat the decimals as approximate. The classification is a heuristic: a family is all URLs sharing a parent path, I sampled child sitemaps at spread intervals rather than taking the first ones, and a retailer with a large catalogue will always look more generated than a magazine. The direction is what matters, and it is not close.

The largest family is usually not the one that sells

Look at the publication row again. Its biggest family by far is /author/ — 21% of everything it submits — followed by /category/ at 5.8%. Those pages exist because the CMS generates them. They are not what the publication sells, and they are not what a reader arrives for. One in five submitted URLs is a page nobody chose to publish.

The commerce rows have a quieter version of the same problem: duplication by prefix. One retailer submitted nearly identical product counts under three regional path prefixes. The template was counted three times, the catalogue once. Another split /products/ and a locale-prefixed /products/ at almost the same count. That is not a content problem, and it is not a design problem. It is a URL policy problem, and a rebuild that ports the routing logic forward reproduces it exactly.

This is stable across categories, which is why I stopped treating it as an SEO curiosity. The sitemap is the honest description of what a site is made of. Most teams have never read it.

The one-hour census

You do not need a tool for this. You need an hour and the willingness to be depressed.

1. Pull the sitemap index, not sitemap.xml alone. Take the child sitemaps at spread intervals if the index is long, so you are not sampling only the product feed.
2. Strip each URL to its path and group by parent path. A spreadsheet SPLIT or a 20-line script both work. You are counting members per family, nothing else.
3. Open the Pages report in Search Console and apply the URL prefix filter once per family. Write down clicks, impressions and average position for each.
4. Sort by clicks, not by size. The big family and the earning family are rarely the same one.
5. Classify each family as commercial, support, duplicate or generated-coverage, then total the commercial share of the whole submitted surface.

That last step is the entire point. If 8% of your sitemap is pages that can end in a purchase or a lead, your problem is not a slow LCP and it is not a tired hero image. It is that 92% of what you submit dilutes the 8% you care about.

The verdict is per family, not per page

Once families are classified, the decision stops being taste. Each family gets one of four verdicts, and each verdict has a test that a non-technical owner can settle in a meeting.

The fourth row is where most cleanup projects go wrong, so it is worth being precise.

"Just noindex the junk" does not do what people think

Google's own crawl budget documentation is unusually direct about this. It warns against noindex for exactly this use case, because the crawler still has to fetch the page to see the tag: "Don't use noindex, as Google will still request, but then drop the page when it sees a noindex meta tag or header in the HTTP response, wasting crawling time."

It is equally direct that blocking does not create a gift for your commercial pages: "Don't use robots.txt to temporarily reallocate crawl budget for other pages... Google won't shift this newly available crawl budget to other pages unless Google is already hitting your site's crawl capacity limit."

And on permanent removal, a 410 or 404 is the strong signal, while blocked URLs stay in the crawl queue longer and get recrawled when the block lifts.

Read those three together and the conclusion is unavoidable: de-indexing generated pages is index hygiene, not a performance lever. It cleans what Search Console reports and it stops the crawler reprocessing pages nobody wants. It does not hand your service pages more crawl. If a vendor is selling you a de-index program as a ranking fix, ask which of those three sentences they think is wrong.

The honest uses of a 410 are narrower and better: kill families that duplicate a canonical, and kill families with no commercial owner. That pair is almost always a meaningful slice of a sitemap — enough to justify a sprint, rarely enough on its own to explain a traffic collapse.

What this changes in a rebuild

This is where the census stops being an SEO exercise and starts setting genuine scope, which is the part that belongs to whoever is building the site. A rebuild's template inventory should equal the number of families that survived the census — not the number of page types the old site happened to have.

Three rules that follow, and that I apply in client work whenever a rebuild is on the table:

• No bespoke template without an owner and a conversion path. If nobody can say what the family is for and what a visitor should do there, it gets the shared template or it gets a 410.
• Every generated family needs a link path to a commercial one. Templates decide whether a page can rank; internal links decide whether it ever will. A catalogue page that links only to sibling catalogue pages is a cul-de-sac with a thousand doors.
• Design effort follows commercial share. If a quarter of the submitted surface earns the clicks, the design review belongs on that quarter. The rest is a template decision, not a design decision.

And the uncomfortable one: if the new site regenerates the same family map, with the same ratio of earning to non-earning URLs, the rebuild is a reskin with better typography. That is the outcome most redesigns actually deliver, and the census is the cheapest way to find out before the contract is signed.

A rough reading of the commercial share, from the shape of the five sitemaps above:

Those bands are judgment, not measurement. The census is the measurement, and it takes an hour. A rebuild takes a quarter. Run the cheap one first — and if you want the rebuild scoped against what the sitemap actually contains rather than what the homepage looks like, that is web design and development work, not a mood board. If you would rather have someone run the census on your sitemap first, tell us what you are planning to rebuild and we will start from the inventory, not from the hero.

Two related pieces if you want to go deeper: the mechanics of getting pages out of the index without breaking anything are covered in what noindex actually does to shareable pages, and if the census convinces you the stack itself is the constraint, composable architecture is the honest comparison. Performance still matters — it just matters after you know which pages deserve to be fast, which is a different conversation from Core Web Vitals.