Published on September 5, 2026
Programmatic sites generate a long tail whether you want one or not. Ours is a taxonomy: records get tagged into dimensions (categories, cities, certifications, tags), and every term in every dimension gets a page. Most of those pages are fine. Some of them carry one record, and a few carry zero, because a term survived an import that its last record did not.
A term page with nothing on it is a doorway page with extra steps. Google has spent this year being much less patient with that shape, and enforcement is domain level now, so the worst pages on a property are not quietly discounted, they drag the rest. So I shipped a rule: a term page below a threshold drops out of the sitemap and gets a noindex robots tag.
The rule worked. It also took 45 live pages with it.
Dimensions already had a field called minItemCount. It looked exactly like the threshold I needed, so isTermIndexable() read it.
It was not that threshold. minItemCount gates compound pages, the intersections like /categories/medical-spa/cities/austin. It answers "how many records must sit in the overlap before that combined page is worth existing", which is a different question from "is this single term page worth indexing". One field, two meanings, and the second meaning was invented by me three months after the first.
The part that made it expensive: schema inference seeds every dimension it proposes with minItemCount: 3. Nobody sets that by hand, it is just what comes back on a proposed schema. So on any tenant that had imported a dataset and taken the inferred taxonomy, and that is how you are meant to use the thing, every simple term page with one or two records was now below a bar it had never been measured against. On our own tenant that was 45 terms. Not the dead ones I was aiming at. Live pages, with content on them, gone from the sitemap on upgrade, silently.
Silently is the word that matters. Nothing failed. The build passed, the sitemap rendered, the pages still returned 200 to a human. The only visible symptom was a sitemap that got shorter, and a sitemap getting shorter is what I had just shipped a feature to do.
The correction was to make the infrastructure rule out less, not more. isTermIndexable() now takes one argument, the term's itemCount, and answers in two lines: a count of null or undefined returns true, and any real count returns itemCount > 0.
That is the whole thing. Core now rules out exactly one case: a term counted at zero. Zero is dead for every dataset anyone will ever load, so it is safe for a platform to decide.
Everything above zero is an editorial call and it does not belong in shared code. Whether a page with two records is thin depends on what the records are. Two records is thin for "restaurants in Austin" and completely fine for "vendors with SecNumCloud certification", where two might be the whole world and the page is the most useful thing on the site. A number picked in core cannot know which one you have. That decision stays per term, as an explicit visibility: 'noindex' somebody chose.
I think this generalises past our codebase. If you run programmatic pages, the pressure right now is all toward pruning, and the instinct is to write one threshold and apply it everywhere. The threshold is the easy half. Knowing which pages it will actually hit is the hard half, and a default someone else seeded is not a bar your content was ever judged against.
The null branch above is load-bearing and it is the part I would have got wrong twice.
The count comes from itemCountCached, which is maintained by the importer. It is absent on terms created before that field existed, and it goes stale when an admin assigns terms by hand in the panel. So "no count" does not mean "no records". It means nobody has counted recently.
If you read missing as zero, every term the importer has not touched drops out of your sitemap, and it drops out for a reason that has nothing to do with the page. That is a worse failure than the one I was fixing, because it scales with how long the site has been running and how much manual curation it has had. The tests say it in one line: absence of data must never deindex.
The general form: any rule that removes pages from the index needs to distinguish "measured and empty" from "not measured". Most caching layers do not make that distinction easy, because for every other purpose a missing count and a zero count are the same thing.
I do not have traffic data on this yet. The fix landed on 9 August, the pages are back in the sitemap, and I am not going to pretend a recovery curve I have not measured. Anyone showing you a before and after chart four weeks after a sitemap change is showing you seasonality.
What I can say is the narrower thing, which is that the system now fails in the direction I want. A page drops out of the index only when something counted it and found nothing, and a gap in the data leaves the page alone. That is the property I would check for in your own pruning logic before you check anything else.
The two commits are b325d76 (the version with the bug) and a59d13a (the correction). The second one is shorter than the first, which is usually how it goes.