LEYUN · SITE SPEC & COMPONENT LIBRARY独立站技术规范手册

Blocking Pagination Pages That Drain Crawl Budget

Section:CRAWL & INDEX · Reviewed:2026-11-20

Key takeawayListing pages that paginate through pages 2, 3, and 4 generate large numbers of pages highly similar to the first screen. These pages bring almost no new search value while consuming real crawl budget and creating duplicate entries in the index.

When this applies

  • E-commerce or catalog sites with thousands of SKUs
  • Sites where listing pages combine sorting, filtering, and pagination
  • Sites whose index contains many "page N" URLs with no traffic
  • Sites where legacy paginated URLs from before an infinite-scroll migration are still being crawled
Pagination handling options compared
HandlingEffect on crawl budgetEffect on indexingBest fit
Keep pagination + self-referencing canonicalModerate consumptionIndexed normally but with low standingEach page has distinct content and ordering
Block pagination in robots.txtNoneIndexed pages gradually removedPaginated pages have no independent value
Infinite scroll + crawlable links retainedLow consumptionDepends on script injection; handle carefullyWhen the front-end rebuild is feasible
Server-side pagination + rel=next/prevModerate consumptionThe old tags are no longer usedSites with a large legacy of paginated URLs
Keep only a limited number of pagesControlled consumptionDeep pages excluded but boundedA compromise option

Step-by-step diagnosis

  1. Count URLs containing the page parameter in the current index and their cumulative crawl count.
  2. Sample these paginated pages: are they duplicative of the first screen, and does the sort order create substantive differences?
  3. For paginations with no independent value: block them in robots.txt first, then submit removal requests in GSC.
  4. For paginations with genuine independent value: keep them indexable but with a self-referencing canonical, never pointing at the first page.
  5. Check whether "load more" buttons link to ?page=N anywhere on the site - these are the main entry points for budget consumption.
  6. Evaluate a move to infinite scroll: ensure paginated content remains reachable by crawlers, as pure front-end injection creates new crawl problems.
  7. Set a monthly cap on acceptable pagination depth (e.g., no deeper than page 3) and uniformly block deeper pages.

Common mistakes

  • Canonicalizing all paginated pages to page one - deep pages differ in content due to sort order; forced convergence loses valid pages.
  • Changing robots.txt without changing site links - recommendation slots still generate ?page= links, so the budget drain continues.
  • Switching to infinite scroll with no remediation - scroll-loaded content is injected by JavaScript and crawlers may never reach the second screen.
  • Skipping cleanup of already-indexed paginated pages - robots enforcement lags, so stale entries persist in the index.
  • Making "view more" links nofollow - this does not affect crawl budget; it only stops the link signal from being passed.

FAQ

Can paginated pages be indexed at all?
Yes, provided each carries independent content and a self-referencing canonical. The problem is that on most sites paginated content is repetitive, so the benefit rarely justifies the cost.
Is infinite scroll the better option?
It is friendlier to crawl budget but introduces new crawlability problems. It only beats pagination when paginated content is directly reachable by the crawler.
How long until the index is clean after blocking pagination?
Usually 4-8 weeks. Already-indexed paginated pages must be re-crawled before the robots directive takes effect, and old entries remain visible meanwhile.
Version notes2026-11-20: first release, compiled from official Google documentation and public technical sources; updated as engine rules change. Engine behaviour details follow the latest official Google documentation.

Note: this page is a summary of publicly documented search-engine rules and is for self-check reference only. Engine behaviour is subject to the latest official Google documentation; verify against your own Search Console data.