Blocking Pagination Pages That Drain Crawl Budget
Key takeawayListing pages that paginate through pages 2, 3, and 4 generate large numbers of pages highly similar to the first screen. These pages bring almost no new search value while consuming real crawl budget and creating duplicate entries in the index.
When this applies
- E-commerce or catalog sites with thousands of SKUs
- Sites where listing pages combine sorting, filtering, and pagination
- Sites whose index contains many "page N" URLs with no traffic
- Sites where legacy paginated URLs from before an infinite-scroll migration are still being crawled
| Handling | Effect on crawl budget | Effect on indexing | Best fit |
|---|---|---|---|
| Keep pagination + self-referencing canonical | Moderate consumption | Indexed normally but with low standing | Each page has distinct content and ordering |
| Block pagination in robots.txt | None | Indexed pages gradually removed | Paginated pages have no independent value |
| Infinite scroll + crawlable links retained | Low consumption | Depends on script injection; handle carefully | When the front-end rebuild is feasible |
| Server-side pagination + rel=next/prev | Moderate consumption | The old tags are no longer used | Sites with a large legacy of paginated URLs |
| Keep only a limited number of pages | Controlled consumption | Deep pages excluded but bounded | A compromise option |
Step-by-step diagnosis
- Count URLs containing the page parameter in the current index and their cumulative crawl count.
- Sample these paginated pages: are they duplicative of the first screen, and does the sort order create substantive differences?
- For paginations with no independent value: block them in robots.txt first, then submit removal requests in GSC.
- For paginations with genuine independent value: keep them indexable but with a self-referencing canonical, never pointing at the first page.
- Check whether "load more" buttons link to ?page=N anywhere on the site - these are the main entry points for budget consumption.
- Evaluate a move to infinite scroll: ensure paginated content remains reachable by crawlers, as pure front-end injection creates new crawl problems.
- Set a monthly cap on acceptable pagination depth (e.g., no deeper than page 3) and uniformly block deeper pages.
Common mistakes
- Canonicalizing all paginated pages to page one - deep pages differ in content due to sort order; forced convergence loses valid pages.
- Changing robots.txt without changing site links - recommendation slots still generate ?page= links, so the budget drain continues.
- Switching to infinite scroll with no remediation - scroll-loaded content is injected by JavaScript and crawlers may never reach the second screen.
- Skipping cleanup of already-indexed paginated pages - robots enforcement lags, so stale entries persist in the index.
- Making "view more" links nofollow - this does not affect crawl budget; it only stops the link signal from being passed.
FAQ
Can paginated pages be indexed at all?
Yes, provided each carries independent content and a self-referencing canonical. The problem is that on most sites paginated content is repetitive, so the benefit rarely justifies the cost.
Is infinite scroll the better option?
It is friendlier to crawl budget but introduces new crawlability problems. It only beats pagination when paginated content is directly reachable by the crawler.
How long until the index is clean after blocking pagination?
Usually 4-8 weeks. Already-indexed paginated pages must be re-crawled before the robots directive takes effect, and old entries remain visible meanwhile.
Version notes2026-11-20: first release, compiled from official Google documentation and public technical sources; updated as engine rules change. Engine behaviour details follow the latest official Google documentation.
Related reading
Why a Site Burns Through Crawl Budget: Causes and an Optimization ChecklistComplete Troubleshooting Guide for "Crawled - Currently Not Indexed" in Google Search Console"Discovered - Currently Not Indexed" in GSC: Causes and FixesFixing Blank Pages Served to AI Crawlers Caused by JavaScript RenderingServer Log Analysis for Independent Sites: Quickly Locating Why Crawlers Stay AwayNote: this page is a summary of publicly documented search-engine rules and is for self-check reference only. Engine behaviour is subject to the latest official Google documentation; verify against your own Search Console data.