Advanced robots.txt Rules: Blocking Parameters, Pagination, and Useless Directories
Key takeawayrobots.txt is a set of path rules grouped by user agent. Its expressive power exceeds what most people assume: it supports path wildcards, the $ end anchor, and priority rules between Allow and Disallow — enough to precisely control which paths are worth crawling.
When this applies
- Sites that need precise control over the crawl scope
- Sites where parameter pages and search result pages consume heavy crawl resources
- Sites with worthless directories (tests, backups, temporary files)
- Sites that need to narrow the crawl surface without blocking the whole domain
| Rule | Meaning | Caveat |
|---|---|---|
| Disallow: /search | Blocks crawling of the search results directory | Already-known URLs there do not disappear automatically |
| Disallow: /*?* | Blocks all URLs with query strings | Blocks pagination too |
| Disallow: /*?*&sort= | Blocks only URLs carrying the sort parameter | Needs to be paired with canonicals |
| Disallow: /tag/ | Blocks the tag directory | Do not write this if tag pages have value |
| Allow: /product/ | Allows a specific path under a Disallow | Allow wins over an equally long Disallow |
| Disallow: /*.pdf$ | Blocks crawling of PDF files | Mind the semantics of the $ end anchor |
| Disallow: /admin | Blocks the admin directory | The admin area should not be crawled anyway |
| User-agent: * / Allow: / | Default allow-all | Must come first, or be isolated in its own group |
Step-by-step diagnosis
- Back up the current robots.txt, then verify each rule's hit behavior in GSC's robots.txt tester.
- List the path categories on the site that produce no standalone content value (search, tests, backups, parameter combinations, tags).
- Add rules category by category and validate each one's actual effect in the tester as you go.
- Allow the sitemap's URL paths — never block the sitemap file itself.
- Also Disallow real duplicate files that exist on the server (.env, .git, backup archives).
- After changes, resubmit robots.txt in GSC and watch how the crawled URL distribution shifts over two weeks.
- Pair with noindex for already-indexed blocked pages — robots blocks crawling; it does not remove the index.
Common mistakes
- Using robots.txt instead of noindex to remove already-indexed URLs — the two mechanisms do entirely different things.
- Blocking the whole site or whole directories in robots.txt (e.g. Disallow: /) — recovery after a mistake is extremely slow.
- Writing Disallow: /*?* and forgetting the pagination pages you actually need indexed.
- Placing the sitemap under a Disallowed directory.
- Not knowing that Allow wins over an equally long Disallow, so rules behave opposite to expectations.
- Writing absolute URLs instead of domain-relative paths in rules.
FAQ
Do robots.txt-blocked pages still appear in search results?
Yes. robots.txt prevents crawling, not indexing. Already-indexed pages must be removed via noindex plus recrawling.
After a Disallow: / mistake, can the site recover?
Yes — fix the rules and request recrawling. But the index may have been progressively emptied in the meantime, and recovery takes time.
Can robots.txt use regular expressions?
No. It supports prefix matching plus two wildcards (* and $), not regex.
Version notes2026-11-20: first release, compiled from official Google documentation and public technical sources; updated as engine rules change. Engine behaviour details follow the latest official Google documentation.
Related reading
Shopify's Built-in SEO Defects and How to Work Around ThemHow WordPress Plugins End Up Blocking CrawlersDomain Migration with 301 Redirects: Preventing Signal LossIndexing Collapsed After a Redesign: How to RecoverMultiple Subdomains Under One Domain: The Dilution Risk ExplainedNote: this page is a summary of publicly documented search-engine rules and is for self-check reference only. Engine behaviour is subject to the latest official Google documentation; verify against your own Search Console data.