LEYUN · SITE SPEC & COMPONENT LIBRARY独立站技术规范手册

Advanced robots.txt Rules: Blocking Parameters, Pagination, and Useless Directories

Section:PLATFORM & ARCHITECTURE · Reviewed:2026-11-20

Key takeawayrobots.txt is a set of path rules grouped by user agent. Its expressive power exceeds what most people assume: it supports path wildcards, the $ end anchor, and priority rules between Allow and Disallow — enough to precisely control which paths are worth crawling.

When this applies

  • Sites that need precise control over the crawl scope
  • Sites where parameter pages and search result pages consume heavy crawl resources
  • Sites with worthless directories (tests, backups, temporary files)
  • Sites that need to narrow the crawl surface without blocking the whole domain
Common robots.txt rules and their effects
RuleMeaningCaveat
Disallow: /searchBlocks crawling of the search results directoryAlready-known URLs there do not disappear automatically
Disallow: /*?*Blocks all URLs with query stringsBlocks pagination too
Disallow: /*?*&sort=Blocks only URLs carrying the sort parameterNeeds to be paired with canonicals
Disallow: /tag/Blocks the tag directoryDo not write this if tag pages have value
Allow: /product/Allows a specific path under a DisallowAllow wins over an equally long Disallow
Disallow: /*.pdf$Blocks crawling of PDF filesMind the semantics of the $ end anchor
Disallow: /adminBlocks the admin directoryThe admin area should not be crawled anyway
User-agent: * / Allow: /Default allow-allMust come first, or be isolated in its own group

Step-by-step diagnosis

  1. Back up the current robots.txt, then verify each rule's hit behavior in GSC's robots.txt tester.
  2. List the path categories on the site that produce no standalone content value (search, tests, backups, parameter combinations, tags).
  3. Add rules category by category and validate each one's actual effect in the tester as you go.
  4. Allow the sitemap's URL paths — never block the sitemap file itself.
  5. Also Disallow real duplicate files that exist on the server (.env, .git, backup archives).
  6. After changes, resubmit robots.txt in GSC and watch how the crawled URL distribution shifts over two weeks.
  7. Pair with noindex for already-indexed blocked pages — robots blocks crawling; it does not remove the index.

Common mistakes

  • Using robots.txt instead of noindex to remove already-indexed URLs — the two mechanisms do entirely different things.
  • Blocking the whole site or whole directories in robots.txt (e.g. Disallow: /) — recovery after a mistake is extremely slow.
  • Writing Disallow: /*?* and forgetting the pagination pages you actually need indexed.
  • Placing the sitemap under a Disallowed directory.
  • Not knowing that Allow wins over an equally long Disallow, so rules behave opposite to expectations.
  • Writing absolute URLs instead of domain-relative paths in rules.

FAQ

Do robots.txt-blocked pages still appear in search results?
Yes. robots.txt prevents crawling, not indexing. Already-indexed pages must be removed via noindex plus recrawling.
After a Disallow: / mistake, can the site recover?
Yes — fix the rules and request recrawling. But the index may have been progressively emptied in the meantime, and recovery takes time.
Can robots.txt use regular expressions?
No. It supports prefix matching plus two wildcards (* and $), not regex.
Version notes2026-11-20: first release, compiled from official Google documentation and public technical sources; updated as engine rules change. Engine behaviour details follow the latest official Google documentation.

Note: this page is a summary of publicly documented search-engine rules and is for self-check reference only. Engine behaviour is subject to the latest official Google documentation; verify against your own Search Console data.