LEYUN · SITE SPEC & COMPONENT LIBRARY独立站技术规范手册

Server Log Analysis for Independent Sites: Quickly Locating Why Crawlers Stay Away

Section:CRAWL & INDEX · Reviewed:2026-11-20

Key takeawaySearch engine dashboards tell you the outcome; server logs tell you the process. When coverage reports look abnormal, only by slicing access logs along four dimensions - user agent, path, status code, and response time - can you tell whether the crawler never came, was refused, or came but could not fetch effectively.

When this applies

  • Sites with unexplained anomalies in the GSC Pages report
  • Sites that need to determine whether a class of pages is being systematically blocked
  • Sites whose CDN, WAF, and server logs are scattered across systems and need a unified conclusion
  • Sites preparing for a migration or major redesign that need baseline data first
Four slicing dimensions for log analysis
DimensionMetric to observeAbnormal signalTypical conclusion
User agent distributionGooglebot share of requestsShare drops sharply or hits zeroCrawler blocked or rate-reduced
Status code distributionShare of 404/403/429/5xx403 or 429 notably highWAF or rate limiting hitting the crawler by mistake
Response timeTTFB distribution of Googlebot requestsP95 above 2 secondsSlow responses causing rate reduction
Path distributionConcentration of visited URLsTop 10 paths account for over 70% of requestsBudget consumed by repeated crawling
Crawl intervalTime between consecutive visits to the same URLGrows from minutes to hoursCrawler actively reducing its rate
Entry pathsPaths the crawler takes to reach inner pagesInner pages discoverable only via the sitemapInternal link structure is broken

Step-by-step diagnosis

  1. Confirm log completeness: the site must keep raw access logs including user agent and response time, not just aggregated reports.
  2. Filter logs by user agent for Googlebot and each AI crawler, and count their daily requests and distinct URLs reached.
  3. Plot both numbers as time series and look for inflection points - they usually correspond to a deployment or configuration change.
  4. Tally the status code distribution of Googlebot requests; investigate WAF and rate-limiting rules first when 403/429 dominates.
  5. Extract the TTFB distribution of Googlebot requests and compare P50 against P95 to locate slow page types.
  6. Measure the concentration of visited URLs to identify path patterns being crawled repeatedly.
  7. Turn conclusions into concrete rules: allow crawler IPs, correct robots.txt, converge the URL set, or improve response speed.

Common mistakes

  • Reading only GSC and not the logs - dashboard data is the outcome, logs are the process; the troubleshooting order cannot be reversed.
  • Skipping user-agent filtering - ordinary visitor traffic drowns out crawler patterns.
  • Ignoring response time - slowness is the most common and most hidden cause of crawl rate reduction.
  • Substituting analytics tools for raw logs - aggregated data hides status codes and user agents.
  • Looking only at the origin server and not the CDN path - sometimes the crawler never reaches the origin at all.

FAQ

How long should logs be retained?
At least 30 days, ideally 90. Indexing problems often need to be traced back to when a change occurred, and that change may have happened weeks earlier.
Which log should be analyzed behind a CDN?
CDN edge logs are the most truthful. Origin logs miss requests answered directly by the CDN and therefore underestimate actual crawling.
Are AI crawlers worth analyzing separately?
Yes. GPTBot, ClaudeBot, and PerplexityBot crawl in patterns clearly different from Googlebot, and their tolerance for client-side rendering is lower, so they hit shell pages more often.
Version notes2026-11-20: first release, compiled from official Google documentation and public technical sources; updated as engine rules change. Engine behaviour details follow the latest official Google documentation.

Note: this page is a summary of publicly documented search-engine rules and is for self-check reference only. Engine behaviour is subject to the latest official Google documentation; verify against your own Search Console data.