Server Log Analysis for Independent Sites: Quickly Locating Why Crawlers Stay Away
Key takeawaySearch engine dashboards tell you the outcome; server logs tell you the process. When coverage reports look abnormal, only by slicing access logs along four dimensions - user agent, path, status code, and response time - can you tell whether the crawler never came, was refused, or came but could not fetch effectively.
When this applies
- Sites with unexplained anomalies in the GSC Pages report
- Sites that need to determine whether a class of pages is being systematically blocked
- Sites whose CDN, WAF, and server logs are scattered across systems and need a unified conclusion
- Sites preparing for a migration or major redesign that need baseline data first
| Dimension | Metric to observe | Abnormal signal | Typical conclusion |
|---|---|---|---|
| User agent distribution | Googlebot share of requests | Share drops sharply or hits zero | Crawler blocked or rate-reduced |
| Status code distribution | Share of 404/403/429/5xx | 403 or 429 notably high | WAF or rate limiting hitting the crawler by mistake |
| Response time | TTFB distribution of Googlebot requests | P95 above 2 seconds | Slow responses causing rate reduction |
| Path distribution | Concentration of visited URLs | Top 10 paths account for over 70% of requests | Budget consumed by repeated crawling |
| Crawl interval | Time between consecutive visits to the same URL | Grows from minutes to hours | Crawler actively reducing its rate |
| Entry paths | Paths the crawler takes to reach inner pages | Inner pages discoverable only via the sitemap | Internal link structure is broken |
Step-by-step diagnosis
- Confirm log completeness: the site must keep raw access logs including user agent and response time, not just aggregated reports.
- Filter logs by user agent for Googlebot and each AI crawler, and count their daily requests and distinct URLs reached.
- Plot both numbers as time series and look for inflection points - they usually correspond to a deployment or configuration change.
- Tally the status code distribution of Googlebot requests; investigate WAF and rate-limiting rules first when 403/429 dominates.
- Extract the TTFB distribution of Googlebot requests and compare P50 against P95 to locate slow page types.
- Measure the concentration of visited URLs to identify path patterns being crawled repeatedly.
- Turn conclusions into concrete rules: allow crawler IPs, correct robots.txt, converge the URL set, or improve response speed.
Common mistakes
- Reading only GSC and not the logs - dashboard data is the outcome, logs are the process; the troubleshooting order cannot be reversed.
- Skipping user-agent filtering - ordinary visitor traffic drowns out crawler patterns.
- Ignoring response time - slowness is the most common and most hidden cause of crawl rate reduction.
- Substituting analytics tools for raw logs - aggregated data hides status codes and user agents.
- Looking only at the origin server and not the CDN path - sometimes the crawler never reaches the origin at all.
FAQ
How long should logs be retained?
At least 30 days, ideally 90. Indexing problems often need to be traced back to when a change occurred, and that change may have happened weeks earlier.
Which log should be analyzed behind a CDN?
CDN edge logs are the most truthful. Origin logs miss requests answered directly by the CDN and therefore underestimate actual crawling.
Are AI crawlers worth analyzing separately?
Yes. GPTBot, ClaudeBot, and PerplexityBot crawl in patterns clearly different from Googlebot, and their tolerance for client-side rendering is lower, so they hit shell pages more often.
Version notes2026-11-20: first release, compiled from official Google documentation and public technical sources; updated as engine rules change. Engine behaviour details follow the latest official Google documentation.
Related reading
Why a Site Burns Through Crawl Budget: Causes and an Optimization ChecklistComplete Troubleshooting Guide for "Crawled - Currently Not Indexed" in Google Search Console"Discovered - Currently Not Indexed" in GSC: Causes and FixesFixing Blank Pages Served to AI Crawlers Caused by JavaScript RenderingDynamic URL Parameters Flooding Google's Index with Low-Quality PagesNote: this page is a summary of publicly documented search-engine rules and is for self-check reference only. Engine behaviour is subject to the latest official Google documentation; verify against your own Search Console data.