What can different types of crawling activity tell website operators?

ProcrustesPR

Crawling activity can help website operators understand which crawlers are accessing their content and whether those visits serve search retrieval, model training, or user-triggered live fetching. These records can also offer clues to crawling efficiency problems. To interpret them, you first need to verify crawler identities and distinguish between types of activity. Being crawled does not mean content has been indexed, incorporated into a model, or generated traffic, and logs cannot directly reveal the intent behind users’ questions.

ProcrustesPRProcrustesPRDo we still need SEO now that AI answers questions?

Verify who is crawling before counting requests

The activity records discussed here include fetch requests in website logs and summaries of crawler activity compiled after identity verification. A request logged under the Googlebot name tells you only that it claims to come from Google. Until verified, it should not be counted as an actual Google crawl.

In a 2026 article, Duane Forrester used two weeks of records from a new website as an example: after verification, ClaudeBot accounted for 166 crawls and Googlebot for 107. This shows the mix of crawlers on that site at that time, but does not establish a market-wide trend or a ratio every website should aim for.

Retrieval, training, and live fetching answer different questions

  • Retrieval crawlers: Build indexes that help assistants find recent sources, affecting whether content is available for near-term retrieval. A crawl record does not mean the content has appeared in an assistant’s answer.
  • Training crawlers: Collect content that may be used in future models, with any effects taking longer to emerge. Seeing this activity does not mean the content has been incorporated into a model, nor can it be counted as referral traffic.
  • Live assistant fetching: Occurs during a real user’s conversation with an assistant. It differs from scheduled background crawling and should be counted separately.

For example, ChatGPT-User and GPTBot represent different types of activity, as do Claude-User and ClaudeBot. Once a live fetch is verified, you can tell that a page was accessed while a user was interacting with an assistant. However, logs cannot tell you whether the user searched for the website by name or asked about a broader topic that led the assistant to retrieve the page.

A gap between crawling and indexing can offer clues for investigation

Traditional search crawler activity can also help operators assess resource use. In a 2015 article, Fili Wiese suggested, based on his experience, that a ratio of roughly 10:1 between pages crawled daily and pages indexed by Google was a reasonable benchmark for large commercial websites. This was an experience-based judgment for a particular period and type of website, not Google’s current official standard.

The article argued that a less favorable ratio could indicate inefficient resource use. Underlying site structure or content issues could, in turn, affect organic search performance. However, the ratio is only a starting point for investigation; it cannot establish the cause on its own. The next step is to examine logs to identify which pages Googlebot is actually crawling and investigate further, rather than focusing solely on whether the total crawl count is high or low.

ProcrustesPRProcrustesPRWhat is AI visibility?

Different types of crawling activity: perspectives from across the web

Can crawler activity shares on other websites serve as a benchmark for your own?

They can help you spot crawlers you might otherwise overlook, but should not be treated directly as targets for individual websites. BrightEdge’s internal data showed that, during the tracking period from November 2025 to March 2026, Applebot accounted for about 30% of AI search crawling activity, while ByteSpider’s activity grew by 138%. These figures describe that dataset and period; they do not imply that every website experienced the same shares or growth rates.

Should website operators prioritize measuring retrieval crawling or training crawling?

Both are worth tracking separately, but the available material does not establish which should take priority for every website. Duane Forrester argues that training crawling may matter more than whether content can currently be found through retrieval, yet is measured less often. This is his judgment, not a proven priority that applies to all websites. The effects of using content for training take longer to emerge and are not directly visible in standard dashboards, so its value cannot be measured solely through short-term traffic.

Can user demand be estimated from assistant live-fetch user-agent names in logs alone?

Names alone should not be used to estimate demand, because requests may use a spoofed assistant user-agent name. Their identities need to be verified first. In Forrester’s single-site case study, only 6 of 33 requests using live-fetch user-agent names came from vendor-published IP addresses. This result applies only to the requests that could be checked in that instance; it cannot be used to estimate a general impersonation rate. Even after successful verification, the records still cannot reveal exactly what users asked.

ProcrustesPRHow does data quality affect a website’s readiness for the agentic web?

References

  1. 81.8% Of My ‘AI Assistant’ Traffic Was Fake. The Googlebot Number Was Worsewww.searchenginejournal.com · 2026-06-25T13:30:24+00:00 — Discusses: Should website operators prioritize measuring retrieval crawling or training crawling?; Can user demand be estimated from assistant live-fetch user-agent names in logs alone?
  2. Top 5 Google SEO Performance Indicatorswww.semrush.com · 2015-09-08T13:00:54Z — Source used in the article above.
  3. The CMO And CIO Friction Point: Navigating The AI Agent And AEO Ecosystemwww.searchenginejournal.com · 2026-06-09T08:00:22+00:00 — Discusses: Can crawler activity shares on other websites serve as a benchmark for your own?

Related articles

Do we still need SEO now that AI answers questions?What is AI visibility?How does data quality affect a website’s readiness for the agentic web?How will AI change SEO work and search traffic?What Is Search Everywhere Optimization, and Why Do You Need It?Why do publishers need SEO, and what challenges does AI bring?Which SEO Best Practices Matter Most? Applying Them to AI Search and E-commerceWhat Role Does Organic Search Play in B2B and AI Search?
ProcrustesPR
ProcrustesPR