What are common web crawlers used for, and how should websites set access rules?

ProcrustesPR

Websites should not allow or block all crawlers indiscriminately. Instead, first determine whether a crawler is used for search indexing, model training, or real-time fetching on a user’s behalf, then decide which content it may access and how to verify its identity. Different bots from the same company may have different purposes and controls, and not all fetchers follow robots.txt. The descriptions below draw on information compiled in 2026. For the iThome article, official documentation was reviewed on September 6; the rules may have changed since then.

ProcrustesPRProcrustesPRDo we still need SEO now that AI answers questions?

Classify by purpose first, not just by company name

Seer Interactive’s 2026 update notes that major AI providers now use multiple crawlers with different purposes. Start with these three broad purposes, then check the documentation for each crawler rather than judging by its name alone:

  • Model training: Web pages fetched by GPTBot may be used to improve future models; ClaudeBot is used in connection with model training.
  • Search and indexing: OAI-SearchBot supports visibility in ChatGPT search, while Claude-SearchBot helps improve search result quality. According to the documentation cited by iThome at the time, PerplexityBot is used for indexing, not for training foundation models.
  • User-triggered fetching: ChatGPT-User, Claude-User, and Perplexity-User fetch content in response to user questions or actions and should not be confused with automated training crawlers.

However, a name used in access settings does not necessarily represent a separate crawler. According to the documentation review discussed above, Google-Extended and Applebot-Extended are tokens that control how content is used, not independent fetchers. Google-Extended settings do not affect visibility in Google Search.

After identifying the name, verify the visitor’s identity

A User-Agent (UA) is a self-declared identifier in a request and can be spoofed. Seeing the name GPTBot is not enough to confirm that a request comes from OpenAI. Seer therefore recommends considering how to verify identity alongside deciding whom to allow and which content they may access.

The iThome author recommends matching distinctive portions of UA strings while enforcing word boundaries, applying more specific rules first, and accounting for case differences. For example, matching an overly broad term such as Google could group different bots together. These are the author’s identification recommendations, not a shared standard established by all platforms, and they cannot replace identity verification.

The verification information available also varies by crawler. The Applebot documentation reviewed in the article provides IP lists and reverse lookup methods. Before implementing verification, check which crawlers a list covers and whether it is up to date. Do not assume that every name compiled by the community has been officially confirmed.

Define access by bot and content type

Start by clarifying your requirements. If you want to retain visibility in ChatGPT search but do not want your content used to train OpenAI foundation models, the official guidance cited by iThome says you can allow OAI-SearchBot while blocking GPTBot. The two settings are independent.

Content should also be assessed separately. Seer recommends deciding access by page type, such as marketing copy, reviews, pricing, and inventory, rather than applying the same rule across the entire site. Finally, check for exceptions: according to iThome’s summary at the time, robots.txt may not apply to ChatGPT-User fetching, and Perplexity-User generally ignores it. Adding a disallow rule therefore does not necessarily mean all access has been blocked.

ProcrustesPRProcrustesPRWhat is AI visibility?

Web crawler access rules: perspectives from across the web

What should you check when a search crawler encounters a 401 error?

A 401 status means the request is unauthorized, which may be related to password protection, IP restrictions, or crawler-specific settings. If the page is not intended to be public, first confirm that its authentication settings serve that purpose. If you want search engines to access it, test the page and check the IP and user-agent rules. After removing any unintended restrictions, you can request recrawling in Google Search Console.

Does robots.txt apply to all Google fetchers?

There is no blanket answer. According to the iThome author's summary of documentation reviewed on September 6, 2026, Google distinguishes between common crawlers, special-case crawlers, and user-triggered fetchers. Google-Safety ignores robots.txt, and user-triggered fetches generally ignore it as well. Check the latest documentation for each fetcher before configuring access; do not assume that all products from the same company follow the same rules.

Can you rely solely on IP blocking to reduce crawling frequency?

You should not rely solely on IP blocking to reduce crawling frequency. According to the Anthropic documentation reviewed by iThome, the company supports the Crawl-delay directive and warns that IP blocking may not reliably identify and block crawlers or keep them blocked over time. This guidance applies to Anthropic; it does not mean that other providers support the same directive.

What limitations remain when using official IP lists to verify crawlers?

Lists may differ in which crawlers they cover and how up to date they are, so one list cannot be used to verify every crawler. When the iThome author checked on September 6, 2026, the dedicated lists reviewed for several AI providers contained only IPv4 addresses, while Google's common crawler list included IPv6. Websites supporting IPv6 would therefore have verification blind spots if they relied solely on those dedicated lists. This was a snapshot from that date; check the latest information before implementation.

ProcrustesPRHow does data quality affect a website’s readiness for the agentic web?

References

  1. Google Search Console Errors: Identifying and Fixing Themwww.semrush.com · 2024-03-20T10:51:00Z — Discusses: What should you check when a search crawler encounters a 401 error?
  2. AI 爬蟲圖鑑:GPTBot、ClaudeBot、PerplexityBot 等等,認清那些半夜敲門的陌生人ithelp.ithome.com.tw — Discusses: Does robots.txt apply to all Google fetchers?; Can you rely solely on IP blocking to reduce crawling frequency?
  3. Here’s Why You Should (Still) Allow GPTBot to Crawl Your Sitewww.seerinteractive.com · 2026-06-15T00:00:00-05:00 — Source used in the article above.

Related articles

Do we still need SEO now that AI answers questions?What is AI visibility?How does data quality affect a website’s readiness for the agentic web?How will AI change SEO work and search traffic?What Is Search Everywhere Optimization, and Why Do You Need It?Why do publishers need SEO, and what challenges does AI bring?Which SEO Best Practices Matter Most? Applying Them to AI Search and E-commerceWhat Role Does Organic Search Play in B2B and AI Search?
ProcrustesPR
ProcrustesPR