Websites should not allow or block all crawlers indiscriminately. Instead, first determine whether a crawler is used for search indexing, model training, or real-time fetching on a user’s behalf, then decide which content it may access and how to verify its identity. Different bots from the same company may have different purposes and controls, and not all fetchers follow robots.txt. The descriptions below draw on information compiled in 2026. For the iThome article, official documentation was reviewed on September 6; the rules may have changed since then.
Classify by purpose first, not just by company name
Seer Interactive’s 2026 update notes that major AI providers now use multiple crawlers with different purposes. Start with these three broad purposes, then check the documentation for each crawler rather than judging by its name alone:
- Model training: Web pages fetched by GPTBot may be used to improve future models; ClaudeBot is used in connection with model training.
- Search and indexing: OAI-SearchBot supports visibility in ChatGPT search, while Claude-SearchBot helps improve search result quality. According to the documentation cited by iThome at the time, PerplexityBot is used for indexing, not for training foundation models.
- User-triggered fetching: ChatGPT-User, Claude-User, and Perplexity-User fetch content in response to user questions or actions and should not be confused with automated training crawlers.
However, a name used in access settings does not necessarily represent a separate crawler. According to the documentation review discussed above, Google-Extended and Applebot-Extended are tokens that control how content is used, not independent fetchers. Google-Extended settings do not affect visibility in Google Search.
After identifying the name, verify the visitor’s identity
A User-Agent (UA) is a self-declared identifier in a request and can be spoofed. Seeing the name GPTBot is not enough to confirm that a request comes from OpenAI. Seer therefore recommends considering how to verify identity alongside deciding whom to allow and which content they may access.
The iThome author recommends matching distinctive portions of UA strings while enforcing word boundaries, applying more specific rules first, and accounting for case differences. For example, matching an overly broad term such as Google could group different bots together. These are the author’s identification recommendations, not a shared standard established by all platforms, and they cannot replace identity verification.
The verification information available also varies by crawler. The Applebot documentation reviewed in the article provides IP lists and reverse lookup methods. Before implementing verification, check which crawlers a list covers and whether it is up to date. Do not assume that every name compiled by the community has been officially confirmed.
Define access by bot and content type
Start by clarifying your requirements. If you want to retain visibility in ChatGPT search but do not want your content used to train OpenAI foundation models, the official guidance cited by iThome says you can allow OAI-SearchBot while blocking GPTBot. The two settings are independent.
Content should also be assessed separately. Seer recommends deciding access by page type, such as marketing copy, reviews, pricing, and inventory, rather than applying the same rule across the entire site. Finally, check for exceptions: according to iThome’s summary at the time, robots.txt may not apply to ChatGPT-User fetching, and Perplexity-User generally ignores it. Adding a disallow rule therefore does not necessarily mean all access has been blocked.
Web crawler access rules: perspectives from across the web
What should you check when a search crawler encounters a 401 error?
A 401 status means the request is unauthorized, which may be related to password protection, IP restrictions, or crawler-specific settings. If the page is not intended to be public, first confirm that its authentication settings serve that purpose. If you want search engines to access it, test the page and check the IP and user-agent rules. After removing any unintended restrictions, you can request recrawling in Google Search Console.
Does robots.txt apply to all Google fetchers?
There is no blanket answer. According to the iThome author's summary of documentation reviewed on September 6, 2026, Google distinguishes between common crawlers, special-case crawlers, and user-triggered fetchers. Google-Safety ignores robots.txt, and user-triggered fetches generally ignore it as well. Check the latest documentation for each fetcher before configuring access; do not assume that all products from the same company follow the same rules.
Can you rely solely on IP blocking to reduce crawling frequency?
You should not rely solely on IP blocking to reduce crawling frequency. According to the Anthropic documentation reviewed by iThome, the company supports the Crawl-delay directive and warns that IP blocking may not reliably identify and block crawlers or keep them blocked over time. This guidance applies to Anthropic; it does not mean that other providers support the same directive.
What limitations remain when using official IP lists to verify crawlers?
Lists may differ in which crawlers they cover and how up to date they are, so one list cannot be used to verify every crawler. When the iThome author checked on September 6, 2026, the dedicated lists reviewed for several AI providers contained only IPv4 addresses, while Google's common crawler list included IPv6. Websites supporting IPv6 would therefore have verification blind spots if they relied solely on those dedicated lists. This was a snapshot from that date; check the latest information before implementation.
References
- Google Search Console Errors: Identifying and Fixing Themwww.semrush.com · 2024-03-20T10:51:00Z — Discusses: What should you check when a search crawler encounters a 401 error?
- AI 爬蟲圖鑑:GPTBot、ClaudeBot、PerplexityBot 等等,認清那些半夜敲門的陌生人ithelp.ithome.com.tw — Discusses: Does robots.txt apply to all Google fetchers?; Can you rely solely on IP blocking to reduce crawling frequency?
- Here’s Why You Should (Still) Allow GPTBot to Crawl Your Sitewww.seerinteractive.com · 2026-06-15T00:00:00-05:00 — Source used in the article above.
