Verifying a crawler requires more than checking its User-Agent name. Follow the provider’s verification method to check against official IP lists or perform reverse and forward DNS lookups, then use the results to decide whether to allow access. A check that finds no match, a check that cannot be completed, and a confirmed identity must be treated differently. An unconfirmed identity does not establish impersonation. When enforcing policies, first check actual traffic and preserve access for essential tools, then make changes in stages and monitor for unintended blocking.
Read the logs first: a claimed identity is not proof
A User-Agent is a string supplied by the requester. It is useful for classification but does not prove identity. For example, even if a request claims to be GPTBot, you still need to check its source rather than assume it comes from OpenAI. In a 2026 article, Seer Interactive recommends reviewing crawler logs before changing policies to identify which crawlers actually visit and whether current rules match the access the site intends to allow.
What should you check in IP lists and DNS?
Official IP lists can be used to check request sources, but providers differ in the verification methods they offer and the level of detail in their lists. If using forward-confirmed reverse DNS (FCrDNS), follow these three steps:
- Perform a reverse PTR lookup on the source IP to obtain a hostname.
- Confirm that the hostname belongs to a verification domain publicly specified by the provider.
- Perform a forward lookup on that hostname and confirm that the results include the original IP.
Check the full domain name, not just whether it contains a familiar string: evil-googlebot.com is not googlebot.com, and x.googlebot.com.attacker.net does not belong to googlebot.com either. A similar-looking reverse DNS hostname is not enough to prove identity, and a provider’s PTR domains should not be inferred from its IP lists. An ASN can offer clues about network ownership, but it cannot independently prove which company a crawler belongs to.
Unconfirmed does not mean spoofed
A technical article published on iThome proposes three verification states to help avoid concluding that a crawler is impersonating another when the evidence is insufficient. This is the article author’s method, not an official rule shared by all platforms:
- Verified: The relevant checks were completed and the criteria were met.
- Unconfirmed: The checks were completed, but the results did not match the available data.
- Unknown: A list could not be retrieved, DNS timed out, or no verification method was available.
If an address is absent from a list, that only means the particular list does not include it; it does not establish impersonation. Keep IP, DNS, and ASN evidence separate so that inconsistent results can be investigated.
Use identity checks to inform access rules
After confirming the source, you still need to decide which content to make accessible. Seer recommends gradually opening access to marketing content, product descriptions, and reviews. Pricing, inventory, and account data should be assessed separately, with access restricted through explicit agreements rather than granted solely on the basis of a crawler’s name.
Before blocking unidentified bots, add the IP addresses of SEO and website monitoring tools that your organization and its partners rely on to an allowlist.
After making changes, keep checking for unexpected drops in traffic from particular sources and investigate whether legitimate tools have been inadvertently blocked. Also designate one person to coordinate decisions across IT, merchandising, and legal teams.
Crawler identity verification and access policies: perspectives from across the web
How should you troubleshoot crawler access when Search Console reports a 401?
First confirm whether the page is intended to require a login, then check authentication settings and whether the server’s IP blocking and User-Agent access rules match the site’s intentions. Manually test affected pages to locate the restriction. If Googlebot was intended to have access, you can request recrawling through Search Console after making corrections. A 401 may also result from deliberate protection, so it should not always be treated as an error to be resolved by granting access.
Should AMP and non-AMP pages use the same crawler policy?
Google’s guidance on paywalled content recommends applying the same bot access policy to AMP and non-AMP pages; otherwise, Search Console may report content mismatch errors. If you use an AMP paywall, also confirm that the authorization endpoint grants content access to the relevant bots from Google and other service providers in accordance with the publisher’s arrangements.
Can you block crawling simply by blocking addresses on official IP lists?
Official IP lists should not be treated as a reliable mechanism for opting out of crawling. A technical article on iThome notes that IP blocking may prevent crawlers from reading even robots.txt. Anthropic’s guidance also cautions that alternative methods, such as blocking its bots’ IP addresses, may not reliably ensure that an opt-out works correctly or remains effective. These lists are better suited to identifying sources; actual restrictions should still be informed by each provider’s policies and verification methods.
How soon after changing robots.txt can you check whether crawlers are complying?
Allow at least 24 hours before checking, but do not treat this as a guaranteed timeframe for changes to take effect across all providers. The cited article quotes OpenAI’s guidance that system adjustments for search results take about 24 hours. The article’s author found no timeframe for changes to take effect in the Anthropic documentation they reviewed. Check both whether the crawler avoids disallowed pages and whether the same system can successfully crawl allowed pages under similar conditions. This helps avoid mistaking a failed crawl for compliance.
Can managed bot verification address the limitations of identification by name alone?
Yes, it can provide additional evidence of identity, but a matching name does not confirm identity. Seer Interactive describes how managed bot verification services can combine verified bot lists with behavioral scoring to help identify impersonators. Web Bot Auth uses cryptographic signatures to prove identity. The article describes it as an early-stage approach, so you should not assume that all crawlers or platforms support it.
References
- 訂閱和付費牆內容標記 | Google 搜尋中心 | Documentation | Google for Developersdevelopers.google.com — Discusses: Should AMP and non-AMP pages use the same crawler policy?
- Google Search Console Errors: Identifying and Fixing Themwww.semrush.com · 2024-03-20T10:51:00Z — Discusses: How should you troubleshoot crawler access when Search Console reports a 401?
- Here’s Why You Should (Still) Allow GPTBot to Crawl Your Sitewww.seerinteractive.com · 2026-06-15T00:00:00-05:00 — Discusses: Can managed bot verification address the limitations of identification by name alone?
- UA 會騙人:用 IP 與 ASN 撕下 AI 爬蟲的偽裝,實測那些冒名頂替的騙子ithelp.ithome.com.tw — Discusses: Can you block crawling simply by blocking addresses on official IP lists?
- robots.txt 的 AI 條目:你立下的規矩,到底有誰在乎?教你親自驗證他們的忠誠度ithelp.ithome.com.tw — Discusses: How soon after changing robots.txt can you check whether crawlers are complying?
