How can you test whether a crawler follows the rules? Check the logs, then compare allowed and disallowed pages

ProcrustesPR

Testing whether a crawler follows explicitly stated rules requires more than checking whether it fetched disallowed pages. This article focuses on robots.txt: first check whether the crawler read the rules, then have the same system attempt to retrieve allowed and disallowed pages under comparable conditions. Check the results against server logs and unique page content. Successfully retrieving allowed pages while not retrieving disallowed pages provides evidence of compliance; retrieving neither leaves the result inconclusive.

ProcrustesPRProcrustesPRHow should I divide my DIY SEO budget?

First confirm that it read the rules—but do not confuse reading with compliance

Find requests for robots.txt in the access logs and compile their timestamps, IP addresses, and User-Agent strings to see which crawlers fetched the file and how often. This answers “Did it retrieve the rules?” but not “Did its subsequent behavior follow them?” You still need to check the request logs for the target pages.

After updating the rules, allow for caching and the time needed for changes to take effect. An article on crawler testing quotes a specification stating that crawlers should not use a cached version for more than 24 hours, with an exception when robots.txt is unreachable. The article therefore recommends waiting at least 24 hours after making changes before testing. This is a testing recommendation, not a guarantee that changes take effect within a day on every platform. The roughly 24-hour timeframe from OpenAI quoted in the article applies specifically to search results. The author also says they found no timeframe for changes to take effect in Anthropic’s documentation.

Use four test conditions to rule out retrieval failure

The July 2026 arXiv paper cited in the same article states that not retrieving disallowed pages is meaningful only when the same system can successfully retrieve allowed pages under comparable conditions. You can set up four test conditions:

  • Allow all: allow all crawlers to fetch the target page.
  • Disallow all: disallow all crawlers from fetching the target page.
  • Allow only the tested crawler: allow only the crawler being tested to fetch the target page.
  • Disallow only the tested crawler: disallow only the crawler being tested from fetching the target page.

A compliant crawler would be expected to retrieve the page in the first and third conditions, but not in the second and fourth. Here, “disallow” refers to robots.txt rules, not an actual server-side connection block. If even an allowed page is not retrieved, the cause could be WAF blocking, a failed retrieval attempt, or a lack of intent to crawl it. In that case, not retrieving the disallowed page either does not establish compliance.

Verify using unique content, not just plausible-looking answers

Place a different random test string on each target page, then check the content retrieved by the tested system against the logs. This helps determine whether the content came from the target page, preexisting knowledge, or secondhand accounts. The 2025 Cloudflare test described in the article used brand-new, unindexed domains with no publicly discoverable entry points to reduce confusion with existing content sources. However, a system’s ability to reproduce the content alone does not justify concluding that it gained access illegally.

Finally, robots.txt does not forcibly block access. A 2021 article notes that noncompliant bots may ignore prohibitions, while restricting search crawlers may also affect indexing and visibility. Test results therefore apply only to the system tested, the version of the rules, and the test conditions. They are not a guarantee that the crawler will always comply.

ProcrustesPRProcrustesPRWhy can I not find my published article on Google?

Testing crawler compliance: perspectives from across the web

How can you verify a crawler’s identity in the logs?

First check whether the crawler being tested provides an official IP list or verification method, then compare that information with the logs. An article compiling crawler information reports that Applebot provides an IP list and a reverse-lookup verification method, and CCBot also publishes its IP addresses. However, as of September 6, 2026, the author had not found official IP lists or webmaster guidance from xAI, ByteDance, or DeepSeek. Missing information limits identity verification, but does not by itself establish that a crawler has violated the rules.

Can the User-Agent identify requests that fetch robots.txt?

Sometimes, but pay attention to which crawler is involved and the period covered by the records. A crawler registry records that, starting August 3, 2026, GPTBot and OAI-SearchBot include a robots.txt marker in their User-Agent when fetching robots.txt. This can help identify such requests, but it is behavior recorded by the author—not an indication that all crawlers use the same marker. The marker alone does not establish that subsequent crawling follows the rules.

ProcrustesPREight SEO essentials for beginners: Website speed

References

  1. AI 爬蟲圖鑑:GPTBot、ClaudeBot、PerplexityBot 等等,認清那些半夜敲門的陌生人ithelp.ithome.com.tw — Discusses: How can you verify a crawler’s identity in the logs?
  2. What Is a Website Crawler? How It Works + Tools to Trywww.semrush.com · 2021-05-19T07:18:00Z — Source used in the article above.
  3. robots.txt 的 AI 條目:你立下的規矩,到底有誰在乎?教你親自驗證他們的忠誠度ithelp.ithome.com.tw — Discusses: Can the User-Agent identify requests that fetch robots.txt?

Related articles

How should I divide my DIY SEO budget?Why can I not find my published article on Google?Eight SEO essentials for beginners: Website speedEight SEO essentials for beginners: Getting your pages indexedEight SEO essentials for beginners: Page titles and descriptionsThe Five Most Common Mistakes SEO Beginners MakeHow to Set Up Heading Tags for SEO: Define Topics and Hierarchy Before Adding KeywordsWhat are the benefits and development challenges of SPAs?
ProcrustesPR
ProcrustesPR