Can probe test results represent what real crawlers encounter?

ProcrustesPR

Not directly. Probes show only how specific test requests are handled; even with the same User-Agent (UA), curl is not equivalent to a real crawler. The reference material recommends using probes to check for differences before and after configuration changes, then interpreting the results alongside real traffic, events, and stored records. A probe receiving a 403 is not enough to prove that a crawler was blocked or that a Worker did not run.

ProcrustesPRProcrustesPRDo we still need SEO now that AI answers questions?

Rerun the same UA matrix to compare results before and after changes

A UA matrix tests a set of different UAs. The reference material recommends saving the curl UA matrix output before making changes, then rerunning the same loop afterward, with only one variable changed at a time:

  • Save the results before the change as ua-matrix-before.txt.
  • After the change, use the same set of UAs and save the output to ua-matrix-after.txt.
  • Use diff to compare the two files and check whether responses to the test requests have changed.

An empty diff means only that this set of probes observed no change between the two runs. The change may not have affected the conditions being tested, so this result alone cannot establish whether the configuration is effective. The reference material does not provide the full UA list or request parameters, so this is a comparison procedure, not a fully reproducible test report.

Requests with the same UA may still be handled differently

The author reporting the client cases illustrates the distinction using anonymous curl requests and IP-verified crawlers. The same UA name does not mean the requests share the same origin or verified identity, so they may be handled differently. Probes alone therefore cannot establish what real crawlers encounter, but this does not mean all real crawlers will be allowed through.

The author reports that their team made two mistaken judgments in client cases. The first time, they claimed a site was blocking AI crawlers because a probe received a 403, only to later find actual traffic from ChatGPT-User, OAI-SearchBot, and others. The second time, they concluded that the tracking code had not run, but then found their own Worker’s header in the 403 response and more than 10,000 events for that site that day. These are individual cases reported by the author; they cannot be used to estimate a general success rate.

What evidence should you check beyond the status code?

  • Response headers: The Worker’s header in the reported case supports the conclusion that the code ran, but does not mean the entire request succeeded. The reference material states that cf-mitigated: challenge confirms a challenge signal; its absence does not rule out a challenge.
  • Security events: The reference material recommends checking the Service field in Security Events to help identify which security feature intervened, rather than looking only at the 403.
  • Actual records: Cross-check crawler events and stored records to see whether other evidence supports the probe alert.

The team reporting these cases therefore adopted an internal policy of suppressing probe alerts when recent traffic was still present. This is the team’s approach to avoiding false alarms based on a single signal, not an official platform rule. Traffic confirms only that some requests were recorded; it does not establish that all crawlers can access all paths without obstruction.

ProcrustesPRProcrustesPRWhy is AI not mentioning my brand?

Probe test results: perspectives from across the web

Does setting curl's UA to a crawler's name simulate a real crawler?

No. Setting the UA name alone does not simulate a real crawler. The reference material notes that anonymous curl requests and IP-verified real crawlers may be handled differently. Even tests using the same set of UAs show only what the test requests encountered. To confirm actual activity from a particular crawler, you also need to check events and traffic logs.

If a test response has no challenge header, can you rule out a challenge?

No. The reference material states that cf-mitigated: challenge in a response confirms a challenge signal, but its absence does not establish whether a challenge occurred. Check response headers and status codes together. To understand what real crawlers encountered, you still need to check actual events rather than assume a single probe result applies to them.

Does finding traffic from some crawlers resolve all concerns about blocking?

No. Traffic from some crawlers does not mean all crawlers or requests can get through. In a client case reported by the author's team, a probe received a 403, but checking events revealed traffic from several crawlers. That was enough to refute their judgment at the time that everything was blocked. Those records do not prove that other crawlers or paths were also unobstructed. You still need to check events and stored records for the specific crawlers or paths you are concerned about.

ProcrustesPRDoes an AI brand mention bring website visitors?

References

  1. Cloudflare 新制下既有 zone 的 AI 自查清單ithelp.ithome.com.tw — Discusses: Does setting curl's UA to a crawler's name simulate a real crawler?
  2. robots.txt 明明放行了,Cloudflare 卻冷冷回個 403?揪出 WAF 與 Bot Fight Mode 的隱形暗礁ithelp.ithome.com.tw — Discusses: Does setting curl's UA to a crawler's name simulate a real crawler?; If a test response has no challenge header, can you rule out a challenge?

Related articles

Do we still need SEO now that AI answers questions?Why is AI not mentioning my brand?Does an AI brand mention bring website visitors?What is AI visibility?Does every brand really need to rush into GEO in 2026?How do you get AI to mention your brand? The key factors in GEOHow does data quality affect a website’s readiness for the agentic web?What Should Your SEO Strategy Aim for in 2026? Define Business Outcomes First, Then Choose Where to Compete in Search
ProcrustesPR
ProcrustesPR