How to block or restrict AI crawlers: identify their purpose, then choose the level of control

ProcrustesPR

To restrict AI crawlers that follow the rules, you can use robots.txt. To reject requests from crawlers that ignore those rules, or requests to fetch content on a user’s behalf, you need controls at the server or web application firewall (WAF) level. Blocking everything is not the only option: you can also limit crawl frequency, make only some content available, or require approval for access. The key is to establish which crawlers can read which content, and how to verify their identity, rather than simply turning on a “block AI” switch.

ProcrustesPRProcrustesPRDo we still need SEO now that AI answers questions?

Identify each crawler’s purpose to avoid blocking access you want to keep

In a 2026 update, Seer Interactive noted that a single provider may operate different bots for model training, search indexing, and fetching content at a user’s request. Keeping content out of training and keeping it out of AI search are therefore two goals that need to be addressed separately.

Different types of content also call for different access decisions. For example, you might consider allowing access to public articles and product descriptions, while assessing real-time prices, inventory, and account data separately. Seer recommends having platform or data agreements in place before granting access to sensitive data. These recommendations are a starting point to adapt to a website’s goals and risks, not official rules that every website must follow.

robots.txt communicates restrictions; a WAF blocks requests

These two approaches serve different purposes: configuring robots.txt does not mean you have enforced a block. A July 2026 overview from Search Engine Journal distinguished between training and search crawlers that follow the rules and content-fetching requests that need to be handled at the server or WAF level.

  • robots.txt: Sets access restrictions for crawlers that follow the file’s rules; it does not guarantee that all automated requests will stop.
  • WAF: Can use an allowlist to admit only specified crawlers, or a blocklist to reject unwanted crawlers.
  • Server rules: Determine whether to deny access based on characteristics such as the source of automated requests and request headers, with specific conditions tailored to the website’s architecture.

Teams managing enterprise websites can start by reviewing their existing WAF settings and blocklist management processes, then have the infrastructure team make adjustments. Changing SEO settings alone should not be taken as proof that restrictions are in effect.

Verify identity, and keep rate limits and partial access as options

A User-Agent tells you the identity a request claims, not whether that identity has been verified. Seer cautions that deciding whether to allow access based solely on crawler names may let impersonators bypass restrictions. Before enabling blocks, you should also add monitoring and SEO tools used by your organization and its partners to the allowlist to avoid blocking them inadvertently.

Allowing or denying access are not the only options. Search Engine Journal recommends a tiered approach based on value, cost, and risk:

  • If the value outweighs the operational burden, retain access.
  • If the value is unclear but costs are low, continue monitoring, apply rate limits, or restrict which areas can be accessed.
  • If access creates significant costs, involves sensitive content, and offers no demonstrated value, consider blocking it.

Review crawler logs before making changes. Afterward, keep watching for unusual traffic patterns and check whether legitimate tools are being blocked. A one-time setup cannot guarantee lasting effectiveness, because crawler identities, purposes, and website needs can all change.

ProcrustesPRProcrustesPRWhat is AI visibility?

Restricting AI crawlers: perspectives from across the web

Do Cloudflare's older AI crawler blocking settings need to be reviewed?

Yes, especially for websites that previously enabled the older “Block AI bots” setting. Reports from September 2026 indicated that, starting September 15, the setting for blocking training crawlers also covered Googlebot and Bingbot, which serve both search indexing and training purposes. If you want to retain search visibility, review your website's actual settings and consider adjustments; do not rely solely on descriptions of the toggle in older articles.

Can you simply block the IP addresses on officially published crawler lists?

Blocking the IP addresses on an official list does not guarantee that crawling will remain blocked over time. An iThome overview recommends using these lists to help identify sources, since the lists may change and may not distinguish between different crawlers from the same provider. Anthropic also cautions that blocking its bots' IP addresses may not remain effective at preventing crawling and may also stop crawlers from reading robots.txt.

Can a crawler name change affect existing blocking rules?

Yes. If rules have the old User-Agent value hard-coded, update them and check that they still track and block the crawler correctly. Search Engine Journal reported in July 2026 that the old value, Google-NotebookLM, would be supported only through August 2026. The documentation listed Google-GeminiNotebook as the new value. Firewall or server configurations using the old value should be adjusted accordingly.

Will blocking training crawlers now prevent existing content from being used for training?

That cannot be guaranteed. Blocking future crawling and addressing content already stored are separate issues. Reports in 2026 noted publishers' concerns that years of content might remain in archives and be available for training. Even if blocking is configured, you therefore cannot conclude that previously collected or archived content has been removed.

ProcrustesPRHow does data quality affect a website’s readiness for the agentic web?

References

  1. Google NotebookLM Rebrand May Expose Your Site To More AI Scrapingwww.searchenginejournal.com · 2026-07-18T10:09:29+00:00 — Discusses: Can a crawler name change affect existing blocking rules?
  2. Here’s Why You Should (Still) Allow GPTBot to Crawl Your Sitewww.seerinteractive.com · 2026-06-15T00:00:00-05:00 — Source used in the article above.
  3. Cloudflare 新制下既有 zone 的 AI 自查清單ithelp.ithome.com.tw — Discusses: Do Cloudflare's older AI crawler blocking settings need to be reviewed?
  4. UA 會騙人:用 IP 與 ASN 撕下 AI 爬蟲的偽裝,實測那些冒名頂替的騙子ithelp.ithome.com.tw — Discusses: Can you simply block the IP addresses on officially published crawler lists?
  5. Should I Block AI Crawlers Or Measure Their Value First? – Ask An SEOwww.searchenginejournal.com · 2026-07-02T09:30:54+00:00 — Source used in the article above.
  6. US Publishers Demand Common Crawl Stop Scraping Their Contentwww.searchenginejournal.com · 2026-06-10T00:45:20+00:00 — Discusses: Will blocking training crawlers now prevent existing content from being used for training?
  7. ChatGPT’s Index, GA Benchmarks, Google Refiles Complaint – SEO Pulsewww.searchenginejournal.com · 2026-08-15T12:00:14+00:00 — Discusses: Do Cloudflare's older AI crawler blocking settings need to be reviewed?

Related articles

Do we still need SEO now that AI answers questions?What is AI visibility?How does data quality affect a website’s readiness for the agentic web?How will AI change SEO work and search traffic?What Is Search Everywhere Optimization, and Why Do You Need It?Why do publishers need SEO, and what challenges does AI bring?Which SEO Best Practices Matter Most? Applying Them to AI Search and E-commerceWhat Role Does Organic Search Play in B2B and AI Search?
ProcrustesPR
ProcrustesPR