How can you check whether search engines and AI crawlers can access your website?

ProcrustesPR

Checking whether search engines and AI crawlers can access your website requires more than confirming that it loads normally: check robots.txt, login requirements, security settings, and how content is rendered, then separately check indexing restrictions such as noindex. For Google, use the URL Inspection tool to test rendering; for AI crawlers, check the rules and security service settings that apply to each crawler. Allowing crawling does not guarantee indexing or inclusion in AI answers.

ProcrustesPRProcrustesPRDo we still need SEO now that AI answers questions?

Test important pages first, not just whether the homepage loads

Google’s course structured data documentation recommends using the URL Inspection tool after deploying a few pages to test how Google renders them and check for barriers such as robots.txt, noindex, or login requirements. This is the deployment checklist in that documentation; passing these checks does not guarantee visibility in search.

  • Select important content pages you want people to find, and check the content Google actually renders.
  • Check for any noindex directives left in place by mistake, and make sure public content does not require a login.
  • After making corrections, you can request recrawling, but do not treat a submitted request as confirmation that indexing is complete; the documentation notes that Google may take a few days to find and crawl new pages.

Check robots.txt rules for each crawler and path

The key question is not whether robots.txt exists, but whether the crawlers you want to allow are prohibited from accessing important pages. A September 2026 interview roundup from Search Engine Journal also lists robots.txt as a basic control to check early on.

The same website can give different instructions to different crawlers. For example, according to the Amazon robots.txt file the publication retrieved on September 22, 2026, three explicitly named Meta crawlers were instructed not to access any pages. Two other Meta crawlers without explicitly named sections fell under the general rules, which allowed access to product pages but not cart, login, or account paths. This is only a snapshot from that date, not evidence of Amazon’s current settings or a basis for assuming that all AI crawlers are treated alike.

Check security services, but do not remove privacy protections for visibility

Beyond robots.txt, check the server, application, and security services. In the interview mentioned above, Darren Shaw described encountering a Shopify site where Cloudflare settings blocked AI crawlers. He therefore recommended first checking whether the site uses Cloudflare, then reviewing the relevant settings. This was an individual case, not an indication that all sites using Cloudflare block these crawlers.

Only fix unintended blocks on public content. Russ Jeffery noted that robots.txt provides advisory instructions for crawlers that follow the rules; genuinely private content still requires access restrictions at the server or application level.

Make sure crawlers receive the content, not just the page framework

After removing access barriers, check whether important text can be displayed without relying on client-side JavaScript. The interview roundup recommends prioritizing accessible server-side HTML and checking sitemaps and how content is rendered.

There is no need to treat creating a Markdown version as a prerequisite: in that interview, Jeffery said he had not yet found an AI search engine that relied on Markdown versions of pages. That was an observation at the time, not a permanent rule for every platform; the priority remains making important pages accessible and ensuring their content is readable.

ProcrustesPRProcrustesPRWhat is AI visibility?

Website access for search engines and AI crawlers: perspectives from across the web

What should you check first when important pages return errors?

First, check whether important pages return 404 or server errors, as these may prevent crawlers from retrieving the content. After fixing them, also check how the content is rendered: although Google can process JavaScript, heavy reliance on client-side rendering may still delay or limit crawling. Where feasible, include key content in the initial HTML.

Which pages should a sitemap include?

Prioritize important pages you want indexed, exclude URLs that are broken, redirect, or have a noindex directive, and make sure the sitemap reflects the current site structure. Google’s course structured data documentation also recommends submitting a sitemap to help Google keep track of future changes. However, submitting a sitemap does not replace checking page accessibility and indexing status.

Do frequent AI crawler visits mean a page is bringing in visitors?

Not necessarily. Frequent crawler visits do not mean a page is bringing in human visitors. Before deciding what to do next, check whether major AI crawlers can access the site, then compare which pages are crawled frequently with which pages bring in human referral traffic. Pages that are crawled heavily but bring in no human visitors may need a different approach from those that attract both crawlers and visitors from your target audience.

ProcrustesPRHow does data quality affect a website’s readiness for the agentic web?

References

  1. 使用課程清單的結構定義 | Google 搜尋中心  |  Documentation  |  Google for Developersdevelopers.google.com — Discusses: Which pages should a sitemap include?
  2. Amazon Blocked Meta’s Muse, And Robots.txt Had Nothing To Saywww.searchenginejournal.com · 2026-09-29T14:30:54+00:00 — Source used in the article above.
  3. How To Build Local Pages AI Systems Can Find And Trustwww.searchenginejournal.com · 2026-09-21T12:00:25+00:00 — Source used in the article above.
  4. How to Perform a Complete SEO Audit in 20 Stepswww.semrush.com · 2020-06-01T11:00:00Z — Discusses: What should you check first when important pages return errors?; Which pages should a sitemap include?
  5. New AI Search & SEO KPIs: 4 Signals That Guide Real Decisions [Watch Now]www.searchenginejournal.com · 2026-09-11T07:45:21+00:00 — Discusses: Do frequent AI crawler visits mean a page is bringing in visitors?

Related articles

Do we still need SEO now that AI answers questions?What is AI visibility?How does data quality affect a website’s readiness for the agentic web?How will AI change SEO work and search traffic?What Is Search Everywhere Optimization, and Why Do You Need It?Why do publishers need SEO, and what challenges does AI bring?Which SEO Best Practices Matter Most? Applying Them to AI Search and E-commerceWhat Role Does Organic Search Play in B2B and AI Search?
ProcrustesPR
ProcrustesPR