How can you check actual crawling activity and page crawlability?

ProcrustesPR

Check two things separately: whether crawlers have actually visited, and whether pages can be accessed and read. To check actual crawling activity, use Google Search Console’s Crawl Stats for Google and analyze server logs for other crawlers. To check whether pages can be crawled, review access restrictions, robots.txt, server responses, and page content. A successful tool test does not mean a search engine has already crawled the page, and a single rejected test is not enough to prove that all crawlers are blocked.

ProcrustesPRProcrustesPRDo we still need SEO now that AI answers questions?

Actual visits: check crawl statistics, then compare logs

To find out whether Google has crawled the site recently, start with Search Console’s Crawl Stats. Semrush’s 2023 overview lists crawl requests, download volume, average response time, and host status among the information available. These details can help you spot changes in activity and decide what to investigate. However, the report does not cover activity from all search engines or AI crawlers.

  • Has activity changed? Look at how requests changed over the period covered by the report, rather than looking only at a single total.
  • Are there connection problems? Review information about robots.txt fetching, DNS, and server connectivity.
  • Which requests warrant investigation? Review requests by categories such as response code, file type, and request purpose.

To examine other crawlers’ activity, you will need server logs. However, a user-agent name in a log is only the identity claimed by the request; the name alone does not verify that the request came from an official crawler. A Search Engine Journal author’s 2026 observations of a single site also highlight a limitation: logs cannot reveal exactly what users asked, so they should not be used to infer search intent.

When you encounter a 403: distinguish test tools from actual crawlers

An iThome article describes how the author’s team mistakenly concluded that a site was blocking AI crawlers after their own test probe received a 403 response. When they reviewed actual event records, they found continued crawler activity. In another case, the team found a marker from their own Worker in the response headers, overturning their conclusion that the tracking code had not run.

These cases offer a reminder: a restriction affecting a probe does not mean that actual crawlers face the same restriction. When a test fails, cross-check the response headers and actual event records. Still, do not assume that all 403 responses are harmless; you need to identify which request was rejected and at what stage.

Page crawlability: check restrictions and content

First, determine whether the target page is restricted by robots.txt, a login requirement, HTTP authentication, or an IP allowlist. Also distinguish crawling from indexing: Tenten’s 2023 article notes that even if robots.txt prevents Google from accessing a URL, the URL may still be indexed through links elsewhere. Blocking crawling therefore does not guarantee removal from search results.

If important content is generated by JavaScript, also compare the original HTML, the HTML after JavaScript execution, and the content visible on screen. The method Screaming Frog described in 2021 checks whether important text and links appear in the rendered DOM. This can help pinpoint problems with reading page content, but it cannot replace records of actual search engine visits.

ProcrustesPRProcrustesPRWhat is AI visibility?

Crawling activity and page crawlability: perspectives from across the web

How can you check for inappropriate nofollow and noindex settings?

Use a site audit tool to list internal links with nofollow and pages set to noindex, then check each setting against its intended purpose. Inappropriate use of nofollow may result in search engines not crawling the linked destination. If a page is intended to rank in search results, check whether noindex was applied by mistake. Using noindex on paid advertising landing pages or thank-you pages may be intentional, so it should not always be removed.

How should you check large numbers of parameterized URLs or duplicate content?

First, inventory the URL variants generated by parameters and assess whether there is search demand for their content before deciding which are worth crawling. Large numbers of useless variants may consume crawl resources. Audit tools can identify duplicate pages: if they were created in error, they can be redirected to the main URL; if they need to be retained, a canonical tag can indicate the main page you want indexed. They should not all be deleted indiscriminately.

What troubleshooting clues can site audit reports provide?

Regular site audits can help detect crawlability problems early. In Site Audit reports, for example, you can review the pages the tool crawled and issues such as redirects, duplicate content, or slow loading, then open the affected pages to investigate further. Enabling JavaScript rendering may improve accuracy, but it may also lengthen the audit. Choose based on the site’s content and your checking needs.

How can you distinguish different purposes when analyzing AI fetching activity?

First, distinguish real-time assistant fetches triggered by users from scheduled background crawler activity. Then consider that background crawlers may serve search retrieval or model training. Retrieval and training differ in purpose and in the outcomes you can observe: retrieved content may be used in near-term answers, while training content may be used in future models. The effects of training cannot be seen directly in ordinary dashboards. Real-time fetch records also cannot reveal exactly what a user asked.

ProcrustesPRHow does data quality affect a website’s readiness for the agentic web?

References

  1. robots.txt 明明放行了,Cloudflare 卻冷冷回個 403?揪出 WAF 與 Bot Fight Mode 的隱形暗礁ithelp.ithome.com.tw — Source used in the article above.
  2. 技術 SEO 初學者指南 (by Ahrefs)tenten.co · 2023-07-30T15:00:24.000Z — Source used in the article above.
  3. 81.8% Of My ‘AI Assistant’ Traffic Was Fake. The Googlebot Number Was Worsewww.searchenginejournal.com · 2026-06-25T13:30:24+00:00 — Discusses: How can you distinguish different purposes when analyzing AI fetching activity?
  4. Conducting an SEO Content Audit: 6 Things You Might Be Overlookingwww.screamingfrog.co.uk · 2021-11-15T13:37:48+00:00 — Source used in the article above.
  5. 15 Crawlability Problems & How to Fix Themwww.semrush.com · 2016-09-15T13:09:15Z — Discusses: How can you check for inappropriate nofollow and noindex settings?; How should you check large numbers of parameterized URLs or duplicate content?
  6. Crawl Budget: What Is It and Why Is It Important for SEO?www.semrush.com · 2023-09-25T10:14:00Z — Discusses: What troubleshooting clues can site audit reports provide?

Related articles

Do we still need SEO now that AI answers questions?What is AI visibility?How does data quality affect a website’s readiness for the agentic web?How will AI change SEO work and search traffic?What Is Search Everywhere Optimization, and Why Do You Need It?Why do publishers need SEO, and what challenges does AI bring?Which SEO Best Practices Matter Most? Applying Them to AI Search and E-commerceWhat Role Does Organic Search Play in B2B and AI Search?
ProcrustesPR
ProcrustesPR