To test and audit robots.txt, first confirm which file the site actually serves, then check whether each target crawler is allowed or blocked from accessing each test URL as intended. Use URL inspection and server logs to verify the results. Passing a rule test does not guarantee that a page will be indexed or that every crawler will comply. Google’s official documentation notes that URLs blocked by robots.txt may still appear in search results because of external links, so first clarify whether you want to restrict crawling or indexing.
Start by listing which pages should allow crawling and which should block it
Open the site’s /robots.txt in a browser to confirm what is publicly served, rather than relying only on the draft in your editor. Next, check the crawlers specified by User-agent and the paths specified by Allow and Disallow in each rule block. If different crawlers need different access restrictions, check the rule blocks that apply to each.
- Pages you want search engines to access: check whether any rules unintentionally block them.
- Directories you deliberately block from crawling: confirm that the scope of the block meets your needs, rather than merely checking that the file exists.
- Pages you want removed from search results: adding Disallow alone is not enough; if you use noindex, Googlebot must be able to read that directive.
For example, blocking a page from crawling while also adding noindex to it may prevent Googlebot from reading the noindex directive at all. This does not “double the restrictions”; blocking crawling may instead prevent noindex from taking effect.
Test representative URLs, not just syntax
Include both URLs you expect to allow and URLs you expect to block in your tests. A 2024 Backlinko article recommends using Screaming Frog to test robots.txt in a pre-launch test environment, with the user agent set to Googlebot to simulate crawling and identify access issues. However, these simulated results should not be treated as Googlebot’s actual behavior.
Tests should also cover the resources a page depends on. If CSS or images are blocked, page rendering may be affected, so access to the main URL alone is not enough to declare the test successful. When a page appears incorrectly in search results, Google’s official guidance recommends using the URL Inspection tool to check whether Googlebot can access it, then checking robots.txt and the page’s noindex directive.
When results differ, revisit the logs and test conditions
In a 2017 test, Screaming Frog found that Search Console’s robots.txt testing tool at the time did not match rules using percent encoding. However, in tests involving actual Googlebot crawling, requests were logged only for unblocked pages over a one-week period. This was an observation from a specific period, not evidence that current tools still have the same issue. The case illustrates why tool results should be cross-checked against actual request logs.
During an audit, you can record the test URLs, expected results, and observation period, then check server logs for requests to disallowed paths. Especially when dealing with AI crawlers, remember that robots.txt only tells crawlers which paths they may or may not access. Crawlers can ignore those rules; the file is not an enforceable safeguard. When sensitive content needs protection, a test showing that access is blocked is not enough to consider security verification complete.
Testing and auditing robots.txt: perspectives from across the web
After uploading robots.txt, how can you confirm that a search engine has retrieved the file?
You can check the robots.txt report in Search Console to confirm the fetch status and look for errors. If the fetch was unsuccessful, investigate and fix the cause. Continue checking regularly for formatting issues and invalid rules, because even small errors can affect the site's indexability. A successful fetch does not mean that every URL behaves as intended; the rules still need to be tested separately.
What should you do if a URL submitted in a sitemap is blocked by robots.txt?
First, confirm whether the block was intentional. If the URL needs to be crawled, test which Disallow rule is blocking it, then adjust the rule or move the page out of the blocked directory. Not every excluded URL needs to be unblocked; assess each one according to the page's purpose.
Can AI-generated robots.txt rules go straight into production?
They should not be applied without review. AI can help generate rules, but if the requirements are not described precisely enough, the results may differ from what you actually need. First, check the crawlers and paths you want to restrict and verify the rules with a testing tool. Only then should you apply the verified rules to the site's file.
If logs show that a crawler has read robots.txt, does that prove it follows the rules?
References
- 網頁程式開發人員的搜尋引擎最佳化 (SEO) 指南 | Google 搜尋中心 | Documentation | Google for Developersdevelopers.google.com — Source used in the article above.
- robots.txt 簡介與指南 | Google 搜尋中心 | Documentation | Google for Developersdevelopers.google.com — Source used in the article above.
- Google Updates Spam Guide For Shared Pages, Generated Content & Appswww.searchenginejournal.com · 2026-10-26T20:57:22+00:00 — Source used in the article above.
- Google Search Console robots.txt Tester Inconsistencieswww.screamingfrog.co.uk · 2017-09-28T08:19:35+00:00 — Source used in the article above.
- Robots.txt Explained: Syntax, Best Practices, & SEOwww.semrush.com · 2018-07-10T11:00:00Z — Discusses: After uploading robots.txt, how can you confirm that a search engine has retrieved the file?
- 檢索預算(Crawl Budget)是什麼?中大型網站不可不知的SEO項目!www.awoo.ai · 2022-05-24T06:21:55+00:00 — Discusses: What should you do if a URL submitted in a sitemap is blocked by robots.txt?
- Website Migration SEO Checklist (88 Steps + Free Template)backlinko.com · 2024-04-01T17:23:50-05:00 — Source used in the article above.
- How to Create and Optimize Your Robots.txt Fileneilpatel.com · 2026-05-26T19:00:00+00:00 — Source used in the article above.
- robots.txt 的 AI 條目:你立下的規矩,到底有誰在乎?教你親自驗證他們的忠誠度ithelp.ithome.com.tw — Discusses: If logs show that a crawler has read robots.txt, does that prove it follows the rules?
- ChatGPT x SEO 13 個應用案例大公開,用 AI 輕鬆提升網站 SEO 執行效率!www.awoo.ai · 2023-03-14T06:49:06+00:00 — Discusses: Can AI-generated robots.txt rules go straight into production?
- Should I Block AI Crawlers At Robots.txt Or Server Level? – Ask An SEOwww.searchenginejournal.com · 2026-09-03T12:00:18+00:00 — Discusses: If logs show that a crawler has read robots.txt, does that prove it follows the rules?



