To create a robots.txt file, first decide which crawlers may access which paths. Then save the rules as a plain-text file in the website’s root directory and check that important pages and required resources are not accidentally blocked. It controls crawling; it does not guarantee exclusion from search indexes, nor is it a security barrier that protects sensitive data. If your goal is to keep a page out of Google search results, use a noindex directive that Googlebot can read. You must not also use robots.txt to prevent it from accessing that page.
Define your goals before creating a minimal set of rules
Start by listing which content you want search engines to access, which paths you do not want crawled, and whether different crawlers need different settings. User-agent specifies which crawlers the rules apply to, while Allow and Disallow specify paths that may and may not be crawled, respectively. You can create separate rule blocks for crawlers that need different settings.
For example, if your only goal is to ask compliant crawlers not to crawl your site’s internal search path, you could write:
User-agent: *
Disallow: /search/
The asterisk means the rules apply to all crawlers. This example restricts crawling of the /search/ path; it does not guarantee that related URLs will stay out of search results. You can use comments beginning with # to record the reasons for your settings for future reference.
The file should be named robots.txt and placed in the website’s root directory as UTF-8 plain text. WordPress may provide a virtual robots.txt file. Depending on the site’s configuration, you may need to use a plugin to modify it or upload a physical file to override it. Do not assume there are no rules simply because you cannot find a physical file.
Check whether the rules do what you intend, not just whether the file exists
Open /robots.txt at the root of your website to confirm the content actually being served, then check each of the following against what you intend:
- Important pages: Are articles, products, or regular website pages accidentally blocked by Disallow?
- Resources needed to render the page: Are CSS, JavaScript, or APIs needed by the page blocked? Do not just check whether the HTML page can be crawled.
- Target crawlers: Check the applicable User-agent block. If you want your content to be available for use in ChatGPT search, check whether OAI-SearchBot is blocked, and also check your CDN’s bot rules.
Use Google’s URL Inspection tool to test how important pages render and confirm that blocked resources are not preventing content from displaying. In one demonstration by Semrush and Bill Widmer, Site Audit was initially blocked by the rules and could crawl successfully only after they were changed. This shows why checks must account for the actual website and tool settings. A single successful test is not proof that all crawlers can access the site.
Keep crawling restrictions, indexing controls, and security separate
A URL that is blocked from crawling may still be discovered by Google through links from other websites and appear in search results. Google’s official documentation also explicitly states that if robots.txt blocks crawling, Google cannot read the page’s robots meta tag or X-Robots-Tag header, and the associated indexing rules will be ignored.
Therefore, do not treat Disallow as a way to hide internal reports, sensitive data, or test sites. robots.txt itself cannot force all visitors to follow its rules. To use it safely, first be clear about your goal: restricting crawling, keeping content out of search results, or actually restricting access to content.
robots.txt: perspectives from across the web
How can you check server logs to see whether a crawler has read robots.txt?
Look for requests to /robots.txt in your access logs, check the crawler’s identity and how often it fetches the file, and compare those requests with the times you changed the rules. This can help establish whether the crawler retrieved the rules, but a retrieval record alone does not prove that it subsequently followed them. Also keep in mind that different services may take different amounts of time to update their rules.
Do rules automatically generated by website services need to be checked separately?
Yes. If your hosting provider or another service offers automatic synchronization, check the robots.txt file it actually publishes to confirm that the defaults match how you want your site crawled and its data used. An article covering Cloudflare’s announcement of a new feature warned that default settings could lead site owners to publish policies they had not personally written or read. However, the article discussed the product as announced at that time; it does not mean that every website currently uses those settings.
How can you check a test site that requires login or restricts crawling?
Provided you are authorized to test the site, first check whether its robots.txt restricts the tool you are using, then configure the tool or provide login credentials as needed. An older Screaming Frog tutorial described a setting for ignoring robots.txt and an authentication dialog. This is a testing method for a specific tool, not evidence that other crawlers can also get past the restrictions. Follow the procedures appropriate to the version you are using.
References
- 漫遊器中繼標記規格 | Google 搜尋中心 | Documentation | Google for Developersdevelopers.google.com — Source used in the article above.
- Robots.txt Explained: Syntax, Best Practices, & SEOwww.semrush.com · 2018-07-10T11:00:00Z — Source used in the article above.
- 10 Features In The SEO Spider You Should Really Knowwww.screamingfrog.co.uk · 2015-01-22T11:48:45+00:00 — Discusses: How can you check a test site that requires login or restricts crawling?
- Spam Guide Update, Restaurant Controls & AI Mode Revenue – SEO Pulsewww.searchenginejournal.com · 2026-10-31T12:00:00+00:00 — Source used in the article above.
- robots.txt 的 AI 條目:你立下的規矩,到底有誰在乎?教你親自驗證他們的忠誠度ithelp.ithome.com.tw — Discusses: How can you check server logs to see whether a crawler has read robots.txt?
- ChatGPT is unable to see client's Google Reviews?localsearchforum.com — Source used in the article above.
- How to Create and Optimize Your Robots.txt Fileneilpatel.com · 2026-05-26T19:00:00+00:00 — Source used in the article above.
- Amazon Blocked Meta’s Muse, And Robots.txt Had Nothing To Saywww.searchenginejournal.com · 2026-09-29T14:30:54+00:00 — Source used in the article above.
- What is technical SEO? Basics and best practiceswww.semrush.com · 2023-02-01T15:56:00Z — Source used in the article above.
- Cloudflare Will Write Your Robots.txt, And It Has A Pointwww.searchenginejournal.com · 2026-09-18T14:30:00+00:00 — Discusses: Do rules automatically generated by website services need to be checked separately?



