How to create and configure robots.txt: first decide what should be crawlable

ProcrustesPR

Before creating robots.txt, clarify your goal: restricting crawling, keeping pages out of search results, or protecting private content. To control crawling, create a UTF-8 plain-text file named robots.txt, use User-agent to specify the crawler, and set paths with Disallow and Allow. Place the file in the root directory of the relevant host. robots.txt does not prevent indexing and cannot replace password protection.

ProcrustesPRProcrustesPRHow Long Does It Take to Evaluate SEO Results?

First decide whether to restrict crawling or exclude pages from search results

Before configuring the file, list your public content, paths you want crawlers to skip, and pages that should not be public. Then choose the appropriate control for each purpose:

  • To ask crawlers to skip certain paths: use crawling rules in robots.txt.
  • To let Google read a page without indexing it: use noindex.
  • To keep content private: use password protection rather than simply listing it under Disallow.

Google’s official documentation cautions that robots.txt cannot guarantee that a particular page will stay out of search results. If a page has a noindex directive but robots.txt blocks crawling, Google cannot read that page’s indexing instructions. So “I don’t want this to appear in search” should not automatically become “block crawling.”

Create a plain-text file and check which host it applies to

Create the file with a plain-text editor. It must be named robots.txt and use UTF-8 encoding. Do not save it in a word processor’s proprietary format; crawlers will not read the file from just any folder.

For example, the file for an HTTPS site should be at https://example.com/robots.txt, not in an article subdirectory. Its scope is limited to a specific protocol, subdomain, and port. Do not assume that the main site’s settings automatically cover other hosts, such as m.example.com.

Start with minimal rules and add restrictions one at a time

User-agent specifies which crawlers the rules apply to; an asterisk is the general wildcard for crawlers. If you want to allow crawling of all paths, you can use the following basic configuration, leaving the value after Disallow blank:

User-agent: *
Disallow:

If you instead want to prevent crawling of the checkout directory, change the example above to:

User-agent: *
Disallow: /checkout/

This example restricts crawling of the /checkout/ path; it does not ask search engines to remove the entire site. A blank Disallow: is different from Disallow: /: the latter prevents the applicable crawlers from crawling any paths and should not be applied as-is to a public site you want people to find through search.

Allow permits crawling of paths, while Disallow blocks it. If different crawlers need different settings, create separate User-agent blocks. Confirm the target and purpose of each block before adding rules, so that different requirements do not get mixed together.

Update the file using the method supported by your host or CMS

Once the file is ready, you can update it in the root directory using FTP, your hosting file manager, or an applicable CMS feature. WordPress may generate a virtual robots.txt, so not finding a physical file does not mean the site is not serving one. Making changes may require a plugin or manually uploading a file to override it; check which method applies to your site’s setup.

ProcrustesPRProcrustesPRWhere should I start if I want to do SEO myself?

robots.txt: perspectives from across the web

How can you check that the rules work as intended after uploading?

After uploading, check the file's fetch status and any errors in the robots.txt report in Google Search Console. When testing a specific path, make sure you are using the intended URL and simulating the intended crawler. If you change rules in a testing tool, save those changes to the robots.txt file on your website as well; edits on the testing page do not directly update the original file. Continue to check the file's formatting regularly.

Do you have to include a sitemap in robots.txt?

No. You can use the Sitemap directive to provide the location of an XML sitemap, making it easier for crawlers to find important pages. This is especially worth considering for large websites. However, a sitemap helps with URL discovery; it is not a required rule in every robots.txt file, and adding one should not be treated as a guarantee of indexing.

Can you set different rules for AI crawlers?

Yes. Based on your site's AI access policy, you can create separate User-agent blocks for agents or control tokens that support robots.txt controls, such as GPTBot, ClaudeBot, Google-Extended, or CCBot. Before setting the rules, understand what each is used for and read its documentation. For example, Google says that some user-triggered fetchers generally ignore robots.txt, so you should not assume the rules cover every form of AI access.

How can you make maintenance easier as the number of rules grows?

Use comments beginning with a hash sign to document the purpose of rules—for example, why a particular path is restricted—so other maintainers can later assess whether a rule is still needed. This is especially useful for growing teams or sites with complex structures. Also check the rules against official specifications to avoid unintended restrictions caused by syntax or matching behavior.

ProcrustesPRShould I rewrite or update an old article that does not rank?

References

  1. 漫遊器中繼標記規格 | Google 搜尋中心  |  Documentation  |  Google for Developersdevelopers.google.com — Source used in the article above.
  2. 網頁程式開發人員的搜尋引擎最佳化 (SEO) 指南 | Google 搜尋中心  |  Documentation  |  Google for Developersdevelopers.google.com — Source used in the article above.
  3. Recap of the February 2026 SEO Update by Yoastyoast.com · 2026-02-26T16:10:27+00:00 — Discusses: Can you set different rules for AI crawlers?
  4. How to Create and Optimize Your Robots.txt Fileneilpatel.com · 2026-05-26T19:00:00+00:00 — Discusses: Do you have to include a sitemap in robots.txt?; Can you set different rules for AI crawlers?
  5. Robots.txt Explained: Syntax, Best Practices, & SEOwww.semrush.com · 2018-07-10T11:00:00Z — Discusses: How can you check that the rules work as intended after uploading?; How can you make maintenance easier as the number of rules grows?
  6. robots.txt是什麼? robots.txt設定、用途與使用範例分享!welly.tw · 2022-01-13T05:20:36.169Z — Discusses: How can you check that the rules work as intended after uploading?
  7. Google Says Why It May Ignore Robots.txt And Negatively Impact SEOwww.searchenginejournal.com · 2026-07-25T10:17:52+00:00 — Discusses: How can you make maintenance easier as the number of rules grows?
  8. Google Explains 503s, Robots.txt Limits & Bracket URLs – SEO Pulsewww.searchenginejournal.com · 2026-10-24T12:00:00+00:00 — Discusses: Can you set different rules for AI crawlers?

Related articles

How Long Does It Take to Evaluate SEO Results?Where should I start if I want to do SEO myself?Should I rewrite or update an old article that does not rank?What should I check when organic traffic suddenly drops?Why can I not find my published article on Google?Will Reposting Social Content on My Website Cause a Duplicate-Content Penalty?Eight SEO essentials for beginners: Getting your pages indexedWhat Is SEO? Understanding Optimization Priorities Through How Search Engines Work
ProcrustesPR
ProcrustesPR