How do you configure robots.txt to allow crawling while restricting certain directories?

ProcrustesPR

To allow crawling of your website while restricting only a few directories, you can allow crawling by default and list each disallowed path: use User-agent: * to apply the rules to all crawlers, then use Disallow to list the directories you do not want crawled. There is no need to add Allow: /. This approach is a useful starting point, but there is no blocklist that suits every website. Choose which directories to restrict based on your site’s content, and remember that robots.txt cannot control access to it.

ProcrustesPRProcrustesPRHow Long Does It Take to Evaluate SEO Results?

Basic configuration: list only the directories you want to restrict

Suppose you want to restrict only the /example/ and /test/ directories while leaving everything else open to crawling. The file could contain:

User-agent: *
Disallow: /example/
Disallow: /test/

  • Applicable crawlers: User-agent: * means this group of rules applies to all crawlers, though compliance depends on each crawler.
  • Restricted directories: The two Disallow lines specify /example/ and /test/, respectively; they do not block the entire website.
  • Other paths: In this simple configuration, paths that are not disallowed remain open to crawling and do not need to be listed individually.

These directory names are only examples; your website may not have these paths. Before applying the rules, replace them with the directories you actually need to restrict. If you do not need to restrict anything, there is no need to add blocking rules just to have a “basic configuration.”

Put the file in the right place, and do not start by blocking the entire site

robots.txt should be placed in the website’s root directory, not inside the folder you want to restrict. A 2018 article by awoo also notes that the filename should be entirely lowercase—robots.txt—not Robots.txt or robots.TXT.

If you only need to restrict a few directories, there is no need to start with Disallow: / and then allow content individually. Disallow: / disallows crawling of the entire website; it does not mean “allow the homepage.” Using it incorrectly could prevent Googlebot from reading content you want to make discoverable. Simply list the paths you need to disallow, without adding unnecessary rules.

Restricting crawling does not ensure privacy or guarantee exclusion from search results

robots.txt tells crawlers which content they are allowed to crawl; noindex asks them not to index content. These serve different purposes: disallowing crawling of a directory does not guarantee that it will stay out of search results.

If a page should remain private, answers in public discussions explicitly recommend authentication, such as a username and password, rather than relying solely on robots.txt. Listing a path under Disallow does not prevent users from visiting it directly.

When checking your configuration, review at least two types of URLs: public pages you want search engines to read, and pages within directories you want to restrict. Check that public pages are not accidentally blocked and that the paths you want to restrict are covered by the rules. This check only confirms whether the configuration meets your needs; it does not guarantee indexing or rankings.

ProcrustesPRProcrustesPRWhere should I start if I want to do SEO myself?

Configuring robots.txt: perspectives from across the web

Can you allow a subdirectory within a restricted directory?

You can consider using a more specific Allow rule to define an exception within a restricted directory. For example, in a group of rules applying to all crawlers, list Disallow: /example/ followed by Allow: /example/foobar/. Public discussions indicate that Google and Bing determine precedence based on path specificity. However, support for Allow may vary among crawlers, so do not assume that all crawlers will handle it the same way.

Can you set rules for a specific search engine's crawler?

Yes. In addition to using User-agent: * to target all crawlers, you can use User-agent: Googlebot or User-agent: Bingbot to target a specific crawler, then list the paths you want to restrict. This is useful when different crawlers require different treatment—for example, if you want to restrict a particular bot that crawls too frequently. If you have no such need, there is no reason to create separate rules for each crawler.

Do you need to create robots.txt from scratch when using a website platform?

Not necessarily. First check whether the platform already has a configuration. In its 2025 introduction to SHOPLINE, awoo noted that the platform's built-in robots.txt restricted some system pages and pages with URL parameters by default and provided the sitemap location. This described that platform at the time; it does not mean all platforms have the same defaults. Before making changes, review the existing rules and your actual directory requirements to avoid accidentally blocking important content.

ProcrustesPRShould I rewrite or update an old article that does not rank?

References

  1. How do you configure robots.txt to allow crawling of the site except for a few directories?webmasters.stackexchange.com — Discusses: Can you allow a subdirectory within a restricted directory?
  2. 如何讓網頁被 Google 搜尋?3 步驟診斷+6 個未收錄原因welly.tw · 2025-01-22T08:05:53.698Z — Source used in the article above.
  3. SHOPLINE SEO 有哪些功能?帶你一次了解 SHOPLINE SEO 的特色與操作www.awoo.ai · 2025-02-08T03:57:09+00:00 — Discusses: Do you need to create robots.txt from scratch when using a website platform?
  4. Is it possible to block an entire site to Google then list exceptions?webmasters.stackexchange.com — Discusses: Can you allow a subdirectory within a restricted directory?
  5. Robots.txt Explained: Syntax, Best Practices, & SEOwww.semrush.com · 2018-07-10T11:00:00Z — Discusses: Can you set rules for a specific search engine's crawler?; Do you need to create robots.txt from scratch when using a website platform?
  6. Robots.txt用途與使用範例教學,釐清SEO收錄觀念!www.awoo.ai · 2018-03-29T08:57:00+00:00 — Source used in the article above.
  7. How do you configure robots.txt to allow crawling of the site except for a few directories?webmasters.stackexchange.com — Discusses: Can you allow a subdirectory within a restricted directory?
  8. SEO: Duplicate content caused by pagination/tag/index pageswebmasters.stackexchange.com — Source used in the article above.

Related articles

How Long Does It Take to Evaluate SEO Results?Where should I start if I want to do SEO myself?Should I rewrite or update an old article that does not rank?What should I check when organic traffic suddenly drops?Why can I not find my published article on Google?Will Reposting Social Content on My Website Cause a Duplicate-Content Penalty?Eight SEO essentials for beginners: Getting your pages indexedWhat Is SEO? Understanding Optimization Priorities Through How Search Engines Work
ProcrustesPR
ProcrustesPR