How do you exclude an entire directory and its subpages using robots.txt?

ProcrustesPR

To restrict crawling of /first_url, /second_url, and their subpages, specify the applicable User-agent in robots.txt, then add Disallow: /first_url and Disallow: /second_url on separate lines. This is the format suggested in public discussions; if you use /first_url/*, there is no need to add a slash after the asterisk. Before configuring these rules, confirm the actual paths and the crawlers they should apply to, and remember: restricting crawling does not prevent indexing or stop people from accessing the content.

ProcrustesPRProcrustesPRHow Long Does It Take to Evaluate SEO Results?

Specify the crawler, then list the two paths to exclude

To set a general restriction for crawlers that comply with the protocol, add the following three lines to the plain-text robots.txt file in your website’s root directory, with one directive per line:

User-agent: *
Disallow: /first_url
Disallow: /second_url

User-agent: * sets general rules for crawlers; the two Disallow directives list the paths you do not want crawled. One answer in a public discussion states that, in this example, /first_url alone can restrict crawling of the main path and its subpages without a trailing wildcard. When applying these rules, check the target URLs rather than entering paths based solely on website menu labels.

  • Confirm that subpages such as /first_url/product1 are actually under the target path.
  • If you want to restrict only one crawler, enter its identifier rather than simply using an asterisk.
  • If the existing file already has blocks dedicated to different crawlers, review those as well; crawlers that need different restrictions can be configured separately.

The slash after the asterisk is unnecessary

A reader proposed Disallow: /first_url/*/, adding another slash after the asterisk. For the same scenario, another public answer recommends /first_url/* and /second_url/* and explicitly states that the final slash is unnecessary. Therefore, /*/ should not be treated as a required format for “including all subdirectories.”

A separate discussion demonstrates excluding a directory with Disallow: /example/. Note that a slash after a directory name is not the same as an extra slash after an asterisk. Do not assume that /first_url and /first_url/ are exactly the same target without checking. The URLs to restrict here also include the main URL without a trailing slash, so checking only the subpages is not enough.

Clarify your goal before blocking crawling

If the goal is to reduce crawling of large numbers of tag pages or an entire path, Disallow is an option. A 2021 article also cautions that any indexing issues should be addressed first: stopping crawling is not a substitute for removing pages from search results.

If the goal is to protect sensitive content, robots.txt alone is not enough. It only governs crawlers willing to follow the rules and does not prevent ordinary visitors from opening pages directly. Confidential information should be protected with access controls such as authentication. Before deployment, separately identify “which URLs should not be crawled,” “which content should not be indexed,” and “which information must not be public,” rather than trying to address three different needs with the same configuration.

ProcrustesPRProcrustesPRWhere should I start if I want to do SEO myself?

Excluding directories with robots.txt: perspectives from across the web

If pages in the directory are already indexed, what should be addressed first?

If you also want to address indexing issues with pages already in search results, the relevant article recommends dealing with indexing first, then using robots.txt to exclude large numbers of paths. A separate discussion notes that once robots.txt restricts crawling, crawlers may be unable to read the noindex directive on a page, so the two settings need to work together. That discussion also cautions that noindex itself does not guarantee that a page will not be indexed.

Do you always need to add an asterisk after the path when subpages have query parameters?

Not necessarily; public discussions offer differing views. One answer says that matching /url/test?test=1 requires Disallow: /url/*. Another answer about a similar path states that a trailing wildcard is ignored and the path alone is sufficient. These discussions do not establish that an asterisk is required in every case. When configuring the rules, check how they work for the target crawler and URLs containing query parameters.

Does restricting a specific AI crawler also control how content is subsequently used?

Not necessarily. You can list directories you do not want a particular crawler to crawl using Disallow in its User-agent block. However, standard robots.txt rules govern whether protocol-compliant crawlers may crawl content, not directly how that content is used after crawling. If you want to express restrictions on other uses, you need to separately verify which controls are applicable.

ProcrustesPRShould I rewrite or update an old article that does not rank?

References

  1. Conducting an SEO Content Audit: 6 Things You Might Be Overlookingwww.screamingfrog.co.uk · 2021-11-15T13:37:48+00:00 — Discusses: If pages in the directory are already indexed, what should be addressed first?
  2. Google Explains 503s, Robots.txt Limits & Bracket URLs – SEO Pulsewww.searchenginejournal.com · 2026-10-24T12:00:00+00:00 — Discusses: Does restricting a specific AI crawler also control how content is subsequently used?
  3. 用 robots.txt、noindex 限制搜尋引擎抓網站資料www.da-vinci.com.tw · 2022-09-11T23:38:01+08:00 — Discusses: If pages in the directory are already indexed, what should be addressed first?
  4. How to Create and Optimize Your Robots.txt Fileneilpatel.com · 2026-05-26T19:00:00+00:00 — Source used in the article above.
  5. How do I disallow an entire directory with robots.txt?webmasters.stackexchange.com — Discusses: Do you always need to add an asterisk after the path when subpages have query parameters?
  6. How do I disallow an entire directory with robots.txt?webmasters.stackexchange.com — Discusses: Do you always need to add an asterisk after the path when subpages have query parameters?
  7. Should I Block AI Crawlers At Robots.txt Or Server Level? – Ask An SEOwww.searchenginejournal.com · 2026-09-03T12:00:18+00:00 — Discusses: Does restricting a specific AI crawler also control how content is subsequently used?
  8. How do you configure robots.txt to allow crawling of the site except for a few directories?webmasters.stackexchange.com — Source used in the article above.
  9. How do I disallow an entire directory with robots.txt?webmasters.stackexchange.com — Source used in the article above.

Related articles

How Long Does It Take to Evaluate SEO Results?Where should I start if I want to do SEO myself?Should I rewrite or update an old article that does not rank?What should I check when organic traffic suddenly drops?Why can I not find my published article on Google?Will Reposting Social Content on My Website Cause a Duplicate-Content Penalty?Eight SEO essentials for beginners: Getting your pages indexedWhat Is SEO? Understanding Optimization Priorities Through How Search Engines Work
ProcrustesPR
ProcrustesPR