Yes. You can use Disallow to block an entire site or folder and Allow to make exceptions for specific paths, provided the target crawler supports Allow and gives precedence to rules with more specific paths. Public discussions include examples of this approach. However, these rules control whether content can be crawled; they do not guarantee that only specified pages will be indexed. To allow the homepage, you also need to check its actual URL and any redirect destination.
Block the entire site, then allow specific paths
A discussion on Webmasters Stack Exchange provides a configuration with several exceptions. Two are shown here:
User-agent: *
Disallow: /
Allow: /index.html
Allow: /example/
This configuration uses Disallow: / to block the entire site, then Allow to open the /index.html and /example/ paths to crawling. Using Google and Bing as examples, the answer explains that rules with longer, more specific paths take precedence over those with shorter paths. It is not a matter of later rules overriding earlier ones. The key is therefore not to put Allow last, but to check the exception paths and how the crawler interprets the rules. This is the explanation given in that discussion, not a rule that necessarily applies to all crawlers.
Block a folder while allowing exceptions within it
Applying the same principle to a folder gives:
User-agent: *
Disallow: /docs/
Allow: /docs/public.html
If the crawler follows the precedence rules described above, /docs/public.html is more specific than /docs/. This lets you block /docs/ while allowing /docs/public.html to be crawled. This example applies the same principle; it has not been tested for a specific website.
The sources do not all agree about Allow. One quoted passage states that there is no Allow field and recommends moving the files to be blocked into another directory. A reply says this claim is only partly correct and points to information about Allow. These statements should not be presented as a consensus, nor should the claim that there is no Allow field be treated as a general rule that applies today.
Allowing the homepage filename does not necessarily allow the actual homepage
Another discussion notes that if /index.php redirects to /, allowing only /index.php may still leave the redirect destination blocked. Even if the two URLs display the same page, they cannot be treated as the same path when configuring the rules.
- First, check whether the homepage actually uses /, /index.html, or another URL.
- If there is a redirect, also check whether its destination is blocked.
- Do not simply copy the example homepage filename into your own site’s configuration.
Handle crawl restrictions and exclusion from indexing separately
Before configuring the rules, clarify your goal: do you want to prevent crawlers from fetching content, or keep pages out of search results? Allow exceptions adjust what can be crawled; this set of rules alone cannot keep pages out of the index.
A 2021 Screaming Frog article describes using Disallow to restrict crawling across large numbers of pages or entire path sections, but emphasizes addressing indexing issues first. A 2024 Harris article also distinguishes crawling from indexing and suggests considering meta robots to control indexing when you do not want pages to appear in search results.
robots.txt blocks and exceptions: perspectives from across the web
Does every folder you want to block need its own robots.txt?
No. A public discussion notes that a single robots.txt can specify allowed or blocked folders using paths that begin with a slash; another source explains that this file is typically placed in the site's root directory. When configuring it, verify the paths and target crawler, and check that you have not accidentally blocked content that needs to be crawled.
If access is allowed by default, when do you still need Allow?
You need Allow when you block access with a broader rule but want to make an exception for a path within that blocked area. One discussion notes that paths that have not been disallowed can be crawled by default, so some configurations only need Disallow. For an Allow exception to work, however, the target crawler must support Allow and give precedence to rules with more specific paths.
Can you safely use wildcards in exception rules?
If a rule contains wildcards, you should not infer the outcome solely from the general principle of path length. Documentation quoted in the supplied discussion states that precedence for rules containing wildcards is undefined. That is the position of the material cited in that discussion, not necessarily the current rule for all crawlers. Verify the configuration against the target crawler's documentation and test results.
References
- Robots.txt and specific file inclusionwebmasters.stackexchange.com — Source used in the article above.
- Robots.txt In SEO ?www.warriorforum.com — Discusses: Does every folder you want to block need its own robots.txt?
- Conducting an SEO Content Audit: 6 Things You Might Be Overlookingwww.screamingfrog.co.uk · 2021-11-15T13:37:48+00:00 — Source used in the article above.
- Robots.txt and specific file inclusionwebmasters.stackexchange.com — Source used in the article above.
- 技術性SEO:打造高品質網站,提升搜尋引擎排名tenten.co · 2024-12-25T18:03:57.000Z — Discusses: Does every folder you want to block need its own robots.txt?
- robots.txt、meta robots設定:讓網頁不被Google搜尋到www.yesharris.com · 2024-08-15T14:20:11+08:00 — Source used in the article above.
- Will googlebot index my index.php even with an htaccess redirect?webmasters.stackexchange.com — Source used in the article above.
- Is it possible to block an entire site to Google then list exceptions?webmasters.stackexchange.com — Discusses: If access is allowed by default, when do you still need Allow?
- Is it possible to block an entire site to Google then list exceptions?webmasters.stackexchange.com — Discusses: If access is allowed by default, when do you still need Allow?; Can you safely use wildcards in exception rules?



