If you want search engines to crawl your website while restricting only a few folders, use User-agent: * to set general rules, then list the paths you do not want crawled with Disallow. If you have nothing to exclude, leave the value after Disallow: blank to indicate that crawling is not prohibited. Do not simply overwrite your existing configuration with this example. Before making changes, check which crawlers you want to allow, any existing crawler-specific rules, and other access restrictions. Allowing crawling does not guarantee indexing.
Start with the fewest rules needed to define access
Place robots.txt in the website’s root directory as a UTF-8 plain-text file. If there are no other rules and you want to allow crawling throughout the site, you can start with these two lines:
User-agent: *
Disallow:
Leaving the value after Disallow: blank means no paths are blocked from crawling. If you only want to exclude two folders, you can change this to:
User-agent: *
Disallow: /example/
Disallow: /checkout/
These paths are examples, not folders that every website should block. Replace them with the directories you actually need to exclude. Other paths remain open unless other restrictions apply. Be careful not to change a blank Disallow: to Disallow: /, which prohibits crawling of the entire site.
Identify the crawler, then review existing rule blocks
User-agent specifies which crawler the rules apply to. If a particular crawler needs different rules, you can create a separate block for it. Seeing one block that allows crawling does not mean the target crawler will necessarily have access.
- First, list the crawlers you want to allow, such as Googlebot, then check the relevant rules for each.
- Check whether existing Disallow rules cover paths for articles, products, or other content you want to make public.
- If crawler-specific blocks already exist, review them as well rather than simply appending general allow rules to the end of the file.
Public discussions differ on whether to use Allow: /. Some people report that their sites were crawled successfully after using it, while others recommend leaving the value after Disallow: blank. Reports of successful crawling reflect individual experiences only. Likewise, a 2015 comment about Allow support does not establish how all crawlers handle it today. This article uses a blank Disallow: as a simple starting point, not to suggest that Allow never works.
After making changes, check URLs—not just the file’s text
First, check the content actually returned at your website’s /robots.txt. Then use Google Search Console’s URL Inspection tool to check representative pages you want crawlers to access. If problems remain, check these three types of restrictions separately:
- Crawling restrictions: Does robots.txt block the page or its directory?
- Indexing restrictions: Does the page still have a noindex setting? Restricting indexing is not the same as blocking crawling.
- Access restrictions: Does the page require a login, preventing crawlers from retrieving its content?
Do not rely on robots.txt to keep content private. Public discussions point out that blocking crawling does not prevent people from visiting a URL directly. Private content should require a login or other authentication; listing its path in Disallow is not enough.
Configuring robots.txt: perspectives from across the web
Can you allow a specific subdirectory within a blocked folder?
Yes, you can use Allow to create an exception for a specific subdirectory, but check how the target crawler handles these directives. To block most of a directory while leaving part of it open, list the parent directory with Disallow, then specify the subdirectory with Allow. Before publishing, test both the paths that should be allowed and those that should remain blocked to check that the rules work as intended.
Will pages blocked from crawling stay out of Google search results?
Not necessarily. Some sources note that even if Google cannot crawl a page, it may still index it if links point to it. You therefore cannot rely solely on robots.txt rules prohibiting crawling to conclude that a page will not appear in search results. Crawling and indexing are different things, and you should also check the page's indexing settings.
How can you test new rules before publishing them?
You can start by loading the existing file or proposed content into a testing tool that supports custom robots.txt rules, then test URLs that should be allowed and those that should be restricted. You can also crawl the entire site using custom rules. This can help identify unintended blocks before publication, but you still need to select the crawler you care about and representative paths for testing.
What should you check if robots.txt allows access but crawlers still cannot get in?
Recheck the applicable robots.txt rules, then check whether a CDN or other edge protection is blocking the crawler's requests. Sources note that robots.txt and edge protection settings operate at different layers and may become inconsistent over time. Even if robots.txt allows crawling, other protections may still block the requests. Also check the robots.txt content the website actually returns.
Do AI search crawlers need to be checked separately from other crawlers?
If you want your website to appear in AI search experiences, it is worth checking each relevant search crawler to ensure it has not been accidentally blocked. Sources note that search crawlers and crawlers used for model training serve different purposes and can be managed separately. First identify which category you want to allow, then check the crawler's identity and applicable rules rather than treating all AI crawlers as the same kind.
References
- Cloudflare 新制下既有 zone 的 AI 自查清單ithelp.ithome.com.tw — Discusses: What should you check if robots.txt allows access but crawlers still cannot get in?
- Why alexa doesn't crawl my site?webmasters.stackexchange.com — Source used in the article above.
- Cloudflare Will Write Your Robots.txt, And It Has A Pointwww.searchenginejournal.com · 2026-09-18T14:30:00+00:00 — Discusses: What should you check if robots.txt allows access but crawlers still cannot get in?
- 如何讓網頁被 Google 搜尋?3 步驟診斷+6 個未收錄原因welly.tw · 2025-01-22T08:05:53.698Z — Discusses: Will pages blocked from crawling stay out of Google search results?
- How to Create and Optimize Your Robots.txt Fileneilpatel.com · 2026-05-26T19:00:00+00:00 — Source used in the article above.
- Reviving Retired Search Console Reportswww.screamingfrog.co.uk · 2019-04-08T13:18:41+00:00 — Discusses: Can you allow a specific subdirectory within a blocked folder?; How can you test new rules before publishing them?
- Why alexa doesn't crawl my site?webmasters.stackexchange.com — Source used in the article above.
- 技術 SEO 初學者指南 (by Ahrefs)tenten.co · 2023-07-30T15:00:24.000Z — Discusses: Will pages blocked from crawling stay out of Google search results?
- 2025 Website Launch Checklist (Pre-Launch and Post-Launch)www.semrush.com · 2023-09-13T14:54:00Z — Source used in the article above.
- 10 SEO best practices for Google and AI searchwww.semrush.com · 2020-05-25T11:00:00Z — Discusses: Do AI search crawlers need to be checked separately from other crawlers?
- Comprehensive SEO Checklist (2025)neilpatel.com · 2024-05-30T19:00:00+00:00 — Source used in the article above.
- How do you configure robots.txt to allow crawling of the site except for a few directories?webmasters.stackexchange.com — Source used in the article above.
- Google Rank Data Issues, Lighthouse Update & AI Opt-Out – SEO Pulsewww.searchenginejournal.com · 2026-11-07T13:00:00+00:00 — Discusses: Do AI search crawlers need to be checked separately from other crawlers?
- How do you configure robots.txt to allow crawling of the site except for a few directories?webmasters.stackexchange.com — Discusses: Can you allow a specific subdirectory within a blocked folder?



