An XML sitemap alone cannot make Google or other search engines crawl only the pages it lists. A sitemap tells search engines what content is available on a site; it is not an allowlist that limits crawling. Leaving a URL out does not prohibit search engines from crawling it. To restrict crawling of particular pages or directories, you must set explicit controls rather than simply remove their URLs from the sitemap. Whether content is indexed is also a separate question from whether crawling is allowed.
Sitemaps provide clues, not boundaries
A public answer on Webmasters Stack Exchange states directly that an XML sitemap cannot specify “do not crawl pages that are not listed.” A 2025 Backlinko article describes a sitemap as a file that lists important pages and guides crawlers to key content, noting that including important pages helps speed up indexing. Together, these explanations show what a sitemap can and cannot do: it helps search engines find important content but does not stop them from crawling other pages.
For example, a sitemap might list only product information pages and articles, leaving out test pages. That list provides the URLs of the product pages and articles, but it does not prohibit search engines from crawling the test pages. Listing pages you want discovered is different from specifying which pages must not be crawled.
Before excluding pages, clarify what you want to restrict
The public answer mentioned above suggests using robots.txt to set separate restrictions for pages or directories you do not want crawled. It also mentions using meta tags for content you do not want crawled or indexed. These tools are not interchangeable: the choice depends on your goal. Nor is this overview a complete setup guide for every search engine.
- To help important pages get discovered: Include their URLs in an XML sitemap to help search engines find them.
- To prevent particular paths from being crawled: Set crawl restrictions separately; leaving URLs out of the sitemap is not enough.
- To keep content out of the index: Check the appropriate indexing controls rather than assuming that crawl restrictions also prevent indexing.
Before organizing your sitemap, list “pages you want discovered” and “pages that need restrictions” separately. Handle each group according to its purpose rather than expecting a sitemap to do both jobs.
An HTML sitemap cannot restrict crawling either
A 2020 Semrush article distinguishes between two types of sitemaps: an HTML sitemap is a web page with organized links that help visitors navigate, while an XML sitemap is primarily intended for search engines. An HTML version can help readers find categories and articles, but switching sitemap formats cannot replace explicit crawl restrictions. The key is not which format you choose: sitemaps help visitors or search engines find content, but they cannot prohibit crawling.
Sitemaps and crawl scope: perspectives from across the web
Will Google crawl every URL listed in a sitemap?
There is no guarantee. Public discussions note that Google has not promised to crawl every URL in an XML sitemap. Including important pages helps search engines find content more quickly and may also speed up indexing, but it does not guarantee that those pages will be crawled or indexed.
How can robots.txt specify crawl restrictions for a particular directory?
Use User-agent to identify the crawlers concerned and Disallow to specify the directory you do not want crawled. A public answer recommends using robots.txt to set separate restrictions for pages or directories you do not want crawled. A 2018 tutorial gives the example of User-agent: * paired with Disallow: /members/ to restrict crawling of that directory by all search engines. Before applying these settings, check the target search engine’s current specifications. Restricting crawling is not the same as keeping content out of the index.
Is the Sitemap entry in robots.txt used to block pages?
No. A Sitemap entry identifies the location of a sitemap. To specify crawl restrictions for paths, the relevant tutorial instead uses User-agent to identify the crawlers concerned and Disallow or Allow to specify paths. The 2018 article also describes Sitemap as an independent entry unaffected by those rule groups. This historical explanation should not replace each search engine’s current specifications.
References
- Should I include everything in the sitemap or only new content?webmasters.stackexchange.com — Discusses: How can robots.txt specify crawl restrictions for a particular directory?
- How Do Search Engines Work? (Quick Guide for Beginners)backlinko.com · 2025-01-21T15:28:48-05:00 — Discusses: Will Google crawl every URL listed in a sitemap?
- Robots.txt用途與使用範例教學,釐清SEO收錄觀念!www.awoo.ai · 2018-03-29T08:57:00+00:00 — Discusses: How can robots.txt specify crawl restrictions for a particular directory?; Is the Sitemap entry in robots.txt used to block pages?
- HTML Sitemap: The Benefits for SEO and Userswww.semrush.com · 2020-12-17T14:29:00Z — Source used in the article above.
- URLs in Google Web Index Has Dropped to 0!! Why?webmasters.stackexchange.com — Discusses: Will Google crawl every URL listed in a sitemap?
- How to get your website indexed by Googlewww.semrush.com · 2020-12-28T14:21:00Z — Discusses: Will Google crawl every URL listed in a sitemap?



