Generally, no. The practical guidance cited below recommends leaving pages deliberately blocked by robots.txt out of an XML sitemap. Take 404Error.aspx as an example: if you do not want users to find the page through search, you can exclude it from the sitemap. That does not, however, prevent indexing. If it is an important page that should appear in search results, check whether it was blocked by mistake rather than simply deleting its sitemap entry.
Sitemap and robots.txt settings should match the page’s purpose
A 2018 article on Neil Patel’s website notes that including a URL in a sitemap while blocking crawling sends conflicting signals. Semrush material dated 2021 also recommends excluding pages blocked by robots.txt or marked noindex. These are practical recommendations; this article has not verified whether they reflect current official Google rules.
- Pages you want users to find through search:Include these content pages in the sitemap and avoid accidentally blocking them or marking them noindex.
- Pages used only for functions within the site:If a page does not need to appear in search results, it is generally left out of the sitemap. You still need to distinguish between restricting crawling and restricting indexing.
Blocked pages can still appear in search results
A discussion on Webmasters Stack Exchange notes that robots.txt restricts crawling and cannot, on its own, ensure that a URL is not indexed. A 2015 Screaming Frog article also notes that blocked URLs with internal or external links pointing to them may still be indexed and appear in search results. A Disallow rule therefore does not mean a URL has been removed from search results.
If the goal is to prevent indexing, noindex is the directive intended for that purpose. But if robots.txt still blocks crawling, do not assume that adding noindex solves the problem. The 2018 article cited above explicitly warns against using noindex and a robots.txt block on the same page.
Removing a URL from a sitemap does not prevent indexing either. Another public discussion notes that a sitemap cannot tell search engines not to crawl URLs that are not listed.
Handle 404Error.aspx according to its actual status
This case provides only a filename, not an HTTP response status. The name alone does not establish that the page actually returns a 404. Consider the following cases:
- It actually returns a 404:A 2022 awoo article recommends updating the sitemap to exclude nonexistent URLs and checking whether links pointing to them need to be adjusted.
- It is a deliberately blocked utility page:Remove it from the sitemap, but do not assume this prevents indexing.
- An important page was blocked by mistake:Check the Disallow rules first. The awoo material suggests modifying the blocking rules or moving the page out of the blocked directory.
Robots.txt blocking and sitemap inclusion: perspectives from across the web
What happens if links still point to a blocked URL?
Screaming Frog noted in 2015 that blocked URLs with internal or external links pointing to them may still accumulate PageRank and appear in search results. Because Google cannot crawl the page's content or see its outgoing links to pass PageRank onward, such URLs may become dead ends in the link structure. The article recommends examining the number of blocked URLs, their links, and their depth within the site together. This is a possibility to consider during analysis, not a claim that every blocked page will cause problems.
Why can't you assume that noindex on a blocked page will take effect?
Because a robots.txt block prevents Google from crawling the page's content, it also prevents Google from reading an on-page noindex directive. A 2018 article likewise warns against using noindex and a robots.txt block on the same page. If you use an on-page noindex directive to address indexing, you cannot keep crawling blocked and assume the search engine has read the directive.
References
- Should I include everything in the sitemap or only new content?webmasters.stackexchange.com — Source used in the article above.
- What Is a Sitemap? Website Sitemaps Explainedwww.semrush.com · 2021-05-24T11:55:00Z — Source used in the article above.
- How can I clone or mirror a site without SEO penalties for duplicate content?webmasters.stackexchange.com — Source used in the article above.
- Want More Traffic? Deindex Your Pages. Here’s Why.neilpatel.com · 2018-03-06T19:00:42+00:00 — Discusses: Why can't you assume that noindex on a blocked page will take effect?
- Screaming Frog SEO Spider Update – Version 5.0www.screamingfrog.co.uk · 2015-09-04T13:41:52+00:00 — Discusses: What happens if links still point to a blocked URL?; Why can't you assume that noindex on a blocked page will take effect?
- 檢索預算(Crawl Budget)是什麼?中大型網站不可不知的SEO項目!www.awoo.ai · 2022-05-24T06:21:55+00:00 — Source used in the article above.



