To keep publicly accessible content out of search results, you can set noindex through a page’s meta tag or an HTTP response header. Search engines must be able to read the rule for it to take effect. Blocking crawling with robots.txt is not a substitute for preventing indexing, and you should not use it to block crawlers that need to read noindex. If the content contains internal information, consider access controls such as requiring a login; keeping content out of search indexes does not make it confidential.
Choose a noindex method based on the content type
Google’s documentation states that meta tags and HTTP response headers have the same effect in preventing indexing. The choice depends on the content type and which parts of the website you can modify.
- HTML pages: Place a meta tag in the head section using name=”robots” and content=”noindex” to tell search engines that support this rule not to index the page.
- Google only: Set the meta tag’s name attribute to googlebot; this does not ask all search engines to exclude the page.
- PDFs, videos, or images: Send X-Robots-Tag: noindex in an HTTP response header, without relying on an HTML tag.
If you use a CMS such as Wix, WordPress, or Blogger, the platform may offer search engine settings or other ways to configure indexing without editing HTML directly. You still need to check exactly which settings are available. Different search engines may interpret rules differently, so there is no guarantee that all search services will handle them the same way.
robots.txt can block crawlers from reading noindex
Do not use robots.txt to block resources you want excluded through noindex. For example, if you add noindex to a page but prohibit Googlebot from accessing it, the crawler cannot see the new rule. If other pages link to that URL, it may still appear in search results.
Google’s documentation states that when Googlebot can access a page and read its noindex rule, the page will be excluded from search results, even if other websites link to it. Google also does not support noindex in robots.txt, so putting it there cannot replace a meta tag or response header.
Sensitive content needs access controls
Before choosing a method, decide whether the content can be public but should stay out of search results, or whether unauthorized people must not be able to read it. noindex controls search indexing; it does not replace login authentication. A technical community discussion recommends access controls, such as requiring a login, for content that should not be public, including administrative areas. This is advice about access security, not an official Google indexing rule.
Search results do not update as soon as setup is complete
Google must crawl the page again to discover a newly added noindex rule. Google’s official documentation states that this may take several months, depending on the page’s importance on the internet. You can request recrawling through the URL Inspection tool, but this does not guarantee that the page will disappear from search results immediately. If you need immediate removal from Google search results, the documentation points to separate removal instructions; a standard noindex setup is not an instant removal tool.
Keeping content out of search indexes: perspectives from across the web
What are the limitations of configuring pages individually when you want to keep many pages out of search indexes?
The main limitation is that configuring each page individually may be impractical when you have many pages. Page-by-page settings still let you control whether individual pages are indexed and are worth considering when you do not have root access to the server. Some CMS platforms may offer search engine settings or other ways to add tags. First check which settings your website supports, then assess how to handle large numbers of pages.
How can you start identifying potentially unwanted indexed pages?
You can use a search engine's site operator to view some of a domain's indexed URLs, then check the results for pages you do not want to appear. You can also compare the indexed page count in Search Console with the actual number of valid pages. A site query shows only some URLs, while comparing counts can help reveal anomalies; you still need to identify the specific pages affected.
Which pages should you check first to see whether they need to be excluded?
Unfinished pages and pages being tested are worth checking first: incomplete or incorrect information may confuse users, and test site content may duplicate content on the eventual live site. Before applying exclusion settings, confirm which pages they will affect. Take particular care with pages whose existing search rankings are important, so you do not accidentally remove them from the index.
References
- 使用「noindex」禁止 Google 搜尋建立索引 | Google 搜尋中心 | Documentation | Google for Developersdevelopers.google.com — Discusses: What are the limitations of configuring pages individually when you want to keep many pages out of search indexes?
- How to stop certain urls from being indexedwebmasters.stackexchange.com — Source used in the article above.
- 使用”noindex”等4種方式禁止特定網頁被收錄,提升網站整體seo品質www.awoo.ai · 2018-01-15T09:00:23+00:00 — Discusses: What are the limitations of configuring pages individually when you want to keep many pages out of search indexes?; How can you start identifying potentially unwanted indexed pages?



