noindex is suitable for pages or resources you definitely do not want appearing in search results but still need to keep, such as thank-you pages shown after an action is completed, test pages, and downloads you do not intend search users to find directly. The key question is whether the page should serve as an entry point from search, not simply how many keywords it contains. You can implement noindex using an HTML meta tag or an HTTP response header. According to Google’s documentation, crawlers must be able to access the content and read the rule, so you must not also block it with robots.txt.
Choose based on each page’s purpose, rather than excluding entire categories
Semrush’s 2024 article lists thank-you pages, test environments, and downloads provided after a form is completed. Its 2023 article also identifies internal search results, login pages, and filtered or sorted versions of product listings as candidates for exclusion. Each situation needs individual assessment; this is not an official checklist every website must follow.
- Thank-you pages: If a page is only shown after a purchase or download is completed, arriving directly from search could confuse users about the process.
- Downloadable resources: If users are meant to complete a form before receiving a resource, consider excluding the resource itself so that search results do not provide a way to bypass the form.
- Test URLs: If their sole purpose is to let the team edit and check content, rather than help the public find information through search, they can be considered for exclusion.
Should a form page use noindex if it has no keywords?
That condition alone is not enough to decide. First, distinguish between the page containing the form and the thank-you or download pages shown after submission. The reasons above explain why you might exclude thank-you and download pages; they do not establish a general rule that form pages with few keywords should be excluded.
For example, an ebook registration page and the page where users obtain the file after completing the form serve different purposes. If the registration page is important and you want readers to find it through search, you should not apply noindex to it simply because the subsequent download page needs to be excluded.
Where to set noindex for HTML pages and other files
According to Google’s documentation, the two implementation methods have the same effect, so you can choose based on the content type:
- Standard HTML pages: Add <meta name=”robots” content=”noindex”> to the head section.
- Non-HTML resources, such as PDFs, images, or videos: Return X-Robots-Tag: noindex in the HTTP response header.
If you use a CMS, its search engine settings may also let you add the meta tag without editing the HTML directly. noindex can be combined with nofollow, but excluding content from indexing does not necessarily require nofollow.
After implementation, make sure Google can actually read the rule
Google does not support specifying noindex in robots.txt. If you block the same URL with robots.txt, Googlebot may not see the rule on the page, and the URL may still appear in search results.
You can use Search Console’s URL Inspection tool to check whether the HTML received by Googlebot contains noindex, then track the page’s indexing status through the Page indexing report. Applying the rule does not immediately remove the page from search results: Google must recrawl it to discover the change. Its official documentation cautions that some pages may not be revisited for several months.
When to use noindex: perspectives from across the web
Should all duplicate pages or pages with only minor differences use noindex?
Not necessarily. A/B test pages with identical content and filtered or sorted variations of the same product listing are all candidates for evaluation: first identify which URLs you want to serve as entry points from search, then consider excluding the remaining versions. Screaming Frog suggests that large numbers of low-value or duplicate URLs may cause index bloat, but this does not mean every similar page necessarily harms rankings.
Does applying noindex to internal or sensitive information keep it confidential?
noindex should not be treated as a confidentiality measure. It tells search engines that support the rule not to index the content; it does not restrict people's access to it. Different search engines may also interpret it differently. Sensitive content, such as user data and internal documents, should have appropriate access restrictions rather than relying solely on the page's absence from search results.
What should you check when an important page does not appear in search results?
First, check whether noindex was applied accidentally, and examine robots.txt, 4XX errors, site structure, and other issues that could prevent crawling or indexing. If the page is intended to serve as an entry point from search, address unintended exclusions rather than retaining noindex just to reduce the number of indexed pages. Reasons for exclusion can vary, so assess each URL individually.
References
- 使用「noindex」禁止 Google 搜尋建立索引 | Google 搜尋中心 | Documentation | Google for Developersdevelopers.google.com — Discusses: Does applying noindex to internal or sensitive information keep it confidential?
- 使用”noindex”等4種方式禁止特定網頁被收錄,提升網站整體seo品質www.awoo.ai · 2018-01-15T09:00:23+00:00 — Discusses: Does applying noindex to internal or sensitive information keep it confidential?
- Google Search Console教學:GSC設定、功能完整教學!welly.tw · 2022-01-03T02:54:11.238Z — Discusses: What should you check when an important page does not appear in search results?
- What Is Noindex Used for? An Overview + Best Practiceswww.semrush.com · 2024-07-31T12:14:00Z — Source used in the article above.
- Conducting an SEO Content Audit: 6 Things You Might Be Overlookingwww.screamingfrog.co.uk · 2021-11-15T13:37:48+00:00 — Discusses: Should all duplicate pages or pages with only minor differences use noindex?
- 什麼是技術性搜尋引擎最佳化(Technical SEO)?基礎知識和 10 項最佳方法tenten.co · 2023-07-30T15:42:50.000Z — Source used in the article above.
- AI Overviews: What Are They & How to Optimize for Themwww.semrush.com · 2024-05-31T14:20:00Z — Discusses: What should you check when an important page does not appear in search results?
- How to Get Google to Instantly Index Your New Websiteneilpatel.com · 2021-02-16T15:00:00+00:00 — Discusses: Should all duplicate pages or pages with only minor differences use noindex?
- High number of excluded (not indexed) pages in Google Search Consolewww.warriorforum.com — Discusses: What should you check when an important page does not appear in search results?
- What is technical SEO? Basics and best practiceswww.semrush.com · 2023-02-01T15:56:00Z — Discusses: Should all duplicate pages or pages with only minor differences use noindex?

