Why keep some pages out of the search index, and which pages are suitable for exclusion?

ProcrustesPR

Some pages need to remain on a website but are not suitable entry points from search. Reasons for excluding them from the index may include keeping duplicate content out of search results, directing visitors through a specific process to obtain information, or preventing unfinished pages from appearing in search. Pages worth reviewing include thank-you pages, duplicate versions, test pages, and advertising landing pages not intended to attract organic search traffic. However, fewer indexed pages are not necessarily better, and removing pages from the index does not guarantee higher rankings. Decisions still depend on each page’s purpose and the website’s goals.

ProcrustesPRProcrustesPRHow can I tell whether an SEO tactic breaks spam rules?

A useful page does not necessarily need to appear in search results

For example, a print-friendly version makes printing easier for readers, but if its content is identical to the standard version, there may be no need for both to appear in search results. Neil Patel’s article uses this example to recommend keeping the standard version indexed and excluding the print-friendly version. The point is to choose which version visitors find through search, not to dismiss the value of the printing feature.

When too many URLs that should stay out of the index are indexed, the result is what Screaming Frog calls “index bloat.” The article notes that this is particularly common on large websites. It is worth checking for, but that does not mean all large websites should exclude substantial numbers of pages from the index.

Start with pages tied to specific steps, duplicate pages, and unfinished pages

  • Thank-you pages shown after an action: If a thank-you page hosts an ebook that visitors should receive only after submitting a form or making a purchase, visitors arriving directly from search may bypass the required steps and potentially distort lead data. This reasoning applies to thank-you pages that host such content; the page’s name alone is not enough to decide whether to exclude it from the index.
  • Different versions with identical content: As with print-friendly versions, consider whether every A/B test page needs to be indexed when the versions differ only in visual design and share the same main content.
  • Unfinished pages or pages still being tested: If they are not yet ready for the public, consider excluding them from the index so they do not become entry points from search.
  • Advertising landing pages: These are candidates for exclusion only if they are not intended to attract organic search traffic. They should not automatically be excluded simply because they are used for PPC advertising.

Temporarily unavailable products can still have search value

Out-of-stock products are an important counterexample. Google’s guidance explicitly advises against using the removals tool to remove out-of-stock products from search. Instead, it recommends keeping the pages accessible and marking the products as out of stock. Customers still need firsthand product information, and information on other websites may be incomplete or incorrect.

When deciding whether to exclude a page from the index, consider not only whether it can lead to an immediate purchase, but also whether it still answers visitors’ questions.

Excluding a page from the index and blocking crawling are different decisions

Once you have decided that a page should not appear in search results, you need to understand what each tool does. Semrush explains that robots.txt controls crawling but does not directly determine whether a URL can be indexed. Even when crawling is blocked, search engines may still discover and index the URL through external links. If the goal is to exclude a page from the index, the article recommends using noindex; Disallow should not be treated as an equivalent alternative.

ProcrustesPRProcrustesPRShould I rewrite or update an old article that does not rank?

Excluding pages from the search index: perspectives from across the web

Should all community member profiles be indexed by search engines?

Not necessarily. If a website has many community profiles, it can assess which ones are worthwhile entry points from search. Neil Patel’s article describes how Moz excluded some low-point member profiles to make more active members easier to find. However, this was an approach taken by a particular website, not a reason for other communities to apply the same threshold, nor a guarantee of increased traffic.

Do internal search results pages need to be indexed?

Not necessarily. It depends on whether those results pages are suitable entry points from search. If internal search generates URLs for different queries, you can assess those pages individually. Welly shares a case in which a website applied noindex to its internal search results URLs out of concern that URLs generated by malicious queries might be indexed. This addressed that website’s particular risk; it does not mean every website must exclude internal search pages.

Does having fewer indexed pages than the website’s total page count indicate a problem?

Not necessarily. Some pages may intentionally be kept out of search results, so an indexed page count below the total page count does not, on its own, indicate a problem. Neil Patel’s article advises against panicking immediately. A more useful approach is to first identify which pages are intended to attract search traffic, then check whether those important pages are indexed.

ProcrustesPRWill Reposting Social Content on My Website Cause a Duplicate-Content Penalty?

References

  1. 暫停或停用網站 | Google 搜尋中心  |  Documentation  |  Google for Developersdevelopers.google.com — Source used in the article above.
  2. Want More Traffic? Deindex Your Pages. Here’s Why.neilpatel.com · 2018-03-06T19:00:42+00:00 — Discusses: Should all community member profiles be indexed by search engines?
  3. Crawlability & Indexability: What They Are & How They Affect SEOwww.semrush.com · 2016-11-01T13:00:00Z — Source used in the article above.
  4. Conducting an SEO Content Audit: 6 Things You Might Be Overlookingwww.screamingfrog.co.uk · 2021-11-15T13:37:48+00:00 — Source used in the article above.
  5. Noindex是什麼?了解3種設定方法與使用時機,讓你提升網站排名!welly.tw · 2024-11-11T08:07:06.770Z — Discusses: Do internal search results pages need to be indexed?
  6. How to Get Google to Instantly Index Your New Websiteneilpatel.com · 2021-02-16T15:00:00+00:00 — Discusses: Does having fewer indexed pages than the website’s total page count indicate a problem?

Related articles

How can I tell whether an SEO tactic breaks spam rules?Should I rewrite or update an old article that does not rank?Will Reposting Social Content on My Website Cause a Duplicate-Content Penalty?What types of URLs are there, and what are they used for?What are the common mistakes, limitations, and costs of using subdomains?Can two URLs point to the same website?How to improve on-site content and structure: audit pages first, then address content gaps and internal linksHow can you use multiple domains to build content and traffic while assessing the risks?
ProcrustesPR
ProcrustesPR