To keep a page out of search results, set noindex on the page itself rather than merely adding nofollow to links pointing to it. For duplicate content, first determine whether multiple versions need to remain available, then decide whether to specify a canonical URL or consolidate the content. For example, if a restaurant website’s location and budget filter pages are only for users to interact with and bookmark—not to find through search—choose page-level noindex rather than nofollow on links. Having duplicate content does not mean it is better to let search engines index every version.
nofollow on links alone cannot keep pages out of the index
noindex means “do not include this page in the index.” Tenten’s technical SEO overview also warns that leaving development-stage noindex directives in place can exclude important pages after a site goes live. A restaurant website should therefore handle pages according to their intended purpose:
- Cuisine listings intended to serve as entry points from search: check for mistakenly applied noindex directives.
- Location and budget filter pages that should not appear in search results: set noindex on the target pages rather than only modifying the links leading to them.
A respondent in a Stack Exchange discussion chose noindex because adding nofollow to their own links would not stop others from linking to the page. This is a practical consideration, not an official rule. In the combination noindex, follow, it is noindex that excludes the page from indexing; the combination does not guarantee that search engines will continue to follow links indefinitely.
robots.txt controls crawling; it does not guarantee exclusion from search results
Semrush explains that even if a URL is blocked by a disallow rule in robots.txt, Google may still index it and display it in search results if links to it exist elsewhere. “Preventing bots from crawling” and “keeping a page out of search results” therefore cannot be treated as the same thing.
If the goal is to keep restaurant filter pages out of the index, setting robots.txt blocking rules alone does not mean the job is done. Advice that treats blocking crawling as a direct way to prevent indexing overlooks this limitation and should not be accepted uncritically.
Choose a primary version of duplicate content rather than deleting it indiscriminately
Neil Patel’s article notes that duplicate versions, such as store pages or print-friendly pages, may serve legitimate purposes. The issue is that search engines may not know which version the website wants them to prioritize. If multiple URLs provide identical or similar content, consider using canonical to specify the primary URL. If separate versions do not need to remain available, consider consolidating the content and using 301 redirects.
Check when each approach applies. For multi-regional websites offering similar content in the same language, Google recommends selecting a preferred version and using canonical together with hreflang. This does not mean ordinary restaurant location filter pages also need hreflang.
Google’s ecommerce guidance also recommends noindex for empty categories; it recommends returning a 404 only when an empty category has been automatically removed from internal search and is no longer available for browsing. Pages that users can still interact with should not simply be treated as pages that no longer exist.
Handling unwanted indexing and duplicate content: perspectives from across the web
Should a sitemap include filter pages you do not want to appear in search results?
No. Include pages you want people to find through search in the sitemap, and leave out individual pages you do not want to appear in search results. Using a new blog as an example, Yoast recommends retaining relevant pages such as posts while excluding thank-you pages shown after someone subscribes. This is a page-by-page decision, rather than a recommendation to remove an entire type of page from the sitemap.
If tag listings overlap with multiple archive pages, can they all point to the same canonical URL?
Not as a blanket rule. In a public discussion, a contributor noted that a tag page may list multiple posts that overlap with one or more archive pages, so there may not be a single corresponding canonical URL. First compare the actual content and purpose of each listing, then determine whether there is a suitable primary version. Do not arbitrarily choose a page simply because some posts appear in more than one listing.
If new URLs for rewritten content have not yet been indexed, should they immediately replace all the old pages?
There is no need to rush into replacing everything. If the new pages have been crawled but not indexed, and their content is genuinely better, the discussion suggests incorporating their strengths into the already indexed old pages—or, conversely, removing the old pages and redirecting them to the new ones. Test each approach on a small number of pages before deciding whether to apply it more broadly. This is practical advice for rewritten pages, not a guarantee that either approach will work.
Before a website goes live, how can you avoid excluding pages you want people to find through search?
Check whether important pages still carry development-stage noindex directives. If a directive was left in place by mistake, remove it first, then request reindexing through Google Search Console. This check is particularly relevant to websites that used noindex in a testing environment: if the directive carries over to the live site, content intended to be discoverable through search may also be excluded.
References
- 管理多地區和多語言版本的網站 | Google 搜尋中心 | Documentation | Google for Developersdevelopers.google.com — Source used in the article above.
- 電子商務網址結構的最佳做法 | Google 搜尋中心 | Documentation | Google for Developersdevelopers.google.com — Source used in the article above.
- Internal links for a websitewebmasters.stackexchange.com — Discusses: If tag listings overlap with multiple archive pages, can they all point to the same canonical URL?
- What Are Canonical Tags?neilpatel.com · 2022-09-10T07:00:00+00:00 — Source used in the article above.
- 技術性SEO:打造高品質網站,提升搜尋引擎排名tenten.co · 2024-12-25T18:03:57.000Z — Discusses: Before a website goes live, how can you avoid excluding pages you want people to find through search?
- What is an XML sitemap and why should you have one?yoast.com · 2026-03-11T13:52:01+00:00 — Discusses: Should a sitemap include filter pages you do not want to appear in search results?
- Internal links for a websitewebmasters.stackexchange.com — Source used in the article above.
- Robots.txt Explained: Syntax, Best Practices, & SEOwww.semrush.com · 2018-07-10T11:00:00Z — Source used in the article above.
- SEO Localization: Tactics for Targeted Global Online Visibilitywww.semrush.com · 2023-11-09T10:27:00Z — Source used in the article above.
- Why did google stop indexing only my Blog pages?localsearchforum.com — Discusses: If new URLs for rewritten content have not yet been indexed, should they immediately replace all the old pages?

