To keep a specific page out of search results, first determine whether you control it and whether it still needs to be publicly accessible. For pages you manage and want to keep, use noindex and allow search engines to read the directive. For pages that need restricted access, use password protection. For permanently deleted pages, return an appropriate status code. Disallow in robots.txt only restricts crawling; it does not prevent indexing. If the page belongs to someone else’s website, you cannot exclude it simply by applying settings on your own site.
Keeping a page without having it appear in search: use noindex
Noindex tells search engines not to index a specific page. It suits situations where content needs to remain on the website but should not serve as an entry point from search. There is no need to take the entire page offline for that purpose. Both Welly’s 2024 article and Semrush’s 2024 explanation support using this type of directive to control page indexing.
- For example, if content about a past event needs to remain available as a record but should not appear in search results, consider adding noindex to that page.
- Neil Patel’s 2025 article also recommends considering noindex for URLs generated by filtering product categories by criteria such as size.
However, this does not mean all paginated pages should also be excluded. The same article notes that search engines may discover new content through pagination. Filtered URLs and paginated pages that lead to other content should be assessed separately.
Already indexed: do not let robots.txt block access to noindex
Even when crawling is prohibited, Google may still discover a page’s URL through links on other websites and include it in search results. If a URL continues to appear, simply adding more Disallow rules is not the answer.
Neil Patel’s 2026 article suggests several approaches for pages that are already indexed but blocked from crawling:
- If you want to keep the page: remove the Disallow rule blocking it so Google can read the noindex directive.
- If you do not want public access: use password protection instead.
- If the page is no longer needed: remove it. His 2025 article recommends returning a 404 or 410 status code for permanently deleted pages.
The key is to match the approach to the page’s status rather than stack settings that interfere with one another. Adding a directive also does not mean search results have already been updated.
Excluding a page from search results does not restrict who can read it
Noindex addresses indexing; password protection addresses access. If the goal is to prevent anyone other than specified users from reading the content, choose access restrictions rather than merely hoping the page will not be found through search. Clarifying this distinction helps avoid confusing “do not index” with “not publicly accessible.”
Your settings cannot exclude pages on someone else’s website
If the target is a review or report on another website, your own noindex or robots.txt settings cannot control that page. In its 2017 discussion of negative online reviews, awoo also cautioned that publishing large volumes of positive articles, paying for sponsored content, or creating multiple content platforms could not reliably keep negative reviews out of the first two pages of search results. This was an analysis of the limitations of ranking manipulation at the time, not a statement of current removal rules. “Trying to push a page down in the rankings” must not be confused with “removing a specific page from the index.”
Keeping pages out of search results: perspectives from across the web
Can I use noindex if I only want to keep content out of AI Overviews?
Yes, but noindex affects more than AI Overviews. Semrush’s 2024 article notes that it also removes the page from regular search results. If you want to retain visibility in regular search, weigh that impact first.
Can robots.txt completely exclude a page embedded in an iframe from search results?
Robots.txt should not be treated as a guarantee of complete exclusion. Although some discussions recommend blocking pages embedded in iframes, other sources note that Google may still discover and index their URLs through other links even when crawling is prohibited. If you control the embedded page and want to keep it, consider adding noindex and allowing Google to read the directive.
References
- Best way to stop a country dropdown from affecting google SEOwebmasters.stackexchange.com — Discusses: Can robots.txt completely exclude a page embedded in an iframe from search results?
- Crawl Depth: 10-Point Guide for SEOsneilpatel.com · 2025-04-02T16:00:00+00:00 — Source used in the article above.
- 網路負評處理/消毒大作戰!SEO業界專家怎麼做?www.awoo.ai · 2017-08-14T08:49:05+00:00 — Source used in the article above.
- AI Overviews: What Are They & How to Optimize for Themwww.semrush.com · 2024-05-31T14:20:00Z — Discusses: Can I use noindex if I only want to keep content out of AI Overviews?
- Noindex是什麼?了解3種設定方法與使用時機,讓你提升網站排名!welly.tw · 2024-11-11T08:07:06.770Z — Source used in the article above.
- How to Create and Optimize Your Robots.txt Fileneilpatel.com · 2026-05-26T19:00:00+00:00 — Discusses: Can robots.txt completely exclude a page embedded in an iframe from search results?
- What Is Googlebot? How Google's Web Crawler Workswww.semrush.com · 2024-08-23T14:32:00Z — Discusses: Can robots.txt completely exclude a page embedded in an iframe from search results?



