To allow Google to crawl a page without showing it in search results, use noindex and make sure Googlebot can read the directive; blocking it with robots.txt alone is not enough. robots.txt controls crawling—it does not guarantee that a URL will not be indexed. If content should be accessible only to certain users, use login or password protection rather than merely hiding it from search results.
Blocking crawling does not prevent indexing
Crawling means a search engine accesses a page; indexing means it adds a page or URL to its search index. These are different processes. Even if Google cannot crawl a page blocked by robots.txt, it may still discover its URL through links from other websites and display that URL in search results without a page description.
So, if you have decided to keep a form page out of search results, adding Disallow alone does not complete the job. robots.txt is useful for managing crawl traffic, such as preventing large numbers of requests from overloading a server, but it cannot ensure that a URL stays out of search results.
Choose based on whether the page should stay available and public
- The page will remain available to users, but should not be found through search: Set noindex using the page’s robots meta tag or the X-Robots-Tag HTTP response header.
- The content is restricted to authorized users: Use login or password protection. Keeping content out of search results does not mean that only authorized users can access it.
- The content no longer needs to be available: You can remove the page entirely. There is no need to keep content just to apply an indexing exclusion.
For example, if you simply do not want people to find a public form through search, consider noindex. If private content is involved, focus on restricting access, not just on indexing settings.
Before using noindex, make sure crawlers can read it
A common conflict occurs when the same page is both blocked by robots.txt and marked noindex. If Googlebot cannot crawl the page, it cannot read the noindex directive on it. Do not assume that combining the two settings provides extra protection.
If a URL is already indexed and blocked by Disallow, you can remove the block for that URL so Google can read noindex. If allowing access is inappropriate, use password protection or remove the page instead. After making the changes, use the URL Inspection tool to check whether Googlebot can access the page and whether the page contains noindex.
Links may still be discovered on a page that is not indexed
Whether a page is indexed and whether a search engine follows its links are separate questions. Asking Google not to display a page therefore does not mean asking it to ignore the links on that page.
But do not make a noindex page the only route to important content. A 2024 Semrush article cautions that search engines may not follow links on these pages indefinitely, so relying on them over the long term may make other pages harder to discover. Important pages should therefore have other internal links pointing to them.
Preventing indexing: perspectives from across the web
What exactly does “do not index this page, but still follow its links” mean?
It means keeping the current page out of search results without asking search engines to ignore the links on it. Welly SEO suggests considering noindex, follow for similar or less important pages that contain valuable links. However, a 2024 Semrush article cautions that search engines may not follow links on these pages indefinitely. Important pages should have other internal links pointing to them rather than relying on such a page as their only entry point.
How can you temporarily hide a URL from Google search results?
Consider using the Removals tool in Search Console to temporarily hide a URL within a property you own. Keep in mind that URL paths are case-sensitive. HTTP/HTTPS and www/non-www variants are matched, but other subdomains are not. Temporarily hiding a URL does not permanently prevent indexing. If a page needs to remain available over the long term without appearing in search results, further measures are needed.
Before applying exclusion settings, how can you avoid excluding important pages by mistake?
First, confirm exactly which URLs the rules cover, especially when applying settings in bulk, so you do not also exclude pages whose search traffic you want to retain. awoo cautions that incorrect settings may affect existing rankings. After applying the settings, use Google’s URL Inspection tool to confirm whether the page is accessible and check that it contains the intended noindex directive. Blocking resources the page depends on may also hinder Google’s understanding of the page.
References
- 網頁程式開發人員的搜尋引擎最佳化 (SEO) 指南 | Google 搜尋中心 | Documentation | Google for Developersdevelopers.google.com — Discusses: How can you temporarily hide a URL from Google search results?; Before applying exclusion settings, how can you avoid excluding important pages by mistake?
- robots.txt 簡介與指南 | Google 搜尋中心 | Documentation | Google for Developersdevelopers.google.com — Discusses: Before applying exclusion settings, how can you avoid excluding important pages by mistake?
- Spam Guide Update, Restaurant Controls & AI Mode Revenue – SEO Pulsewww.searchenginejournal.com · 2026-10-31T12:00:00+00:00 — Source used in the article above.
- 使用”noindex”等4種方式禁止特定網頁被收錄,提升網站整體seo品質www.awoo.ai · 2018-01-15T09:00:23+00:00 — Discusses: Before applying exclusion settings, how can you avoid excluding important pages by mistake?
- Robots.txt Explained: Syntax, Best Practices, & SEOwww.semrush.com · 2018-07-10T11:00:00Z — Source used in the article above.
- How to Create and Optimize Your Robots.txt Fileneilpatel.com · 2026-05-26T19:00:00+00:00 — Source used in the article above.
- Noindex是什麼?了解3種設定方法與使用時機,讓你提升網站排名!welly.tw · 2024-11-11T08:07:06.770Z — Discusses: What exactly does “do not index this page, but still follow its links” mean?
- What Is Noindex Used for? An Overview + Best Practiceswww.semrush.com · 2024-07-31T12:14:00Z — Discusses: What exactly does “do not index this page, but still follow its links” mean?
- How to hide links from search engineswww.warriorforum.com — Discusses: How can you temporarily hide a URL from Google search results?



