Use robots.txt to restrict crawling; use meta robots with noindex to keep a page out of Google search results, and make sure Google can crawl that page. These controls operate at different stages. They are not interchangeable, and neither is universally better. robots.txt lets you manage crawl restrictions by path in one place, while meta robots lets you set indexing and other directives for individual HTML pages. The following primarily covers Google’s rules; other search engines may interpret directives differently.
Blocking crawling with robots.txt does not prevent a URL from being indexed
robots.txt can tell crawlers which paths not to crawl, but it cannot guarantee that a URL will not be indexed. Google’s official documentation explains that even when Google cannot read a page, it may still discover its URL through links from other websites and use information such as the anchor text of those links to generate a search result title.
Before choosing an approach, clarify your goal: do you want to stop crawlers from reading the content, or keep the URL out of search results? If you do not want a batch of pages indexed, blocking them with Disallow alone will not reliably exclude them from the index.
meta robots controls individual pages, and crawlers must be able to read noindex
On an HTML page, you can place a noindex meta tag in the head section, for example: <meta name=”robots” content=”noindex”>. According to Google’s official documentation, once Googlebot crawls the page and reads this rule, Google will exclude the page from search results, even if other websites still link to it.
For noindex to take effect, do not block the page with robots.txt. The issue is not which directive takes priority: a blocked crawler cannot see the tag on the page at all. A website can use both controls, but they must be configured according to each page’s purpose.
Choose based on your needs, not just ease of configuration
- Keeping shopping cart and checkout pages out of search results: Consider noindex while keeping the pages crawlable. This is a suggestion for a particular use case, not a mandatory rule for all e-commerce pages.
- Restricting crawling of certain paths: Use robots.txt to manage these restrictions centrally. Large websites can also adjust restrictions based on server load and crawling needs, but robots.txt is not a substitute for excluding pages from the index.
- Keeping non-HTML resources, such as PDFs and images, out of the index: Use the HTTP response header X-Robots-Tag: noindex. Google states that setting noindex through a header has the same effect as using a meta tag.
After setup, wait for Google to crawl the page again
Adding noindex does not make a page disappear from search results immediately. Google must crawl the page again to see the new rule. Its official documentation notes that Google may not revisit some pages for several months. You can request another crawl through the URL Inspection tool, and should also check that robots.txt is not blocking the page. Google does not support specifying noindex in robots.txt, so adding noindex to that file will not exclude a page from the index.
meta robots and robots.txt: perspectives from across the web
How should you configure a page if you do not want it indexed but still want search engines to follow its links?
Consider setting noindex on the page without adding a page-level nofollow directive. This indicates that you do not want the page indexed but allows links on it to be followed. For example, internal search results pages may not be suitable landing pages for search traffic, yet may still contain links worth crawling. This choice depends on the page's purpose and does not guarantee that linked pages will be crawled or indexed.
Is noindex enough to keep test pages or an admin area private?
Do not treat noindex as access protection.Consider noindex if you do not want test pages, login pages, or admin URLs to appear in search results. However, important pages or files that need to remain confidential should also be password-protected. Search engines may interpret directives differently, and restricting indexing is a different requirement from restricting access.
If many pages need noindex, must you edit each one manually?
Not necessarily. Each target page needs noindex, but you do not have to edit each page's code manually. Depending on your website's system, you can use page settings or have code automatically add noindex to pages in each section. After setup, confirm that crawlers can access the target pages so they can read the directive. Different search engines may also interpret it differently.
References
- 使用「noindex」禁止 Google 搜尋建立索引 | Google 搜尋中心 | Documentation | Google for Developersdevelopers.google.com — Discusses: Is noindex enough to keep test pages or an admin area private?; If many pages need noindex, must you edit each one manually?
- 影響 Google 搜尋中的標題連結 | Google 搜尋中心 | Documentation | Google for Developersdevelopers.google.com — Source used in the article above.
- robots.txt、meta robots設定:讓網頁不被Google搜尋到www.yesharris.com · 2024-08-15T14:20:11+08:00 — Discusses: How should you configure a page if you do not want it indexed but still want search engines to follow its links?; If many pages need noindex, must you edit each one manually?
- robots.txt是什麼? robots.txt設定、用途與使用範例分享!welly.tw · 2022-01-13T05:20:36.169Z — Discusses: How should you configure a page if you do not want it indexed but still want search engines to follow its links?
- 用 robots.txt、noindex 限制搜尋引擎抓網站資料www.da-vinci.com.tw · 2022-09-11T23:38:01+08:00 — Discusses: Is noindex enough to keep test pages or an admin area private?; If many pages need noindex, must you edit each one manually?



