Search results and tag pages should not all be blocked with robots.txt by default. If the goal is to reduce crawler access to large numbers of problematic dynamic URLs, consider blocking crawling. If the goal is to keep pages out of search results, use noindex and allow crawlers to read the directive. Assess tag pages separately based on their purpose and content value; being a listing page or having a high bounce rate is not enough to justify blocking them.
Search results pages: first ask whether you want to stop crawling or prevent indexing
Google’s documentation suggests considering robots.txt if you discover that crawlers are accessing problematic URLs. These may include URLs for search results, sorting, and filtering, as well as URLs that could be generated without limit. This advice addresses crawling problems; it does not mean all dynamic pages must be blocked.
- If you do not want pages to appear in search results: Use noindex, but do not also use robots.txt to prevent crawlers from accessing those pages, or they will be unable to read the directive.
- If you do not want large numbers of URLs to keep being crawled: Consider using the Disallow directive in robots.txt. Address any indexing issues first; stopping crawling does not mean pages have been removed from the index.
For example, on-site search results may still be useful to visitors without being suitable landing pages for people arriving from external search engines. Whether to prevent indexing and whether to block crawling should therefore be considered separately.
Tag pages: assess whether they help visitors find what they need
Tag pages do not necessarily need the same settings as search results pages generated by arbitrary queries. A 2019 awoo article noted that tags help users find content, but tag pages typically have less content and may be less competitive than category, article, or product pages. This is content strategy advice, not a Google rule requiring tag pages to be blocked.
For an e-commerce tag page listing related products, for example, check whether the tag description fits the products, whether the products are appropriately grouped, and whether the page makes it easier for visitors to find what they need. Assess what the page actually does for visitors rather than deciding whether to keep it based on its URL alone.
An e-commerce site owner asked about tag pages that had introductions and explanations of product uses, yet had a bounce rate above 90%. This was only the site owner’s account and concern. There was no evidence that Google had judged the pages to be low quality or lowered their rankings as a result; that figure should not be treated as a threshold for blocking pages.
Resolve indexing issues before dealing with large numbers of pages that offer no value
A 2021 Screaming Frog article described robots.txt as an option when crawlers need to be blocked from an entire path or large numbers of pages. The article focused particularly on crawl resources for large sites and emphasized resolving indexing issues first.
A 2025 awoo report also presented a case involving more than 100 million pages: tag pages with no value were set to noindex, while API pages were blocked with robots.txt. This case illustrates that different tools can be used for different pages; it does not support blocking all tag pages indiscriminately.
Search results and tag pages: perspectives from across the web
Can on-site search be exploited for spam?
Yes, this is possible. Spammers may use search boxes to generate URLs containing spam keywords, then create large numbers of links to those pages. If the pages can be indexed, this may create quality issues, and the pages may even appear in search results. Google staff have said that pages may be flagged as hacked in such cases; this does not mean all on-site search pages have been hacked.
What if search results pages are unsuitable as landing pages, but you still want crawlers to read their links?
Consider using noindex on the pages while allowing crawlers to access them and follow their links. One practical guide uses on-site search results as an example: results from a previous visitor’s query may be unsuitable as a landing page for a new visitor, while the site still wants crawlers to read the page and its links. This approach requires crawlers to be able to read the directive, so the page must not also be blocked with robots.txt.
Does a high bounce rate mean tag pages should be blocked?
Bounce rate alone is not enough to draw that conclusion. In a public discussion, one respondent argued that even when tag pages have a high bounce rate, some visitors may still continue browsing the site, and therefore recommended keeping them crawlable for now. However, this was a judgment based on their experience, not a guarantee of quality. For e-commerce tag pages, also check whether the descriptions fit the products and whether visitors can use the pages to find relevant products.
References
- Google 搜尋的網址結構最佳做法 | Google 搜尋中心 | Documentation | Google for Developersdevelopers.google.com — Source used in the article above.
- 分類 vs. 標籤 ,兩個重要性超強但常被忽略的 SEO 關鍵www.awoo.ai · 2019-09-05T07:14:22+00:00 — Discusses: Does a high bounce rate mean tag pages should be blocked?
- robots.txt、meta robots設定:讓網頁不被Google搜尋到www.yesharris.com · 2024-08-15T14:20:11+08:00 — Discusses: What if search results pages are unsuitable as landing pages, but you still want crawlers to read their links?
- Google May Treat Search Box Pages As A Site Quality Issuewww.searchenginejournal.com · 2026-07-31T11:24:14+00:00 — Discusses: Can on-site search be exploited for spam?
- Conducting an SEO Content Audit: 6 Things You Might Be Overlookingwww.screamingfrog.co.uk · 2021-11-15T13:37:48+00:00 — Source used in the article above.
- 【Search Central Live Deep Dive】Day1:SEO is Dead?AI 搜尋時代的 SEO 新解讀!|awoo 活動實記www.awoo.ai · 2025-07-24T06:56:02+00:00 — Source used in the article above.
- Google Says Why It May Ignore Robots.txt And Negatively Impact SEOwww.searchenginejournal.com · 2026-07-25T10:17:52+00:00 — Discusses: What if search results pages are unsuitable as landing pages, but you still want crawlers to read their links?
- Search and Tags for robots.txtwebmasters.stackexchange.com — Discusses: Does a high bounce rate mean tag pages should be blocked?
- Search and Tags for robots.txtwebmasters.stackexchange.com — Source used in the article above.



