For pages intended solely for direct sharing, collaboration, or practical functions—not discovery through public search—Google’s guidelines recommend applying noindex or blocking crawling by default in robots.txt. However, the two have different effects: if the goal is to keep a URL out of search results, use noindex and allow Googlebot to crawl the page and read the rule. Blocking crawling through robots.txt alone does not guarantee that the URL will stay out of search results.
Start with the purpose—not every shareable page needs to be excluded
A section Google added to its guidelines in October 2026 addresses services that let users create shareable pages, generated outputs, or interactive tools. It calls for first considering whether the content is meant to be found through search engines. What matters is the page’s purpose, not whether it has a share button. Content should not automatically be excluded just because it is generated.
- If a generated results page exists only to share results with people who have the link, and is not intended for public search, this recommendation applies.
- If a collaboration tool page is intended only for use by participants, its default settings should likewise reflect that it is not meant to be found through search.
- If a page is intended to be discovered through search, the exclusion recommendation above should not be applied merely because the page can be shared.
Decide what you want to prevent: indexing or crawling
The official guidelines list two approaches: set noindex using a robots meta tag or an X-Robots-Tag HTTP response header, or block crawling in robots.txt. These are not different places to put the same rule; they make different requests of search engines.
- noindex: Prevents a page from appearing in search results, but Googlebot must be able to crawl the page and read the rule for it to take effect.
- robots.txt: Prevents Google from crawling blocked pages. If other websites link to a URL, Google may still discover it, and the URL may appear in search results.
Blocking crawling therefore does not mean a URL has been excluded from search results. If you rely on noindex, you cannot also block Googlebot through robots.txt and expect it to read the noindex rule on the page. Although the guidelines list both options, choose based on your goal: preventing crawling or keeping the URL out of search results.
After setup, check whether indexing and search activity match the service’s purpose
Google also recommends verifying your site or subdomain in Search Console to receive email notifications about manual actions and security issues. Regularly review the Page indexing and Search performance reports, too. Look for unusual spikes in indexed pages, impressions, or queries that do not align with the platform’s purpose.
Platforms can also track the number of newly published public pages, watch for a small number of users generating large numbers of indexable URLs, and set up alerts for unusual activity. This monitoring helps identify changes that warrant investigation, but a single increase in volume does not by itself prove spam or a policy violation.
Indexing pages intended for direct sharing: perspectives from across the web
How should indexing be handled for new users’ pages?
Google recommends initially adding a noindex robots meta tag to posts by new users who have not yet established a reputation, then allowing search engines to index those posts once the users have built a reputation. This is a way to reduce the risk of user-generated spam. For pages intended solely for direct sharing, whether the page is meant to be discoverable through public search must still be considered separately; user reputation alone should not determine the decision.
References
- 避免您的網站使用者產生的垃圾內容 | Google 搜尋中心 | Documentation | Google for Developersdevelopers.google.com — Discusses: How should indexing be handled for new users’ pages?
- Spam Guide Update, Restaurant Controls & AI Mode Revenue – SEO Pulsewww.searchenginejournal.com · 2026-10-31T12:00:00+00:00 — Source used in the article above.
- Google Updates Spam Guide For Shared Pages, Generated Content & Appswww.searchenginejournal.com · 2026-10-26T20:57:22+00:00 — Source used in the article above.


