There are two main ways to set up noindex: add a meta tag to the head section of an HTML page, or configure the server to return an X-Robots-Tag HTTP response header. After deployment, you also need to confirm that search engines can crawl the URL and read the directive; simply checking a setting in the admin interface is not enough to verify it. The following focuses on verifying that Google can read the directive. Other search engines may interpret noindex differently, so this does not guarantee that all search services will handle it the same way.
HTML pages: place the tag in the head section
For a standard web page, you can add <meta name=”robots” content=”noindex”> to the head section of the HTML. This is an indexing directive for the page, not body text intended for readers. If you want to target only Google’s web crawler, Google’s documentation provides this syntax: <meta name=”googlebot” content=”noindex”>. These tags target different crawlers, so choose according to your needs.
If you use a CMS and cannot easily edit the HTML directly, look for the system’s search engine settings or SEO plugin features, then save and publish your changes. The location of these settings varies by system. What matters is not the button’s label, but whether the publicly served page actually contains the tag. Do not assume Googlebot can see noindex just because the option is checked in the editing interface.
Resources such as PDFs: use an HTTP response header
For non-HTML resources such as PDFs, images, and videos, you can use X-Robots-Tag: noindex. According to Google’s documentation, an HTTP response header can also serve as an alternative to a page’s meta tag, so its use is not limited to non-HTML files.
- HTML article pages: add a meta tag directly to the head section.
- PDF files: configure the server to add X-Robots-Tag to the file’s HTTP response.
- Header-based deployment: verify the actual response headers, not just the page’s HTML.
Server-level configuration is more complex. Semrush’s tutorial warns that even a small syntax error can cause a site to malfunction and recommends getting help from a developer. Its Apache example applies to PDFs across the entire site; do not copy it into another environment without checking its scope.
Verification: can Googlebot access the resource and see the directive?
Google must crawl a resource before it can read its meta tag or HTTP header. If robots.txt blocks Googlebot from accessing the URL, it may also prevent Googlebot from seeing noindex. To let Google read the directive, remove that block first rather than treating blocked crawling as confirmation that the directive has been read.
- Confirm that robots.txt does not block the target URL, so Googlebot can access it.
- For HTML tags, use Search Console’s URL Inspection tool to examine the HTML received by Googlebot and confirm that noindex is visible.
- For header-based settings, confirm that the target resource’s HTTP response includes X-Robots-Tag: noindex. You can also use the Page indexing report to monitor pages where Googlebot has detected noindex.
This verifies whether the setting is being served and can be read; it does not mean that search results have already updated to reflect it. Diagnosing cases where noindex has been deployed but the indexing status has not yet changed is a topic for a separate article.
Setting up and verifying noindex: perspectives from across the web
Should all duplicate pages generated by a CMS be set to noindex?
Not necessarily. First check whether the CMS generates identical content at different URLs, then decide which pages need to be discoverable through search. Neil Patel's article lists canonical tags, noindex, and rewriting content as separate options, and considers noindex particularly suitable for pages you do not want indexed, such as print versions. You do not need to apply the same approach to every duplicate URL.
Should noindex be applied to all paginated content?
No—not as a blanket rule. First assess the site's content and navigation needs. Semrush argues that noindex on paginated pages may prevent search engines from discovering new content through links and may also affect the flow of link equity. For pagination, it recommends self-referencing canonical tags on each page, clear navigation between pages, and a consistent URL structure instead.
When should you use noindex for pages intended only for sharing by link or collaboration?
First decide whether you want people to find the page through search. According to a report on guidance updated in October 2026, pages intended for direct sharing, collaboration, or use as tools can use noindex by default. The report also mentions blocking crawling with robots.txt, but that is a different approach from allowing Google to read noindex. If you choose noindex, you must not also block Googlebot from accessing that URL.
References
- 使用「noindex」禁止 Google 搜尋建立索引 | Google 搜尋中心 | Documentation | Google for Developersdevelopers.google.com — Discusses: When should you use noindex for pages intended only for sharing by link or collaboration?
- Google Updates Spam Guide For Shared Pages, Generated Content & Appswww.searchenginejournal.com · 2026-10-26T20:57:22+00:00 — Discusses: When should you use noindex for pages intended only for sharing by link or collaboration?
- Duplicate Content: 5 Myths and 5 Facts About How It Impacts SEOneilpatel.com · 2022-02-22T21:00:00+00:00 — Discusses: Should all duplicate pages generated by a CMS be set to noindex?
- Pagination and SEO: A Complete Guide to Best Practiceswww.semrush.com · 2024-01-26T14:45:00Z — Discusses: Should noindex be applied to all paginated content?
- What Is Noindex Used for? An Overview + Best Practiceswww.semrush.com · 2024-07-31T12:14:00Z — Source used in the article above.

