What is the difference between HTTP robots headers and HTML meta tags?

ProcrustesPR

The main differences are where you set the directives and which resources they apply to—not how effective they are. A robots meta tag goes inside an HTML page; X-Robots-Tag goes in the HTTP response headers and applies not only to web pages but also to non-HTML resources such as PDFs, images, and videos. According to Google’s documentation, setting noindex through either method has the same effect. Choose based on the content type and how your website is configured.

ProcrustesPRProcrustesPRHow Long Does It Take to Evaluate SEO Results?

Where to set directives: the page’s head section or HTTP response headers

The recommended location for a robots meta tag is the page’s head section. For example, adding <meta name=”robots” content=”noindex”> asks search engines that support this directive not to index the page. This method works well for configuring individual HTML pages. If you use a CMS, you may also be able to add the tag through its search engine settings or another mechanism, without editing the source code directly.

X-Robots-Tag does not go inside the HTML content. Instead, it is returned with the HTTP response for that URL. For example, the response header X-Robots-Tag: noindex likewise asks search engines not to index the resource. An HTTP header is not the HTML head section: these are two different places to set directives.

Use cases: individual pages, non-HTML resources, and bulk management

According to Google’s specifications, directives available for robots meta tags can also be set through X-Robots-Tag. The choice therefore depends on which resources you need to manage and where it makes sense to maintain the rules:

  • Individual HTML pages: You can add a robots meta tag to a page, for example to keep that page out of search results.
  • PDFs, images, or videos: These resources cannot contain a robots meta tag the way an HTML page can. Use an X-Robots-Tag response header on the resource itself instead.
  • Bulk configuration for files of the same type: Google’s documentation provides Apache and NGINX examples for applying directives by file type. This lets you add rules to responses for PDFs or images across a site without editing each file’s contents.

X-Robots-Tag can also be used for HTML pages; it is not limited to non-HTML files. It makes applying rules in bulk convenient, but that does not make it more effective than a meta tag at preventing indexing.

A shared requirement: Google must be able to read the directive first

Whichever method you use to set noindex, Googlebot must be allowed to crawl the URL. If you also block access through robots.txt, the crawler cannot read the page’s tags or response headers, and the URL may still appear in search results because of links from other pages. Google also does not support specifying noindex in robots.txt.

For example, to remove an image from the index using an HTTP header, you must allow Googlebot to crawl the image URL; allowing it to access only the article containing the image is not enough. After a directive is added, Google still needs to crawl the resource again to discover the change. Completing the setup therefore does not mean the content will disappear from search results immediately. These effects are based on Google’s documentation; other search engines may interpret the directives differently.

ProcrustesPRProcrustesPRWhere should I start if I want to do SEO myself?

HTTP robots headers and HTML meta tags: perspectives from across the web

Do you need to use both methods on the same page?

Usually not. According to Semrush, either method is sufficient, and using both does not increase the likelihood that crawlers will follow the directives. If a website uses both because of different management workflows, check that the directives actually returned in the page's meta tags and response headers are consistent to avoid confusion during maintenance.

Does setting noimageindex only on an article page guarantee that Google will not index its images?

No. Google explains that noimageindex on a page restricts indexing of images embedded on that page, but if the same image appears on other pages, it may still be indexed through those pages. To address the image itself, you can set noindex in the image's response headers and allow Googlebot to crawl the image URL so it can read the directive.

What should you check if content still appears in Google search results after you set noindex?

First, confirm that the relevant URL is not blocked by robots.txt. Then check the meta tags in the page's source code and the response headers returned by an actual request for the expected directives. If you are dealing with an image, check the settings for both the image and the page containing it separately. Google may not have crawled the resource again since the change. You can use the URL Inspection tool to request another crawl, but that does not mean the result will be removed immediately.

Besides noindex, what else can you configure using these two methods?

For example, you can set notranslate to request that translation features in Google Search be disabled. It can be included in a robots or googlebot meta tag, or sent through an X-Robots-Tag response header. When choosing, pay attention to which crawler the directive targets: a googlebot tag targets Google, and support by other search engines should not be assumed.

ProcrustesPRShould I rewrite or update an old article that does not rank?

References

  1. 翻譯 Google 搜尋結果 | Google 搜尋中心  |  Documentation  |  Google for Developersdevelopers.google.com — Discusses: Besides noindex, what else can you configure using these two methods?
  2. 從 Google 搜尋中移除自己網站上的圖片 | Google 搜尋中心  |  Documentation  |  Google for Developersdevelopers.google.com — Discusses: Does setting noimageindex only on an article page guarantee that Google will not index its images?
  3. 使用「noindex」禁止 Google 搜尋建立索引 | Google 搜尋中心  |  Documentation  |  Google for Developersdevelopers.google.com — Discusses: What should you check if content still appears in Google search results after you set noindex?
  4. 漫遊器中繼標記規格 | Google 搜尋中心  |  Documentation  |  Google for Developersdevelopers.google.com — Source used in the article above.
  5. How Can I Tell If My Images Are Optimized For Multimodal Search? – Ask an SEOwww.searchenginejournal.com · 2026-10-29T15:30:40+00:00 — Discusses: What should you check if content still appears in Google search results after you set noindex?
  6. Meta Robots Tag & X-Robots-Tag Explainedwww.semrush.com · 2020-12-16T14:16:00Z — Discusses: Do you need to use both methods on the same page?

Related articles

How Long Does It Take to Evaluate SEO Results?Where should I start if I want to do SEO myself?Should I rewrite or update an old article that does not rank?What should I check when organic traffic suddenly drops?Why can I not find my published article on Google?Will Reposting Social Content on My Website Cause a Duplicate-Content Penalty?Eight SEO essentials for beginners: Getting your pages indexedWhat Is SEO? Understanding Optimization Priorities Through How Search Engines Work
ProcrustesPR
ProcrustesPR