When choosing robots settings, first distinguish between crawling, where crawlers read content, and indexing, where content is included in a search engine’s index. Then consider the resource format. To keep an HTML page out of search results, you can add a meta robots tag with noindex in the head section. For non-HTML resources such as PDFs and images, you can set X-Robots-Tag in the HTTP response headers. To restrict crawling of specific paths, consider robots.txt instead: restricting crawling is not the same as restricting indexing.
First, decide which pages do not need to appear in search results
Do not apply restrictions to every page simply because your site needs SEO. A 2024 article by yesharris recommends keeping pages with SEO value crawlable. For pages that search engines can still read but that you want to keep out of search results, it recommends using meta robots to control indexing. This is the author’s practical recommendation, not an official rule that every website must follow.
- Cart and checkout pages: The article lists these as examples of pages that can usually be set to noindex to keep them out of search results.
- Paths you do not want crawled: Consider robots.txt separately; noindex does not prevent crawlers from reading the content.
- Links on a page: Whether to restrict crawlers from following links is a separate decision. Do not automatically add nofollow just because you want to prevent indexing.
Choose where to configure the directives based on the resource format
The Semrush article explains which method to use for each resource format: meta robots is simpler for HTML pages, while X-Robots-Tag is used for non-HTML resources such as PDFs and images. X-Robots-Tag can also be used for HTML and allows rules to be applied in bulk by file type.
Usually, choosing one method is enough. According to the article, using both meta robots and X-Robots-Tag does not make crawlers more likely to follow the directives, so there is no need to duplicate the settings to “make them more effective.”
Start with a single page, and distinguish code changes from admin settings
If the goal is simply to keep a particular HTML page from being indexed, you can place the following in that page’s head section:
<meta name=”robots” content=”noindex”>
This example specifies only noindex. Whether to add nofollow depends on whether you also want to restrict crawlers from following links on the page.
If you use a content management system, first check the page’s SEO settings or look for a suitable plugin. Semrush suggests that using a plugin is usually simpler and avoids manual code changes, though the choice depends on your system. The article is dated 2020, so its admin navigation instructions may not apply to the current interface.
To set noindex for non-HTML resources, configure it in the HTTP response headers, not in the page content. The methods in the Semrush article require changes to Apache or Nginx server settings, rather than changes in an individual page’s editor. For changes to Shopify theme files, the article also advises caution and consulting a developer if necessary.
Robots tags: perspectives from across the web
Does adding follow force crawlers to follow links?
No. The follow directive does not force crawlers to follow links. One discussion argues that crawlers will crawl pages and follow links on their own unless prohibited, and that robots tags are mainly used to restrict behavior. Another participant agrees that follow does not force Google to follow links. Whether to restrict crawlers from following links on a page should therefore depend on what you want to allow.
Can you add noindex only to specific pages in Shopify?
Yes. Semrush's example adds a condition to the theme.liquid file that checks the page handle and outputs a noindex tag only for matching pages. This lets you configure different pages separately. It requires changes to the theme code, so proceed carefully and consult a developer if necessary. The methods in the article may not reflect the current admin interface.
How do you add robots directives to HTTP response headers?
Configure your server to include X-Robots-Tag in the HTTP response headers, for example through an Apache or Nginx configuration file. Semrush gives the example X-Robots-Tag: noindex, nofollow, which applies the same rules as meta robots to non-HTML resources such as PDFs and images. This example restricts both indexing and crawlers following links. If you need only one restriction, use the relevant directive rather than copying both.
References
- Meta Robots Tag & X-Robots-Tag Explainedwww.semrush.com · 2020-12-16T14:16:00Z — Discusses: Can you add noindex only to specific pages in Shopify?; How do you add robots directives to HTTP response headers?
- robots.txt、meta robots設定:讓網頁不被Google搜尋到www.yesharris.com · 2024-08-15T14:20:11+08:00 — Source used in the article above.
- New META TAGS with positive effects for seo ranking in 2011 and beyondwebmasters.stackexchange.com — Discusses: Does adding follow force crawlers to follow links?
- New META TAGS with positive effects for seo ranking in 2011 and beyondwebmasters.stackexchange.com — Discusses: Does adding follow force crawlers to follow links?



