What happens when you block pages that already rank using robots.txt?

ProcrustesPR

Using robots.txt to block pages that already rank restricts crawling but does not directly remove them from the index. The pages may still appear in search results, but their rankings are not guaranteed to stay the same. In an Ahrefs test involving two pages, ranking changes were limited, but the way the pages appeared in search, their featured snippets, and their clicks all changed. So, “still appearing in search” does not mean there has been no impact. Blocking crawling should not replace measures intended to remove pages from search results.

ProcrustesPRProcrustesPRHow Long Does It Take to Evaluate SEO Results?

Stopping crawling does not immediately remove pages from the index

Semrush’s 2024 guide to filtered pages explains that robots.txt blocks crawling but does not directly remove indexed pages from search results. To exclude a page from search results, you can use noindex, but search engines must be able to crawl the page to read that tag.

This also explains why “blocking first and adding a tag later” may not work. A Stack Exchange answer warns that if a page is already indexed, blocking crawling while adding a meta tag prevents search engines from seeing the update. Combining the two settings does not necessarily make them more effective; they may instead interfere with each other.

A two-page test: limited ranking changes, but a different appearance in search

In a 2023 article, Ahrefs’ Patrick Stox described a test in which he blocked two articles that already ranked. He began on January 30 of that year and observed the pages for nearly five months. The results included:

  • Rankings dropped by one or two positions for some keywords and rose for others; the overall change in estimated traffic was smaller than the author expected.
  • Both pages lost all their featured snippets. Their search results no longer displayed their custom titles, and the original descriptions were replaced with a message saying no information was available.
  • Both impressions and clicks declined, but clicks fell more sharply. The author thought the appearance of the search results might have hurt clicks more than the ranking declines did. This was his interpretation, not an established causal relationship.

The author speculated that Google might still be using previously crawled content to rank the pages, but could not confirm this. He also suspected that prolonged blocking might cause Google to lose trust in the old content, although he saw no evidence of that during the test. After the block was removed, one article quickly regained some featured snippets. These observations are still not enough to establish that blocking necessarily makes all pages ineligible for featured snippets.

These are past test results for just two pages, not a guarantee about rankings today. The author also explicitly advised against doing this and warned that other sites might see different results.

First, clarify the goal: keep rankings or leave search results?

  • To keep appearing in search: First check whether important pages have been blocked by mistake. Semrush’s 2024 explanation notes that intentional blocking is not necessarily a problem, but blocking pages that should remain accessible needs to be addressed.
  • To remove low-value pages from search: Semrush’s guide to filtered pages recommends first applying noindex to low-value filtered pages that already rank, while allowing search engines to crawl the pages and read the tag. Once the pages have been removed from search results, crawling can be restricted with robots.txt.

For example, product filters can generate large numbers of similar URLs. Conserving the resources used to crawl those URLs and removing them from existing search results are two separate goals. A single blocking rule should not be treated as a way to achieve both.

ProcrustesPRProcrustesPRWhere should I start if I want to do SEO myself?

Blocking ranked pages with robots.txt: perspectives from across the web

After a page is blocked, can links from other websites still affect whether it is discovered?

Yes, they can. robots.txt restricts crawling; it does not directly control indexing. Even if a page is blocked, search engines may still discover and index its URL through links from other websites. Blocking crawling is therefore not a reliable way to prevent search engines from discovering a URL, and actual outcomes may vary by search engine.

Do you still need to check robots.txt if the page itself is not blocked?

Yes, especially if the main content relies on JavaScript to load. If robots.txt blocks essential JavaScript files, crawlers may not see the content those files load. Check for blocked resources and, if necessary, ask developers to identify which files are essential for rendering the content. This is a separate issue from whether the page URL itself is blocked.

Why is it important to be especially careful when defining what to block?

If a rule covers important pages you intended to keep visible in search, its effects may extend beyond the intended URLs. A 2018 article warns that an incorrect folder-level setting could affect indexing across the entire site and its existing rankings. Whether blocking is a problem still depends on whether it serves the intended purpose, so first confirm which pages the rules actually cover.

ProcrustesPRShould I rewrite or update an old article that does not rank?

References

  1. Crawlability & Indexability: What They Are & How They Affect SEOwww.semrush.com · 2016-11-01T13:00:00Z — Discusses: After a page is blocked, can links from other websites still affect whether it is discovered?
  2. robots.txt是什麼? robots.txt設定、用途與使用範例分享!welly.tw · 2022-01-13T05:20:36.169Z — Discusses: After a page is blocked, can links from other websites still affect whether it is discovered?
  3. The Story of Blocking 2 High-Ranking Pages With Robots.txtahrefs.com · 2023-07-10T02:11:55+00:00 — Source used in the article above.
  4. 使用”noindex”等4種方式禁止特定網頁被收錄,提升網站整體seo品質www.awoo.ai · 2018-01-15T09:00:23+00:00 — Discusses: Why is it important to be especially careful when defining what to block?
  5. Faceted Navigation: An SEO Win Waiting to Happenwww.semrush.com · 2024-12-20T15:47:00Z — Source used in the article above.
  6. Google Search Console Errors: Identifying and Fixing Themwww.semrush.com · 2024-03-20T10:51:00Z — Discusses: Why is it important to be especially careful when defining what to block?
  7. 15 Crawlability Problems & How to Fix Themwww.semrush.com · 2016-09-15T13:09:15Z — Discusses: Do you still need to check robots.txt if the page itself is not blocked?
  8. How to stop certain urls from being indexedwebmasters.stackexchange.com — Source used in the article above.

Related articles

How Long Does It Take to Evaluate SEO Results?Where should I start if I want to do SEO myself?Should I rewrite or update an old article that does not rank?What should I check when organic traffic suddenly drops?Why can I not find my published article on Google?Will Reposting Social Content on My Website Cause a Duplicate-Content Penalty?Eight SEO essentials for beginners: Getting your pages indexedWhat Is SEO? Understanding Optimization Priorities Through How Search Engines Work
ProcrustesPR
ProcrustesPR