Can search results show only a website’s homepage? First, distinguish indexing from crawling

ProcrustesPR

If your goal is to stop pages other than the homepage from appearing as standalone search results, you can keep the homepage indexable and apply noindex to the other pages, rather than just blocking crawling through robots.txt. This does not, however, guarantee that the homepage will appear in searches. If you want to hide the links to internal pages displayed beneath the homepage result, the sources cited here do not provide enough information to explain how to control those links separately.

ProcrustesPRProcrustesPRHow Long Does It Take to Evaluate SEO Results?

To exclude internal pages, control indexing

An answer on Webmasters Stack Exchange recommends applying noindex to pages you do not want indexed through a robots meta tag or the X-Robots-Tag HTTP response header. Another answer also considers page-by-page settings more precise than using robots.txt to address this issue.

  • Homepage: Do not apply the noindex setting intended for internal pages.
  • Internal pages such as About Us: If you are sure you do not want them appearing in search results, apply noindex to each page.
  • Pages visitors should still be able to browse: Excluding a page from indexing is not the same as deleting it. You can keep it available to visitors.

noindex does not “hide internal pages only when someone searches for the domain name.” It asks search engines not to index those pages at all. Before using it, confirm that you do not need readers to find those pages through search.

Blocking crawling through robots.txt does not remove search results

Semrush’s 2024 explanation states that even if robots.txt blocks a page, Google may still discover its URL through links on other pages, index it, and display it in search results. You therefore cannot assume that adding every internal page path to Disallow will leave only the homepage in search results.

Opinions on the forums differ. On Warrior Forum, paulgl argues that only taking a page offline or password-protecting it can ensure it does not appear in search results, and questions the effectiveness of noindex. Other answers and the article cited above support using noindex to control indexing. paulgl’s personal assertion is not an official Google rule. Likewise, noindex is not password protection and cannot be used to restrict visitor access.

Before configuring WordPress, confirm the homepage URL and which pages to exclude

It is particularly easy to use the wrong URL when making a crawling exception for the homepage. A case on Webmasters Stack Exchange highlights the problem: if /index.php redirects to /, allowing crawling of /index.php while blocking / could also block the homepage visitors actually see. Do not base the settings solely on a filename that looks like the homepage.

For category, tag, and archive pages, a 2022 Semrush article demonstrated how to turn off search visibility for specific page types through Yoast SEO. This recommendation applies when those pages offer little value to search users; it does not mean every website should exclude them. The menu paths in that article reflect the interface at the time and should not be treated as current instructions. Even after configuring category pages, you still need to check the settings for individual internal pages such as About Us.

ProcrustesPRProcrustesPRWhere should I start if I want to do SEO myself?

Homepage-only search results: perspectives from across the web

How should I configure internal pages if I do not want them indexed but still want their links followed?

An answer on Webmasters Stack Exchange recommends using noindex, follow on pages you do not want indexed. If you also do not want crawlers to follow links on a page, you can replace follow with nofollow. This is a page-by-page option for situations where you want to keep pages available to visitors without having them appear in search results. Whether to allow link following depends on the purpose of each page.

Will existing search results disappear immediately after I apply these settings?

There is no guarantee they will disappear immediately. An answer on Webmasters Stack Exchange suggests that search results for cached pages may gradually disappear if a crawler finds no activity at those URLs for an extended period. However, the answer gives no specific timeframe, and the assessment applies only to crawlers that follow the relevant instructions. This is a participant’s judgment based on experience, not a removal deadline or guarantee.

Will using noindex on internal pages affect AI Overviews?

Semrush’s 2024 explanation states that applying noindex to a page prevents it from appearing in AI Overviews but also removes it from regular search results. If you only want to restrict AI Overviews while retaining internal pages’ visibility in regular search, consider this trade-off before using it. It is not a setting that targets AI Overviews alone.

ProcrustesPRShould I rewrite or update an old article that does not rank?

References

  1. Will googlebot index my index.php even with an htaccess redirect?webmasters.stackexchange.com — Source used in the article above.
  2. Is it possible to noindex whole website except one page?www.warriorforum.com — Source used in the article above.
  3. AI Overviews: What Are They & How to Optimize for Themwww.semrush.com · 2024-05-31T14:20:00Z — Discusses: Will using noindex on internal pages affect AI Overviews?
  4. Multiple domain links on Google from one WordPress sitewebmasters.stackexchange.com — Discusses: How should I configure internal pages if I do not want them indexed but still want their links followed?; Will existing search results disappear immediately after I apply these settings?
  5. Which is the best option Meta Robot tags or robots.txt?www.warriorforum.com — Source used in the article above.
  6. The Ultimate Guide to WordPress SEOwww.semrush.com · 2022-11-18T14:32:00Z — Source used in the article above.
  7. Multiple domain links on Google from one WordPress sitewebmasters.stackexchange.com — Source used in the article above.
  8. What Is Googlebot? How Google's Web Crawler Workswww.semrush.com · 2024-08-23T14:32:00Z — Source used in the article above.

Related articles

How Long Does It Take to Evaluate SEO Results?Where should I start if I want to do SEO myself?Should I rewrite or update an old article that does not rank?What should I check when organic traffic suddenly drops?Why can I not find my published article on Google?Will Reposting Social Content on My Website Cause a Duplicate-Content Penalty?Eight SEO essentials for beginners: Getting your pages indexedWhat Is SEO? Understanding Optimization Priorities Through How Search Engines Work
ProcrustesPR
ProcrustesPR