How can you control and check search engine crawling and indexing?

ProcrustesPR

To control search engine crawling and indexing, first distinguish between whether a page can be crawled and whether it is indexed. robots.txt restricts crawling, while noindex excludes pages from indexing. Content that needs to remain private should have access restrictions. After setting up these controls, use a website crawler to find configuration errors and crawl statistics and server logs to see what search engines actually access. Then use URL Inspection to check indexing status and Google’s selected canonical URL. Saving the settings alone does not mean the job is done.

ProcrustesPRProcrustesPRHow can I tell whether an SEO tactic breaks spam rules?

Blocking crawling does not keep a URL out of search results

In robots.txt, User-Agent specifies the crawler and Disallow specifies the paths to exclude, but not all crawlers follow these rules. tenten’s 2023 guide also notes that even if Google cannot access a page, its URL may still be indexed if it is linked to elsewhere.

  • To restrict crawling: Check which crawlers and paths your robots.txt rules apply to.
  • To exclude a page from indexing: Check whether the page’s robots meta tag or the X-Robots-Tag in its HTTP response uses noindex, rather than simply blocking the path.

These controls address different issues. Do not treat a crawler’s inability to access a URL as proof that the URL has disappeared from search results.

Public pages and restricted content need different controls

Semrush uses thank-you pages shown after a purchase or form submission as examples of pages that can use noindex to stay out of search results. For pages you want people to find, check whether this setting has been added by mistake.

Members-only content, intranets, and websites under development need access restrictions, not just controls to keep them out of search results. The tenten guide lists login restrictions, HTTP authentication, and IP allowlists as ways to limit access to designated users. Keeping content out of search results and preventing unauthorized access are two different goals.

For pages with multiple versions, check which URL Google has selected

When the same content has multiple versions, the canonical tag is just one signal Google uses to select the canonical URL; it does not determine the result on its own. According to the tenten guide, Google also considers signals such as duplicate pages, internal links, redirects, and URLs in sitemaps.

After specifying the primary version, use Search Console’s URL Inspection tool to check whether Google’s selected canonical URL matches your expectations.

Review configuration, crawl records, and indexing results separately

  • Find configuration issues: Welly’s 2022 tool overview notes that Screaming Frog can simulate crawling to find broken links and incorrect noindex and canonical settings.
  • Observe actual crawling: The tenten guide recommends using Search Console’s crawl statistics to monitor Google’s crawling activity. To analyze access by other crawlers, check server logs.
  • Verify indexing: Then check the target URL’s indexing status and canonical URL. A simulated crawl report and a search engine’s indexing results answer different questions.
ProcrustesPRProcrustesPRShould I rewrite or update an old article that does not rank?

Search engine crawling and indexing: perspectives from across the web

Do you have to submit a sitemap to get pages indexed or improve their rankings?

Not necessarily. A 2017 SEO tutorial notes that sitemaps can help search engines crawl content but do not directly improve rankings. If a website can already be crawled without difficulty, submitting a sitemap may not be necessary. A sitemap is a tool for helping search engines discover content, not a guarantee of rankings or indexing.

Does a sudden drop in crawl volume mean something is wrong with the website?

Not necessarily, but it warrants further investigation. A 2017 tutorial suggests that fewer files being crawled could be related to less frequent updates, the search engine's assessment of the website's importance, or problems accessing the website. These are possible causes; crawl volume alone is not enough to draw a conclusion. Start by comparing crawl statistics with changes to the website, then check for access barriers.

Could redirecting every page to a consent page first affect indexing?

Possibly. One webmaster described a setup in which every URL redirected to the same warning page until visitors gave their consent. At the time, Webmaster Tools showed that the same content had been indexed for each page. This is a single case and does not establish that all consent mechanisms behave this way. If your website uses similar redirects, check whether search engines actually retrieve the consent page or the original page's content.

How should different URLs used in A/B testing be handled?

One discussion recommends adding a canonical tag to each test version that points to the original primary version. Using noindex on the non-primary versions was also suggested, but that respondent considered it less ideal. These are recommendations for A/B testing, not a claim that the tags will necessarily determine Google's choice. Other canonicalization signals may still affect how multiple versions are handled.

ProcrustesPRWill Reposting Social Content on My Website Cause a Duplicate-Content Penalty?

References

  1. Robots.txt, 決定 SEO 的 25 項標準與通訊協定系列 VIIithelp.ithome.com.tw — Source used in the article above.
  2. 如何做好網站 SEO優化?從SEO健診清單開始(2018版)www.yesharris.com · 2017-11-27T21:06:56+08:00 — Discusses: Do you have to submit a sitemap to get pages indexed or improve their rankings?; Does a sudden drop in crawl volume mean something is wrong with the website?
  3. 技術 SEO 初學者指南 (by Ahrefs)tenten.co · 2023-07-30T15:00:24.000Z — Discusses: How should different URLs used in A/B testing be handled?
  4. Screaming Frog是什麼?5分鐘教會你檢測網站SEO!welly.tw · 2022-04-25T03:52:32.809Z — Source used in the article above.
  5. Meta Tags: What They Are & How to Use Them for SEOwww.semrush.com · 2020-12-07T14:31:00Z — Source used in the article above.
  6. Should I disallow robots on A/B test pages?webmasters.stackexchange.com — Discusses: How should different URLs used in A/B testing be handled?
  7. Adult website crawlability issuewebmasters.stackexchange.com — Discusses: Could redirecting every page to a consent page first affect indexing?

Related articles

How can I tell whether an SEO tactic breaks spam rules?Should I rewrite or update an old article that does not rank?Will Reposting Social Content on My Website Cause a Duplicate-Content Penalty?What types of URLs are there, and what are they used for?What are the common mistakes, limitations, and costs of using subdomains?Can two URLs point to the same website?How to improve on-site content and structure: audit pages first, then address content gaps and internal linksHow can you use multiple domains to build content and traffic while assessing the risks?
ProcrustesPR
ProcrustesPR