To control search engine crawling and indexing, first distinguish between whether a page can be crawled and whether it is indexed. robots.txt restricts crawling, while noindex excludes pages from indexing. Content that needs to remain private should have access restrictions. After setting up these controls, use a website crawler to find configuration errors and crawl statistics and server logs to see what search engines actually access. Then use URL Inspection to check indexing status and Google’s selected canonical URL. Saving the settings alone does not mean the job is done.
Blocking crawling does not keep a URL out of search results
In robots.txt, User-Agent specifies the crawler and Disallow specifies the paths to exclude, but not all crawlers follow these rules. tenten’s 2023 guide also notes that even if Google cannot access a page, its URL may still be indexed if it is linked to elsewhere.
- To restrict crawling: Check which crawlers and paths your robots.txt rules apply to.
- To exclude a page from indexing: Check whether the page’s robots meta tag or the X-Robots-Tag in its HTTP response uses noindex, rather than simply blocking the path.
These controls address different issues. Do not treat a crawler’s inability to access a URL as proof that the URL has disappeared from search results.
Public pages and restricted content need different controls
Semrush uses thank-you pages shown after a purchase or form submission as examples of pages that can use noindex to stay out of search results. For pages you want people to find, check whether this setting has been added by mistake.
Members-only content, intranets, and websites under development need access restrictions, not just controls to keep them out of search results. The tenten guide lists login restrictions, HTTP authentication, and IP allowlists as ways to limit access to designated users. Keeping content out of search results and preventing unauthorized access are two different goals.
For pages with multiple versions, check which URL Google has selected
When the same content has multiple versions, the canonical tag is just one signal Google uses to select the canonical URL; it does not determine the result on its own. According to the tenten guide, Google also considers signals such as duplicate pages, internal links, redirects, and URLs in sitemaps.
After specifying the primary version, use Search Console’s URL Inspection tool to check whether Google’s selected canonical URL matches your expectations.
Review configuration, crawl records, and indexing results separately
- Find configuration issues: Welly’s 2022 tool overview notes that Screaming Frog can simulate crawling to find broken links and incorrect noindex and canonical settings.
- Observe actual crawling: The tenten guide recommends using Search Console’s crawl statistics to monitor Google’s crawling activity. To analyze access by other crawlers, check server logs.
- Verify indexing: Then check the target URL’s indexing status and canonical URL. A simulated crawl report and a search engine’s indexing results answer different questions.
Search engine crawling and indexing: perspectives from across the web
Do you have to submit a sitemap to get pages indexed or improve their rankings?
Not necessarily. A 2017 SEO tutorial notes that sitemaps can help search engines crawl content but do not directly improve rankings. If a website can already be crawled without difficulty, submitting a sitemap may not be necessary. A sitemap is a tool for helping search engines discover content, not a guarantee of rankings or indexing.
Does a sudden drop in crawl volume mean something is wrong with the website?
Not necessarily, but it warrants further investigation. A 2017 tutorial suggests that fewer files being crawled could be related to less frequent updates, the search engine's assessment of the website's importance, or problems accessing the website. These are possible causes; crawl volume alone is not enough to draw a conclusion. Start by comparing crawl statistics with changes to the website, then check for access barriers.
Could redirecting every page to a consent page first affect indexing?
Possibly. One webmaster described a setup in which every URL redirected to the same warning page until visitors gave their consent. At the time, Webmaster Tools showed that the same content had been indexed for each page. This is a single case and does not establish that all consent mechanisms behave this way. If your website uses similar redirects, check whether search engines actually retrieve the consent page or the original page's content.
How should different URLs used in A/B testing be handled?
One discussion recommends adding a canonical tag to each test version that points to the original primary version. Using noindex on the non-primary versions was also suggested, but that respondent considered it less ideal. These are recommendations for A/B testing, not a claim that the tags will necessarily determine Google's choice. Other canonicalization signals may still affect how multiple versions are handled.
References
- Robots.txt, 決定 SEO 的 25 項標準與通訊協定系列 VIIithelp.ithome.com.tw — Source used in the article above.
- 如何做好網站 SEO優化?從SEO健診清單開始(2018版)www.yesharris.com · 2017-11-27T21:06:56+08:00 — Discusses: Do you have to submit a sitemap to get pages indexed or improve their rankings?; Does a sudden drop in crawl volume mean something is wrong with the website?
- 技術 SEO 初學者指南 (by Ahrefs)tenten.co · 2023-07-30T15:00:24.000Z — Discusses: How should different URLs used in A/B testing be handled?
- Screaming Frog是什麼?5分鐘教會你檢測網站SEO!welly.tw · 2022-04-25T03:52:32.809Z — Source used in the article above.
- Meta Tags: What They Are & How to Use Them for SEOwww.semrush.com · 2020-12-07T14:31:00Z — Source used in the article above.
- Should I disallow robots on A/B test pages?webmasters.stackexchange.com — Discusses: How should different URLs used in A/B testing be handled?
- Adult website crawlability issuewebmasters.stackexchange.com — Discusses: Could redirecting every page to a consent page first affect indexing?

