How to check for website crawling barriers: robots.txt, .htaccess, and WordPress visibility settings

ProcrustesPR

When checking for barriers to search engine access and crawling, first confirm whether Googlebot can access the target URL. Then review robots.txt, the page’s noindex directives, and server-side access restrictions and redirects. If you use WordPress, also check its search engine visibility settings. Blocking crawling, preventing indexing, and restricting visitor access are different things. The absence of a robots.txt block alone does not guarantee that a site will be indexed.

ProcrustesPRProcrustesPRHow Long Does It Take to Evaluate SEO Results?

Check URL access first, then indexing restrictions

Google recommends first using the URL Inspection tool to confirm whether Googlebot can access the page, then testing robots.txt and checking the HTML meta tags for noindex. This separates an inability to retrieve content from a restriction on indexing it.

  • For public pages you want search engines to index: confirm that login requirements or noindex directives have not been added by mistake.
  • For pages that may be crawled but should not appear in search results: use noindex.
  • For content that ordinary visitors or crawlers should not be able to access: use access restrictions such as password protection.

Check robots.txt for unintended blocks; do not use it to keep content private

Check the rules against the target URLs to see whether they accidentally block pages or directories intended to be public. Google explicitly distinguishes between crawling and indexing: robots.txt is used to block crawling, but it cannot guarantee that a URL will not appear in search results. If the goal is to prevent indexing, blocking crawling is not a substitute.

For example, placing sensitive documents in a single directory and blocking crawling still does not prevent access to those documents. Nor should you layer multiple rules simply to be safe. Google warns that crawling and indexing rules can counteract one another, so check the purpose of each setting.

Check .htaccess for access restrictions and redirects

Beyond robots.txt, also review the server configuration. Public discussions suggest checking .htaccess or other server settings for redirects related to the target URLs. This is a troubleshooting lead, not evidence that redirects cause every indexing failure.

A 2018 awoo article uses Apache as an example to explain how .htaccess can be used to password-protect a directory. If content intended to be public is subject to this kind of restriction, check whether the site actually needs that restriction. If the content is sensitive, however, do not simply remove its protection to let search engines crawl it.

Check WordPress Reading settings, but do not treat the option as a guarantee

If you want search engines to index your WordPress site, go to Settings → Reading and confirm that “Discourage search engines from indexing this site” is unchecked. This recommendation comes from a community reply on the WordPress forum, not official WordPress documentation for a specific version.

A 2018 article on Neil Patel’s website also notes that the search engine visibility option is only a request to search engines; whether they honor it is up to them. The option therefore cannot replace password protection or prove that existing search results have been removed. After confirming the setting, return to the target URL and check its actual accessibility and indexing restrictions.

ProcrustesPRProcrustesPRWhere should I start if I want to do SEO myself?

Search engine access and website crawling barriers: perspectives from across the web

How can you test whether robots.txt is accidentally blocking important URLs?

Start by testing the target URLs intended to be public, then simulate a site-wide crawl if needed to determine the scope of the rules' effects. A 2019 Screaming Frog article explains that you can import an existing robots.txt file or custom rules, test whether an individual URL is blocked, and crawl using custom rules. Testing before making changes helps prevent incorrect rules from affecting large numbers of pages.

Can you set separate robots.txt rules for Google's mobile and desktop crawlers?

You cannot set separate robots.txt rules for these two types of Googlebot. Google explains that although you can distinguish the mobile and desktop crawlers by the user-agent header in their requests, both use the same user-agent token in robots.txt—the name used to identify which rules apply. Rules targeting that token therefore affect both crawlers, so troubleshooting needs to account for both.

What exactly is configured in this site's .htaccess file?

This cannot currently be determined: the supplied material does not include the site's actual .htaccess contents. Public discussions suggest checking that file or other server settings for redirects related to the target URLs. This can serve as a troubleshooting lead, but it does not establish that redirects are the reason the pages have not been indexed. The actual configuration still needs to be checked against the affected URLs.

ProcrustesPRShould I rewrite or update an old article that does not rank?

References

  1. 軟體應用程式 (SoftwareApplication) 結構定義 | Google 搜尋中心  |  Documentation  |  Google for Developersdevelopers.google.com — Source used in the article above.
  2. Googlebot 簡介 | Google 搜尋中心  |  Documentation  |  Google for Developersdevelopers.google.com — Discusses: Can you set separate robots.txt rules for Google's mobile and desktop crawlers?
  3. 網頁程式開發人員的搜尋引擎最佳化 (SEO) 指南 | Google 搜尋中心  |  Documentation  |  Google for Developersdevelopers.google.com — Source used in the article above.
  4. Want More Traffic? Deindex Your Pages. Here’s Why.neilpatel.com · 2018-03-06T19:00:42+00:00 — Source used in the article above.
  5. 使用”noindex”等4種方式禁止特定網頁被收錄,提升網站整體seo品質www.awoo.ai · 2018-01-15T09:00:23+00:00 — Source used in the article above.
  6. Reviving Retired Search Console Reportswww.screamingfrog.co.uk · 2019-04-08T13:18:41+00:00 — Discusses: How can you test whether robots.txt is accidentally blocking important URLs?
  7. High number of excluded (not indexed) pages in Google Search Consolewww.warriorforum.com — Discusses: What exactly is configured in this site's .htaccess file?
  8. Google search by low competition keywordswww.warriorforum.com — Source used in the article above.

Related articles

How Long Does It Take to Evaluate SEO Results?Where should I start if I want to do SEO myself?Should I rewrite or update an old article that does not rank?What should I check when organic traffic suddenly drops?Why can I not find my published article on Google?Will Reposting Social Content on My Website Cause a Duplicate-Content Penalty?Eight SEO essentials for beginners: Getting your pages indexedWhat Is SEO? Understanding Optimization Priorities Through How Search Engines Work
ProcrustesPR
ProcrustesPR