To troubleshoot website indexing issues, first check whether Google has crawled the page, then look at why it is not indexed. The page may not have been crawled yet, or it may have been crawled but not indexed. Other possibilities include exclusion settings, an invalid URL, or Google choosing another version to index. “Not indexed” is not a single type of failure and does not, by itself, mean a website has been penalized. No symptoms or indexing data for a specific website are available here, so the following is a troubleshooting process, not a diagnosis of any individual website.
Start with the reasons pages are not indexed
Check the reasons and affected URLs in Search Console’s Page indexing report, then address each issue accordingly. The Semrush article referenced here is dated 2020. The following draws on its troubleshooting approach, but its interface descriptions should not be treated as current instructions.
- Discovered – currently not indexed: Google knows the URL exists but has not yet crawled it. The source notes that this may reflect Google’s concerns about the load on the website. Check internal links and review whether duplicate pages or pages with thin content are consuming crawl resources.
- Crawled – currently not indexed: Google has visited the page but has not included it in the index. The source identifies content quality as a common area to investigate, but this does not establish that quality is the cause. Review whether the content is original and fully answers readers’ questions.
Check whether blocking settings match the page’s purpose
First confirm whether the page actually needs to appear in search results, then check its settings. Do not remove all restrictions simply to reduce the number of excluded pages.
- robots.txt: Check whether any rules prevent Googlebot from crawling pages intended to be public. Remove or adjust those rules if they were set by mistake.
- noindex: This tag instructs search engines not to index a page. Remove it only if the page is intended to be indexed and the tag was added by mistake.
- canonical: Confirm that each version points to the preferred URL. If a product page incorrectly points to another page, correct the canonical URL setting rather than repeatedly requesting indexing.
Check links and URLs so important content can be found
Check orphan pages with no internal links pointing to them, as well as pages buried beneath multiple levels of navigation. Add internal links from relevant pages to important or new pages, and adjust the site structure if necessary. The source suggests checking pages that take more than three clicks to reach. This does not mean that pages beyond three clicks will necessarily remain unindexed.
If a URL returns a 404, the appropriate action depends on what happened to the content. Restore content deleted by mistake, or correct a mistyped URL. If the content has moved, update internal links and consider a 301 redirect from the old URL to the current version. If the content no longer exists, relevant links can be changed to point to suitable replacement content.
Consolidate duplicate versions, then monitor the results of your fixes
Identical or highly similar content spread across multiple URLs may lead search engines to index an unintended version. Unnecessary copies can be consolidated into the main page, with their URLs redirected to it using 301 redirects. For copies that must remain available, use canonical tags to indicate the preferred version.
After making corrections, you can request indexing and use the URL Inspection tool regularly to monitor progress, but submitting a request does not guarantee immediate indexing. Regular site audits can also help identify new issues with links, exclusion settings, and URLs.
Website indexing FAQ: perspectives from across the web
What else can you check when non-HTML resources, such as PDFs, are not indexed?
Check whether the X-Robots-Tag in the response places restrictions on the resource. Site audit tools may also flag related issues. The source notes that these tags are commonly used for non-HTML documents such as PDFs. First confirm whether the resource needs to be indexed, then decide whether the restrictions need adjusting rather than removing them indiscriminately.
Why might a page still be difficult to discover even when internal links point to it?
Check whether internal links pointing to the page use nofollow. The source notes that Google may ignore these links, so even if a link is visible, crawlers may still have difficulty discovering the page through it. This is only a troubleshooting clue: it does not mean all nofollow links will be ignored, and this setting alone cannot establish why a page is not indexed.
Beyond exclusion settings, what other technical issues are worth auditing?
Check mobile usability, loading speed, and redirect issues. A site audit tool can help identify problems that may affect crawling and show which pages are affected. The source suggests these as common areas to investigate, but a slow-loading page or a redirect does not, by itself, prove that either issue caused the page to remain unindexed.
References
- How to get your website indexed by Googlewww.semrush.com · 2020-12-28T14:21:00Z — Discusses: What else can you check when non-HTML resources, such as PDFs, are not indexed?; Why might a page still be difficult to discover even when internal links point to it?

