How can you check and fix crawling, indexing, and link issues?

ProcrustesPR

When troubleshooting crawling, indexing, and link issues, start with three questions: Can search engines access the page? Have they indexed it? Do internal links point to the right destination? Taken together, the sources below suggest addressing server errors and unintended blocks first, then fixing broken links, redirect problems, and indexing issues affecting important pages. This is a suggested troubleshooting sequence, not an official search engine requirement. A page’s absence from the index cannot automatically be attributed to insufficient links.

ProcrustesPRProcrustesPRHow Long Does It Take to Evaluate SEO Results?

Check indexing status, not just the sitemap

A 2026 Yoast article recommends adding your XML sitemap to Google Search Console to help Google find the sitemap, then monitoring the numbers of discovered and indexed URLs. If there is a substantial gap, investigate the cause. Submitting a sitemap does not mean all its URLs have been indexed.

A 2024 Semrush article recommends checking indexing data separately in Google Search Console and Bing Webmaster Tools rather than assuming both index the same pages. For unindexed pages, first check whether crawling is blocked, a noindex directive is present, or the page has been crawled but is currently not indexed. Then decide how to address the issue.

Resolve access barriers first, then check exclusion settings

Semrush’s 2024 article recommends the following approaches. Before changing settings, confirm each page’s purpose so you do not treat deliberately excluded content as an error.

  • 5xx server errors: Compile the affected URLs and error codes and pass them to developers to resolve so search engines can access the content properly.
  • robots.txt blocks: Check whether Disallow unintentionally blocks important directories or pages. The goal is to correct unintended blocks, not remove all restrictions.
  • noindex tags: Remove a noindex tag only if it was mistakenly added to an important page you want indexed. The two settings serve different purposes: noindex instructs search engines not to include a page in search results, while robots.txt restricts crawling.

Fix broken links and shorten redirect paths

For broken internal links, the same Semrush article recommends restoring the original page or using a 301 redirect from its URL to a similar, relevant page. For broken external links, update the URL or link to a similar resource. Remove the link if no suitable replacement is available.

For example, if page A redirects to page B, which then redirects to page C, the link or redirect can point directly to the final destination, page C. If page B instead redirects back to page A, the loop needs to be broken.

If the HTTP, HTTPS, www, or non-www versions each load independently without redirecting to a single preferred version, the article recommends choosing a primary HTTPS version and using 301 redirects to send the other versions to it.

If a page has been crawled but not indexed, check its content and internal links

For important pages marked “Crawled – currently not indexed,” Semrush’s 2024 article recommends prioritizing content updates. It lists insufficient quality and excessive similarity to existing pages as common causes. However, that does not mean an individual page necessarily has these problems; its indexing status alone does not establish the cause.

For internal links, check how many point to important pages, whether any pages are orphaned with no internal links pointing to them, and how many clicks it takes to reach those pages. Semrush’s 2023 article lists these as audit items that can help identify pages worth investigating. However, needing more clicks to reach a page does not, on its own, mean the page cannot be indexed.

ProcrustesPRProcrustesPRWhere should I start if I want to do SEO myself?

Crawling, indexing, and link issues: perspectives from across the web

What should you prioritize in regular link audits?

Start with the internal link errors listed in the audit report, then review warnings and notices according to their impact. Semrush's 2023 article considers the latter two categories generally less urgent. An audit can also examine the number of links on individual pages and nofollow links to assess whether they hinder visitor navigation, crawling, or the flow of link equity. Not every alert needs to be treated as equally serious.

Should you wait for visitors to report 4xx errors before addressing them?

No. Semrush recommends proactively auditing 4xx errors, identifying the affected pages and specific error codes, and applying fixes appropriate to each error to prevent problems from affecting site performance or the visitor experience. Regular scans can help detect new issues early. However, 4xx covers different errors, so the category alone does not mean every URL needs the same fix.

Can IndexNow replace indexing checks in individual search engines?

No. Semrush's 2024 article suggests using IndexNow to submit URLs to Bing and other search engines that support the protocol to encourage faster indexing, but this does not guarantee inclusion in the index. You should still check indexing status separately in each search engine rather than assuming they index the same pages. Submitting a URL and confirming that it has been indexed are separate steps.

ProcrustesPRShould I rewrite or update an old article that does not rank?

References

  1. What is an XML sitemap and why should you have one?yoast.com · 2026-03-11T13:52:01+00:00 — Source used in the article above.
  2. What Is a 400 Bad Request? Definition, Causes, & How to Fixwww.semrush.com · 2021-09-20T20:22:00Z — Discusses: Should you wait for visitors to report 4xx errors before addressing them?
  3. HTML Anchor Element: What Are Anchor Links & How to Use Themwww.semrush.com · 2021-10-18T20:43:00Z — Discusses: Should you wait for visitors to report 4xx errors before addressing them?
  4. What Is Silo SEO and Is There a Better Option?www.semrush.com · 2023-09-27T11:14:00Z — Discusses: What should you prioritize in regular link audits?
  5. The technical SEO checklist for search engines and AI searchwww.semrush.com · 2024-09-18T14:13:00Z — Discusses: Can IndexNow replace indexing checks in individual search engines?

Related articles

How Long Does It Take to Evaluate SEO Results?Where should I start if I want to do SEO myself?Should I rewrite or update an old article that does not rank?What should I check when organic traffic suddenly drops?Why can I not find my published article on Google?Will Reposting Social Content on My Website Cause a Duplicate-Content Penalty?Eight SEO essentials for beginners: Getting your pages indexedWhat Is SEO? Understanding Optimization Priorities Through How Search Engines Work
ProcrustesPR
ProcrustesPR