How can you improve website crawling and indexing? Remove access barriers first, then improve content discovery

ProcrustesPR

To improve website crawling and indexing, first confirm that search engines can access important pages and resources. Then help them discover content through crawlable internal links, pagination, and sitemaps. You also need to distinguish between settings that block crawling and those that prevent indexing. The following focuses on Google: being crawlable does not guarantee indexing. Even meeting Search Essentials requirements does not guarantee that every page will be indexed.

ProcrustesPRProcrustesPRHow Long Does It Take to Evaluate SEO Results?

Check crawl records rather than guessing from the proportion of unindexed pages

For a site with around 34,000 pages, roughly half of which show “Discovered – currently not indexed,” those details alone cannot tell you whether the problem is content quality, server load, or crawl budget. Start troubleshooting by checking whether Googlebot has actually accessed important URLs, rather than treating all unindexed pages as the same type of problem.

  • Review server logs to confirm whether specific URLs have been crawled; a crawl record still does not mean a page has been indexed.
  • Check whether the server is unable to respond to crawl requests; Googlebot reduces crawling when it detects this situation.
  • Consider when pages were published: Google typically takes several days to notice new pages, so do not expect crawling to be completed on the day of publication.

Confirm that important content and resources are accessible

For ordinary pages you want included in public search results, check for login restrictions or robots.txt blocks. Also confirm that important resources such as images and CSS are accessible. Use the URL Inspection tool in Search Console to examine the rendered result and confirm that Google sees the content you expect.

Important text should be present in the DOM, and you should use semantic HTML wherever possible. For example, a product category page should include text descriptions, not just images of shirts. Do not rely solely on canvas or CSS content to display key text. Content added through CSS content is not part of the DOM, and Google’s documentation states that Search ignores this type of content.

Provide crawlable links to subsequent pages

Google generally follows URLs in the href attributes of a elements, but it does not click “Load more” buttons and generally does not trigger JavaScript that requires user interaction. Just because users can see all products does not mean crawlers can find them.

  • If you use infinite scrolling, provide a paginated version with crawlable links from each page to the next.
  • Give each page a unique URL, such as ?page=2, rather than distinguishing page numbers only with URL fragments such as #page2. Google ignores fragments and may therefore miss subsequent content.

Use a sitemap to help Google find content, and distinguish crawling from indexing settings

A sitemap can list important pages and update information. It is especially helpful for large sites, sites with complex hierarchies, or sites whose content is difficult to find through links, but it does not guarantee that the listed pages will be indexed. Update the sitemap when adding pages, and retain internal links to important pages.

If sorting or filtering creates unnecessary duplicate listings, first clarify your goal: use robots.txt to request that search engines avoid crawling URLs that match particular patterns, or use noindex to prevent indexing. Do not treat robots.txt as a tool for preventing indexing. Rather than pursuing indexing for every page, first decide which URLs actually need to appear in search results.

ProcrustesPRProcrustesPRWhere should I start if I want to do SEO myself?

Website crawling and indexing: perspectives from across the web

Should a website with around 34,000 pages prioritize crawl budget?

Not necessarily. Having around 34,000 pages, or a particular proportion of unindexed pages, does not by itself mean that a site has insufficient crawl budget. Google's guidance on managing crawl budget describes particularly large sites whose content changes regularly or frequently. Check actual crawl records and server conditions first, then decide whether crawl budget should be a priority.

What should you consider if you want Google to index paywalled content?

If you want Google to crawl and index paywalled content, you must allow Googlebot and, where applicable, Googlebot-News to access the pages. You can use the URL Inspection tool to test how Google crawls and renders a URL and confirm whether the content you want it to read is visible; allowing access does not guarantee indexing.

Does pagination still need to rely on rel=next and rel=prev?

Not for Google. Google no longer uses rel=next and rel=prev to identify relationships between consecutive pages. Rather than relying solely on these tags to help Google find subsequent pages, provide crawlable links to the next page. Other search engines may still use these tags, so whether you keep them can depend on which search engines you want your site to support.

Can you pay Google to crawl faster and get every page indexed?

No. Google states that it does not accept payment to increase a site's crawl frequency or improve its rankings. Even if a page meets Search Essentials requirements, there is no guarantee that it will be crawled, indexed, or displayed in search results. Treat claims that payment can get every page indexed with particular caution.

ProcrustesPRShould I rewrite or update an old article that does not rank?

References

  1. 什麼是 Sitemap | Google 搜尋中心  |  Documentation  |  Google for Developersdevelopers.google.com — Source used in the article above.
  2. 技術性搜尋引擎最佳化 (SEO) 技巧與策略 | Google 搜尋中心  |  Documentation  |  Google for Developersdevelopers.google.com — Discusses: Should a website with around 34,000 pages prioritize crawl budget?
  3. Google 建議的分頁最佳做法 | Google 搜尋中心  |  Documentation  |  Google for Developersdevelopers.google.com — Discusses: Does pagination still need to rely on rel=next and rel=prev?
  4. 訂閱和付費牆內容標記 | Google 搜尋中心  |  Documentation  |  Google for Developersdevelopers.google.com — Discusses: What should you consider if you want Google to index paywalled content?
  5. 詳盡指南:Google 搜尋的運作方式 | Google 搜尋中心  |  Documentation  |  Google for Developersdevelopers.google.com — Discusses: What should you consider if you want Google to index paywalled content?; Can you pay Google to crawl faster and get every page indexed?
  6. 網頁程式開發人員的搜尋引擎最佳化 (SEO) 指南 | Google 搜尋中心  |  Documentation  |  Google for Developersdevelopers.google.com — Source used in the article above.
  7. 排解 Google 搜尋檢索錯誤 | Google 搜尋中心  |  Documentation  |  Google for Developersdevelopers.google.com — Discusses: Should a website with around 34,000 pages prioritize crawl budget?

Related articles

How Long Does It Take to Evaluate SEO Results?Where should I start if I want to do SEO myself?Should I rewrite or update an old article that does not rank?What should I check when organic traffic suddenly drops?Why can I not find my published article on Google?Will Reposting Social Content on My Website Cause a Duplicate-Content Penalty?Eight SEO essentials for beginners: Getting your pages indexedWhat Is SEO? Understanding Optimization Priorities Through How Search Engines Work
ProcrustesPR
ProcrustesPR