URL structures that can cause crawling problems include large numbers of similar URLs generated by multiple filters and unnecessary parameters, endlessly extending calendars, and overly long redirect chains. Relative links can also cause problems under certain error-response conditions. The goal is not simply to shorten URLs, but to reduce unnecessary URL variations, prevent links from continually generating nonexistent URLs, and shorten the path to the destination page. Google’s documentation notes that complex URLs can waste Googlebot’s bandwidth and even prevent a site’s content from being fully indexed, but not all URLs with parameters are problematic.
Filter combinations and unnecessary parameters multiply similar listings
If a hotel website offers filters such as “budget,” “beach,” and “with a fitness center,” more combinations can mean more listing URLs with only minor differences in content. Googlebot generally needs to visit only a small number of listings to reach individual hotel pages, rather than crawl every combination.
- Remove unnecessary parameters that do not change the content, rather than removing all parameters indiscriminately.
- Check whether referral, sorting, and session parameters generate large numbers of URLs. Avoid putting session IDs in URLs where possible, and consider using cookies instead.
- If the server treats paths as case-insensitive, use consistent capitalization, because Google treats /APPLE and /apple as different URLs.
Relative links on error pages can continually generate nonexistent paths
This problem occurs under specific conditions: the server does not return the correct HTTP status code for nonexistent pages, and the error pages contain links relative to a parent directory. A link such as ../../category/stuff can resolve to another nonexistent URL when encountered at an invalid path, allowing these URLs to multiply indefinitely.
Google’s documentation recommends using root-relative URLs, such as /category/stuff, instead of parent-relative URLs. Root-relative URLs start at the site’s root directory. You should also check the status codes returned for nonexistent pages. This does not mean that all relative links are problematic.
Limit crawling of problematic URLs in infinite calendars and dynamic listings
Without start and end limits, a dynamic calendar can continually generate links to dates in the past or future. Google’s documentation recommends adding nofollow to links pointing to dynamically generated future calendar pages.
If you find that Google Search is crawling problematic URLs, the documentation also recommends using robots.txt to block Googlebot’s access and considering blocking search results, infinite calendars, and sorting and filtering URLs. These recommendations concern problematic URLs; they do not mean that all dynamic URLs should be blocked.
Shorten redirect chains by sending the starting URL directly to the destination
A 2021 Semrush article illustrates a redirect chain as A → B → C, noting that too many successive redirects can cause crawling problems and may increase load times. Its example fix sends A directly to C, skipping B. This approach to shortening redirect chains is worth considering, but the older article’s statements about redirect counts should not be treated as current Google rules.
URL structure and crawling: perspectives from across the web
What should you do if an important page has been crawled but not yet indexed?
First, check whether the page has thin content and add internal links from relevant pages. For example, if a product page has little text, consider putting relevant information shown in its images into words. These steps apply to important pages that need to be indexed. Neither being crawled nor improving content and links guarantees indexing.
What can you check if a page has been discovered but not yet crawled?
Check whether the site has too many broken links or unnecessary redirects, whether the sitemap contains large numbers of unimportant URLs, and whether the site hierarchy is too deep. These issues may cause crawling resources to be spent on other URLs or make the target page harder to reach as crawlers explore the site. However, the status alone does not establish which issue caused the delay.
How can you avoid blocking important pages when blocking problematic URLs?
First, confirm whether the blocked pages need to be crawled. If a page is important, you can adjust the relevant blocking directives or move it out of the blocked directory. Google's documentation recommends considering robots.txt when you find that problematic URLs are being crawled, but the scope of blocking should still be checked against the actual URLs rather than applied indiscriminately.
References
- Google 搜尋的網址結構最佳做法 | Google 搜尋中心 | Documentation | Google for Developersdevelopers.google.com — Discusses: How can you avoid blocking important pages when blocking problematic URLs?
- 檢索預算(Crawl Budget)是什麼?中大型網站不可不知的SEO項目!www.awoo.ai · 2022-05-24T06:21:55+00:00 — Discusses: What should you do if an important page has been crawled but not yet indexed?; What can you check if a page has been discovered but not yet crawled?
- JavaScript Redirect: How to Redirect to a New URLwww.semrush.com · 2021-10-14T17:25:00Z — Source used in the article above.

