When crawling a large ecommerce site, first limit page types, category paths, and URL parameters based on the audit objectives, rather than trying to crawl every URL from the outset. Then group the results by template or page type to examine where issues occur and review crawl depth. The approach below is based on a 2024 ecommerce audit article on Screaming Frog’s official website; the specific interface will depend on the version you use.
Decide what to check before excluding unnecessary pages
Crawling product pages on large ecommerce sites often requires substantial memory. If the audit only covers category pages, you can exclude product pages. If the focus is on commercial pages, you can also exclude editorial content such as blog posts. These exclusions only define the scope of this audit. They do not mean the excluded pages are unimportant, nor do they instruct search engines to stop indexing them.
- Check category pages only: set exclusion rules based on the actual paths used for product pages. For example, if all product pages sit under /p/, you can exclude that path.
- Audit departments in batches: crawl paths such as /women/, /men/, and /kids/ separately.
- Audit product categories in batches: alternatively, crawl category paths for clothing, swimwear, or accessories separately.
These paths are only examples. Rules must fit the site’s structure and audit objectives and cannot be applied unchanged to every store.
Manage filter parameters to keep the scope from expanding to millions of URLs
Product attributes such as size, color, and brand can generate large numbers of URLs with parameters. The article notes that a crawl may include millions of URLs if the URLs generated by these attributes are not excluded, though not every ecommerce site will reach that scale.
The article lists three ways to control this: remove all URL parameters, exclude specific parameters using regular expressions, or limit the number of query strings. The choice depends on what the audit is intended to examine. For example, if you are only checking category pages without filters, you could consider excluding the relevant parameters. If the goal is to audit filtered pages, however, you cannot first remove all the URLs you need to inspect.
Group the results to see where issues are concentrated
Setting the crawl scope determines which URLs to crawl; analyzing the results reveals which page types have issues. The Segments feature described in the article groups URLs by template, page type, or site section, so you can review each group separately rather than search through a single mixed list.
- Review the distribution of issues: use segment information in Issues to identify which groups a particular issue affects.
- Check crawl depth: review category pages, product pages, and other page types separately rather than relying solely on sitewide results.
- Retain grouping information: the article explains that segment labels are also included in exported data, making subsequent analysis easier.
Custom extraction can also retrieve data from URLs to help assess content quality at scale. However, the excerpt discussed here does not specify extraction fields or thresholds for identifying low-quality content. Enabling this feature alone does not establish that a group of pages is low quality.
Controlling crawl scope and analyzing results for large ecommerce sites: perspectives from across the web
How long does it usually take to crawl a large ecommerce site?
Crawl times vary with site size and can range from minutes to hours. In a 2021 tool tutorial, the author reported working with large ecommerce sites containing millions of URLs, where crawls took several hours. This is an individual’s experience, not an indication that every large store will take the same amount of time. Allow time when planning an audit, then adjust based on how the actual site’s crawl progresses.
Why should you pay attention to pagination when analyzing results?
A 2019 SEO article cautioned that poorly handled pagination on large ecommerce sites could result in pages in a series being indexed individually, leading to index bloat. It could also consume a large site’s crawl budget and cause pages to compete with one another for positions in search results. This is particularly worth checking when products come in multiple sizes and colors. However, pagination or separate indexing alone does not establish that a problem exists.
Besides large numbers of filter URLs, how else can you help search engines discover products?
A forum comment suggested creating pages that bring together related products, such as listing books by the same author on one page, so search engines can discover individual books from that page rather than relying solely on large numbers of filter URLs. This is a suggestion for product discovery, not a rule that guarantees search engine indexing. Its suitability still depends on how the site organizes products and provides links.
References
- Using a slash before a question mark in a URL - Does it make a difference to rankings?www.warriorforum.com — Discusses: Besides large numbers of filter URLs, how else can you help search engines discover products?
- How to Create an SEO-Boosting XML Sitemap in 20 Minutes (or Less)neilpatel.com · 2021-09-30T22:00:00+00:00 — Discusses: How long does it usually take to crawl a large ecommerce site?
- Pagination: You’re Doing It Wrong! (Part 1)www.semrush.com · 2019-01-21T09:00:00Z — Discusses: Why should you pay attention to pagination when analyzing results?
- Advanced Screaming Frog Crawling Tips and Use Cases for Ecommerce Auditingwww.screamingfrog.co.uk · 2024-04-08T11:06:08+00:00 — Source used in the article above.

