Before a crawl, define which content to check, then configure the low word count threshold, near-duplicate checks, and searches for errors in HTML. The settings and menu paths below come from an article published on the Screaming Frog website in 2023; they do not establish what menus or defaults are available in the current version. The word count and similarity thresholds are the author’s suggested screening criteria, not universal standards for judging content quality.
Define the main content first to avoid counting sitewide elements
Before assessing whether a page has enough content, decide which parts of the page to include in the checks. The article gives the menu path Configuration > Content and recommends adjusting the settings to suit the site’s structure rather than simply using the defaults.
- On their own website, the author excludes the nav and footer sections so that sitewide navigation and footers are not counted as content.
- Whether to exclude aside depends on how the site uses that element; there is no single approach that fits every site.
- If the article section contains all the main content, you can include only that section. You can also customize which sections to include or exclude when they do not use standard semantic HTML.
Set a low word count threshold, but do not automatically classify short content as low quality
The article recommends going to Configuration > Spider > Preferences and scrolling to the bottom to set Low Content Word Count. The default at the time was 0, which the author considered unhelpful for these checks, so they recommended adjusting the threshold.
The author also emphasizes that there is no single “correct word count,” suggesting only that fewer than 100 words may be a useful indicator of thin content. This refers to the tool’s word count; it does not mean that Traditional Chinese content with fewer than 100 characters fails the check. Nor should this rule of thumb be treated as an official quality standard.
Enable near-duplicate checks, then run the analysis after crawling
According to the article, go to Configuration > Content > Duplicates, select Enable Near Duplicates, and choose a similarity threshold:
- 90%: The default recorded in the article at the time, which the author considered sufficient for detecting near duplicates.
- 50%: The author’s recommended setting for finding content that is “too similar.”
Choose between these thresholds based on the purpose of the check; neither should be treated as a requirement for every website. Enabling the feature before crawling is not enough: the article notes that you must run Crawl Analysis after the crawl finishes to perform this check.
Search for errors in the HTML, not just on the visible page
The article recommends configuring error searches through Configuration > Custom > Search. Some PHP errors may be hidden in the HTML, such as loading an undefined PHP variable in a script section. The page may still look normal, but the author warns that other functionality could be affected, so the HTML should also be searched.
If you know which part of the page contains the error, you can restrict the search to that area to find pages with the same type of error. The article does not provide a universal search string, so this advice is not a complete set of error-search rules ready to use on every website.
Content quality checks before a crawl: perspectives from across the web
Can you still define the scope of checks if the site's main content does not use standard semantic tags?
Yes. You can customize which sections to include or exclude based on the site's structure, rather than relying solely on standard semantic HTML. Before assessing content quantity and quality, identify where the main content is located. If sitewide navigation or footers are not part of the content you want to check, you can exclude them. Including only the article section is appropriate only if it actually contains all the main content.
Can a low word count threshold directly determine page quality?
No. It should be used only as a screening indicator. The author suggests that fewer than 100 words may indicate thin content, while emphasizing that there is no fixed, correct word count. Define which content will be counted before setting the threshold. This advice also cannot be directly converted into a character-count rule for Traditional Chinese.
Which similarity threshold should you choose for near-duplicate checks?
It depends on the purpose of the check. The author of the 2023 article considered the then-default threshold of 90% sufficient for detecting near duplicates, but recommended 50% for finding content that is "too similar." These are the author's recommendations, not universal standards. After enabling the feature and setting the threshold, you must also run Crawl Analysis once the crawl has finished to perform the check.
If a page displays normally, is it still worth searching for errors in its HTML?
Yes. The article notes that some PHP errors may be hidden in the HTML without affecting how the page displays in a browser, yet may cause problems elsewhere. If you know which part of the page contains the error, you can restrict the search to that area to find pages with the same type of error. The article does not provide a universal search string.
References
- Using Screaming Frog to Improve Programmatic SEO Contentwww.screamingfrog.co.uk · 2023-06-22T10:54:32+00:00 — Discusses: Can you still define the scope of checks if the site's main content does not use standard semantic tags?; Can a low word count threshold directly determine page quality?

