How do you check for duplicate pages and broken links?

ProcrustesPR

To check for duplicate pages, use search results and indexing data to find URLs worth checking, then use a website crawler to compare body content, titles, and descriptions. To check for broken links, identify both the URLs that no longer work and the pages linking to them. Both checks can be part of the same website audit, but assess the results separately: for similar content, check which parts are duplicated; for broken links, return to the pages containing them to make fixes.

ProcrustesPRProcrustesPRHow can I tell whether an SEO tactic breaks spam rules?

Find URLs worth checking, then compare the actual content

A 2022 article on Neil Patel’s website recommends searching Google with site: followed by your domain to see whether unexpected pages appear. You can also compare the number of indexed pages in Search Console with the number you expect to be indexed. These methods can help you find clues, but differences in page counts alone cannot establish which pages have duplicate content.

For a closer comparison, you can use Siteliner or Screaming Frog. A 2023 Semrush article gives this example: if two similar leather phone cases share the same product description, changing a few words may still leave them with duplicate content. So do not just check whether URLs or product names differ—you also need to read the body content.

Check duplicate body content separately from duplicate titles

A 2023 Screaming Frog tutorial provides a more specific workflow. The interface labels below are those used in that article; check them against the version you are using:

  • In the Content report, use Exact Duplicates and Near Duplicates to review identical and nearly identical pages, respectively.
  • Check the No. Near Duplicates column to identify pages with content similar to multiple other pages.
  • Select a URL, then open Duplicate Details to compare content differences between similar pages.
  • Check Page Titles, H1, and H2 separately. Missing or duplicate meta descriptions can also be included in the review.

When organizing the results, specify whether the duplication affects body content, titles, or descriptions rather than grouping everything into one category. Once you confirm duplicate body content, identify the primary version you want to keep before deciding how to proceed. The Semrush article suggests two approaches: set a canonical on duplicate pages pointing to the primary version, or rewrite the content to make each page distinct.

Find both the broken URLs and the pages linking to them

A 2021 Semrush Site Audit tutorial notes that the broken links report lists both the URLs that no longer work and the pages containing those links. Use both pieces of information to fix the links one by one, rather than simply listing broken URLs. If you only want to check a single article, a 2023 article on Neil Patel’s website also introduces browser extensions that scan links one page at a time.

  • If the original resource has simply moved, find its new location and update the link.
  • If the original resource is gone, link to an alternative resource with comparable content and quality.
  • If there is no suitable replacement, remove the broken link rather than linking to an unrelated page.

This sequence of fixes comes from the recommendations in the 2023 article on Neil Patel’s website. These steps provide practical guidance, but fixing broken links does not guarantee higher rankings.

ProcrustesPRProcrustesPRShould I rewrite or update an old article that does not rank?

Checking for duplicate pages and broken links: perspectives from across the web

Which duplicate pages is Search Console useful for checking?

Start with the Pages report under Indexing to find URLs flagged as duplicates, then compare their actual content with the version you intend to keep. A 2024 tutorial recommends this approach but also notes that it requires indexed pages. An unindexed test site therefore cannot rely on this method alone.

How can you check links before a website goes live if it cannot be indexed?

You can check links before launch using a crawler that can bypass indexing restrictions and access the test site. A Semrush case study used this approach because the test site was not yet indexable. The issues found were then passed to developers to fix, followed by another check before launch.

How can you keep finding new broken links?

Schedule regular website audits rather than checking only once when the site is built. Semrush’s Site Audit tutorial notes that you can schedule daily or weekly audits and review broken links and the pages containing them in the reports. This is useful when you need to monitor website changes on an ongoing basis.

ProcrustesPRWill Reposting Social Content on My Website Cause a Duplicate-Content Penalty?

References

  1. Ecommerce SEO Checklist: 22 Tips to Grow Your Traffic & Saleswww.semrush.com · 2023-11-29T12:22:00Z — Source used in the article above.
  2. Screaming Frog是什麼?5分鐘教會你檢測網站SEO!welly.tw · 2022-04-25T03:52:32.809Z — Source used in the article above.
  3. What Are Canonical Tags?neilpatel.com · 2022-09-10T07:00:00+00:00 — Source used in the article above.
  4. 重複內容影響SEO成效, 如何正確使用Canonical?www.da-vinci.com.tw · 2024-08-15T20:49:01+08:00 — Discusses: Which duplicate pages is Search Console useful for checking?
  5. 8 Steps to Perform a Technical SEO Audit for a Client’s Blogwww.semrush.com · 2021-11-22T10:50:00Z — Discusses: How can you check links before a website goes live if it cannot be indexed?
  6. How Updating Your Old Content Can Help With Your Rankingsneilpatel.com · 2023-04-03T19:00:00+00:00 — Source used in the article above.
  7. Using Screaming Frog to Improve Programmatic SEO Contentwww.screamingfrog.co.uk · 2023-06-22T10:54:32+00:00 — Source used in the article above.
  8. HTML Anchor Element: A Simple Guide for Beginnerswww.semrush.com · 2021-05-19T08:59:00Z — Discusses: How can you keep finding new broken links?

Related articles

How can I tell whether an SEO tactic breaks spam rules?Should I rewrite or update an old article that does not rank?Will Reposting Social Content on My Website Cause a Duplicate-Content Penalty?What types of URLs are there, and what are they used for?What are the common mistakes, limitations, and costs of using subdomains?Can two URLs point to the same website?How to improve on-site content and structure: audit pages first, then address content gaps and internal linksHow can you use multiple domains to build content and traffic while assessing the risks?
ProcrustesPR
ProcrustesPR