How can you prevent or handle unwanted duplicate HTTPS pages?

ProcrustesPR

To handle unwanted duplicate HTTPS pages, first identify which URLs serve duplicate content and which version you want to keep. Then fix the links pointing to those pages and, as needed, use 301 permanent redirects or canonical tags to specify the preferred version. Do not assume a page should be removed simply because its URL uses HTTPS. The advice to redirect HTTPS back to HTTP comes from a specific situation described in a 2011 article and should not be applied directly to websites today.

ProcrustesPRProcrustesPRHow can I tell whether an SEO tactic breaks spam rules?

Confirm the duplicate content, then trace the links to those URLs

Semrush’s 2020 article recommends first identifying duplicate pages through a site audit, then deciding how to handle them. For example, if HTTP and HTTPS URLs with the same path serve identical content, determine which version you intend to keep rather than assuming one is problematic simply because the protocols differ. Finding duplicate content in an audit does not, by itself, confirm that search engines have indexed both versions.

Screaming Frog’s 2011 article also advises tracing internal or external links to duplicate HTTPS pages when they appear. Start by checking links on login pages, shopping carts, and other crawlable pages. If you cannot find the source within the site, consider the possibility of external links.

Prevention: point links to the version you intend to keep

The Screaming Frog article notes that relative URLs such as “/product/” on HTTPS pages do not specify a protocol. They generally inherit HTTPS from the page they appear on, though settings such as a base tag may change this behavior. As a result, even if a link was intended to point to another version, crawlers may still follow it to an HTTPS URL.

To avoid this, check whether links point to the intended version. If necessary, use full URLs that include the protocol to point explicitly to the preferred version. This simply sets the links’ destinations; it does not mean product pages must use HTTP, nor does it replace the need to handle existing duplicate pages.

Choose an approach based on whether the pages need to be kept

  • The duplicate version is no longer needed: Semrush’s 2020 article recommends redirecting it to the original page with a 301 redirect to consolidate unwanted duplicate URLs.
  • You need to specify a preferred version: Use a canonical tag to tell search engines which page you prefer. The 2011 article also lists this as an option to consider when a 301 redirect is not possible.
  • Both pages need to provide their own content: Make their content distinct rather than keeping identical content and expecting them to be treated as different pages.

The direction of redirects must match the actual migration goal. Semrush’s 2021 article lists migration from HTTP to HTTPS as a common use of 301 redirects. By contrast, the HTTPS-to-HTTP redirects described in Screaming Frog’s 2011 article addressed a situation in which HTTP was the preferred version at the time. The 2011 article also cautions against using robots.txt to block pages you plan to redirect: blocking crawling and consolidating duplicate URLs are not the same thing.

ProcrustesPRProcrustesPRShould I rewrite or update an old article that does not rank?

Duplicate HTTPS pages: perspectives from across the web

What can you do if you do not want shopping cart or login pages to appear in search indexes?

A meta noindex tag is one option to consider. Screaming Frog's 2011 article described it as the ideal approach for keeping shopping cart or login pages out of search indexes. The article also mentioned using robots.txt to block crawling, but cautioned against blocking HTTPS pages that you wanted indexed or that might already have link equity. These were recommendations at the time; the article alone cannot confirm search platforms' current rules.

Do internal links to login pages and shopping carts need nofollow?

Not necessarily. Screaming Frog's 2011 article suggested considering nofollow for internal links pointing to login pages or shopping carts, but said this was generally unnecessary if the other steps in the article could be followed. This was simply an option at the time, not a current search platform requirement, and it should not be taken to mean that all duplicate HTTPS pages need nofollow.

ProcrustesPRWill Reposting Social Content on My Website Cause a Duplicate-Content Penalty?

References

  1. How to Set Up 301 Redirects in an .htaccess Filewww.semrush.com · 2021-09-17T14:20:00Z — Source used in the article above.
  2. 5 Easy Steps To Fix Secure Page (https) Duplicate Contentwww.screamingfrog.co.uk · 2011-03-23T14:57:50+00:00 — Discusses: What can you do if you do not want shopping cart or login pages to appear in search indexes?; Do internal links to login pages and shopping carts need nofollow?
  3. SEO mistakes: Common SEO issues & how to fix themwww.semrush.com · 2020-04-16T12:17:00Z — Source used in the article above.

Related articles

How can I tell whether an SEO tactic breaks spam rules?Should I rewrite or update an old article that does not rank?Will Reposting Social Content on My Website Cause a Duplicate-Content Penalty?What types of URLs are there, and what are they used for?What are the common mistakes, limitations, and costs of using subdomains?Can two URLs point to the same website?How to improve on-site content and structure: audit pages first, then address content gaps and internal linksHow can you use multiple domains to build content and traffic while assessing the risks?
ProcrustesPR
ProcrustesPR