Before addressing duplicate URLs, decide what you want indexed: should search results use a particular primary URL, or should a page be excluded from the index altogether? When multiple URLs serve identical or similar content, a canonical tag can indicate the preferred version; consider noindex for pages you do not want indexed. robots.txt controls crawling and cannot replace settings that exclude pages from the index. Compare the actual content, too—different URL formats alone do not mean the content is duplicated.
Find potentially duplicate URLs, then check their content and decide what should be indexed
A 2026 article on Neil Patel’s website lists several situations where duplicate content may occur: URLs with tracking codes or filter parameters, HTTP and HTTPS versions, versions with and without www, pagination, and content syndicated across websites. These are starting points for investigation, not reasons to consolidate every paginated or filtered page.
- Check whether different URLs serve identical or similar content, rather than just comparing their paths.
- Confirm whether you need to keep multiple versions and which URL you want to appear in search results.
- Check whether the content management system keeps generating additional versions, so you address more than just the URLs already found.
Another article, published in 2022, suggested using Google or free online duplicate-content checkers to find clues, as well as checking the content management system. These suggestions reflect the options available at the time; they do not mean that any particular tool still offers the same functionality today.
If categories can share one URL, there is no need for a separate page for each
A photo belonging to both the “rabbits” and “fluffy” categories does not necessarily need two photo pages. A discussion on Webmasters Stack Exchange proposed one approach: display the photo on both category pages, but always link to /gallery/rabbits/bunny1 under the primary category, rather than also creating /gallery/fluffy/bunny1.
This is a site-structure approach suggested by the community, not a mandatory search engine rule. The point is to have different categories share a single content URL, not to claim that adding another keyword to a URL improves rankings.
Canonical indicates a preferred version; noindex indicates that a page should not be indexed
According to the article on Neil Patel’s website, a canonical tag is appropriate when multiple URLs serve duplicate or similar content and the website has a preferred version. Without it, Google chooses which URL to index and consolidates ranking signals for that URL. Its choice may not match the website’s preference. A canonical tag therefore signals a preference; it does not guarantee that the specified version will be indexed.
The article also explains which settings to use for different goals and what to watch for:
- Choosing a primary version: Check that the canonical tag points to the intended page. Avoid pointing to URLs that return a non-200 status code, and make sure the same page does not have conflicting canonical settings.
- Keeping a page out of the index: Consider noindex, for example on test pages or thank-you pages that you have confirmed do not need visibility in search. Do not apply it indiscriminately based only on page type.
- Restricting crawling: Use robots.txt, but blocked URLs may still be indexed because of links elsewhere. Blocking crawling does not mean a URL will stay out of search results.
Duplicate URLs and indexing directives: perspectives from across the web
Can a page have a canonical pointing to itself?
Yes. The article on Neil Patel’s website states that a page can use a self-referencing canonical tag and says that Google’s guidance recommends it. If a page has already been selected as the preferred version, this setting can indicate that its own URL is the primary URL. Still, check that the setting matches what the website wants indexed. The tag does not guarantee that search engines will honor that preference.
If you do not want a page indexed, must you also use nofollow?
Not necessarily. noindex indicates that a page should not be indexed, while nofollow concerns how links are handled. They can be used together, but serve different purposes. The article on Neil Patel’s website also lists links you do not want to endorse as a use case for nofollow. Decide whether nofollow is needed based on the links, rather than automatically adding it just because a page uses noindex.
Can a sitemap replace indexing directives?
No, not as a direct substitute: they serve different purposes. The cited article describes a sitemap as a file listing important pages to help search engines discover and crawl them more efficiently. By contrast, noindex indicates that a particular page should not be indexed. If you want important content to be easier to discover, consider a sitemap. If you want a page kept out of the index, decide separately which indexing directive to use.
References
- URL paths and their effect on SEOwebmasters.stackexchange.com — Source used in the article above.
- URL paths and their effect on SEOwebmasters.stackexchange.com — Source used in the article above.
- Robots.txt Explained: Syntax, Best Practices, & SEOwww.semrush.com · 2018-07-10T11:00:00Z — Discusses: If you do not want a page indexed, must you also use nofollow?; Can a sitemap replace indexing directives?
- Meta Tags for SEO: What They Are, Why They Matter, and How to Use Themneilpatel.com · 2026-09-04T16:00:00+00:00 — Discusses: Can a page have a canonical pointing to itself?; If you do not want a page indexed, must you also use nofollow?
- Duplicate Content: 5 Myths and 5 Facts About How It Impacts SEOneilpatel.com · 2022-02-22T21:00:00+00:00 — Source used in the article above.

