An XML sitemap should include canonical URLs that you want search engines to index, that offer value, and that are accessible. It should not simply include every link a generator finds, nor should it be limited to the main navigation. Older articles that still meet these criteria do not need to be excluded just because they can already be discovered through internal links. A sitemap provides supplementary information to help search engines discover content; omission does not prevent indexing, and inclusion does not guarantee it.
First, decide which pages deserve to appear in search results
Focus on each page’s purpose and status, rather than its age or position in the site hierarchy. Product pages, articles, and other important landing pages should all be checked against the same criteria. Based on relevant recommendations from Tenten and Semrush, you can start with this checklist:
- Keep the primary version of each URL you want indexed, and avoid including alternative versions of the same content.
- Confirm that the page returns a 200 status code rather than a redirect, a page-not-found response, or a server error.
- Exclude URLs marked noindex or blocked from crawling by robots.txt to avoid conflicts between the list and the access settings.
For example, after an article moves to a new URL, remove the old URL that redirects, then check whether the new URL meets the inclusion criteria rather than keeping both versions.
Older articles can remain listed even if they are already discoverable
Public discussions offer differing views on whether sitemaps are necessary: some argue that internal links are sufficient for search engines to discover content, so there is no need to list every page. Another recommendation is that, if you use a sitemap, it should cover all content you want crawled and indexed, not just new articles.
These are separate questions: whether a site needs a sitemap is not the same as whether a sitemap should contain only new content. For an established blog, older articles that remain valuable and that you want to appear in search results can stay listed. The existence of internal links should not be treated as a reason they must be removed, nor should a sitemap be treated as a replacement for internal links.
Sorting and pagination URLs should not be kept or removed indiscriminately
Not every link a generator finds represents a page worth indexing separately. Public discussions suggest avoiding URLs for different sort orders because they may simply present the same content in a different sequence.
Pagination requires a separate assessment: Screaming Frog advises checking for tag pages and pagination URLs you do not want included, while another discussion advocates allowing paginated pages in the main article listing to be indexed. A blanket rule to remove all pagination URLs is therefore not appropriate. First, determine whether the page is worth retaining and whether it is the version you want indexed, then decide whether to include it.
Apply your filtering criteria in the generator settings
When choosing or reviewing a sitemap feature, check whether it supports excluding individual pages and entire folders, rather than requiring you to accept the full automatically generated list. A Search Engine Journal article identifies this type of control as a consideration when choosing a system, but that does not mean every platform offers it.
After configuring the settings, inspect the URLs actually generated. Update the sitemap when pages are added, removed, or substantially revised. If the system does not update it automatically, arrange manual maintenance so that the listed URLs stay aligned with the site’s current state.
XML sitemaps: perspectives from across the web
If a URL is removed from the sitemap, will search engines stop indexing it?
Not necessarily. A sitemap supplements regular crawling, and URLs not listed in it may still be discovered and indexed through other means. Conversely, inclusion does not guarantee indexing. If the goal is to prevent a page from being indexed, removing its URL from the sitemap is not enough; you also need to check the page’s indexing and crawling settings.
How can you configure a sitemap for different language or regional versions?
If a page has corresponding language or regional versions, you can provide hreflang information in the XML sitemap. One approach is to create a separate URL entry for each version, with each entry identifying itself and all corresponding versions to help search engines understand how they relate. Pages without such localized versions do not need additional variants created for this purpose.
What date and time should lastmod contain?
If you provide lastmod, use the date and time of the page’s last meaningful update. Do not change it simply because the sitemap was regenerated. Search engines can use lastmod to help prioritize crawling, so the site’s update process should ensure that this timestamp reflects actual changes to the page.
How should you split a sitemap when there are many URLs?
A single sitemap must not exceed 50 MB uncompressed or 50,000 URLs; if either limit is exceeded, split it into multiple sitemaps. You can also organize sitemaps by content type, such as pages, articles, or videos, then create an index file listing the individual sitemaps and submit that file’s URL. Grouping sitemaps by type makes them easier to manage and helps you identify what each contains, but you still need to check that the URLs in each sitemap meet the inclusion criteria.
References
- What Is a Sitemap? Website Sitemaps Explainedwww.semrush.com · 2021-05-24T11:55:00Z — Discusses: How should you split a sitemap when there are many URLs?
- 寫給電商的技術SEO指南 | 2023年檢查清單tenten.co · 2023-09-04T07:11:05.000Z — Source used in the article above.
- Your CMS Is Still Failing SEO Basics – 11 Fixes To Demandwww.searchenginejournal.com · 2026-11-04T15:00:12+00:00 — Source used in the article above.
- Conducting an SEO Content Audit: 6 Things You Might Be Overlookingwww.screamingfrog.co.uk · 2021-11-15T13:37:48+00:00 — Source used in the article above.
- How to Submit a Sitemap to Google (in 3 Simple Steps)www.semrush.com · 2021-06-16T21:05:00Z — Discusses: If a URL is removed from the sitemap, will search engines stop indexing it?; How can you configure a sitemap for different language or regional versions?
- 301 Redirects: How to Use Them & How They Affect SEOwww.semrush.com · 2020-05-08T15:32:00Z — Source used in the article above.
- Should I include everything in the sitemap or only new content?webmasters.stackexchange.com — Source used in the article above.
- Should I include everything in the sitemap or only new content?webmasters.stackexchange.com — Discusses: If a URL is removed from the sitemap, will search engines stop indexing it?
- Should I include everything in the sitemap or only new content?webmasters.stackexchange.com — Source used in the article above.
- Should I include everything in the sitemap or only new content?webmasters.stackexchange.com — Discusses: If a URL is removed from the sitemap, will search engines stop indexing it?
- How to create a good sitemap for dynamic websitewebmasters.stackexchange.com — Source used in the article above.
- Hreflang Attributes & Tags Explained for Beginnerswww.semrush.com · 2019-06-21T14:30:00Z — Discusses: How can you configure a sitemap for different language or regional versions?

