Sitemaps can help you quickly take stock of content volume, compare category shares, and explore publishing trends and topic distribution when the URLs and dates provide enough information. Visual sitemaps showing information architecture can also help assess navigation and identify content gaps. But neither type provides a complete picture of a website: URL counts do not represent impact, last-modified dates are not publication dates, and missing or broken URLs also limit interpretation.
First, distinguish between analyzing a content inventory and reviewing site structure
A 2023 NN/g article introduced sitemaps that show content hierarchy: nodes represent different levels, and annotations can describe page functions and distinguish category labels from actual pages. These diagrams are useful for examining how navigation connects different sections and where content is missing.
XML sitemaps, by contrast, provide lists of URLs and may include additional fields such as last-modified dates. A BuzzFeed analysis published by Semrush in 2020 used this information to organize content data. The two serve different purposes: the former shows how content is organized, while the latter provides URLs and metadata that can be aggregated. They are not interchangeable for analysis.
Extract categories from URLs, then compare content volume and topics
How much you can learn about content from URLs depends on the information they contain. In the case study above, the sitemap file paths included categories, making it possible to categorize and count URLs. Article URLs also offered clues such as authors, languages, and title fragments. Depending on the available data, you can proceed step by step:
- Retain each page URL and the sitemap it belongs to, then extract categories from meaningful paths.
- Calculate URL counts and shares for each category to compare content volume within the list.
- Analyze word frequencies in title fragments. For example, select URLs containing “quiz” and examine co-occurring words to explore the topics covered by quiz content.
Once you understand the data’s limitations, you can also explore topic modeling, entity extraction, or clustering. But if URLs contain only product IDs with no semantic meaning, they offer few direct clues about the content. These methods cannot be applied unchanged to every website.
Account for date uncertainty when using lastmod to examine trends
If the data includes lastmod, you can count URLs by the year or month of that date to track changes in volume. However, it records the last modification time, not a reliable initial publication date. The 2020 case study used it to approximate publication dates, but this does not establish that each piece of content went live on that date.
Missing values also need to be examined separately: in that BuzzFeed dataset, 2.1% of URLs lacked lastmod values. All belonged to the video category and could not be included in the date analysis. Those video URLs also provided no content information suitable for analysis. Their share of traffic or revenue was unknown, so their small share of URLs did not establish that their impact was minor.
Counts and historical clues still need corroboration
A category’s share of content does not equate to its impact. In the case study, Tasty accounted for a very small share of the content-count chart, but the author noted at the time that its Facebook page had nearly 100 million followers—a reminder not to assess reach solely by article counts.
Broken URLs are a limitation of the data, but they can also preserve historical clues. Spot checks in the case study returned 404 responses for some URLs, showing that not every page on the list was still accessible. Yet those broken URLs preserved some historical clues because they remained in the sitemap. Old URLs already removed from the sitemap, however, could not be identified from that list. Sitemaps are therefore useful for an initial overview, but important conclusions should still be checked against other sources wherever possible.
Sitemap content analysis and its limitations: perspectives from across the web
Can a sitemap reveal why article volume suddenly increased?
The increase alone cannot establish the cause, but author information and monthly trends can help suggest explanations to investigate. Semrush’s 2020 case study found that many articles came from news organizations and therefore suggested that the increase in April 2011 was more likely related to content partnerships. This remained an inference from the data, not a proven causal relationship. Confirming the partnerships and their effects would require additional sources.
If spot checks find some broken URLs, can you estimate the proportion of broken URLs in the entire list?
A small number of spot checks alone cannot reliably estimate the proportion of broken URLs in the entire list. In the Semrush case study, spot checks of some URLs with HuffPost listed as the author returned 404 responses. However, the author did not check every URL and noted that the entire dataset would need to be checked to quantify the problem. Even a complete check of the list would exclude old URLs already removed from the sitemap.
When using last-modified dates to estimate publishing trends, can you assume that all changes occurred shortly after publication?
This assumption cannot be applied to every website. The author of the Semrush case study observed that changes were usually corrections made within a day or two and that most pages were not updated. But this was the author’s observation, not sufficient evidence that other websites behave the same way. When using last-modified dates as approximations of publication dates, you should account for possible timing errors caused by later updates.
References
- Information Architecture vs. Sitemapswww.nngroup.com · 2023-09-03T16:00:00+0000 — Source used in the article above.
- Content Analysis with XML Sitemaps and Pythonwww.semrush.com · 2020-05-08T07:00:00Z — Discusses: Can a sitemap reveal why article volume suddenly increased?; If spot checks find some broken URLs, can you estimate the proportion of broken URLs in the entire list?


