You can add it, but it is neither the only way to provide a sitemap to Google nor a required setting. According to Google’s official documentation, you can specify a sitemap URL in robots.txt or submit it through Search Console or its API. Adding this line lets Google discover your sitemap the next time it crawls robots.txt. If you already use another submission method, this step is optional. Whichever method you use, submission is only a hint and does not guarantee that Google will download the sitemap or crawl the URLs it contains.
Choose a submission method based on your needs
Here, robots.txt provides the sitemap’s location. Search Console also offers reports showing when Googlebot accessed the sitemap and any potential processing errors. Choose based on what you need:
- To provide the sitemap’s location through robots.txt: add a Sitemap declaration.
- To check access times and processing errors: submit through Search Console and review its reports.
- To submit programmatically: use the Search Console API.
These are all methods listed in the official documentation, so leaving a sitemap URL out of robots.txt does not mean Google cannot access it.
How to configure it: add the sitemap URL on a single line
According to Google’s documentation, you can add the following line anywhere in robots.txt, replacing the example URL with your sitemap’s actual URL:
Sitemap: https://example.com/my_sitemap.xml
If you have multiple sitemaps, you can list them on separate lines. The documentation states that there is no limit to the number of sitemaps you can include in robots.txt. Google will find these locations the next time it crawls robots.txt, but that does not guarantee it will download the sitemaps or crawl the pages they list.
Hosting sitemaps for multiple sites: requirements differ by submission method
A sitemap does not have to be hosted on the site whose URLs it lists. According to the official documentation, sitemaps for multiple sites can be hosted centrally, but the requirements differ by submission method:
- Through Search Console: You must verify ownership of all included sites. You can then create one or more sitemaps containing URLs from multiple sites.
- Through robots.txt: You must create a separate sitemap for each site, containing only that site’s URLs, and point to the corresponding sitemap in each site’s robots.txt.
For example, a sitemap for example.com can be hosted at sitemaps.example.com. If you submit it through robots.txt, example.com’s robots.txt should point to that file, and the sitemap must contain only example.com URLs. The approach described above that combines URLs from multiple sites does not apply here.
Public sitemap URLs and fetch errors are separate issues
A 2021 article by Da Vinci Digital Technology recommended avoiding public disclosure of sitemap URLs because they could reveal important files and the number of pages on a site. However, the same article also recommended automatically generating sitemaps and pointing to them from robots.txt. These recommendations are not entirely consistent and require a trade-off; they should not be read as a Google rule against providing sitemap locations in robots.txt.
Separately, Search Engine Journal reported in October 2026 that Search Console may still display “Couldn’t fetch” even when a sitemap is valid, publicly accessible, and its URL is correctly listed in robots.txt. Adding the declaration therefore does not guarantee that this error will be resolved. If a problem occurs, review the error information in Search Console rather than repeatedly adding the URL.
Sitemap URLs in robots.txt: questions and perspectives from across the web
Does adding a sitemap URL determine which pages can be crawled?
No. A Sitemap declaration only provides the sitemap's location; it does not set crawl rules. User-agent specifies which crawler the rules apply to, while Allow and Disallow specify paths that crawlers may crawl or are asked not to crawl. You cannot adjust these path rules through the Sitemap line alone. Providing the sitemap's location also does not guarantee that every URL in it will be crawled.
If a site generates its sitemap automatically, do I still need to provide its location manually?
Automatic sitemap generation alone does not tell you whether you still need to provide its location separately. Generating a sitemap and providing its location are different steps. A 2021 Da Vinci article recommended having the site generate its sitemap automatically and then pointing to it from robots.txt. Google also lists Search Console and API submission methods, so manually adding the URL to robots.txt is not the only option. Whichever method you use, submission is only a hint. Even an automatically updated sitemap does not guarantee that Google will download it or crawl the URLs it contains.
If the URL is already listed but Search Console still shows “Couldn't fetch,” does that mean the declaration must be wrong?
Not necessarily. Search Engine Journal reported in October 2026 that Search Console may still display “Couldn't fetch” even when a sitemap is valid, publicly accessible, and correctly linked in robots.txt. This message alone is therefore not enough to conclude that the declaration is incorrect. You can use Search Console to check access times and potential processing errors, then investigate further.
References
- 建立並提交 Sitemap | Google 搜尋中心 | Documentation | Google for Developersdevelopers.google.com — Discusses: Does adding a sitemap URL determine which pages can be crawled?; If a site generates its sitemap automatically, do I still need to provide its location manually?
- Google Explains The “Couldn’t Fetch” Sitemap Errorswww.searchenginejournal.com · 2026-10-06T09:52:43+00:00 — Discusses: If the URL is already listed but Search Console still shows “Couldn't fetch,” does that mean the declaration must be wrong?
- Sitemap是什麼?一次搞懂網站地圖提交www.da-vinci.com.tw · 2021-08-17T20:33:00+08:00 — Discusses: If a site generates its sitemap automatically, do I still need to provide its location manually?
- robots.txt是什麼? robots.txt設定、用途與使用範例分享!welly.tw · 2022-01-13T05:20:36.169Z — Discusses: Does adding a sitemap URL determine which pages can be crawled?



