To verify Googlebot, you cannot rely solely on the User-Agent in the request headers. Google recommends performing a reverse DNS lookup on the source IP address or checking it against Googlebot’s IP ranges. Before restricting access, be clear about your goal: to prevent crawling, use robots.txt; to keep a page out of the search index, use noindex; to prevent unauthorized users from reading content, use access controls such as password protection. These three measures are not interchangeable. In particular, blocking crawling does not guarantee that a URL will not appear in search results.
Verify the source first—do not block based on a name alone
Google warns that other crawlers often spoof Googlebot’s HTTP User-Agent. A log entry identifying a request as Googlebot is therefore not enough to establish that it came from Google. Before deciding to block suspicious traffic, verify the source of the request:
- Use a reverse DNS lookup to verify the request’s source IP address.
- Alternatively, check the source IP address against Googlebot’s IP ranges.
Blocking Googlebot does more than prevent a crawler from reading a single page, which is why verification matters. Google states that blocking it affects Google Search, including Discover and various search features, as well as products such as Images, Videos, and News.
Handle crawling, indexing, and confidentiality separately
When choosing how to restrict access, start with the outcome you want rather than relying on robots.txt for every requirement:
- Prevent crawling: Use robots.txt to instruct Googlebot not to crawl specified content. The URL may still appear in search results, however.
- Prevent indexing: Use noindex to control whether a page is indexed, not who is authorized to read its content.
- Restrict access: Use methods such as password protection to limit content to authorized users, rather than merely issuing instructions to crawlers.
Keeping links private is not a reliable way to maintain confidentiality either. Google’s documentation notes that when users follow links from a secret site to other websites, the source URL may appear in referrer information, which the other site could log or even publish.
You can restrict video access while still allowing Googlebot to crawl
If you want to prevent others from accessing videos while still allowing Google to crawl them, Google’s video sitemap documentation recommends verifying Googlebot. This guidance applies specifically to video content and player URLs. It does not mean that submitting a sitemap lets Google bypass a site’s restrictions.
The documentation requires files referenced in a video sitemap to be accessible to Googlebot. They must not be blocked by robots.txt or a firewall, nor require Googlebot to log in. Any access restrictions must therefore still allow Googlebot to crawl these files.
Before blocking resources, make sure the page can still load
In November 2026, Search Engine Journal reported Mueller’s explanation of age-verification pop-ups: if the associated embedded resource is blocked through robots.txt, Googlebot will not fetch it or use it to render the page. However, the site must still load properly without that resource.
This guidance applies under specific conditions; it does not mean that blocking any JavaScript is safe. If a single JavaScript file handles many functions, Mueller advises against blocking the entire file.
Restricting and verifying Googlebot: common questions
What should you check when Search Console reports a 401?
A 401 means the request is unauthorized. If you intended Googlebot to be able to access the page, check whether the page requires a login, whether IP-based access restrictions are blocking Googlebot, and whether crawler-specific settings are misconfigured, then test access again. If the page is intended only for authorized users, first confirm that the restriction matches your intent. You do not need to remove protection simply because a 401 appears.
What happens when you add password protection to a private page that Google has already indexed?
If you intend to restrict access by both general users and crawlers, password protection can limit viewing to authorized users while also preventing Googlebot from accessing the page. Semrush states that if the page was previously indexed, Google will eventually remove it from search results. However, this guidance does not specify how long removal takes, so you should not assume the page will disappear from search results immediately after password protection is enabled.
Does a mismatched verification signal prove that a request is impersonating Googlebot?
You should not conclude that a request is impersonating Googlebot based on a single mismatched signal. Google recommends a reverse DNS lookup on the source IP address or checking it against Googlebot's IP ranges. In its discussion of crawler verification, iThome cautions that a failed list retrieval or a DNS timeout should be treated as an unknown result, and that an unconfirmed identity does not mean impersonation. An ASN can provide supporting evidence, but cannot establish identity on its own; different signals should be recorded separately.
References
- 影片 Sitemap 和範例 | Google 搜尋中心 | Documentation | Google for Developersdevelopers.google.com — Source used in the article above.
- Googlebot 簡介 | Google 搜尋中心 | Documentation | Google for Developersdevelopers.google.com — Discusses: Does a mismatched verification signal prove that a request is impersonating Googlebot?
- Google Search Console Errors: Identifying and Fixing Themwww.semrush.com · 2024-03-20T10:51:00Z — Discusses: What should you check when Search Console reports a 401?
- Google Says It’s Okay To Block Some JavaScript Resources With Robots.txtwww.searchenginejournal.com · 2026-11-03T11:16:50+00:00 — Source used in the article above.
- UA 會騙人:用 IP 與 ASN 撕下 AI 爬蟲的偽裝,實測那些冒名頂替的騙子ithelp.ithome.com.tw — Discusses: Does a mismatched verification signal prove that a request is impersonating Googlebot?
- What Is Googlebot? How Google's Web Crawler Workswww.semrush.com · 2024-08-23T14:32:00Z — Discusses: What happens when you add password protection to a private page that Google has already indexed?



