A link that opens now may not have worked when a search engine crawled it earlier. A 500 error in Webmaster Tools may reflect a server error encountered during the last crawl; successful access now is not enough to conclude that the tool reported it incorrectly. Focus on identifying the affected URLs and error times, then checking them against server logs, rather than immediately changing website settings. The case below does not establish a single cause.
Distinguish between a successful test now and a failed crawl earlier
In the Stack Exchange case, the person asking the question said that several links listed as errors still worked and that testing with the then-available “Fetch as Googlebot” returned 200 OK. They also said that Googlebot was not blocked through robots.txt or HTML meta tags. These details indicate successful access at the time of testing, but do not rule out an earlier 500 error.
One respondent suggested that the site may have temporarily returned 500 errors when Google last crawled it. The next crawl could be expected to succeed only if the problem really was temporary downtime and the site continued to function normally afterward. This explanation still needs to be verified; it is not an established conclusion from the case. Also, “Fetch as Googlebot” was the tool’s name at the time, which does not mean the current interface offers the same functionality.
Cross-check URLs, times, and logs
Do not assume that an entire batch of errors has been resolved just because the homepage or a randomly chosen article opens successfully. Check each affected URL listed by the tool:
- List each affected URL with its error date and time. Keep the time recorded by the tool separate from the time of your current test.
- If you have access to the web server, check the access log to see whether Googlebot visited. These logs generally record visit times.
- Check the server logs to determine whether that request actually received a 500 response, then assess whether there is reason to suspect a reporting problem with the tool.
The original questioner said that their provider did not allow access to IIS logs, which limited their ability to investigate. Without records from the time, the checks above cannot be completed. Being unable to check the logs does not mean the server did not encounter an error.
Once a server error is confirmed, follow the evidence
A 2021 Semrush article noted that server errors may involve hosting, plugins, or configuration. Possible causes of 500 errors include faulty plugins and insufficient memory. The article recommended checking error logs, disabling problematic plugins, or increasing resources when needed.
These are avenues for investigation, not a repair checklist for every website. If a site does not use plugins, a plugin-related explanation does not apply. Without evidence of insufficient resources, there is also no basis for assuming that adding resources will fix the problem. In this case, the first step is still to obtain records that match the affected URLs and error times, then decide which type of problem to address.
Investigating 500 errors: perspectives from across the web
If robots.txt does not block Googlebot, is it still worth checking the file?
You can check the file’s contents, but should not treat it as an established cause of the error. In one case, a respondent found invalid characters at the beginning of robots.txt that were invisible when the file was displayed as UTF-8 in a text editor. They thought these might be related to the 500 error and suggested retyping the first line, saving the file, and validating it again. This was a lead specific to that file, not a claim that all 500 errors have this cause.
If articles are still not indexed after robots.txt is fixed, can temporary downtime be ruled out?
No. This observation alone cannot rule out temporary downtime, but it does not establish downtime as the cause either. The person asking the question said that, after fixing robots.txt, all articles using a particular URL format were still not indexed by Google. This led them to suspect another problem. Another respondent suggested temporary downtime during the last crawl as a possibility. The case does not confirm which explanation is correct.
References
- Why do many valid links in my blog show error 500 in Google WebMaster Tools?webmasters.stackexchange.com — Discusses: If articles are still not indexed after robots.txt is fixed, can temporary downtime be ruled out?
- 使用『site:』檢查新網站有被google索引,但GOOGLE仍搜尋不到公司網站www.mobile01.com — Source used in the article above.
- Why do many valid links in my blog show error 500 in Google WebMaster Tools?webmasters.stackexchange.com — Discusses: If robots.txt does not block Googlebot, is it still worth checking the file?; If articles are still not indexed after robots.txt is fixed, can temporary downtime be ruled out?
- What Are Crawl Errors & How Do They Affect SEO?www.semrush.com · 2021-06-10T16:30:00Z — Source used in the article above.
- Why do many valid links in my blog show error 500 in Google WebMaster Tools?webmasters.stackexchange.com — Source used in the article above.
- Why do many valid links in my blog show error 500 in Google WebMaster Tools?webmasters.stackexchange.com — Source used in the article above.



