How can you analyze server logs for free? Start with manual checks and a small subset of records

ProcrustesPR

If you already have access to server logs, you can start analyzing them for free by examining request records directly, without buying an analyzer first. Use timestamps, requested URLs, and user agents to identify which pages search engine bots visited, how often they visited, and whether they encountered errors or redirects. The free tools discussed below were described in older articles, so there is no guarantee that they are still available or offer the same free allowances.

ProcrustesPRProcrustesPRHow should I divide my DIY SEO budget?

Choose a period to examine and check which fields your logs contain

A Semrush tutorial suggests starting with logs from the past 30 days. This is a convenient starting point, not a mandatory rule. First, confirm the dates covered by the records, then check fields such as timestamps, requested URLs, IP addresses, request types, data transfer volumes, and user agents. The questions you can answer will depend on what the files actually record.

You do not need to organize all traffic from the outset. If your question is whether a search engine bot visited a particular page, start by looking for requests to that URL. To investigate on-site search requests, the advice on Stack Exchange is to find records related to the search form, then inspect the user agents and, if necessary, perform reverse IP lookups for more information.

Start with three types of clues, without a paid analyzer

  • Which URLs were visited: Check the pages you want to examine against the logs for your chosen period, noting which have request records and which do not. Finding no records in these logs does not mean a page has never been crawled.
  • How often requests occurred: Count requests by date or page group to check for sudden increases or decreases in activity. The Semrush tutorial notes that such changes may reflect a shift in crawling behavior or point to an issue worth investigating. Fluctuations alone do not establish the cause.
  • How the server responded: If the logs include status codes, isolate error and redirect records and examine them alongside URLs and timestamps. For example, when investigating a page’s redirect, first confirm the response to that request, then check for subsequent requests to the destination URL.

Consider free tools, but keep the age of the information in mind

A 2023 article from tenten stated that the Screaming Frog SEO Log File Analyser could analyze up to 1,000 lines for free at the time. It was a free option for small-scale analysis then, but you cannot assume that the same allowance is still available. Nor should findings from a small set of records be treated as a complete picture of the site.

A 2016 Semrush article introduced the free, open-source OnCrawl ELK, which could organize data on unique pages crawled by Google, crawl frequency for each page group, status codes, and active and inactive pages. These were the features described at the time. The old article alone cannot confirm whether the tool is still maintained or compatible with current deployment environments, so its installation instructions should not be treated as up to date.

ProcrustesPRProcrustesPRWhy is AI not mentioning my brand?

Analyzing server logs for free: perspectives from across the web

Why check server logs if you already have a web analytics tool?

The two may capture different requests. If a crawler requests only the raw HTML and does not execute JavaScript, it will not trigger client-side analytics tags and therefore will not appear in tag data from GA4, Adobe, or Amplitude. Server logs may provide clues in this situation, but whether you can find those requests still depends on whether they were recorded and whether the logs cover the relevant period.

How can you use logs to trace the source of on-site search requests?

Start by finding requests associated with the search form, then inspect their user agents. If necessary, you can also perform reverse IP lookups for more information. This helps narrow the investigation; it does not mean that a user agent or reverse lookup result alone can establish a requester's identity or intent. The analysis also covers only the records in the available logs.

Can logs prove that a search engine bot did not follow a redirect?

Not necessarily. Logs can help with an investigation, but conclusions should be limited to the observable records. After reviewing logs, one site administrator reported that Google requested an unencoded URL and received a redirect but did not follow it to the encoded version. The administrator believed Google treated it as a redirect to itself. This was an observation and interpretation specific to that site, not a general rule for all redirects.

ProcrustesPRDoes an AI brand mention bring website visitors?

References

  1. Is google submitting my search box form?webmasters.stackexchange.com — Discusses: How can you use logs to trace the source of on-site search requests?
  2. Google isn't indexing URLs after a redirect that differs only in percent-encoding/decoding?webmasters.stackexchange.com — Discusses: Can logs prove that a search engine bot did not follow a redirect?
  3. 免費 SEO 軟體推薦:29 款幫您提升排名的工具tenten.co · 2023-12-10T11:39:17.000Z — Source used in the article above.
  4. What Is Googlebot? How Google's Web Crawler Workswww.semrush.com · 2024-08-23T14:32:00Z — Source used in the article above.
  5. The New Analytics Blind Spot: AI Traffic & AI Agents That Shop Without Visiting Your Sitewww.seerinteractive.com · 2026-07-22T00:00:00-05:00 — Discusses: Why check server logs if you already have a web analytics tool?
  6. Log File Analysis: SEO Advantages and How To Do It for Freewww.semrush.com · 2016-09-07T11:09:17Z — Source used in the article above.
  7. What Is a Log File Analysis? & How to Do It for SEOwww.semrush.com · 2021-01-04T14:45:00Z — Source used in the article above.

Related articles

How should I divide my DIY SEO budget?Why is AI not mentioning my brand?Does an AI brand mention bring website visitors?How Long Does It Take to Evaluate SEO Results?How do I write titles customers want to click?How is organic traffic different from paid traffic?Why do I have keyword rankings but no customers?Are Google Maps rankings the same as website SEO?
ProcrustesPR
ProcrustesPR