Publishers do not have to choose between allowing all access and blocking it entirely. Instead, they should decide which content to make accessible and which to restrict based on its value, their revenue sources, and the crawlers involved. Machine access does not necessarily lead to citations, clicks, or revenue; denying access may also mean sacrificing visibility. A practical starting point is to review actual access logs, then distinguish between three things: crawling directives, removing pages from search results, and restricting access. This helps avoid using the wrong tools.
Check who is reading before letting headlines drive decisions
In an August 2026 Search Engine Journal article, Slobodan Manic recommended that typical websites start by monitoring Meta and use server logs to identify who is actually accessing their content, rather than focusing only on companies in the news. This was the author’s recommendation at the time, not a general rule that all websites should delay taking action.
The crawler activity data he cited came from security provider DataDome’s network. It cannot be assumed to represent the entire web or replace a website’s own logs. For publishers, the useful takeaway is his method of checking the evidence, not necessarily his conclusion about whether to block a particular platform.
Separate crawling, search visibility, and access permissions
Screaming Frog explained in July 2025 that robots.txt can specify whether crawling is allowed or disallowed by crawler name and path, but noncompliant bots may ignore it. In other words, crawling directives are not access protection.
- To ask specific crawlers not to read certain areas: use robots.txt to express restrictions, but this cannot guarantee that all access will be blocked.
- To keep a page out of Google search results: Google’s documentation explains that noindex can be used, but it does not prevent access by users or search engines that do not support the tag.
- To allow only authorized users to read content: the same documentation lists password protection, which allows approved users to browse while blocking crawlers such as Googlebot.
For example, members-only content that requires restricted access is not protected simply because it is absent from search results. Google’s documentation also explicitly warns against using robots.txt to remove pages from search results.
Weigh visibility gains against revenue needs
In July 2026, Yoast author Edwin Toonen argued that most websites should allow AI access to improve visibility. However, blocking may also be reasonable if the content is genuinely unique and proprietary and the site already has a monetization strategy beyond search traffic. He also suggested offering content previews to attract clicks when blocking access. These are strategic recommendations, not platform guarantees.
However, in a June 2026 commentary, Roger Montti noted that asking publishers to allow crawling and create content readers want still did not address concerns about declining search referral traffic. Publishers still need to ask whether visibility opportunities are enough to sustain their business. Machine access alone cannot answer that question.
Do not treat licensing negotiations as the only way forward
In his commentary, Manic argued that major platforms’ licensing deals are concentrated among a small number of large media organizations, leaving smaller publishers in a weaker negotiating position. He therefore advocated building stronger direct relationships with readers and developing publisher-owned channels to reduce dependence on platforms, while acknowledging that this takes time. Access management can determine which content is open and which is restricted, but it is not a complete revenue strategy.
How publishers can respond to automated access by platforms: perspectives from across the web
How can publishers organize openly accessible content to help machines understand it?
They can use structured data to clearly define entities and put important facts, such as prices and specifications, in simple tables that machines can parse easily. They should also check that they are not blocking the crawlers or agents they want to reach. These are technical recommendations for real-time retrieval or model training, intended to make content easier for machines to understand. They do not guarantee citations or traffic.
Have the legal boundaries for restricting automated access been settled?
These individual cases do not establish a settled, general rule for restricting automated access. Search Engine Journal noted in July 2026 that the Amazon–Perplexity case examined whether AI agents were authorized visitors under the U.S. Computer Fraud and Abuse Act, while the Google-related case examined whether anti-bot barriers protected copyright under the Digital Millennium Copyright Act. The two cases involve different legal questions and statutes and do not establish that all blocking methods have the same legal effect.
References
- 從 Google 中移除您的網站資訊 | Google 搜尋中心 | Documentation | Google for Developersdevelopers.google.com — Source used in the article above.
- Google Says AI Visibility Hinges On Content People Actually Want To Readwww.searchenginejournal.com · 2026-06-28T11:00:56+00:00 — Source used in the article above.
- Out of Cite: Why Your PR Wins May Be Missing From AI Searchwww.screamingfrog.co.uk · 2025-07-22T09:49:00+00:00 — Source used in the article above.
- The June 2026 SEO Update by Yoast recapyoast.com · 2026-07-01T15:02:35+00:00 — Source used in the article above.
- Everyone Is Negotiating With Google While Meta Reads The Web For Freewww.searchenginejournal.com · 2026-08-18T17:30:31+00:00 — Source used in the article above.
- Google Lost Its Scraping Case – Now You Have To Pick A Side On The Open Webwww.searchenginejournal.com · 2026-07-28T17:30:13+00:00 — Discusses: Have the legal boundaries for restricting automated access been settled?
- Google Is Becoming A Personalizing Mirror Before You Even Type A Querywww.searchenginejournal.com · 2026-06-18T11:00:31+00:00 — Discusses: How can publishers organize openly accessible content to help machines understand it?
