Do not use robots.txt to block all JavaScript files simply because they are JavaScript. The search crawlers you want to allow should be able to fetch scripts needed to display a page’s main content and links. But allowing every crawler to access every resource on the site is not necessarily better. When explaining the settings in a presentation, start with what the resources do: which scripts generate content that needs to be discoverable in search, and which resources need separate crawling restrictions. Set the rules based on those needs, rather than choosing only between “block everything” and “allow everything.”
First, check whether important content depends on scripts
The key consideration is how much the page depends on scripts. Semrush notes that if important content or internal links appear only after JavaScript runs, blocking resources needed for rendering may prevent Google from seeing that information. Single-page applications require particular attention because navigation and content loading may both happen in the browser.
- Content generation: If scripts load product descriptions, check that the relevant JavaScript is not blocked from crawling.
- Navigation links: If scripts create links to categories or other important pages, also check that those links appear correctly after rendering.
- Other uses: Do not decide whether to allow access based solely on a file extension; first check whether the file is used to display important content.
Websites have different needs, so decisions should depend on what each resource does. The advice above is to allow crawling of necessary resources, not to open every resource on every website.
Allowing crawling does not guarantee indexing
In a 2025 article, Welly describes Google’s process for handling JavaScript pages as crawling, rendering, and indexing. Allowing files to be fetched only makes subsequent processing possible. If core content depends on scripts, Google still needs to run those scripts and render the page to obtain the full content. Whether that content is indexed may also depend on page settings and how the search engine allocates its resources.
Therefore, “JavaScript is no longer blocked” should not be presented as “indexing is guaranteed.” Likewise, you cannot rely on crawl restrictions in robots.txt alone to keep pages out of search results. Semrush notes that even when crawling is prohibited, a URL may still appear in search results because other pages link to it. To control whether a page is indexed, separately review indexing settings such as noindex.
Check rules against the page’s actual needs
Start with an important page. First, check whether the initial HTML already contains the title, main description, and links. Then identify which information appears only after scripts run, and check whether robots.txt blocks the relevant resources. Do not stop at confirming that the page URL can be crawled; check the files it depends on as well.
If key content is missing from the initial HTML, consider making it available earlier as well as allowing necessary scripts to be crawled. Neil Patel recommends considering server-side rendering, while awoo’s 2025 article recommends including information such as titles and main descriptions in the HTML source. These approaches affect when content becomes readable; robots.txt controls whether resources can be crawled. They address different problems and cannot replace one another.
robots.txt and JavaScript: perspectives from across the web
What should you watch for when using a crawling tool to check rendered pages?
When using Screaming Frog SEO Spider’s rendering mode, check whether robots.txt blocks JavaScript and CSS. Also make sure the tool is configured to crawl those resources and external resources. Otherwise, the rendered view may be incomplete. Screaming Frog’s 2016 guidance also warns that scripts executed during rendering may affect analytics tools, so consider excluding scripts that should not run.
Does allowing an AI crawler to fetch scripts mean it can see dynamic content?
Not necessarily. Semrush notes that some AI crawlers do not execute JavaScript and instead rely on the initial HTML, so they may not see content that appears only after rendering. Therefore, even if necessary scripts are allowed, you should still check whether key information is present in the initial response. If the goal is to appear in the ChatGPT search experience, the article also recommends checking that OAI-SearchBot is not blocked.
Can important content wait to load until a user clicks or scrolls?
If you want search engines to see important content, the safer approach is to avoid making it load only after a user clicks or scrolls, and to check that it loads reliably. Semrush cautions against delaying important content or internal links that appear only after rendering, or making them appear only after a user interacts with the page. This is advice about checking content visibility, not a claim that all crawlers handle interactions in the same way.
Can blocking a script that generates links stop search engines from following those links?
This should not be treated as a reliable approach. A respondent in a public discussion speculated that creating links with JavaScript and then blocking that script with robots.txt might, in theory, prevent search engines from following the links. However, they explicitly stated that they had not tested it and thought it might qualify as black-hat or gray-hat SEO. This is only an untested hypothesis, not an established rule that can be relied on today.
References
- How to handle hundred of external links in a site?webmasters.stackexchange.com — Discusses: Can blocking a script that generates links stop search engines from following those links?
- What is technical SEO? Basics and best practiceswww.semrush.com · 2023-02-01T15:56:00Z — Discusses: Does allowing an AI crawler to fetch scripts mean it can see dynamic content?; Can important content wait to load until a user clicks or scrolls?
- Screaming Frog SEO Spider Update – Version 6.0www.screamingfrog.co.uk · 2016-07-21T08:57:40+00:00 — Discusses: What should you watch for when using a crawling tool to check rendered pages?
- JavaScript SEO 是什麼?9 技巧教你優化網頁welly.tw · 2025-01-22T08:30:20.845Z — Source used in the article above.
- Definitive Javascript SEO Guideneilpatel.com · 2023-09-14T19:00:00+00:00 — Source used in the article above.
- JavaScript SEO 完整指南,打造對搜尋引擎友善的網站www.awoo.ai · 2025-06-13T09:26:47+00:00 — Source used in the article above.



