Can AI crawlers see JavaScript sites? How to test and fix it
GPTBot, ClaudeBot and PerplexityBot didn't run JavaScript in Vercel's 2024 data. How to see what a non-JS crawler sees, then fix with SSR or prerendering.
, 6 min read
Mostly no. In a December 2024 study by Vercel and MERJ, GPTBot, ClaudeBot and PerplexityBot did not execute JavaScript, so any text your page builds in the browser was invisible to them. Google is the exception: Gemini uses Googlebot's rendering, which runs JavaScript.
That finding is almost two years old, and crawlers change. The safe reading in 2026 is simple: if a sentence matters, put it in the HTML your server sends.
What the Vercel and MERJ study found
Vercel and MERJ (December 17, 2024) analysed crawler traffic on nextjs.org and the Vercel network, then checked the results against two job board sites. Their main conclusion: "none of the major AI crawlers currently render JavaScript."
The details that matter for a site owner:
- GPTBot, ClaudeBot and PerplexityBot did not render JavaScript. The study says ChatGPT's and Claude's crawlers fetch JavaScript files but "don't execute them." In the sample, JavaScript files were 11.50% of ChatGPT's crawler requests and 23.84% of Claude's.
- Gemini "leverages Googlebot's infrastructure, enabling full JavaScript rendering," according to the same post.
- AppleBot renders JavaScript "through a browser-based crawler, similar to Googlebot."
- Microsoft Copilot was left out because it had no unique user agent.
The study also found that 34.82% of ChatGPT's crawler fetches and 34.16% of Claude's hit 404 pages, against 8.22% for Googlebot. Broken URLs and stale redirects cost you with these crawlers too.
Read it with caution
This is one dataset from one hosting network, and it is almost two years old. It lists OpenAI's OAI-SearchBot and ChatGPT-User among the non-rendering crawlers but gives detailed figures only for ChatGPT's and Claude's crawlers. It does not cover Claude-SearchBot, and OpenAI's crawler documentation says nothing about JavaScript rendering. Any of these companies could add rendering without announcing it.
So treat "AI crawlers don't run JavaScript" as the cautious default, not a permanent law. Server-rendered HTML works whether or not a crawler executes scripts, which is why it is the safer bet.
What Google says
Google describes three phases for JavaScript pages in its JavaScript SEO basics: crawl, render, index. Rendering happens in headless Chromium "once Google's resources allow," so there can be a delay. The same page still calls server-side rendering or pre-rendering "a great idea" because "not all bots can run JavaScript."
Google's guide to its AI features says it "is able to process content within JavaScript as long as it isn't blocked," and that a page must be indexed and eligible for a snippet to appear in those features. If your JavaScript files sit behind a robots.txt Disallow, Google cannot render them either.
How to see what a non-JS crawler sees
You don't need special tools. These three checks take a few minutes.
1. View source, not Inspect
Open a key page (pricing, a product page, a docs page) and press Ctrl+U (Cmd+Option+U on a Mac), or type view-source: before the URL. This shows the raw HTML the server sent. The Inspect panel shows the DOM after JavaScript ran, which is what you see, not what GPTBot saw.
Use your browser's find to search for a sentence from the visible page. If it is missing, or the body is little more than <div id="root"></div> and a list of <script> tags, a crawler that doesn't execute JavaScript gets an empty page.
2. Fetch it with curl
From a terminal, request the page with a crawler user agent and count matches for a sentence you expect:
curl -s -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" \
https://yourdomain.com/pricing | grep -c "Plans start at"
0 means the text is not in the HTML. Also try a normal browser user agent: if the bot version returns less content or a challenge page, your CDN or firewall treats bots differently, which is a separate problem. Your request comes from your own IP, not OpenAI's, so treat differences as a hint.
3. Turn JavaScript off
In Chrome DevTools, open the Command Menu (Ctrl+Shift+P, or Cmd+Shift+P on a Mac), type "Disable JavaScript", press Enter and reload. What is left on screen is roughly what a non-rendering crawler can read. Check the main heading, the opening paragraph, prices, feature lists and FAQ answers.
For Google specifically, Search Console's URL Inspection tool shows the HTML that Googlebot rendered.
How to fix it
The goal is the same in every stack: the server sends HTML that already contains your main content. JavaScript can still add interactivity on top. Vercel's post recommends server rendering (SSR, ISR or SSG) for important content and keeping client-side rendering for extras such as view counters, live chat and social feeds.
Server-side rendering or static generation
- Server-side rendering (SSR): the server builds the HTML on each request. Good for pages that change often.
- Static generation (SSG): pages are built to HTML at deploy time. Good for marketing pages, docs and blog posts, and cheap to host.
Most modern frameworks do one of these by default. The usual problem is a page that opted out, or one that loads its main content in the browser after the first render.
| Framework | Default | What to check |
|---|---|---|
| Next.js (App Router) | Pages are Server Components | Content fetched inside useEffect in a 'use client' component is not in the first HTML. Fetch it in the Server Component instead. |
| Nuxt | Universal (server) rendering | Make sure ssr: false is not set in nuxt.config.ts. |
| SvelteKit | Server render, then hydrate | export const ssr = false sends an empty shell. Use export const prerender = true for static pages. |
| Astro | Pages prerendered at build time | Content loaded only by client-side scripts is still invisible. |
| Angular | Client-side unless SSR is enabled | Run ng add @angular/ssr; use RenderMode.Prerender for static routes. |
| React Router | Depends on config | With ssr: false, add a prerender list in react-router.config.ts. |
Prerendering for single-page apps
If you run a plain client-side React, Vue or Angular app and a migration is not on the table, prerender the pages that matter. Build-time prerendering generates an HTML file per route during your build, so every visitor and crawler gets the same content. React Router's prerender option and Angular's prerender mode both do this. You don't need every route: the homepage, pricing, product pages, docs and comparison pages cover most of what an assistant would quote.
The older alternative is dynamic rendering: detecting bots by user agent and sending them a prerendered snapshot from a separate rendering server or a hosted service. Google now says dynamic rendering "was a workaround and not a long-term solution" and recommends server-side rendering, static rendering or hydration instead. It also depends on a bot list that you must keep current as new AI user agents appear.
A short checklist
- View source on your top 5 to 10 pages shows the main heading and first paragraph
- Prices, plan names and feature lists are in the raw HTML, not loaded by a script
- FAQ answers are in the HTML, even if they display in an accordion
- Robots.txt does not block your JavaScript or CSS files from Googlebot
- Old URLs redirect to live pages instead of returning 404
- No
noindexon pages you want found
What this will and won't do
Getting your content into the HTML removes a reason an AI assistant can't use your page. It does not make the assistant cite you. Which sources get cited depends on many things, and most published AEO research on those factors is correlational and comes from vendors. Readable HTML is a precondition, not a ranking lever.
Check your site
The free scan at domyaeo.com flags pages that rely on JavaScript-only rendering and checks whether key pages open with a direct answer. It also reads your robots.txt rules for 14 AI and search crawlers, checks Cloudflare Content-Signal and noindex, and tests how your CDN or firewall responds to AI crawler user agents (a hint, not proof).