Executive Summary · Key Takeaways
The Empty Shell Problem
Most AI website builders generate Client-Side Rendered (CSR) React SPAs. Their raw HTML contains only <div id="root"></div>, leaving non-JS crawlers with zero text.
AI Crawlers Don't Run JS
OpenAI's OAI-SearchBot, Anthropic's ClaudeBot, and PerplexityBot do not execute JavaScript bundles. If your text isn't in the initial HTML response, you do not exist to them.
Pre-Render + Entity Graph
You don't need to abandon React. Adding build-time static pre-rendering (or SSR), route-level metadata, /llms.txt, and linked JSON-LD makes your site 100% readable everywhere.
The "It Looks Great in My Browser" Illusion
Every week in Melbourne, we speak with business owners and founders who have used an AI builder—such as Lovable, Bolt.new, v0, or Replit—to launch their company website. On their MacBook or iPhone, the site looks terrific. The animations are smooth, the Tailwind typography is sharp, and clicking between the Home, Services, and Contact pages feels instantaneous.
Then three months pass. Despite having a well-written site, the business doesn't rank in Google for its own suburb and service keywords. When someone pastes a link to their /services page into LinkedIn, WhatsApp, or Slack, the preview card either fails to load or shows a generic template title. Worse still, when a prospective customer asks ChatGPT, Claude, or Perplexity for recommendations in their industry, the AI assistant has no record of what the company actually does.
The culprit is rarely the visual design or the written copy. It is a fundamental architectural gap between how a human web browser renders a page and how search engines, AI answer engines, and social crawlers fetch a URL.
When you prompt an AI coding tool to "build a modern website for my business," it almost always scaffolds a Client-Side Rendered (CSR) Single-Page Application (SPA) using Vite and React. In a pure CSR architecture, the web server does not send your headings, paragraphs, pricing tables, or FAQs over the network as HTML. Instead, the server sends a tiny, virtually empty HTML shell—often fewer than 20 lines—containing a single empty <div id="root"></div> tag and a reference to a 500 KB JavaScript bundle.
Your Chrome browser downloads that JavaScript file, executes it on your device's CPU, and dynamically paints the text onto your screen a fraction of a second later. Because human eyes never see the empty shell, founders assume the text is "on the page." To an automated crawler that does not execute JavaScript, however, that page is completely blank.
Live Crawler Vision Simulator
Switch between visitor types and toggle the underlying architecture to see the exact HTTP payload and extracted data each bot receives.
<!doctype html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<!-- Hardcoded default title on every URL! -->
<title>Vite + React + TS</title>
</head>
<body>
<!-- ZERO text or headings in HTML -->
<div id="root"></div>
<script type="module" src="/assets/index-8f92a.js"></script>
</body>
</html>Extracted Page Content: 0 words
OAI-SearchBot and ClaudeBot do not execute /assets/index-8f92a.js. They parse the raw HTML, find an empty <div id="root">, and conclude the page has no content.
- Detected H1
- null (None)
- Body Word Count
- 0 words
- Schema Entities
- 0 found
- AI Citation Status
- Skipped
Tip: Press Cmd+Option+U (Mac) or Ctrl+U (Windows) on any website to view its raw pre-rendered HTML source.
How Search Engines vs. AI Answer Engines Actually Crawl Your Site
A common misconception among developers is: "Doesn't Google execute JavaScript now?" The honest answer is: only partially, only on a delayed second queue, and AI search engines do not execute it at all.
When Googlebot discovers a URL on your site, it performs a first-wave crawl of the raw HTML response. If the content is missing because your site relies on client-side JavaScript, the URL is placed into Google's Web Rendering Service (WRS) queue to be rendered by a headless Chromium instance later. For a brand-new domain or a site with dozens of location and service pages, that rendering queue can delay indexing by days or weeks—and if your JavaScript bundle throws an unhandled runtime error or times out fetching an API, the page is dropped silently.
More importantly, the way Australians find businesses has expanded beyond traditional blue links. Buyers now ask ChatGPT, Claude, Perplexity, and Microsoft Copilot questions like "Which Melbourne software companies build custom NDIS platforms?" or "Compare custom ERP developers in Victoria." None of those AI retrieval crawlers run a headless browser when indexing the web.
| Crawler / Bot | Platform | Executes JS? | Raw AI SPA Result | Pre-Rendered / SSR Result |
|---|---|---|---|---|
| Googlebot (Wave 1) | Google Search | Deferred (Wave 2 queue) | Sees empty <div id='root'>; indexing delayed or metadata missed | Instant full-page indexing on first crawl pass |
| OAI-SearchBot / GPTBot | OpenAI (ChatGPT Search) | No | Reads 0 words of body content; cannot cite your business | Extracts full headings, pricing, FAQs & entity facts |
| ClaudeBot | Anthropic (Claude) | No | Blank shell; skips domain when answering user prompts | Full semantic HTML + /llms.txt parsed cleanly |
| PerplexityBot | Perplexity AI | No | Fails to extract article text or service specifications | High citation accuracy with direct source links |
| LinkedInBot / WhatsApp / Slack | Social Link Previews | No | Every shared link shows generic homepage title or broken preview | Rich, page-specific OpenGraph card unfurls immediately |
| Bingbot / Applebot | Microsoft Copilot / Apple Siri | Limited / Partial | Frequently misses deep routes and dynamic meta tags | 100% route coverage and structured schema ingestion |
The 5-Stage Engineering Pipeline: From AI Prototype to Production
Using AI tools to accelerate web development is smart engineering—provided you wrap that code in a proper production build pipeline. Below is the exact five-stage architecture we use at Alltechzone to ensure every React page ships as fast, pre-rendered, machine-verifiable HTML. Click any stage to inspect its deliverables and implementation code.
Build-Time Pre-Rendering or SSR
This is the step 95% of AI-built websites skip. Because GPTBot, ClaudeBot, PerplexityBot, and social link previewers do not execute client-side JavaScript bundles, your build pipeline must output pre-rendered HTML for every route. You can achieve this either by migrating to Next.js/Astro SSR, or by running a headless Chromium pre-render script immediately after vite build.
Engineering Deliverables
// scripts/prerender.mjs — Post-build static HTML snapshot
await page.goto(`http://localhost:4173${route}`, { waitUntil: "networkidle0" });
const html = await page.content();
fs.mkdirSync(path.join("dist", route), { recursive: true });
fs.writeFileSync(path.join("dist", route, "index.html"), html);Choosing the Right Fix: Build-Time Pre-Rendering vs. Next.js SSR vs. Astro
When a client brings us a website generated in Lovable, Bolt, or Vite + React, they often worry they will need to throw away their entire codebase to fix their SEO. In practice, you have three clean architectural options depending on how often your content changes and how many routes you manage:
Build-Time Static Pre-Rendering
Keep your existing Vite + React Router codebase intact. Add a post-build script using Puppeteer or Playwright that visits every route in your sitemap during CI/CD and saves the fully rendered DOM to dist/<route>/index.html.
Ideal for:
Company websites, service & location landing pages, portfolios, and blogs up to ~300 routes.
Next.js App Router (SSR / ISR)
Migrate your React components into Next.js Server Components. Pages are rendered on the server on request (or cached via Incremental Static Regeneration), with native generateMetadata() support for dynamic database rows.
Ideal for:
E-commerce stores, directories, job boards, and platforms with 1,000+ database-driven URLs.
Astro Islands Architecture
Compile all marketing and content pages to pure, zero-JavaScript HTML at build time, while mounting interactive React components ("islands") only where user state is needed (such as calculators or forms).
Ideal for:
Content-heavy publications and marketing sites chasing 100/100 Core Web Vitals scores.
Fixing the "Soft-404" SPA Routing Trap
When deploying a Single-Page Application on Netlify, Vercel, or Cloudflare Pages, developers almost always add a catch-all rewrite rule (/* /index.html 200) so refreshing a sub-page doesn't trigger a server 404.
Without pre-rendering, that rule has a nasty side effect: every mistyped URL on your domain returns an HTTP 200 OK status code with the exact same homepage shell. Google Search Console flags these as Soft 404s and begins distrusting your site's status codes. When you pre-render every valid route into its own static index.html file, real pages serve their exact static HTML immediately before the fallback rule is ever reached.
Engineering for AI Answer Engines: Linked JSON-LD & /llms.txt
Getting your HTML rendered is half the battle. The second half is structuring your facts so Large Language Models and Google's Knowledge Graph can verify who you are without guessing.
AI website generators rarely include structured data, or if prompted, they paste a generic, disconnected snippet. In production, your site should output a unified @graph JSON-LD block where every entity cross-references a stable @id:
- Organization & LocalBusiness: Declares your legal name, Australian Business Number (ABN) or Corporate Identification Number (CIN), physical street address, geo-coordinates, telephone, and verified social profiles (
sameAs). - Service & SoftwareApplication: Links each service page or software platform back to
{ "@id": "https://yourdomain.com/#organization" }as theprovider, defining the exactareaServed. - FAQPage Synced with Visible
<details>Elements: Google's structured data guidelines require every question and answer in yourFAQPageschema to be visibly rendered in the DOM. Using native HTML<details>and<summary>tags ensures collapsed answers are present in the initial HTML for crawlers without requiring JavaScript click events.
Publishing a Production /llms.txt File
Just as /robots.txt tells crawlers what they may access and /sitemap.xml lists every URL, /llms.txt gives AI systems a clean, token-efficient briefing on your entire business in one request. Place it in your public/llms.txt directory so it resolves at the root of your domain:
# Company Name — Brief Positioning Line > One-paragraph factual summary of what your business builds, who it serves, and where it is headquartered. ## Verified Business Facts - Legal Entity: Your Company Name (ABN XX XXX XXX XXX) - Headquarters: Street Address, Suburb, State, Postcode, Country - Phone: +61 X XXXX XXXX | Email: hello@yourdomain.com.au ## Core Services & Canonical URLs - [Custom Software Development](https://yourdomain.com.au/services/custom-software): Plain-English scope summary. - [Mobile App Development](https://yourdomain.com.au/services/mobile-apps): iOS & Android engineering.
The Silent Security Trap in AI-Generated Apps
Beyond search visibility, our engineering audits of AI-built websites regularly uncover three critical security vulnerabilities that put business data at risk:
1. Exposed Secret Keys in VITE_ Env Vars
Any environment variable prefixed with VITE_ or NEXT_PUBLIC_ is compiled directly into the public JavaScript bundle sent to every visitor's browser. Never place OpenAI keys, Stripe secret keys, or Supabase service_role keys in client variables.
2. Missing Supabase Row Level Security (RLS)
Because the Supabase anon key is public by design, your database security depends entirely on Postgres Row Level Security (RLS) policies. AI builders frequently create tables with RLS disabled—or with permissive USING (true) policies that let anyone dump your customer leads table.
3. Unprotected Contact & Booking Endpoints
Forms wired directly from the browser without server-side validation, rate limiting, or honeypot checks quickly become targets for automated spam bots and API credit exhaustion.
The 30-Second Terminal Diagnostic: Test Your Own Website Right Now
You don't need paid SEO software to verify whether your website is readable by AI and search crawlers. Open Terminal on macOS/Linux (or PowerShell/WSL on Windows) and run these three commands, replacing yourdomain.com.au with your live URL:
curl -s -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot" https://yourdomain.com.au | grep -E "<title>|<h1"
curl -s https://yourdomain.com.au/about | grep -i "<title>"
curl -s -o /dev/null -w "%{http_code}\n" https://yourdomain.com.au/llms.txt10-Point AI Website Production Readiness Audit
Tick each requirement your website currently passes to calculate your live Crawler Readiness Score.
Initial HTTP response contains full body copy and headings (not an empty #root div)
Engineering fix: Add build-time pre-rendering (Puppeteer/Playwright static snapshot) or migrate to Next.js SSR/SSG so crawlers receive rendered HTML immediately.
Every sub-page (/about, /services, /contact) returns a unique <title> and <meta name='description'> without JS
Engineering fix: Combine react-helmet-async with build-time pre-rendering per route so every URL has its own baked-in head tags in static HTML.
Self-referencing <link rel='canonical'> is present in raw HTML for every route
Engineering fix: Ensure index.html does not hardcode the homepage canonical across all routes; inject route-specific canonicals during pre-render.
robots.txt explicitly allows OAI-SearchBot, GPTBot, ClaudeBot, PerplexityBot and Google-Extended
Engineering fix: Audit your robots.txt and Cloudflare WAF settings—many managed hosts block AI search bots by default, cutting off ChatGPT and Perplexity citations.
A structured /llms.txt file is published at the domain root with factual entity, service and pricing summaries
Engineering fix: Create public/llms.txt in clean Markdown listing your legal entity, location, core offerings, and canonical URLs for LLM ingestion.
Linked @graph JSON-LD schema (Organization, Service, FAQPage, BreadcrumbList) is embedded in HTML
Engineering fix: Use stable @id URIs (e.g. #organization, #website) to connect your company, founders, locations, and FAQs in a machine-readable graph.
OpenGraph (og:title, og:description, og:image) unfurls accurately on LinkedIn, WhatsApp and Slack
Engineering fix: Social preview bots never run JavaScript. Pre-render every route so sharing a deep link shows that page's actual title and preview image.
Semantic HTML5 landmarks (<header>, <main>, <article>, <nav>, <footer>) and strict H1→H2→H3 hierarchy
Engineering fix: Replace nested <div> soup from AI prompt generators with semantic elements so reader modes and AI chunkers parse content boundaries cleanly.
Zero private API keys exposed in VITE_ / NEXT_PUBLIC_ client bundles and Supabase RLS enabled on all tables
Engineering fix: Move secret API calls to serverless functions or Supabase Edge Functions, and enforce Row Level Security (RLS) policies on every database table.
Automated XML sitemap generated from actual router definitions on every build
Engineering fix: Write a pre-build Node script that parses src/App.tsx routes and outputs public/sitemap.xml so new pages are never orphaned.
Frequently Asked Questions
Why don't AI crawlers like ChatGPT and Claude execute JavaScript like a normal browser?
Running a headless browser (like Chromium) to download, parse, compile, and execute 500 KB to 2 MB of JavaScript takes 10 to 20 times more CPU and memory than fetching raw HTML—and introduces security and timeout risks at scale. While Google invests in a two-wave Web Rendering Service (WRS), AI crawlers such as OAI-SearchBot, GPTBot, ClaudeBot, and PerplexityBot fetch raw HTTP responses only. If your server returns an empty <div id='root'></div> waiting for client-side React to mount, those bots see a blank page.
If I built my site in Lovable, Bolt, v0, or Vite + React, do I have to throw it away and rebuild from scratch?
No. Your React components, Tailwind styling, and UI design are completely reusable. You have two clean engineering paths: 1) Keep your Vite + React architecture and add a build-time pre-rendering pipeline (using Puppeteer or Playwright after vite build) alongside react-helmet-async and an automated sitemap generator, or 2) Port your components into Next.js (App Router) or Astro for native Server-Side Rendering (SSR) and Static Site Generation (SSG).
How can I check right now if my website is invisible to AI crawlers?
Open your terminal (or Command Prompt) and run: curl -s https://yourdomain.com | grep -i '<h1'. Alternatively, open your website in Chrome, right-click anywhere, and select 'View Page Source' (not 'Inspect Element'). If you only see <div id='root'></div> and a <script> tag instead of your actual paragraphs and headings, your site is client-side rendered only and invisible to non-JS crawlers.
What is llms.txt and does my business website really need one?
llms.txt is a plain-text Markdown file placed at the root of your domain (https://yourdomain.com/llms.txt) designed specifically for Large Language Models and AI agents. While traditional sitemaps list URLs, llms.txt provides a concise, authoritative summary of who your company is, your legal registration (ABN/CIN), physical locations, core services, and canonical links—making it effortless for AI assistants to cite your business accurately without hallucinating.
Alltechzone Engineering · Melbourne
Want your AI-built site or React app engineered to rank and convert?
Whether you have an existing Lovable, Bolt, or React project that needs pre-rendering, SSR, schema graphs, and security hardening—or you want a custom platform engineered properly from day one—our Melbourne team can help.