Engineering & SearchProduction Architecture Guide

    Why AI-built websites are invisible to Google and AI crawlers: and how to engineer them for production.

    Tools like Lovable, Bolt, v0, Replit, and Cursor have made it possible to assemble a good-looking React website in a single weekend. Yet when founders hit publish, they discover a painful truth: to ChatGPT, Claude, Perplexity, social preview bots, and Google's first crawl wave, their website is a blank white page.

    By Rickson P. AntonyNovember 2026 · Melbourne Engineering Team

    Executive Summary · Key Takeaways

    01 / THE ROOT CAUSE

    The Empty Shell Problem

    Most AI website builders generate Client-Side Rendered (CSR) React SPAs. Their raw HTML contains only <div id="root"></div>, leaving non-JS crawlers with zero text.

    02 / AI SEARCH BLINDSPOT

    AI Crawlers Don't Run JS

    OpenAI's OAI-SearchBot, Anthropic's ClaudeBot, and PerplexityBot do not execute JavaScript bundles. If your text isn't in the initial HTML response, you do not exist to them.

    03 / THE FIX

    Pre-Render + Entity Graph

    You don't need to abandon React. Adding build-time static pre-rendering (or SSR), route-level metadata, /llms.txt, and linked JSON-LD makes your site 100% readable everywhere.

    Section 01

    The "It Looks Great in My Browser" Illusion

    Every week in Melbourne, we speak with business owners and founders who have used an AI builder—such as Lovable, Bolt.new, v0, or Replit—to launch their company website. On their MacBook or iPhone, the site looks terrific. The animations are smooth, the Tailwind typography is sharp, and clicking between the Home, Services, and Contact pages feels instantaneous.

    Then three months pass. Despite having a well-written site, the business doesn't rank in Google for its own suburb and service keywords. When someone pastes a link to their /services page into LinkedIn, WhatsApp, or Slack, the preview card either fails to load or shows a generic template title. Worse still, when a prospective customer asks ChatGPT, Claude, or Perplexity for recommendations in their industry, the AI assistant has no record of what the company actually does.

    The culprit is rarely the visual design or the written copy. It is a fundamental architectural gap between how a human web browser renders a page and how search engines, AI answer engines, and social crawlers fetch a URL.

    When you prompt an AI coding tool to "build a modern website for my business," it almost always scaffolds a Client-Side Rendered (CSR) Single-Page Application (SPA) using Vite and React. In a pure CSR architecture, the web server does not send your headings, paragraphs, pricing tables, or FAQs over the network as HTML. Instead, the server sends a tiny, virtually empty HTML shell—often fewer than 20 lines—containing a single empty <div id="root"></div> tag and a reference to a 500 KB JavaScript bundle.

    Your Chrome browser downloads that JavaScript file, executes it on your device's CPU, and dynamically paints the text onto your screen a fraction of a second later. Because human eyes never see the empty shell, founders assume the text is "on the page." To an automated crawler that does not execute JavaScript, however, that page is completely blank.

    Interactive Diagnostic LabURL: https://example-agency.com.au/services/custom-erp

    Live Crawler Vision Simulator

    Switch between visitor types and toggle the underlying architecture to see the exact HTTP payload and extracted data each bot receives.

    1. Select Architecture
    2. Inspect Visitor / Crawler
    Initial Server HTML Response412 bytes (Empty Shell)
    <!doctype html>
    <html lang="en">
      <head>
        <meta charset="UTF-8" />
        <!-- Hardcoded default title on every URL! -->
        <title>Vite + React + TS</title>
      </head>
      <body>
        <!-- ZERO text or headings in HTML -->
        <div id="root"></div>
        <script type="module" src="/assets/index-8f92a.js"></script>
      </body>
    </html>
    AI Crawler Extraction (GPTBot / ClaudeBot) Failed to Read

    Extracted Page Content: 0 words

    OAI-SearchBot and ClaudeBot do not execute /assets/index-8f92a.js. They parse the raw HTML, find an empty <div id="root">, and conclude the page has no content.

    Detected H1
    null (None)
    Body Word Count
    0 words
    Schema Entities
    0 found
    AI Citation Status
    Skipped

    Tip: Press Cmd+Option+U (Mac) or Ctrl+U (Windows) on any website to view its raw pre-rendered HTML source.

    Section 02

    How Search Engines vs. AI Answer Engines Actually Crawl Your Site

    A common misconception among developers is: "Doesn't Google execute JavaScript now?" The honest answer is: only partially, only on a delayed second queue, and AI search engines do not execute it at all.

    When Googlebot discovers a URL on your site, it performs a first-wave crawl of the raw HTML response. If the content is missing because your site relies on client-side JavaScript, the URL is placed into Google's Web Rendering Service (WRS) queue to be rendered by a headless Chromium instance later. For a brand-new domain or a site with dozens of location and service pages, that rendering queue can delay indexing by days or weeks—and if your JavaScript bundle throws an unhandled runtime error or times out fetching an API, the page is dropped silently.

    More importantly, the way Australians find businesses has expanded beyond traditional blue links. Buyers now ask ChatGPT, Claude, Perplexity, and Microsoft Copilot questions like "Which Melbourne software companies build custom NDIS platforms?" or "Compare custom ERP developers in Victoria." None of those AI retrieval crawlers run a headless browser when indexing the web.

    Crawler / BotPlatformExecutes JS?Raw AI SPA ResultPre-Rendered / SSR Result
    Googlebot (Wave 1)Google SearchDeferred (Wave 2 queue)Sees empty <div id='root'>; indexing delayed or metadata missedInstant full-page indexing on first crawl pass
    OAI-SearchBot / GPTBotOpenAI (ChatGPT Search)NoReads 0 words of body content; cannot cite your businessExtracts full headings, pricing, FAQs & entity facts
    ClaudeBotAnthropic (Claude)NoBlank shell; skips domain when answering user promptsFull semantic HTML + /llms.txt parsed cleanly
    PerplexityBotPerplexity AINoFails to extract article text or service specificationsHigh citation accuracy with direct source links
    LinkedInBot / WhatsApp / SlackSocial Link PreviewsNoEvery shared link shows generic homepage title or broken previewRich, page-specific OpenGraph card unfurls immediately
    Bingbot / ApplebotMicrosoft Copilot / Apple SiriLimited / PartialFrequently misses deep routes and dynamic meta tags100% route coverage and structured schema ingestion
    Section 03

    The 5-Stage Engineering Pipeline: From AI Prototype to Production

    Using AI tools to accelerate web development is smart engineering—provided you wrap that code in a proper production build pipeline. Below is the exact five-stage architecture we use at Alltechzone to ensure every React page ships as fast, pre-rendered, machine-verifiable HTML. Click any stage to inspect its deliverables and implementation code.

    Stage 03 · The bridge that makes React visible to AI & search bots

    Build-Time Pre-Rendering or SSR

    This is the step 95% of AI-built websites skip. Because GPTBot, ClaudeBot, PerplexityBot, and social link previewers do not execute client-side JavaScript bundles, your build pipeline must output pre-rendered HTML for every route. You can achieve this either by migrating to Next.js/Astro SSR, or by running a headless Chromium pre-render script immediately after vite build.

    Engineering Deliverables

    Post-build pre-renderer spins up a local static server and visits every route in sitemap
    Waits for React and Helmet to settle, then writes dist/<route>/index.html
    Every URL now serves complete, instant HTML on the very first byte—zero JS required for bots
    Implementation PatternStage 03
    // scripts/prerender.mjs — Post-build static HTML snapshot
    await page.goto(`http://localhost:4173${route}`, { waitUntil: "networkidle0" });
    const html = await page.content();
    fs.mkdirSync(path.join("dist", route), { recursive: true });
    fs.writeFileSync(path.join("dist", route, "index.html"), html);
    Section 04

    Choosing the Right Fix: Build-Time Pre-Rendering vs. Next.js SSR vs. Astro

    When a client brings us a website generated in Lovable, Bolt, or Vite + React, they often worry they will need to throw away their entire codebase to fix their SEO. In practice, you have three clean architectural options depending on how often your content changes and how many routes you manage:

    Option 01 · Best for Vite SPAs

    Build-Time Static Pre-Rendering

    Keep your existing Vite + React Router codebase intact. Add a post-build script using Puppeteer or Playwright that visits every route in your sitemap during CI/CD and saves the fully rendered DOM to dist/<route>/index.html.

    Ideal for:

    Company websites, service & location landing pages, portfolios, and blogs up to ~300 routes.

    Option 02 · Best for Dynamic Apps

    Next.js App Router (SSR / ISR)

    Migrate your React components into Next.js Server Components. Pages are rendered on the server on request (or cached via Incremental Static Regeneration), with native generateMetadata() support for dynamic database rows.

    Ideal for:

    E-commerce stores, directories, job boards, and platforms with 1,000+ database-driven URLs.

    Option 03 · Zero-JS Default

    Astro Islands Architecture

    Compile all marketing and content pages to pure, zero-JavaScript HTML at build time, while mounting interactive React components ("islands") only where user state is needed (such as calculators or forms).

    Ideal for:

    Content-heavy publications and marketing sites chasing 100/100 Core Web Vitals scores.

    Fixing the "Soft-404" SPA Routing Trap

    When deploying a Single-Page Application on Netlify, Vercel, or Cloudflare Pages, developers almost always add a catch-all rewrite rule (/* /index.html 200) so refreshing a sub-page doesn't trigger a server 404.

    Without pre-rendering, that rule has a nasty side effect: every mistyped URL on your domain returns an HTTP 200 OK status code with the exact same homepage shell. Google Search Console flags these as Soft 404s and begins distrusting your site's status codes. When you pre-render every valid route into its own static index.html file, real pages serve their exact static HTML immediately before the fallback rule is ever reached.

    Section 05

    Engineering for AI Answer Engines: Linked JSON-LD & /llms.txt

    Getting your HTML rendered is half the battle. The second half is structuring your facts so Large Language Models and Google's Knowledge Graph can verify who you are without guessing.

    AI website generators rarely include structured data, or if prompted, they paste a generic, disconnected snippet. In production, your site should output a unified @graph JSON-LD block where every entity cross-references a stable @id:

    • Organization & LocalBusiness: Declares your legal name, Australian Business Number (ABN) or Corporate Identification Number (CIN), physical street address, geo-coordinates, telephone, and verified social profiles (sameAs).
    • Service & SoftwareApplication: Links each service page or software platform back to { "@id": "https://yourdomain.com/#organization" } as the provider, defining the exact areaServed.
    • FAQPage Synced with Visible <details> Elements: Google's structured data guidelines require every question and answer in your FAQPage schema to be visibly rendered in the DOM. Using native HTML <details> and <summary> tags ensures collapsed answers are present in the initial HTML for crawlers without requiring JavaScript click events.

    Publishing a Production /llms.txt File

    Just as /robots.txt tells crawlers what they may access and /sitemap.xml lists every URL, /llms.txt gives AI systems a clean, token-efficient briefing on your entire business in one request. Place it in your public/llms.txt directory so it resolves at the root of your domain:

    public/llms.txt — Example StructureMarkdown (UTF-8)
    # Company Name — Brief Positioning Line
    
    > One-paragraph factual summary of what your business builds, who it serves, and where it is headquartered.
    
    ## Verified Business Facts
    - Legal Entity: Your Company Name (ABN XX XXX XXX XXX)
    - Headquarters: Street Address, Suburb, State, Postcode, Country
    - Phone: +61 X XXXX XXXX | Email: hello@yourdomain.com.au
    
    ## Core Services & Canonical URLs
    - [Custom Software Development](https://yourdomain.com.au/services/custom-software): Plain-English scope summary.
    - [Mobile App Development](https://yourdomain.com.au/services/mobile-apps): iOS & Android engineering.
    Section 06

    The Silent Security Trap in AI-Generated Apps

    Beyond search visibility, our engineering audits of AI-built websites regularly uncover three critical security vulnerabilities that put business data at risk:

    1. Exposed Secret Keys in VITE_ Env Vars

    Any environment variable prefixed with VITE_ or NEXT_PUBLIC_ is compiled directly into the public JavaScript bundle sent to every visitor's browser. Never place OpenAI keys, Stripe secret keys, or Supabase service_role keys in client variables.

    2. Missing Supabase Row Level Security (RLS)

    Because the Supabase anon key is public by design, your database security depends entirely on Postgres Row Level Security (RLS) policies. AI builders frequently create tables with RLS disabled—or with permissive USING (true) policies that let anyone dump your customer leads table.

    3. Unprotected Contact & Booking Endpoints

    Forms wired directly from the browser without server-side validation, rate limiting, or honeypot checks quickly become targets for automated spam bots and API credit exhaustion.

    Section 07

    The 30-Second Terminal Diagnostic: Test Your Own Website Right Now

    You don't need paid SEO software to verify whether your website is readable by AI and search crawlers. Open Terminal on macOS/Linux (or PowerShell/WSL on Windows) and run these three commands, replacing yourdomain.com.au with your live URL:

    Test 1: Simulate OpenAI's ChatGPT Search Crawler (Check if your H1 & text exist in raw HTML)
    curl -s -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot" https://yourdomain.com.au | grep -E "<title>|<h1"
    Test 2: Verify deep routes return unique titles (not a duplicate homepage shell)
    curl -s https://yourdomain.com.au/about | grep -i "<title>"
    Test 3: Check if JSON-LD structured data and /llms.txt are live
    curl -s -o /dev/null -w "%{http_code}\n" https://yourdomain.com.au/llms.txt
    Interactive Self-Assessment

    10-Point AI Website Production Readiness Audit

    Tick each requirement your website currently passes to calculate your live Crawler Readiness Score.

    0%0/10 Passed
    Rendering & HTMLCritical Impact

    Initial HTTP response contains full body copy and headings (not an empty #root div)

    Engineering fix: Add build-time pre-rendering (Puppeteer/Playwright static snapshot) or migrate to Next.js SSR/SSG so crawlers receive rendered HTML immediately.

    Rendering & HTMLCritical Impact

    Every sub-page (/about, /services, /contact) returns a unique <title> and <meta name='description'> without JS

    Engineering fix: Combine react-helmet-async with build-time pre-rendering per route so every URL has its own baked-in head tags in static HTML.

    Indexing & RoutingHigh Impact

    Self-referencing <link rel='canonical'> is present in raw HTML for every route

    Engineering fix: Ensure index.html does not hardcode the homepage canonical across all routes; inject route-specific canonicals during pre-render.

    AI DiscoverabilityCritical Impact

    robots.txt explicitly allows OAI-SearchBot, GPTBot, ClaudeBot, PerplexityBot and Google-Extended

    Engineering fix: Audit your robots.txt and Cloudflare WAF settings—many managed hosts block AI search bots by default, cutting off ChatGPT and Perplexity citations.

    AI DiscoverabilityHigh Impact

    A structured /llms.txt file is published at the domain root with factual entity, service and pricing summaries

    Engineering fix: Create public/llms.txt in clean Markdown listing your legal entity, location, core offerings, and canonical URLs for LLM ingestion.

    Structured DataHigh Impact

    Linked @graph JSON-LD schema (Organization, Service, FAQPage, BreadcrumbList) is embedded in HTML

    Engineering fix: Use stable @id URIs (e.g. #organization, #website) to connect your company, founders, locations, and FAQs in a machine-readable graph.

    Social & TrustMedium Impact

    OpenGraph (og:title, og:description, og:image) unfurls accurately on LinkedIn, WhatsApp and Slack

    Engineering fix: Social preview bots never run JavaScript. Pre-render every route so sharing a deep link shows that page's actual title and preview image.

    Accessibility & DOMMedium Impact

    Semantic HTML5 landmarks (<header>, <main>, <article>, <nav>, <footer>) and strict H1→H2→H3 hierarchy

    Engineering fix: Replace nested <div> soup from AI prompt generators with semantic elements so reader modes and AI chunkers parse content boundaries cleanly.

    Security & BackendCritical Impact

    Zero private API keys exposed in VITE_ / NEXT_PUBLIC_ client bundles and Supabase RLS enabled on all tables

    Engineering fix: Move secret API calls to serverless functions or Supabase Edge Functions, and enforce Row Level Security (RLS) policies on every database table.

    Indexing & RoutingHigh Impact

    Automated XML sitemap generated from actual router definitions on every build

    Engineering fix: Write a pre-build Node script that parses src/App.tsx routes and outputs public/sitemap.xml so new pages are never orphaned.

    Frequently Asked Questions

    Why don't AI crawlers like ChatGPT and Claude execute JavaScript like a normal browser?

    Running a headless browser (like Chromium) to download, parse, compile, and execute 500 KB to 2 MB of JavaScript takes 10 to 20 times more CPU and memory than fetching raw HTML—and introduces security and timeout risks at scale. While Google invests in a two-wave Web Rendering Service (WRS), AI crawlers such as OAI-SearchBot, GPTBot, ClaudeBot, and PerplexityBot fetch raw HTTP responses only. If your server returns an empty <div id='root'></div> waiting for client-side React to mount, those bots see a blank page.

    If I built my site in Lovable, Bolt, v0, or Vite + React, do I have to throw it away and rebuild from scratch?

    No. Your React components, Tailwind styling, and UI design are completely reusable. You have two clean engineering paths: 1) Keep your Vite + React architecture and add a build-time pre-rendering pipeline (using Puppeteer or Playwright after vite build) alongside react-helmet-async and an automated sitemap generator, or 2) Port your components into Next.js (App Router) or Astro for native Server-Side Rendering (SSR) and Static Site Generation (SSG).

    How can I check right now if my website is invisible to AI crawlers?

    Open your terminal (or Command Prompt) and run: curl -s https://yourdomain.com | grep -i '<h1'. Alternatively, open your website in Chrome, right-click anywhere, and select 'View Page Source' (not 'Inspect Element'). If you only see <div id='root'></div> and a <script> tag instead of your actual paragraphs and headings, your site is client-side rendered only and invisible to non-JS crawlers.

    What is llms.txt and does my business website really need one?

    llms.txt is a plain-text Markdown file placed at the root of your domain (https://yourdomain.com/llms.txt) designed specifically for Large Language Models and AI agents. While traditional sitemaps list URLs, llms.txt provides a concise, authoritative summary of who your company is, your legal registration (ABN/CIN), physical locations, core services, and canonical links—making it effortless for AI assistants to cite your business accurately without hallucinating.

    Alltechzone Engineering · Melbourne

    Want your AI-built site or React app engineered to rank and convert?

    Whether you have an existing Lovable, Bolt, or React project that needs pre-rendering, SSR, schema graphs, and security hardening—or you want a custom platform engineered properly from day one—our Melbourne team can help.

    Alltechzone Support

    Online

    Hi there! 👋
    How can we help you promote your business today?

    Powered by WhatsApp