Answer Engine Optimisation: Getting Cited by AI Search
Answer engine optimization in practice: unblock AI crawlers, ship server-rendered HTML, write extractable chunks and measure AI search visibility properly.
- Check your access logs for
GPTBot,OAI-SearchBot,ClaudeBotandPerplexityBotbefore touching content. Cloudflare began blocking AI crawlers by default for new domains in July 2025, and plenty of sites are blocked without knowing it. - The major AI crawlers do not execute JavaScript. Vercel’s crawler analysis found GPTBot and ClaudeBot fetch HTML and leave. Client-rendered content is invisible to them, full stop.
- Retrieval happens at chunk level, not page level. A heading plus a self-contained 40 to 70 word answer underneath it is the unit that gets cited, so stop writing paragraphs that depend on the previous three to make sense.
llms.txtcosts ten minutes and no major provider has confirmed reading it. Ship it if you want, but do not count it as work.- Measure with referrer data (
chatgpt.com,perplexity.ai,copilot.microsoft.com) plus bot hits in server logs. Treat anything else as a vendor’s guess.
What answer engine optimisation actually is
Classic SEO optimises for a ranked list of ten links. Answer engines optimise for a synthesised paragraph with two to six citations attached. The mechanics differ in one way that changes everything: there’s a retrieval step and a generation step, and you need to win both.
Retrieval pulls candidate passages, often from a mix of a search index (Bing, Google, or a provider’s own crawl) and live fetches triggered by the user’s prompt. Generation decides which of those passages actually get used in the answer and which get a citation marker. You can be retrieved and still not cited. That’s the most common failure mode, and it usually happens because the retrieved passage was vague, hedged, or needed context from elsewhere on the page to make sense.
AEO work splits into three buckets: can they fetch you, can they parse you, and is your text extractable as a standalone claim. The third is a writing problem. The first two are engineering.

Check the logs before you change a single word
Run this against your access logs first. Not after the content refresh. First.
# 90 days of AI crawler activity, grouped by user agent
zcat access.log*.gz | grep -Eo 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|PerplexityBot|Perplexity-User|Google-Extended|Applebot-Extended|CCBot|meta-third-partyagent' \
| sort | uniq -c | sort -rn
status codes GPTBot is actually getting back
zcat access.log*.gz | grep 'GPTBot' | awk '{print $9}' | sort | uniq -c | sort -rn
If that second command shows a wall of 403s or 429s, nothing else in this article matters yet. Cloudflare’s July 2025 change made AI crawler blocking the default for newly onboarded domains, and most bot-management products bundle AI crawlers into their “scrapers” category. We have found this on sites where the marketing team was simultaneously paying an agency for “AI visibility.”
Then look at your robots.txt. Separate the training crawlers from the search crawlers, because they are not the same thing and blocking them has different consequences.
# Training / dataset crawlers: block these if you want, costs you nothing in citations
User-agent: CCBot
User-agent: Bytespider
Disallow: /
Search and live-fetch agents: these are the ones that produce citations
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
GPTBot feeds both training and some retrieval. Judgement call.
User-agent: GPTBot
Allow: /
Sitemap: https://example.com/sitemap.xml
Two things worth getting right. Google-Extended is not a crawler. It’s a token that controls whether your content can be used for Gemini grounding and training, and disallowing it does not remove you from AI Overviews. AI Overviews run off the normal Googlebot index. If you genuinely want out of those, your levers are nosnippet, max-snippet:N, and data-nosnippet on specific elements, which also cut your regular search snippets. That trade is almost never worth it for a commercial site.
AI crawlers do not run your JavaScript
This is the single biggest structural advantage a server-rendered site has right now. Vercel’s crawler study, which instrumented real requests across their network, found that GPTBot and ClaudeBot fetch HTML and never execute the JS bundle. Gemini’s crawling rides on Google infrastructure, so it gets the closest thing to real rendering, and Google’s own rendering queue still runs on a delay.
If your product descriptions, FAQ accordions, pricing tables, or review text arrive via fetch() after hydration, they do not exist for most answer engines. We have audited headless builds where the entire body copy was client-rendered and the crawler received a <div id="root"></div> plus 400KB of JavaScript.
Three practical consequences:
- Tab and accordion content must be in the DOM on first paint. Hidden with CSS is fine. Absent until a click handler fires is not. Same logic applies to carousels, where the content in slides 2 through 6 frequently never ships in the initial HTML.
- SSR or SSG beats CSR for AEO by a wide margin. If you’re weighing architecture, this belongs in the decision alongside the usual concerns we covered in our piece on headless WordPress.
- Static HTML wins by default. This is partly why we still build a lot of marketing sites as flat HTML. The Canvas HTML template ships everything server-rendered because there’s no build-time hydration step to get wrong, and a crawler that doesn’t run JS sees exactly what a human sees.
Speed matters here too, though less than people assume. Live-fetch agents like ChatGPT-User operate under a request timeout. A page that takes 6 seconds to send its first byte sometimes just doesn’t make it into the answer. The LCP work you’ve already done covers most of this.
Write for extraction: the chunk is the unit
Retrieval systems split your page into passages, embed them, and match them against the query. A passage that reads as a complete answer gets cited. A passage that starts with “This approach also means…” gets discarded, because on its own it says nothing.
The pattern that works, tested across a few dozen client pages:
- Question-shaped or claim-shaped H2/H3. “How long does webhook retry backoff take?” not “Retry Considerations.”
- A direct answer in the first 40 to 70 words below it. Lead with the answer. Then qualify.
- Named entities repeated, pronouns avoided. Write “Stripe’s signing secret” in the chunk rather than “its secret,” because the chunk is read without its neighbours.
- Numbers, units and versions inline. “Default retry window is 72 hours across 16 attempts” is quotable. “Retries happen over several days” is not.
Tables are disproportionately effective. A comparison table with clear headers gets lifted into answers far more often than the same information in prose, because the rows survive chunking intact. Make sure they’re real <table> markup with <th> and scope, not flexbox divs, and make sure they survive narrow viewports (the patterns here hold up).
One thing nobody mentions: hedging destroys citability. “It depends on your use case” is honest and useless. Give the number, then give the exception. A model summarising three sources will reach for the one that committed to something.
Structured data and entity consistency
No AI provider has published a ranking factor list that includes schema.org markup. Anyone telling you FAQPage markup gets you into ChatGPT is guessing. What structured data reliably does is disambiguate entities, and that matters because answer engines resolve “who is this company” before they decide whether to trust the page.
Worth shipping, in priority order: Organization with sameAs pointing at your real LinkedIn, GitHub and Crunchbase profiles; Article with author as a Person that has its own URL and credentials; Product with real offers and aggregateRating if the reviews exist; BreadcrumbList for hierarchy.
{
"@context": "https://schema.org",
"@type": "Article",
"headline": "Answer Engine Optimisation: Getting Cited by AI Search",
"datePublished": "2026-01-14T09:00:00+05:30",
"dateModified": "2026-01-14T09:00:00+05:30",
"author": {
"@type": "Person",
"name": "Name Surname",
"url": "https://semicolonweb.com/about",
// jobTitle and knowsAbout help resolve the author as a real entity
"jobTitle": "Lead Developer",
"knowsAbout": ["Web Performance", "Structured Data"]
},
"publisher": { "@type": "Organization", "name": "SemiColonWeb", "url": "https://semicolonweb.com" }
}
Keep dateModified truthful. Bumping it weekly on unchanged content is a tell, and it burns trust in the one signal that genuinely helps with freshness-sensitive queries. If you’re on WordPress and your SEO plugin’s output is thin, we went through the gaps in detail.
llms.txt and other possible theatre
llms.txt was proposed by Jeremy Howard in September 2024: a markdown index at your root that points language models at clean versions of your key pages. It’s a sensible idea. As of now, no major provider has publicly confirmed consuming it.
Ship it if generating it is automated. Skip it if it means manual maintenance. Ten minutes of CI work, possible upside, zero downside. What’s not acceptable is billing it as AEO strategy.
The same scepticism applies to “AI visibility” platforms that score you out of 100. Most are prompting a model a few hundred times a week and counting brand mentions. That’s a real signal, sampled badly, with no stability between runs. Useful as a rough trend. Not a KPI you should be reporting to a board.
Measuring AI search visibility honestly
Three sources, in descending order of reliability.
Referrer data. ChatGPT, Perplexity and Copilot all pass a referrer when a user clicks a citation. In GA4, build an exploration filtered on session source matching a regex:
chatgpt\.com|perplexity\.ai|copilot\.microsoft\.com|gemini\.google\.com|claude\.ai|you\.com
Expect low volume and high quality. On content-heavy client sites we typically see AI referrals at a small single-digit percentage of organic sessions, but with session durations well above the organic average and noticeably better form conversion. Fewer people, further along.
Bot hits in logs. Crawl frequency per URL tells you what the retrieval layer considers worth revisiting. A page OAI-SearchBot fetches weekly is in play. One it hasn’t touched in four months isn’t.
Prompt testing. Pick 20 queries that matter commercially, run them manually once a month, log which domains get cited. Tedious, but it’s the only method that shows you who you’re losing to.
Where to actually start
Four things, if you do nothing else: unblock the search-class crawlers in your WAF and robots.txt, make sure your primary content is in the server-rendered HTML, rewrite your top 20 pages so every H2 is followed by a self-contained answer, and get Organization and Article schema correct with honest dates.
That’s a week of work for most sites, maybe two. Everything beyond it has sharply diminishing returns right now, and the tooling is too young to justify a bigger bet.
Frequently Asked Questions
Is answer engine optimisation different from SEO, or just rebranded?
It overlaps heavily but it is not identical. Crawlability, server-rendered HTML and clean structure serve both, so most good SEO work helps. The genuine differences are chunk-level writing (self-contained passages rather than page-level narrative flow) and the fact that AI crawlers mostly do not execute JavaScript, which makes rendering architecture more consequential than it is for Googlebot.
Should I block GPTBot to stop my content being used for training?
If your worry is training data, block CCBot, Bytespider and Google-Extended, which cost you nothing in citations. Blocking GPTBot is murkier because it contributes to retrieval as well as training. Blocking OAI-SearchBot, ChatGPT-User or PerplexityBot is a direct trade of visibility for control, and for commercial sites that trade rarely pays.
Does FAQPage schema get me into AI answers?
No provider has confirmed that, and we have seen no evidence it moves citation rates on its own. What helps is the content pattern that FAQ markup usually wraps: a question heading with a short, complete answer directly beneath it. Ship the markup because it is cheap and valid, but the writing structure is doing the work.
How much traffic should I expect from AI search in 2026?
On most client sites AI referrals sit in the low single digits as a share of organic sessions, though it is climbing and varies enormously by vertical. Technical, B2B and comparison-heavy content sees more. The sessions convert better than average because the user arrived with their question already answered and clicked through to verify, so judge it on conversions rather than session count.
Will AI search kill organic traffic to informational pages?
It is already reducing clicks on queries with a single factual answer, and that will continue. Pages that mainly restate a definition are the most exposed. Content that survives tends to involve original measurements, named trade-offs, specific configuration, or anything where the reader needs the full context rather than a one-sentence summary.
Pull your logs this afternoon. If the search-class crawlers are getting 403s, you have a one-line fix that’s worth more than any content rewrite, and you’ll know within a fortnight whether the citations start appearing.


