AI search optimisation: what actually works
There is a large and growing market selling AEO, GEO and AI visibility retainers. Some of it is useful. A striking amount of it recommends things the AI providers have explicitly said in writing are not required. This guide sticks to what the providers themselves document, and is clear about where the honest answer is that nobody knows yet.
Quick answer: AI assistants cite you by searching, so the prerequisite is being indexed and genuinely useful. Google states that no special files, AI text files or schema are needed to appear in AI features. The one thing worth checking today is your robots.txt: the training crawlers and the search crawlers are different user agents, and blocking the search ones removes you from the results. Everything else is regular SEO with clearer writing.
The distinction that most robots.txt files get wrong. Left panel costs you nothing. Right panel costs you the citation.
Start with what Google puts in writing
Google's own documentation on AI features in Search is the most useful thing published on this subject, mainly because of what it rules out. Two statements matter, and both are direct quotes from Google.
- You do not need to create new machine readable files, AI text files, or markup to appear in these features.
- There is also no special schema.org structured data that you need to add.
AI Overviews and AI Mode are built on the ordinary Search index, fetched by ordinary Googlebot, and governed by ordinary Search controls. That means the work that earns you a place in AI Overviews is the work that earns you a place in Search: being crawlable, being indexed, and being the best answer available. If a proposal in front of you leads with a special file or a bespoke schema for AI, it is selling something Google has said is unnecessary.
Training crawlers and search crawlers are not the same thing
This is the one genuinely technical decision, and it is the one most sites get wrong, usually by pasting a block-everything robots.txt snippet from a post about protecting content from AI. Each provider runs separate user agents for separate jobs.
| Provider | Trains models | Finds and cites you | Fetches when a user asks |
|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
| Perplexity | None. Perplexity does not train foundation models | PerplexityBot | Perplexity-User |
| Google-Extended, for Gemini | Googlebot, the same one as Search | Googlebot |
Read the columns rather than the rows. You can block the entire first column and lose nothing in AI search. Block anything in the second or third and you have removed yourself from the product. OpenAI also runs OAI-AdsBot for advertising, which is a separate decision again.
Google is explicit that Google-Extended does not impact a site's inclusion in Google Search and is not used as a ranking signal. It is purely a Gemini training and grounding control. Blocking it does not remove you from AI Overviews, because AI Overviews are not fetched by it.
The llms.txt question
llms.txt is a proposed convention: a markdown file at your root listing your content in a form easy for a model to consume. It has become the centrepiece of a lot of AEO offers.
The current state of evidence is that no provider documents reading it, and Google's John Mueller has stated that no AI system currently uses it. Claims that specific assistants read it circulate widely and contradict each other, and none of them trace back to provider documentation.
Our position, and we publish one ourselves: it takes minutes, it does no harm, and if the convention is adopted you are already there. That is the entire case for it. It is a cheap option, not a strategy, and it should be priced accordingly.
What genuinely helps
Setting aside the ceremony, a short list survives contact with the documentation.
- Be indexed. Assistants that cite sources find them by searching. If Google has not indexed a page, it is not a candidate. This is why the honest first step of AI visibility work is the same as the first step of SEO work, and it is covered in why is my website not ranking.
- Answer the question in the opening lines. A model quoting you needs a passage that stands alone. Pages that open with three paragraphs of preamble before the answer are harder to quote than pages that lead with it, for humans as well.
- Be specific enough to be worth citing. Generic advice is already in the model. Numbers, named constraints, dated facts and clear caveats are what a model has to go and fetch.
- Keep the crawlers straight. Audit robots.txt against the table above. This is a fifteen-minute job that is often the only thing standing between a site and the results.
- Use structured data because it is useful, not because AI needs it. Schema still earns rich results in ordinary Search. Google has said it is not required for AI features. Both things are true.
If you want to be excluded rather than included
A legitimate position, particularly for publishers. The controls exist but they are blunter than people expect.
- nosnippet removes the page from AI Overviews and from ordinary Search snippets at the same time. There is no AI-only version.
- data-nosnippet excludes a specific section of a page while leaving the rest usable.
- max-snippet caps how much text can be shown.
- noindex removes the page from Search entirely, and therefore from anything built on Search.
The trade is real and worth stating plainly to anyone considering it: you cannot currently opt out of Google's AI features while keeping your ordinary snippets. Blocking the training crawlers, by contrast, has no Search cost at all.
The honest limits
Three things are worth saying that most articles on this topic will not.
The published statistics are unreliable. The market is full of studies claiming precise citation shares by platform, and they contradict each other badly, often by an order of magnitude. They come from vendors selling tracking tools. We have not used any of them in this guide for that reason.
A citation is not a visit. If an assistant answers the question completely, the reader may never click. For some businesses that is fine, because the brand mention at the point of decision is the value. For a business whose model depends on traffic volume, it is a genuine threat rather than an opportunity, and it deserves to be planned for rather than sold around.
It is early. The behaviour of these systems changes faster than best practice can be written, and anyone presenting a settled methodology is presenting confidence rather than evidence. The defensible strategy is the one that works regardless: be findable, be accurate, be specific, and do not block the crawlers that would cite you.
Where we sit
We do not sell AI search optimisation as a separate retainer, because on the current evidence it is not a separate discipline. It is a crawler audit, which is an hour, plus writing that answers questions directly, which is what good content was supposed to be doing anyway. Charging twice for that would be hard to defend.
What we do is fold it into SEO work: check the user agents, structure pages so the answer comes first, and keep the technical foundation clean enough that every system built on Google's index can see you. Detail on the SEO service page, and SEO costs in the UK covers what the market charges for the underlying work.
Frequently asked questions
Do I need to do anything special to appear in Google's AI Overviews?
Google says no, in writing. Its documentation states that you do not need to create new machine readable files, AI text files, or markup to appear in AI features, and that there is no special schema.org structured data you need to add. AI Overviews and AI Mode draw on the ordinary Search index, so the work that gets you into Search is the work that gets you into them.
What is llms.txt and should I have one?
It is a proposed file at the root of a site that lists your content in a form intended to be easy for language models to read. No AI provider currently documents using it, and Google's John Mueller has said plainly that no AI system currently uses it. It costs almost nothing to publish and it is not harmful, so treat it as a cheap bet on a standard that may or may not arrive. Anyone charging you meaningfully for one is charging for a file that nothing reads yet.
Should I block AI crawlers in robots.txt?
It depends which one, and the distinction is the whole point. Training crawlers such as GPTBot, ClaudeBot and Google-Extended take your content to train models, and blocking them does not affect whether you can be cited in AI search. Search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot are how those products find and link to you, so blocking them removes you from the results. Blocking everything is a common and expensive mistake.
Does blocking Google-Extended hurt my Google rankings?
No, and Google states this explicitly: Google-Extended does not impact a site's inclusion in Google Search, nor is it used as a ranking signal. It controls whether your content helps train and ground Gemini. AI Overviews are fetched by ordinary Googlebot and controlled by ordinary Search controls, which is why blocking Google-Extended does not remove you from them.
How do I keep my content out of AI Overviews but stay in Google Search?
Use the standard Search snippet controls, because AI features respect them. The nosnippet meta tag removes the page from AI Overviews and from ordinary snippets alike, data-nosnippet excludes specific sections of a page, and max-snippet limits how much can be shown. There is no separate AI-only toggle, which means the trade is real: less snippet exposure everywhere, not just in AI results.
Is AEO or GEO a different discipline to SEO?
Mostly not, on current evidence. The providers that cite sources do so by running a search, which means being findable, indexed and genuinely useful is the prerequisite, and that is SEO. The parts that are actually distinct are narrow: making sure the right crawlers are not blocked, writing pages that answer a question cleanly enough to be quoted, and accepting that a citation may not produce a click. Treat anyone selling AEO as a separate retainer with scepticism until they can show you what specifically differs.
Related guides: why is my website not ranking covers the indexing work every AI citation depends on, SEO costs in the UK covers pricing, and how to choose a web design agency covers reading a proposal critically.