LLM SEO is the technical and editorial work of making a website easy for large language model products to find, fetch, read and cite, including the crawlers behind ChatGPT, Claude, Perplexity and Google's AI features. The label is informal and overlaps with GEO and AEO, but it is most useful for the layer beneath them: crawler access, page formats and the way each assistant retrieves pages.
What LLM SEO is
AI products reach web pages by three different routes. Some crawlers collect content that may be used to train models. Some build the index that an assistant's search feature draws on. Others fetch a page on the spot because a user asked something that needs it. The operators describe these as separate jobs with separate controls, and a site can allow one while blocking another.[1][2]
LLM SEO means managing those routes on purpose and making the pages easy to read once they are fetched. It sits below strategy questions such as which outside sources should mention you, and above plain server administration. The evidence base is thinner than for classic SEO, so this page separates what the operators document from what vendors and studies claim.
Not ready to enquire? Get the free AI visibility audit worksheet, a spreadsheet template sent by email.
Ask for the free ranking snapshot
How LLM SEO differs from SEO, GEO and AEO
LLM SEO and SEO. Technical SEO mostly concerns the crawlers of search engines and the index they build. LLM SEO adds operator-specific bots, each with its own purpose and its own robots.txt token. For Google's AI features the position is simple: Google says nothing beyond standard Search eligibility is required, so most LLM SEO work concerns the other assistants.[3]
LLM SEO and GEO. The GEO page is about what you say and where you earn mentions. LLM SEO is about whether the systems can fetch what you say at all. A strong source study means little if the assistant's bot is blocked at your CDN.
LLM SEO and AEO. The AEO page covers writing direct answers for single questions. LLM SEO makes sure those answers are reachable and readable. The service pages, GEO agency, AEO agency and AI SEO agency, describe how the work is packaged, and the AI visibility audit records what assistants say today so that you can tell whether any access change mattered.
How AI products reach your content
The table sets out each major operator's bots, using the operators' own documentation. Read the last column carefully: it records what the operator says about blocking, not a promise about what an assistant will then do.
| Operator | Token | Documented purpose | robots.txt behaviour | Documented effect of blocking |
|---|---|---|---|---|
| OpenAI | OAI-SearchBot | Surfaces sites in ChatGPT search features | Controlled by robots.txt; about 24 hours to apply | Site will not appear in ChatGPT search answers, though plain navigational links may remain[1] |
| OpenAI | GPTBot | Collects content that may train OpenAI foundation models | Controlled by robots.txt | Signals the content should not be used for training[1] |
| OpenAI | ChatGPT-User | Visits pages when a user or a GPT Action asks | robots.txt rules may not apply | Does not decide whether content appears in search[1] |
| Anthropic | Claude-SearchBot | Improves the relevance and accuracy of search results | Controlled by robots.txt | May reduce visibility in search results[4] |
| Anthropic | ClaudeBot | Collects content that may train models | Controlled by robots.txt | Excludes future content from training sets[4] |
| Anthropic | Claude-User | Fetches pages when a Claude user asks | Controlled by robots.txt | May reduce visibility in user-directed searches[4] |
| Perplexity | PerplexityBot | Surfaces and links sites in Perplexity results; not used for foundation model training | Honours robots.txt; allowing it is recommended | Allowing it is how a site can appear in results[2] |
| Perplexity | Perplexity-User | Fetches pages when a user asks a question | Generally ignores robots.txt because a user triggered it | Owners can still control which sites these requests reach[2] |
| Google-Extended | Robots token for Gemini training and for grounding in Gemini Apps and Vertex AI | Token only, with no separate user agent string | Does not affect inclusion in Search or ranking[5] |
Cloudflare put numbers on how lopsided this traffic is. It reported crawl-to-refer ratios, meaning pages crawled for each visit referred back, from its Radar data for January to July 2025. Anthropic's ratio fell from about 286,900 to about 38,100 over those months, OpenAI's moved from about 1,217 to about 1,091, Perplexity's rose from about 55 to about 195, and Google's moved from about 3.8 to about 5.4. Over the trailing 12 months Cloudflare classed 80 percent of AI crawling as training, 18 percent as search and 2 percent as user actions.[6] The data comes from Cloudflare's own network, and it describes crawling against referrals, not whether your pages are cited.
What the documentation and studies support
Allow the search and retrieval bots you want to appear in
OpenAI recommends allowing OAI-SearchBot in robots.txt, and allowing requests from its published IP ranges, to help a site appear in ChatGPT search results. It says a robots.txt update can take about 24 hours to take effect.[1] Perplexity gives the same advice for PerplexityBot and quotes a delay of up to 24 hours.[2] Anthropic says that blocking Claude-SearchBot may reduce visibility in search results and that blocking Claude-User may reduce visibility in user-directed searches.[4] None of these statements promises that allowing a bot leads to citations. They say only that blocking removes the chance.
Keep the important facts in fetchable text
Microsoft advises against hiding key answers in tabs or expandable menus, against relying on PDFs for core information, and against putting critical details only in images without alt text.[7] Google likewise asks that important content be available in text form.[3] A simple test follows from this: request the page without running any scripts and check that your product name, category, main claims and dates are present in the returned text.
Treat training and retrieval as separate choices
OpenAI states that its settings are independent, so a site can allow OAI-SearchBot to appear in search while blocking GPTBot from training.[1] Google-Extended works as a robots.txt token only, and Google says it does not affect inclusion in Search or serve as a ranking signal.[5] Whether to permit training is a business and legal decision about your content. The documentation does not link it to citation.
Use the reports that now exist
Bing Webmaster Tools has offered an AI Performance report in public preview since 10 February 2026, showing citation counts and cited pages for Copilot and Bing's AI summaries, with the caveat that counts do not show placement.[8] For Google, AI feature traffic is counted in the Web search type of the Search Console performance report,[3] and Google also points to a generative AI performance report there.[9]
llms.txt: what is known
The llms.txt idea was proposed by Jeremy Howard on 3 September 2024. It is a Markdown file at a site's root that gives AI agents a short, curated map of the site, with an H1 as its only required element. The proposal calls itself open for community input and does not claim that AI companies have adopted it as a standard. It does note that thousands of sites publish the file and that OpenAI, Anthropic and Google publish one for their developer documentation.[10]
Two datasets test whether it matters for citations. SE Ranking analysed nearly 300,000 domains and found that 10.13 percent had an llms.txt file. It reports no correlation between having the file and how often a domain was cited, and removing the variable from its model made the model more accurate. The article does not say how many AI responses it sampled and describes its results as dependent on the dataset.[11]
Ahrefs examined logs from about 137,000 domains for May 2026, as reported by Search Engine Journal. About 28 percent published an llms.txt file, which is roughly 38,000 domains with valid files. Of those files, 97 percent received no requests at all, from bots or people, and only about 1,100 of the domains saw any traffic to the file. Retrieval bots linked to ChatGPT and Perplexity made about 1 percent of the requests, audit tools about 12 percent, coding agents about 10 percent, training crawlers about 5 percent and AI assistants about 2 percent. Ahrefs calls every figure a ceiling, its customers are more technical than the wider web, and the window was one month.[12]
Google says directly that its Search systems ignore files like llms.txt, so creating one neither helps nor harms visibility in Google Search, though other services may find it useful.[9] My reading is that the file is unproven as a citation lever and cheap to publish. Wise Mango publishes one as a plain index of its pages, as housekeeping and not as a ranking tactic. If you add one, list only canonical pages, keep it current, and judge it by whether your logs show anything fetching it.
robots.txt is a request, not a lock
Treat robots.txt as a statement of preference. Two of the user-triggered fetchers in the table, ChatGPT-User and Perplexity-User, are described by their operators as not bound by it in the usual way.[1][2] Anthropic adds that blocking its IP addresses is not a reliable opt-out because it stops the bots from reading your robots.txt file.[4]
Independent testing found compliance to be uneven. In the Tow Center study, which ran 1,600 queries across eight AI search tools using excerpts from articles by 20 publishers in February 2025, the chatbots did not consistently follow publishers' robots.txt preferences. Perplexity's free version correctly identified all ten excerpts from National Geographic, which blocks its crawlers, although the authors note that other sources of information could explain that result.[13] The sensible conclusion is that anything confidential belongs behind authentication, not behind a robots.txt line.
What does not work, and what is unproven
- llms.txt as a citation lever. Two studies found no sign of it,[11][12] and Google says it ignores the file.[9]
- Blocking by IP address alone. Anthropic says it defeats the purpose because the bots cannot read your instructions.[4]
- Pages written only for bots. Google's guide lists writing specifically for AI systems as unnecessary.[9]
- Counting crawler hits as visibility. Cloudflare's ratios show that crawling and referral are far apart, so a busy crawler in your logs does not mean your pages are cited.[6]
- Relying on robots.txt for secrets. User-triggered fetchers may not follow it, and testing found uneven compliance.[13]
Several questions are open. I found no controlled study showing that allowing a particular bot increases citations. I found none showing that Markdown copies of pages help, although the llms.txt proposal suggests them.[10] And the share of file requests that come from coding agents, about 10 percent in the Ahrefs data, suggests that developer-facing documentation may be a better use of such files than marketing pages, which is a hypothesis and not a result.[12]
A step-by-step method a B2B team can run
- Read your logs. Pull 30 days of server or CDN logs and filter for the user agents in the table plus Googlebot and Bingbot. Note which bots arrive, how often, and which pages they request. Logs are the only way to see whether a firewall rule is blocking a bot.
- Set a policy per purpose. Decide separately for search and retrieval bots, training bots and user-triggered fetchers. A common choice is to allow search and retrieval while treating training as a separate decision. That is a position to take deliberately, not a default this page recommends for everyone.
- Write the robots.txt groups. Name each bot explicitly. An example that allows the three search bots and opts out of training only:
User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / # Optional: opt out of model training only User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: /
- Check the CDN and firewall. OpenAI and Perplexity both recommend allowing their published IP ranges, and Perplexity recommends matching both the user agent string and the IP ranges in a firewall rule.[2] Test from outside your network that each bot name receives a normal page response.
- Test a plain fetch. Request your key pages without scripts and confirm that the product name, category, main claims, dates and contact details appear in the returned text.[7]
- Wait, then confirm. Allow about 24 hours after a robots.txt change, which is what OpenAI and Perplexity both quote, then check the logs again.[1]
- Verify the two webmaster consoles. Set up Bing Webmaster Tools and Search Console, and read their AI reports monthly.[8]
- Decide on llms.txt with a simple rule. Publish one only if the cost is close to zero, list canonical pages only, and expect no change in citations. Check the logs after a quarter to see whether anything fetches it.
- Run a citation test. Use a fixed list of buyer questions, record which URLs each assistant cites, and confirm that the cited pages are the ones you intended. If an assistant cites an outdated URL, fix the redirect or the canonical tag.
- Review monthly. Compare the logs, the console reports and the citation test month on month, and write down any change you made so that you can judge cause and effect honestly.
LLM SEO checklist
- Thirty days of logs reviewed for every bot in the table.
- A written policy for search, training and user-triggered access.
- robots.txt names each bot explicitly instead of relying on a wildcard.
- CDN and firewall rules allow the search bots you want, by user agent and published IP range.
- Key pages return their main facts in plain text without scripts.
- No core fact lives only in a PDF, a tab or an image.
- A 24 hour wait after every robots.txt change before judging it.
- Both webmaster consoles set up, with their AI reports reviewed each month.
- Confidential material sits behind authentication, not only behind robots.txt.
- If an llms.txt file exists, it lists canonical pages only and has an owner.
- A citation test records which URLs assistants cite for your buyer questions.
- A change log ties every access or format change to a date.
Common questions
What is LLM SEO?
It is the work of making a site reachable, readable and citable for AI products, covering crawler access, page format and the retrieval route each assistant uses. The name is informal and overlaps with GEO and AEO.
Is LLM SEO different from GEO and AEO?
Yes, in emphasis. GEO deals with what sources assistants use and weigh, AEO deals with direct answers to single questions, and LLM SEO deals with whether the systems can fetch and read your pages in the first place. Most teams need all three.
Which AI crawlers should I allow?
That depends on your goals. OpenAI says a site that opts out of OAI-SearchBot will not appear in ChatGPT search answers, Perplexity recommends allowing PerplexityBot, and Anthropic says blocking Claude-SearchBot may reduce search visibility. Training bots are a separate decision.
Do I need an llms.txt file?
Not for Google, which says it ignores such files. Two studies found no sign that the file affects citations, and one found that 97 percent of such files received no requests in a month. It is cheap to publish, but do not expect it to change what assistants say.
Will blocking GPTBot remove me from ChatGPT answers?
OpenAI documents GPTBot as a training crawler and OAI-SearchBot as the search crawler, with independent settings. On its account, blocking GPTBot alone does not remove you from ChatGPT search. Check your own logs and answers to confirm.
Is robots.txt enough to keep content away from AI tools?
No. User-triggered fetchers may not follow it, independent testing found uneven compliance, and blocking IP addresses can stop bots from reading your rules. Use authentication for anything confidential.
How can I tell whether AI crawlers visit my site?
Read your server or CDN logs for the bot names in the table. Bing Webmaster Tools and Search Console also report citation and AI feature activity for their own products. Crawling is not the same as citation, so test your buyer questions as well.
Where do I begin with Wise Mango?
Send your site address and software category through the form below. I run a few buyer questions in the major assistants myself and email you a snapshot of what they say, at no cost. Paid work is scoped only after we talk.
Who wrote this
Shruti Kaushik wrote this page as founder of Wise Mango, which is her own independent practice and not a marketplace. It has no partnership with OpenAI, Anthropic, Google, Microsoft or Perplexity. All sources listed below were opened and read on 21 November 2026, and the one news report on a study is labelled as such. See her author page for background.
Sources
- OpenAI crawlers and user agents documentation. OpenAI developer documentation, undated page. Accessed 21 November 2026.
- Perplexity crawlers. Perplexity documentation, undated page. Accessed 21 November 2026.
- AI features and your website. Google Search Central documentation, last updated 10 December 2025. Accessed 21 November 2026.
- Does Anthropic crawl data from the web, and how can site owners block the crawler?. Anthropic support article, undated page. Accessed 21 November 2026.
- Google's common crawlers. Google Search Central documentation, undated page. Accessed 21 November 2026.
- Cloudflare blog post on AI crawlers, training and referrals. Cloudflare, 29 August 2025, using Cloudflare Radar data for January to July 2025. Accessed 21 November 2026.
- Optimizing Your Content for Inclusion in AI Search Answers. Krishna Madhavan, Microsoft Advertising blog, 8 October 2025. Accessed 21 November 2026.
- Introducing AI Performance in Bing Webmaster Tools Public Preview. Microsoft Bing Webmaster Blog, 10 February 2026. Accessed 21 November 2026.
- Optimizing your website for generative AI features on Google Search. Google Search Central documentation, last updated 10 July 2026. Accessed 21 November 2026.
- The /llms.txt file. Jeremy Howard, proposal dated 3 September 2024. Accessed 21 November 2026.
- Does LLMs.txt impact your AI visibility and citations?. SE Ranking blog, analysis of nearly 300,000 domains, date shown on the page: 15 October 2026. Accessed 21 November 2026.
- 97% of llms.txt files got no requests, Ahrefs data shows. Matt G. Southern, Search Engine Journal, 16 June 2026, reporting an Ahrefs study. Accessed 21 November 2026.
- We compared eight AI search engines. They're all bad at citing news.. Klaudia Jaźwińska and Aisvarya Chandrasekar, Columbia Journalism Review, Tow Center, 6 March 2025. Accessed 21 November 2026.

Shruti Kaushik