Do You Need an llms.txt File?

By Mehmet, Top Seed AI · Published Oct 20, 2026

In short: llms.txt is a proposed Markdown file that gives AI agents a short guide to your site. Google says its Search ignores it, none of the other companies we checked documents reading it, and the large studies we found show no clear citation advantage. Adding one is cheap, but nothing we found shows it gets a brand recommended. Fix crawl access, indexing and answer pages first.

What llms.txt is, and what it is not

llms.txt is a proposed Markdown file, placed at yourdomain.com/llms.txt, that gives AI agents a short, curated overview of a site with links to its key pages.

Jeremy Howard proposed it. The page shows a Sept 3, 2024 publication date and is labeled version 2, modified Aug 10, 2026. Only a heading with the site name is required; a summary and lists of linked pages can follow.

What it is not:

• Not robots.txt. The proposal says robots.txt signals which automated access is acceptable, while llms.txt is read on demand when an agent is helping a user.
• Not a rule anyone must follow. The page calls itself a proposal. It does not say any AI company has committed to using the file for other sites, or mention rankings, citations or visibility, and it says use is heaviest for software documentation.

How to check: open yourdomain.com/llms.txt. Some builders and plugins create one automatically (the proposal's page lists Wix, Yoast SEO and AIOSEO), so you may already have one.

What Google says

Google says its Search does not use llms.txt, and that having one neither helps nor harms your visibility in Google Search.

Google's guide to optimizing for generative AI features (last updated Jul 10, 2026) lists llms.txt files and similar special markup among the things to ignore for Google Search, and says creating one for other services is fine. An earlier page, AI features and your website (last updated Dec 10, 2025), says AI Overviews and AI Mode need no new machine-readable files, AI text files or markup, without naming llms.txt.

The one Google representative statement we could open in a primary source is John Mueller's Bluesky post of June 17, 2025: "FWIW no AI system currently uses llms.txt." It is a personal-account post, over a year old, not documentation.

Chrome's Lighthouse documentation (last updated May 5, 2026) includes an llms.txt check in an experimental Agentic Browsing category. It calls the file optional for now and does not mention rankings or citations. That is a developer-tool check, not evidence Google Search uses the file.

What OpenAI, Anthropic, Perplexity and Microsoft say

In the pages we opened, none of these four companies says its crawlers or assistants read llms.txt files on other sites.

Microsoft is the thinnest row: we could not read Bing's Webmaster Guidelines page, which needs JavaScript.

OpenAI, Anthropic and Perplexity each publish an llms.txt file for their own developer documentation (OpenAI, Anthropic, Perplexity). That shows how they serve docs to developers, not what their crawlers do with your file.

What the published studies found

The large studies found no clear link between having the file and AI citations, and Ahrefs found most files draw no requests. All three come from companies that sell related tools, and none is a controlled test.

Adoption depends on who is counted: about 10% to 28% across these three samples, and none is the whole web.

On requests, Ahrefs found that the search crawlers of OpenAI, Perplexity and Anthropic made only a couple of hundred requests combined across thousands of sites, while Anthropic's coding agent, Claude Code, requested the file more often than any AI search or assistant bot.

The nearest case for the file is from Wix's AI Search Lab (page dated Jun 1, 2026); Wix generates the file for its users. It showed llms.txt files surfacing in Google and AI search answers, called its explanation a hypothesis, and put Google's index at about 120,000 such files by May 2026. That concerns the files appearing, not a brand being recommended.

Evidence is thin. The one controlled test we found, a vendor test by Attrifast, used five sites with the file and five without; its author calls the one gain, on Perplexity, suggestive, not conclusive. The correlational studies can miss a small effect.

Cost, risk and what to do first

Per Google, the file does no harm in Google Search, and the evidence does not show it changes what assistants recommend, so the basics come first.

Cost and risk: a short file takes little time; keeping it accurate is the ongoing work. Our judgment, not a study finding: the risks are a stale or auto-generated list pointing to outdated pages, and time not spent on the basics.

What to expect: in Ahrefs's sample the likeliest outcome was zero requests. Our reading: the proposal names documentation as the main use, so a case is more plausible for a developer-docs site than a local business; we found no test of that.

Do these first:

1. Let the search crawlers in. OpenAI says blocking OAI-SearchBot keeps a site out of ChatGPT's search answers, though plain links can still appear; Anthropic says blocking Claude-SearchBot may reduce visibility; Perplexity recommends allowing PerplexityBot. Check robots.txt and bot protection.
2. Make sure key pages are indexed. Google says a page has to be indexed, and able to appear in Search with a snippet, to qualify for its AI features.
3. Give each question one clear page with the answer in the first sentence.
4. Keep facts consistent across your site and listings.

Our guides Why ChatGPT Doesn't Mention Your Brand and How to Get Recommended by AI Assistants cover these steps. We sell AI visibility work that includes them, so weigh our emphasis; each can be done without us.

How to check whether any crawler requests it on your site

Server logs show whether a crawler asks for /llms.txt, not whether anything used it. If this is too technical, ask your host or developer to run steps 2 to 4.

1. Find your logs in your hosting control panel, CDN dashboard or server access logs. If your host keeps little or none, you cannot run this check.
2. Filter for /llms.txt over the last 30 to 90 days, noting date, status code, user agent and IP. Do the same for /robots.txt to see which crawlers visit at all. A 404 means something asked for a file you do not have (ignore your own visits).
3. Group by user agent and compare with the vendors' published names: OAI-SearchBot, GPTBot, ChatGPT-User (OpenAI); ClaudeBot, Claude-SearchBot, Claude-User (Anthropic); PerplexityBot, Perplexity-User (Perplexity); Googlebot.
4. Confirm the source. User-agent strings can be faked, and OpenAI, Perplexity and Anthropic each link to a list of their crawler IP addresses.
5. Read the result carefully. A fetch does not mean the file was read, as Ahrefs notes, and Googlebot requests any URL it finds. In an informal test reported Aug 7, 2026, a joke file called cats.txt was requested by PerplexityBot, GPTBot, ClaudeBot and Googlebot. If you publish a file, repeat the check after 30 days.

What nobody can promise

No one can promise that an llms.txt file will get you cited or recommended, and no company we checked says it will. Be cautious about anyone who sells it as an AI ranking service.

Questions people ask

Does llms.txt help AI citations?

We found no large published test showing that it does. SE Ranking's analysis of nearly 300,000 domains found no correlation with citations. Trakkr's check of 37,894 AI-cited domains found near-identical averages, though one of its two tests showed a small significant difference. Both firms sell related tools and both studies are observational, so neither rules out a small effect.

Should I add llms.txt to my website?

The evidence we found does not make it a priority for being recommended. Google says it neither helps nor harms Search, and no AI company we checked documents reading it. If you add one, keep it accurate. Check crawler access, indexing and your key answer pages first.

What is llms.txt?

It is a proposed Markdown file at yourdomain.com/llms.txt that gives AI agents a short summary of your site and links to key pages. Jeremy Howard published the proposal on Sept 3, 2024. It is not robots.txt: it does not control access, and none of the AI companies we checked says its crawlers follow it.

Does llms.txt help SEO?

Google says Search ignores it and that it neither helps nor harms rankings in Google Search. It is not a ranking file or a crawl-control file. Chrome's Lighthouse has an optional check for it, but that page does not mention rankings.

Sources

• llmstxt.org (Jeremy Howard), The /llms.txt file, v2, published Sept 3, 2024, modified Aug 10, 2026: the proposal itself; its statements about adoption and tools are the author's own and we did not verify them.
• Google, Google's Guide to Optimizing for Generative AI Features on Google Search, last updated Jul 10, 2026: primary documentation.
• Google, AI Features and Your Website, last updated Dec 10, 2025: primary documentation.
• John Mueller (Google), Bluesky post, Jun 17, 2025: primary source; one person's statement, not Google policy.
• Chrome for Developers, Lighthouse llms.txt audit and Lighthouse agentic browsing scoring, both last updated May 5, 2026: primary documentation, experimental category.
• OpenAI, Overview of OpenAI Crawlers, no date shown: primary documentation.
• Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler?, Apr 7, 2026: primary documentation.
• Perplexity, Perplexity Crawlers, no date shown: primary documentation.
• Microsoft Advertising, Optimizing Your Content for Inclusion in AI Search Answers, Oct 8, 2025: primary; blog post by a Microsoft Bing product manager.
• OpenAI, Anthropic and Perplexity developer documentation indexes: developers.openai.com/llms.txt, platform.claude.com/llms.txt, docs.perplexity.ai/llms.txt, opened Oct 20, 2026: primary files.
• Ahrefs, We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read, Jun 15, 2026 (modified Aug 27, 2026): vendor study; server logs of its own analytics customers, one month, adoption figure called an upper bound; no citation analysis.
• SE Ranking, Does LLMs.txt impact your AI visibility and citations?, page dated Oct 15, 2026 (press coverage dates from Nov 2025): vendor study; nearly 300,000 domains; correlational; the page does not state the AI platforms, time window or detection method.
• Search Engine Journal, LLMs.txt Shows No Clear Effect On AI Citations, Based On 300k Domains, Nov 20, 2025: press write-up of the SE Ranking study.
• Trakkr, The llms.txt effect, last updated Mar 14, 2026 (date of the data): vendor study; 37,894 selected AI-cited domains; observational; the page's headline box differs from its body, and we used the body figures.
• Search Engine Journal (Mark Williams-Cook), How Cats.txt Showed LLMs.txt Evidence Is GEO Astrology, Aug 7, 2026: opinion piece with one informal test, no control group described.
• Wix AI Search Lab, Debunking LLMs.txt Myths, page dated Jun 1, 2026 (research from Oct and Nov 2025, updated May 2026): vendor article; qualitative examples and Google index counts; Wix generates the file for its users.
• Attrifast, Is llms.txt Worth It? A 10-Site Controlled Test, May 26, 2026: vendor test (the firm sells an analytics platform); 5 treated and 5 control sites that the author owns or advises; four AI engines; Perplexity +10.2 points (p about 0.04), the other engines not significant.

External figures were checked against the linked pages on Oct 20, 2026. Where a study comes from a company that sells related tools, we say so. None of the outside studies measures your brand.

Related guides

• Why ChatGPT Doesn't Mention Your Brand
• How to Get Recommended by AI Assistants
• AI Visibility vs SEO: What's the Difference?

About this guide. Written by Mehmet, who runs Top Seed AI. We sell AI visibility measurement and improvement, so we have an interest in this topic; the sources above are there so you can check our reading. Published Oct 20, 2026.