Free tool · AI robots.txt and llms.txt Generator
Choose, bot by bot, which AI crawlers can access your website: the ones that cite you in ChatGPT, Claude and Perplexity answers, and the ones that collect content to train AI models. Get a ready-to-paste robots.txt, check your current file and create an llms.txt.
The generator
Start from a preset, then switch any bot on or off. Every bot name below is checked against its company's own documentation.
The checker
Open yourdomain.com/robots.txt, copy everything and paste it below. We'll show what each AI bot is currently allowed to do.
Paste your robots.txt to see what each AI bot can do.
The llms.txt builder
An llms.txt file is a short Markdown summary of your site with links to your key pages, following the format at llmstxt.org.
Send us your site and we'll reply personally with a recommendation on which AI bots to allow, and check for anything else blocking AI from reading your content. Free, with no obligation.
The guide
Add a group to your robots.txt for each AI bot you want to control: a User-agent line with the bot's name, followed by Allow: / to let it in or Disallow: / to keep it out. Several bots can share one group by listing their User-agent lines together.
For example, this lets ChatGPT search show your pages but opts you out of OpenAI's model training:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /robots.txt must live at the root of your domain, such as yourdomain.com/robots.txt. After changing it, Google Search Console's robots.txt report shows whether Google can read the file.
Most AI companies now run separate bots for separate jobs, and you can control each one independently. OpenAI, for example, says you can allow OAI-SearchBot to appear in ChatGPT search while disallowing GPTBot to opt out of training.
| Bot | Company | Type | What it does |
|---|---|---|---|
OAI-SearchBot | OpenAI | AI search and answer engines | Shows and cites your pages in ChatGPT search. |
Claude-SearchBot | Anthropic | AI search and answer engines | Indexes pages to improve Claude's search results. |
PerplexityBot | Perplexity | AI search and answer engines | Surfaces and links your pages in Perplexity answers. Not used to train AI models. |
Amzn-SearchBot | Amazon | AI search and answer engines | Makes your content eligible for Amazon search experiences such as Alexa. |
DuckAssistBot | DuckDuckGo | AI search and answer engines | Crawls pages in real time for DuckDuckGo's AI-assisted answers, which cite their sources. |
ChatGPT-User | OpenAI | User-requested visits | Visits a page when a ChatGPT user asks it to. |
Claude-User | Anthropic | User-requested visits | Visits a page when a Claude user asks a question. |
Perplexity-User | Perplexity | User-requested visits | Visits a page to answer a Perplexity user's question, and links to it. |
MistralAI-User | Mistral AI | User-requested visits | Visits a page when a Mistral user asks a question, and links to the source. |
Meta-ExternalFetcher | Meta | User-requested visits | Fetches individual links at a user's request in Meta's AI products. |
Amzn-User | Amazon | User-requested visits | Fetches pages in response to user requests in Amazon's products. |
GPTBot | OpenAI | AI model training | Collects content that may be used to train OpenAI's models. |
ClaudeBot | Anthropic | AI model training | Collects content that may be used to train Anthropic's models. |
Google-Extended | AI model training | A robots.txt setting, not a crawler. Controls use of your content to train Gemini models and to ground Gemini answers. Doesn't affect Google Search. | |
Applebot-Extended | Apple | AI model training | A robots.txt setting, not a crawler. Controls use of your content to train Apple's AI models. Doesn't affect Apple search results. |
Meta-ExternalAgent | Meta | AI model training | Crawls for uses such as training Meta's AI models and indexing content. |
Amazonbot | Amazon | AI model training | Improves Amazon's products and services, and may be used to train Amazon's AI models. |
CCBot | Common Crawl | AI model training | Builds Common Crawl's open web archive, which is widely used to train AI models. |
Checked against each company's own documentation in October 2026. Bot names change from time to time, so check the linked pages before you rely on any rule.
For most businesses, the answer is to block selectively, not wholesale:
Leave Googlebot alone. According to Google, Google-Extended only controls Gemini training and grounding, and doesn't affect inclusion or ranking in Google Search. Blocking Googlebot itself would remove you from Google, AI Overviews included.
The major AI companies document that their crawlers follow robots.txt, but there are two important caveats.
robots.txt is a request, not a lock. If you need to keep content away from bots entirely, put it behind a login or use your firewall or CDN's bot controls.
User-agent: * rulesA crawler follows only the most specific group that names it. As soon as you give an AI bot its own group, it stops reading your User-agent: * group. If that group blocks private areas such as /admin or /checkout, repeat those Disallow lines in the AI bot's group. The “Private paths” field in the generator does this for you.
llms.txt is a proposed standard for a Markdown file, at yourdomain.com/llms.txt, that gives AI agents a short summary of your site and links to its most useful pages. It was proposed by Jeremy Howard in 2024 and updated to version 2 in August 2026.
The format is simple: an H1 with your site's name (the only required part), a one-line summary in a blockquote, optional context, then sections of links, each headed with an H2. A section called “Optional” holds secondary links an agent can skip.
According to llmstxt.org, llms.txt files are used most heavily for software documentation, where coding agents follow them. There's no evidence it influences what ChatGPT or Google's AI features cite, so treat it as a quick, low-cost extra rather than a priority.
Only if you don't want your content used to train OpenAI's models. Blocking GPTBot doesn't stop ChatGPT search from showing and citing your pages; that's controlled separately by OAI-SearchBot. OpenAI says each setting is independent.
Not if you leave Googlebot alone. Google-Extended only controls whether your content is used for Gemini training and grounding, and Google says it doesn't affect inclusion or ranking in Google Search. Never block Googlebot to avoid AI Overviews: it would remove you from Google Search entirely.
robots.txt is a voluntary standard. OpenAI, Anthropic, Perplexity, Google, Apple, Meta, Amazon and others document that their crawlers follow it, but user-triggered fetchers can be exceptions. If you need a hard block, use a firewall or CDN rule.
Usually up to a day. Meta says its crawlers may cache robots.txt for up to 24 hours, and Perplexity and Amazon give similar timeframes.
There's no evidence it's a ranking or citation factor for AI search. llms.txt is mostly used by coding agents and AI assistants that read documentation. It's quick to add and can help agents find your key pages, but crawler access, clear content and authority matter far more.
Yes, in your server logs or your host's bot reports, by searching for the user-agent names in the table above. Google Analytics won't show them, because crawlers don't run analytics code.
No. Everything is generated and checked in your browser. Nothing you paste or type is sent or saved.
Keep going
Is your website AI-friendly? Fifteen questions show how likely ChatGPT, Perplexity, Gemini and Google AI Overviews are to recommend your brand, with a prioritised fix list.
Tell people who find a security flaw on your site how to report it. Create a security.txt file, check your existing one and get a disclosure policy template.
Track ChatGPT, Perplexity, Gemini, Claude and Copilot visits in Google Analytics 4. Get a tested regex, check it against your own sources, then follow the setup steps.