Free Tool
AI Robots.txt Generator
Most robots.txt generators only know about Googlebot. This one handles every modern AI crawler — OpenAI, Anthropic, Google's Gemini, Perplexity, Apple, and the rest. Toggle each bot, copy the output, ship it.
Presets
OpenAI
GPTBot
Trains future OpenAI models on your content. Block this if you don't want your docs in training data.
OAI-SearchBot
Indexes for ChatGPT Search results. Block to disappear from ChatGPT's search-style answers.
ChatGPT-User
Fetched in real-time when a user asks ChatGPT to read your URL. Almost always allow.
Anthropic
ClaudeBot
Anthropic's training crawler. Block to keep docs out of future Claude training.
anthropic-ai
Older Anthropic UA — kept for backward compatibility.
Claude-Web
Real-time fetches when a Claude user asks it to read a URL.
Google-Extended
Controls inclusion in Gemini training and Google's AI overviews. Doesn't affect normal search ranking.
Perplexity
PerplexityBot
Indexes for Perplexity answers and sources. Allowing means more citation traffic.
Apple
Applebot-Extended
Apple Intelligence training. Doesn't affect Siri search.
Other
CCBot
Common Crawl — feeds many AI training datasets indirectly.
Bytespider
ByteDance / TikTok's crawler. Often misbehaves — many sites block it.
FacebookBot
Meta's training crawler for Llama. Block to keep out.
robots.txt
# Generated by Docsio AI Robots.txt Generator # https://docsio.co/tools/ai-robots-txt User-agent: * Allow: / # OpenAI User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / # Anthropic User-agent: ClaudeBot Allow: / User-agent: anthropic-ai Allow: / User-agent: Claude-Web Allow: / # Google User-agent: Google-Extended Allow: / # Perplexity User-agent: PerplexityBot Allow: / # Apple User-agent: Applebot-Extended Allow: / # Other User-agent: CCBot Allow: / User-agent: Bytespider Disallow: / User-agent: FacebookBot Allow: /
How it works
Toggle
Pick allow or block for each AI bot, or use a preset.
Generate
Output is grouped by provider with a comment header.
Deploy
Save as robots.txt at your site root.
Want this baked into your docs site?
Docsio generates a branded documentation site from your URL in minutes — with llms.txt, schema markup, and AI-friendly structure already built in.
Try Docsio FreeFAQ
AI robots.txt: common questions
What does the AI robots.txt generator do?
It builds a robots.txt file with per-bot rules for every major AI crawler. You toggle each bot on or off, set allow or disallow paths, and copy the output to the root of your site. The file tells crawlers like GPTBot, ClaudeBot, and PerplexityBot whether they can read your pages for training or live retrieval.
Which AI bots does the robots.txt generator cover?
It covers the bots that ship with the largest AI products: GPTBot and ChatGPT-User from OpenAI, ClaudeBot and Claude-Web from Anthropic, PerplexityBot, Google-Extended, Applebot-Extended, Bytespider, Amazonbot, CCBot, and Meta-ExternalAgent. New crawlers are added when vendors publish their user agents.
Should I allow or block AI crawlers?
It depends on your goal. Allow them if you want your docs and content cited by ChatGPT, Claude, and Perplexity, which usually means more discoverability and brand mentions. Block them if your content is paid, gated, or you do not want it used for model training. Many teams allow live retrieval bots and block training crawlers.
Is the AI robots.txt generator really free?
Yes, the generator is free with no signup, no rate limit, and no email gate. You can use it for as many sites as you want. Docsio also auto-generates an AI-friendly robots.txt and llms.txt for every published docs site, so customers do not have to think about crawler rules.
Where do I put the robots.txt file?
Save the output as robots.txt and upload it to the root of your domain, served at https://yoursite.com/robots.txt with content-type text/plain. Crawlers check this exact path before fetching other URLs. If you use a CMS or static site host, drop it in the public or static folder so it ships at the root.
How often should I update my AI robots.txt?
Review it every few months or when a new AI product launches. Vendors add and rename bots more often than search crawlers, so a file that was correct last year may miss newer agents. Regenerating with this tool takes under a minute and keeps your rules aligned with the current crawler list.