Free Tool
AI Robots.txt Generator
Most robots.txt generators only know about Googlebot. This one handles every modern AI crawler — OpenAI, Anthropic, Google's Gemini, Perplexity, Apple, and the rest. Toggle each bot, copy the output, ship it.
Presets
OpenAI
GPTBot
Trains future OpenAI models on your content. Block this if you don't want your docs in training data.
OAI-SearchBot
Indexes for ChatGPT Search results. Block to disappear from ChatGPT's search-style answers.
ChatGPT-User
Fetched in real-time when a user asks ChatGPT to read your URL. Almost always allow.
Anthropic
ClaudeBot
Anthropic's training crawler. Block to keep docs out of future Claude training.
anthropic-ai
Older Anthropic UA — kept for backward compatibility.
Claude-Web
Real-time fetches when a Claude user asks it to read a URL.
Google-Extended
Controls inclusion in Gemini training and Google's AI overviews. Doesn't affect normal search ranking.
Perplexity
PerplexityBot
Indexes for Perplexity answers and sources. Allowing means more citation traffic.
Apple
Applebot-Extended
Apple Intelligence training. Doesn't affect Siri search.
Other
CCBot
Common Crawl — feeds many AI training datasets indirectly.
Bytespider
ByteDance / TikTok's crawler. Often misbehaves — many sites block it.
FacebookBot
Meta's training crawler for Llama. Block to keep out.
robots.txt
# Generated by Docsio AI Robots.txt Generator # https://docsio.co/tools/ai-robots-txt User-agent: * Allow: / # OpenAI User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / # Anthropic User-agent: ClaudeBot Allow: / User-agent: anthropic-ai Allow: / User-agent: Claude-Web Allow: / # Google User-agent: Google-Extended Allow: / # Perplexity User-agent: PerplexityBot Allow: / # Apple User-agent: Applebot-Extended Allow: / # Other User-agent: CCBot Allow: / User-agent: Bytespider Disallow: / User-agent: FacebookBot Allow: /
How it works
Toggle
Pick allow or block for each AI bot, or use a preset.
Generate
Output is grouped by provider with a comment header.
Deploy
Save as robots.txt at your site root.
Want this baked into your docs site?
Docsio generates a branded documentation site from your URL in minutes — with llms.txt, schema markup, and AI-friendly structure already built in.
Try Docsio Free