free tool
AI robots.txt generator
choose which AI crawlers may read your site, for AI search answers, user fetches or model training, and get robots.txt rules to paste. free, and it runs in your browser.
your robots.txt
# AI crawlers that are welcome User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: GPTBot User-agent: Claude-SearchBot User-agent: Claude-User User-agent: ClaudeBot User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot Allow: / Disallow: /admin/ Disallow: /api/ # everyone else User-agent: * Allow: / Disallow: /admin/ Disallow: /api/
check it before you replace a live robots.txt, and keep any rules your site already needs.
search vs training: the choice that matters
AI companies run separate crawlers for separate jobs. the search and user crawlers fetch pages so an assistant can answer a question today. the training crawlers collect pages for future models. if you want AI assistants to recommend you, keep the search and user crawlers allowed. you can decide about training separately: the “allow AI search, block training” preset does exactly that.
more in which AI crawlers to allow in robots.txt. after you change it, check the result with the AI crawler checker.
questions
+ − will blocking AI crawlers hurt my AI visibility?
blocking the search and user crawlers (like OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot) can keep you out of AI answers. blocking only training crawlers (like GPTBot or ClaudeBot) doesn't stop assistants finding you when they search.
+ − what is Google-Extended?
a robots.txt name, not a separate crawler. it controls whether Google may use your pages for Gemini. Google Search crawling is unaffected.
+ − why are my private paths repeated in each group?
a crawler follows only the group that names it and ignores the * group, so any path every crawler should skip has to be listed in each group.
+ − is this list complete?
it covers the main AI crawlers as their companies document them. names change and new ones appear, so check each company's documentation for anything critical.
more free tools
AI crawler checker →
see which AI crawlers (GPTBot, ClaudeBot, PerplexityBot and more) your robots.txt lets in, and whether you have llms.txt and a sitemap.
llms.txt generator →
write a clean llms.txt for your site in a minute: name, summary, key pages. copy or download it.
see your page like AI does →
what an AI crawler reads on any page without JavaScript: text, title, headings, noindex and structured data.
AI traffic tracker →
a ready filter to see visits from ChatGPT, Perplexity, Claude, Gemini and more in Google Analytics 4.
structured data generator →
JSON-LD for your company, your software product and your FAQ, ready to paste.
where to list your SaaS →
launch sites, communities, directories and review sites where AI finds products, sorted, with what each costs.