RankedByAi.dev

free tool

AI robots.txt generator

choose which AI crawlers may read your site, for AI search answers, user fetches or model training, and get robots.txt rules to paste. free, and it runs in your browser.

your robots.txt

# AI crawlers that are welcome
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: GPTBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: ClaudeBot
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Allow: /
Disallow: /admin/
Disallow: /api/

# everyone else
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/

check it before you replace a live robots.txt, and keep any rules your site already needs.

search vs training: the choice that matters

AI companies run separate crawlers for separate jobs. the search and user crawlers fetch pages so an assistant can answer a question today. the training crawlers collect pages for future models. if you want AI assistants to recommend you, keep the search and user crawlers allowed. you can decide about training separately: the “allow AI search, block training” preset does exactly that.

more in which AI crawlers to allow in robots.txt. after you change it, check the result with the AI crawler checker.

questions

+ will blocking AI crawlers hurt my AI visibility?

blocking the search and user crawlers (like OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot) can keep you out of AI answers. blocking only training crawlers (like GPTBot or ClaudeBot) doesn't stop assistants finding you when they search.

+ what is Google-Extended?

a robots.txt name, not a separate crawler. it controls whether Google may use your pages for Gemini. Google Search crawling is unaffected.

+ why are my private paths repeated in each group?

a crawler follows only the group that names it and ignores the * group, so any path every crawler should skip has to be listed in each group.

+ is this list complete?

it covers the main AI crawlers as their companies document them. names change and new ones appear, so check each company's documentation for anything critical.

more free tools