free tool
AI crawler checker
see what your robots.txt tells each AI crawler: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended and more. free, no sign-up.
why it matters
AI assistants like ChatGPT, Claude and Perplexity answer many questions by searching the web and reading a few pages. if your robots.txt turns their crawlers away, they can't read your site, and they can't cite it. many sites block them by accident: a template robots.txt, a plugin, or a “block AI” toggle.
reading the results
- • allowed: nothing in robots.txt stops this crawler.
- • partly blocked: some paths are off limits. fine if they're private pages; check that your product and pricing pages aren't among them.
- • blocked: the whole site is off limits to this crawler.
the crawlers marked AI search answers or fetches for a user are the ones that matter most if you want to be recommended today. those marked model training only affect whether your pages can be used to train future models. read which AI crawlers to allow for the details, or write new rules with the AI robots.txt generator.
questions
+ − what does this check?
it reads your site's robots.txt and works out, for each main AI crawler, whether the whole site is allowed, partly blocked or blocked. it also looks for an llms.txt file and a sitemap.
+ − does "allowed" mean AI will recommend me?
no. it means the crawler is allowed to read your site, which is a precondition. whether an assistant names you depends on what the pages it reads say about you.
+ − should I block GPTBot?
GPTBot collects pages for model training. ChatGPT's search answers use OAI-SearchBot and ChatGPT-User. you can block training and still allow search: see the guide on which AI crawlers to allow.
+ − why does it say allowed when my host blocks AI bots?
this tool reads robots.txt only. a firewall or a "block AI bots" setting at your host or CDN can turn crawlers away regardless. the full RankedByAi.dev report tests what your server answers to the main AI crawlers.
+ − is it free?
yes, with no sign-up. there's a limit on checks per hour to keep it fair.
more free tools
llms.txt generator →
write a clean llms.txt for your site in a minute: name, summary, key pages. copy or download it.
AI robots.txt generator →
choose which AI crawlers may read your site (search, user fetches, training) and get robots.txt rules to paste.
see your page like AI does →
what an AI crawler reads on any page without JavaScript: text, title, headings, noindex and structured data.
AI traffic tracker →
a ready filter to see visits from ChatGPT, Perplexity, Claude, Gemini and more in Google Analytics 4.
structured data generator →
JSON-LD for your company, your software product and your FAQ, ready to paste.
where to list your SaaS →
launch sites, communities, directories and review sites where AI finds products, sorted, with what each costs.