RankedByAi.dev

free tool

AI crawler checker

see what your robots.txt tells each AI crawler: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended and more. free, no sign-up.

why it matters

AI assistants like ChatGPT, Claude and Perplexity answer many questions by searching the web and reading a few pages. if your robots.txt turns their crawlers away, they can't read your site, and they can't cite it. many sites block them by accident: a template robots.txt, a plugin, or a “block AI” toggle.

reading the results

  • • allowed: nothing in robots.txt stops this crawler.
  • • partly blocked: some paths are off limits. fine if they're private pages; check that your product and pricing pages aren't among them.
  • • blocked: the whole site is off limits to this crawler.

the crawlers marked AI search answers or fetches for a user are the ones that matter most if you want to be recommended today. those marked model training only affect whether your pages can be used to train future models. read which AI crawlers to allow for the details, or write new rules with the AI robots.txt generator.

questions

+ what does this check?

it reads your site's robots.txt and works out, for each main AI crawler, whether the whole site is allowed, partly blocked or blocked. it also looks for an llms.txt file and a sitemap.

+ does "allowed" mean AI will recommend me?

no. it means the crawler is allowed to read your site, which is a precondition. whether an assistant names you depends on what the pages it reads say about you.

+ should I block GPTBot?

GPTBot collects pages for model training. ChatGPT's search answers use OAI-SearchBot and ChatGPT-User. you can block training and still allow search: see the guide on which AI crawlers to allow.

+ why does it say allowed when my host blocks AI bots?

this tool reads robots.txt only. a firewall or a "block AI bots" setting at your host or CDN can turn crawlers away regardless. the full RankedByAi.dev report tests what your server answers to the main AI crawlers.

+ is it free?

yes, with no sign-up. there's a limit on checks per hour to keep it fair.

more free tools