Skip to content
BBenchline

Crawlers

Four AI crawlers, and the robots.txt lines that mention them

Benchline fetches /robots.txt and asks a narrow question: is “/” disallowed for GPTBot, ClaudeBot, PerplexityBot, or Google-Extended? It is a policy reading, not a log of who visited.

A specific user-agent group wins over User-agent: *. The longest matching path wins. If an allow and a disallow tie, the allow wins. An empty Disallow: does not block anything. Those are the rules the checker implements, and they match the way operators usually intend a file to work.

If the file cannot be fetched, every crawler row is unknown and worth zero until you run the check again. Unknown is a hole in the evidence, not a verdict that you are blocked.

Run a free check

Brand, website, and an optional category. You get a shareable report with a citability score, a prompt pack, and a paste scorer.

Check a brand
Homepage, robots.txt, and llms.txt only. The report page is public.

We request the homepage, /robots.txt, and /llms.txt.

Fills prompts such as “best {category} tools 2026”.

Free. No account. 8 checks per hour and 30 per day from one network. Turnstile runs only when the site operator configures it.

What each token is for

  • GPTBot is OpenAI’s crawler token for model training. It is not the same string as the agent that fetches a page during a chat.
  • ClaudeBot is Anthropic’s crawler token. Older notes also mention anthropic-ai. This check looks for ClaudeBot.
  • PerplexityBot is Perplexity’s crawler token.
  • Google-Extended is Google’s control for generative use. Leaving it allowed does not change how Googlebot indexes Search.

Confirm the current token in each company’s crawler documentation before you ship a change. Names get revised. The score uses the four strings above.

A conservative allow

This snippet allows the four agents and keeps a private path out of the wildcard group. It is an illustration, not a full policy.

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: *
Allow: /
Disallow: /admin

After you edit the file, run the visibility check again. Pair it with llms.txt if you want a crawler that is allowed in to also know which URLs you prefer.

Questions

Which user-agents does the score include?

GPTBot, ClaudeBot, PerplexityBot, and Google-Extended. Each is its own row. A disallow for one does not automatically fail the others.

What if I have no robots.txt?

Crawlers are allowed by default. The check marks those rows as a pass and says the file is missing. Publishing a file is still useful, because it makes the policy explicit.

Is Google-Extended the same as Googlebot?

No. Google-Extended is a control for Google’s generative use. Googlebot is the Search crawler. This score does not grade Googlebot. Do not add “User-agent: * / Disallow: /” to opt out of one bot.

Does ChatGPT-User count as GPTBot?

No. GPTBot is the training crawler we score. ChatGPT-User is a different agent used when a person asks ChatGPT to browse. Check the vendor’s current docs before you copy a token.

Related guides