AI Crawler robots.txt Tester

Paste your robots.txt, enter a path, and instantly see whether Googlebot, Bingbot and the main AI crawlers such as GPTBot, ClaudeBot and PerplexityBot are allowed or blocked. Everything runs in your browser; nothing is sent anywhere.

User agentWhat it isGroup usedMatching ruleResult

Generate robots.txt rules to block AI crawlers

Tick the crawlers you want to block site-wide. Add the output to your existing robots.txt; do not replace your current rules unless you mean to.


 

What are AI crawlers?

AI crawlers are automated bots operated by AI companies to fetch web pages. Some collect public content to train large language models, some build search indexes that power AI answers, and some fetch a single page on demand when a user asks an assistant to read a link. Each one identifies itself with a user-agent token, and most of the large operators publish their tokens and say they respect robots.txt. That is what makes it possible to allow or block them with a few lines in a file at the root of your domain.

Training bots vs search and retrieval bots

The distinction matters because blocking the wrong bot can cost you visibility. Training crawlers, such as GPTBot from OpenAI, ClaudeBot from Anthropic and CCBot from Common Crawl, gather content that may be used to train future models. Blocking them keeps your content out of future training sets that respect the rule, but it does not remove anything already collected.

Search and retrieval bots work differently. OAI-SearchBot is used to surface sites in ChatGPT search results, PerplexityBot builds Perplexity’s index, and ChatGPT-User fetches pages when a person asks ChatGPT to visit them. If you block these, your pages are less likely to be cited or linked in those AI answers. Some operators also note that user-initiated fetchers may not follow robots.txt in the same way as automated crawlers, so check each vendor’s current documentation.

A few tokens are not crawlers at all. Google-Extended and Applebot-Extended are control tokens: Google and Apple crawl with their normal bots, and these tokens only tell them whether your content may be used for their AI models. Blocking Google-Extended does not affect how Googlebot crawls or ranks your site in Google Search. Note that Google’s AI Overviews are part of Search and are governed by Googlebot, not by Google-Extended.

How robots.txt rules are evaluated

This tester follows the rules in RFC 9309 and Google’s documentation. A crawler looks for the group whose user-agent line matches its own token and ignores all other groups. If no group names it, it falls back to the User-agent: * group. Within the chosen group, the most specific rule wins, meaning the matching Allow or Disallow pattern with the most characters. If an Allow and a Disallow rule are equally long, Allow wins. The * wildcard matches any sequence of characters and a $ at the end anchors the pattern to the end of the URL. An empty Disallow line allows everything.

robots.txt is advisory

robots.txt is a voluntary protocol, not access control. Reputable crawlers honour it, but nothing technically stops a bot from ignoring it, and some scrapers do. For content that must stay private, use authentication. To stop bots that ignore robots.txt, use server or firewall rules, for example in your host, CDN or a security plugin. Also remember that robots.txt blocks crawling, not indexing: a blocked URL can still appear in search results if other sites link to it.

Frequently asked questions

Will blocking GPTBot hurt my Google rankings?

No. GPTBot is OpenAI’s training crawler and has nothing to do with Google Search. Googlebot follows its own group or the * group in your robots.txt.

Does blocking Google-Extended remove my site from AI Overviews?

No. Google-Extended controls whether content is used for Gemini models and grounding. AI Overviews are part of Google Search and are controlled by Googlebot rules.

What is the difference between GPTBot and OAI-SearchBot?

GPTBot collects content that may be used to train OpenAI models. OAI-SearchBot is used to show and link websites in ChatGPT search results. You can block one and allow the other.

Do all AI crawlers obey robots.txt?

No. robots.txt is advisory. Major operators say they respect it, but some bots ignore it. Use server or firewall rules if you need to enforce a block.