AI crawlers: should you block them?

AI companies send three kinds of bots to websites, and they ask for different decisions.

Training crawlers

They collect public pages that may be used to train AI models. Examples: GPTBot, ClaudeBot, CCBot, Bytespider. Some companies use a separate token only for training permission, such as Google-Extended or Applebot-Extended: blocking it does not remove you from Google Search or Apple's search features.

Block them if you do not want your content in future training data. It does not affect classic search results.

Search crawlers

They build the index that AI search and assistants cite, such as OAI-SearchBot, Claude-SearchBot or PerplexityBot.

Allow them if you want to appear, with a link, when people ask an assistant about your topic. Blocking them is like blocking a search engine.

User-triggered fetchers

They fetch a page at the moment someone asks an assistant to read it: ChatGPT-User, Claude-User, Perplexity-User. Several companies say these do not always follow robots.txt, because a person asked for the page.

A sensible default

Many sites allow search and user fetchers and block training crawlers:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

Robots.txt is a request, not a lock. For bots that ignore it, block them on your server or firewall by user agent and, where the company publishes them, by IP range.

Do they show up in your analytics?

Crawlers do not run your analytics script the way a browser does, and Statsdot also leaves known bots out on the server, so they do not inflate your visitors. To see every AI bot with its user agent and robots.txt rules, browse our AI crawler directory.