Anthropic · Training crawlers

ClaudeBot

Collects public web content that may be used to train Anthropic's Claude models. Blocking it signals that your future content should be left out of training data; Claude search and user fetches use other bots.

robots.txt name

ClaudeBot

User agent

ClaudeBot

Category

Training crawlers: Collect public pages that may be used to train AI models.

Follows robots.txt

Yes

Official documentation

https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler

Published IP ranges

https://claude.com/crawling/bots.json

Checked against the official documentation on October 17, 2026.

Block or allow it in robots.txt

Add these lines to the robots.txt file at the root of your domain. To block it everywhere:

User-agent: ClaudeBot
Disallow: /

To allow it everywhere except one folder:

User-agent: ClaudeBot
Disallow: /private/

Is it really this bot?

Anyone can copy a user agent. To be sure a request comes from Anthropic, compare its IP address with the ranges the company publishes.

Other bots from Anthropic

• Claude-User · AI answers
• Claude-SearchBot · Search indexes