Anthropic · Search indexes

Claude-SearchBot

Crawls the web to improve the relevance and accuracy of Claude's search results. Blocking it keeps your content out of that index, which may reduce your visibility in search answers.

robots.txt name

Claude-SearchBot

User agent

Claude-SearchBot

Category

Search indexes: Crawl the web to build the index of a search engine or AI search.

Follows robots.txt

Yes

Official documentation

https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler

Published IP ranges

https://claude.com/crawling/bots.json

Checked against the official documentation on October 17, 2026.

Block or allow it in robots.txt

Add these lines to the robots.txt file at the root of your domain. To block it everywhere:

User-agent: Claude-SearchBot
Disallow: /

To allow it everywhere except one folder:

User-agent: Claude-SearchBot
Disallow: /private/

Is it really this bot?

Anyone can copy a user agent. To be sure a request comes from Anthropic, compare its IP address with the ranges the company publishes.

Other bots from Anthropic

• Claude-User · AI answers
• ClaudeBot · Training crawlers