Common Crawl · Training crawlers
CCBot
Crawler of Common Crawl, a non-profit that maintains an open, freely downloadable archive of the web. The archive is widely used as a source of training data for language models, so blocking CCBot keeps future pages out of it.
- robots.txt name
CCBot- User agent
CCBot/2.0 (https://commoncrawl.org/faq/)- Category
- Training crawlers: Collect public pages that may be used to train AI models.
- Follows robots.txt
- Yes
- Official documentation
- https://commoncrawl.org/ccbot
- Published IP ranges
- https://index.commoncrawl.org/ccbot.json
Checked against the official documentation on October 17, 2026.
Block or allow it in robots.txt
Add these lines to the robots.txt file at the root of your domain. To block it everywhere:
User-agent: CCBot
Disallow: /To allow it everywhere except one folder:
User-agent: CCBot
Disallow: /private/Is it really this bot?
Anyone can copy a user agent. To be sure a request comes from Common Crawl, compare its IP address with the ranges the company publishes.
See which AI crawlers read your site
Statsdot keeps bots out of your visitors and, with AI crawler tracking, shows which crawlers read your pages and which pages they request. No cookies, no personal data.