Ai2 · Training crawlers
AI2Bot
Crawler of Ai2 (Allen Institute for AI), a non-profit research institute. The content it collects is used to train open language models.
- robots.txt name
AI2Bot- User agent
Mozilla/5.0 (compatible) AI2Bot (+https://www.allenai.org/crawler)- Category
- Training crawlers: Collect public pages that may be used to train AI models.
- Follows robots.txt
- Yes
- Official documentation
- https://allenai.org/crawler
Checked against the official documentation on October 17, 2026.
Block or allow it in robots.txt
Add these lines to the robots.txt file at the root of your domain. To block it everywhere:
User-agent: AI2Bot
Disallow: /To allow it everywhere except one folder:
User-agent: AI2Bot
Disallow: /private/Is it really this bot?
Anyone can copy a user agent, and Ai2 does not publish IP ranges for this bot. A reverse DNS lookup of the IP address may help; check the official documentation.
See which AI crawlers read your site
Statsdot keeps bots out of your visitors and, with AI crawler tracking, shows which crawlers read your pages and which pages they request. No cookies, no personal data.