Diffbot · Search indexes
Diffbot
Crawls the web to build the Diffbot Knowledge Graph and its web search services, where sites can be found and cited. Diffbot says it does not crawl pages to train generative AI foundation models.
- robots.txt name
Diffbot- User agent
Diffbot- Category
- Search indexes: Crawl the web to build the index of a search engine or AI search.
- Follows robots.txt
- Partly. Diffbot follows robots.txt by default, but says the rules can be overridden in specific cases, usually because of an agreement with the crawled site.
- Official documentation
- https://docs.diffbot.com/docs/does-crawl-respect-robotstxt
Checked against the official documentation on October 17, 2026.
Block or allow it in robots.txt
Add these lines to the robots.txt file at the root of your domain. To block it everywhere:
User-agent: Diffbot
Disallow: /To allow it everywhere except one folder:
User-agent: Diffbot
Disallow: /private/This bot does not always follow robots.txt. To keep it out for sure, block it on your server or firewall by user agent and, where published, by IP range.
Is it really this bot?
Anyone can copy a user agent, and Diffbot does not publish IP ranges for this bot. A reverse DNS lookup of the IP address may help; check the official documentation.
See which AI crawlers read your site
Statsdot keeps bots out of your visitors and, with AI crawler tracking, shows which crawlers read your pages and which pages they request. No cookies, no personal data.