Firecrawl · Other AI bots
FirecrawlAgent
Firecrawl is a scraping and crawling API that developers use to turn web pages into data for AI applications. It reads robots.txt before crawling a site and applies the rules for FirecrawlAgent and *.
- robots.txt name
FirecrawlAgent- User agent
FirecrawlAgent- Category
- Other AI bots: Other automated agents related to AI services.
- Follows robots.txt
- Yes
- Official documentation
- https://docs.firecrawl.dev/advanced-scraping-guide
Checked against the official documentation on October 17, 2026.
Block or allow it in robots.txt
Add these lines to the robots.txt file at the root of your domain. To block it everywhere:
User-agent: FirecrawlAgent
Disallow: /To allow it everywhere except one folder:
User-agent: FirecrawlAgent
Disallow: /private/Is it really this bot?
Anyone can copy a user agent, and Firecrawl does not publish IP ranges for this bot. A reverse DNS lookup of the IP address may help; check the official documentation.
See which AI crawlers read your site
Statsdot keeps bots out of your visitors and, with AI crawler tracking, shows which crawlers read your pages and which pages they request. No cookies, no personal data.