Skip to content

Meta · Search indexes

Meta-WebIndexer

Crawls the web to improve the quality of Meta AI search results. Meta says allowing it helps Meta AI cite and link to your content in its answers.

robots.txt name
meta-webindexer
User agent
meta-webindexer/1.1
Category
Search indexes: Crawl the web to build the index of a search engine or AI search.
Follows robots.txt
Yes
Official documentation
https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/

Checked against the official documentation on October 17, 2026.

Block or allow it in robots.txt

Add these lines to the robots.txt file at the root of your domain. To block it everywhere:

User-agent: meta-webindexer
Disallow: /

To allow it everywhere except one folder:

User-agent: meta-webindexer
Disallow: /private/

Is it really this bot?

Anyone can copy a user agent, and Meta does not publish IP ranges for this bot. A reverse DNS lookup of the IP address may help; check the official documentation.

Other bots from Meta

See which AI crawlers read your site

Statsdot keeps bots out of your visitors and, with AI crawler tracking, shows which crawlers read your pages and which pages they request. No cookies, no personal data.

Try Statsdot free for 30 days How AI crawler tracking works