ByteDance · Training crawlers

Bytespider

Crawler of ByteDance, the owner of TikTok. Bot directories report that it downloads training data for ByteDance's language models, including those behind its Doubao assistant.

robots.txt name

Bytespider

User agent

Mozilla/5.0 (compatible; Bytespider; https://zhanzhang.toutiao.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.0.0 Safari/537.36

Category

Training crawlers: Collect public pages that may be used to train AI models.

Follows robots.txt

Partly. Press reports in 2024 said it did not always follow robots.txt; block it on the server as well if it matters to you.

Official documentation

https://knownagents.com/agents/bytespider

Checked against the official documentation on October 17, 2026.

Block or allow it in robots.txt

Add these lines to the robots.txt file at the root of your domain. To block it everywhere:

User-agent: Bytespider
Disallow: /

To allow it everywhere except one folder:

User-agent: Bytespider
Disallow: /private/

This bot does not always follow robots.txt. To keep it out for sure, block it on your server or firewall by user agent and, where published, by IP range.

Is it really this bot?

Anyone can copy a user agent, and ByteDance does not publish IP ranges for this bot. A reverse DNS lookup of the IP address may help; check the official documentation.