What is ClaudeBot?
Collects public web content that could be used to train Anthropic's generative AI models.
- Operator
- Anthropic
- Purpose
- AI training
- Feeds
- Training of Anthropic generative AI models
- robots.txt token
ClaudeBot- Follows robots.txt
- Yes
- IP ranges
- Published list
- Source
- Anthropic documentation
ClaudeBot user agent
Anthropic doesn't publish a full user-agent string for ClaudeBot.
Match it in robots.txt with the token ClaudeBot. User agents can be faked, so check requests against Anthropic's published IP ranges before trusting them. A source IP on Anthropic's published list (one list shared by ClaudeBot, Claude-User and Claude-SearchBot) indicates the request comes from Anthropic; no reverse-DNS method is documented. Anthropic warns that blocking its IPs instead of using robots.txt may not reliably opt you out, because it prevents the bot from reading robots.txt.
Should you block ClaudeBot?
Restricting ClaudeBot signals that the site's future materials should be excluded from Anthropic's AI model training datasets.
Yes. Anthropic says ClaudeBot follows robots.txt. “Anthropic’s Bots respect “do not crawl” signals by honoring industry standard directives in robots.txt.” Crawl-delay is also supported.
Block ClaudeBot
User-agent: ClaudeBot Disallow: /
Allow ClaudeBot
User-agent: ClaudeBot Allow: /
Other Anthropic crawlers
Anthropic uses separate user agents for separate jobs, so you can allow one and block another.
- Claude-User (User-triggered fetch): Fetches web pages when a Claude user asks a question that needs them.
- Claude-SearchBot (AI search index): Crawls and analyzes web content to improve the relevance and accuracy of search results for Claude users.
ClaudeBot FAQ
What is ClaudeBot?
Collects public web content that could be used to train Anthropic's generative AI models.
What is the ClaudeBot user agent?
Anthropic doesn't publish a full user-agent string. In robots.txt, use the token "ClaudeBot".
Does ClaudeBot respect robots.txt?
Yes. Anthropic says ClaudeBot follows robots.txt.
How do I block ClaudeBot?
Add "User-agent: ClaudeBot" followed by "Disallow: /" to the robots.txt file at the root of your site. Restricting ClaudeBot signals that the site's future materials should be excluded from Anthropic's AI model training datasets.
Crawlable isn't the same as cited
Letting AI crawlers in is step one. Citedify checks whether ChatGPT, Claude, Perplexity and Google AI Overviews actually recommend your brand, and what to fix when they don't.
Facts on this page come from Anthropic's documentation, checked 2026-09-28. Sources: support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler, claude.com/crawling/bots.json