Training crawler
ClaudeBot: what it is.
ClaudeBot is Anthropic's crawler for web content that may help train its generative AI models.
Identity
- Operator: Anthropic
- User-agent token:
ClaudeBot - robots.txt token:
ClaudeBot - In Geordy's crawler directory: Collects pages for Anthropic model training. See the full list.
What it does, per Anthropic
Anthropic says ClaudeBot collects web content that may help train its generative AI models.
Anthropic says restricting ClaudeBot signals that your future content should be excluded from training datasets.
Anthropic documents two other agents separately: Claude-User, which accesses websites when a Claude user asks a question that requires it, and Claude-SearchBot, which crawls the web to improve search result quality and relevance.
robots.txt
Allow ClaudeBot
User-agent: ClaudeBot
Allow: /Block ClaudeBot
User-agent: ClaudeBot
Disallow: /The block lines below are the ones Anthropic prints. Anthropic says to place them in the robots.txt file at the top-level directory, and to do the same for each subdomain you want to opt out. Its page prints no allow example, so that one is standard robots.txt syntax.
Anthropic says it supports the non-standard Crawl-delay extension, for example Crawl-delay: 1 under User-agent: ClaudeBot.
Anthropic does not recommend blocking by IP address, because that may stop it reading your robots.txt and may not reliably opt you out. It says a request from an address on its published list (claude.com/crawling/bots.json) comes from Anthropic.
What Geordy observes
Share of eligible sites where ClaudeBot requested each Geordy-served file at least once, over the 90 days to 12 November 2026.
| Format | Evidence | Sites with a request | Median requests per site per day | Rank |
|---|---|---|---|---|
| Markdown | Established pattern | under 5% | 0 | Similar to others |
| Schema JSON (JSON-LD) | Established pattern | under 5% | 0 | Similar to others |
| YAML | Established pattern | under 5% | 0 | Similar to others |
| RSS Feed | Established pattern | about 5% | 0 | Similar to others |
| OG.json | Established pattern | 0% | 0 | Similar to others |
| LLMs.txt | Established pattern | 0% | 0 | Similar to others |
| Humans.txt | Established pattern | 0% | 0 | Similar to others |
| Manifest.json | Established pattern | 0% | 0 | Similar to others |
Figures are rounded and cover only sites that served the file for the whole window. Requests are to Geordy-served files, identified by user agent; a request is not evidence of use or citation. Pairs with too few independent sites are not shown. See how we measure.
Check your site
See whether your robots.txt currently allows or blocks ClaudeBot, and what else AI crawlers find on your site.
Sources
Last reviewed: 19 October 2026. Every statement about what ClaudeBot does comes from the operator's own documentation: