Search indexing crawler
Claude-SearchBot: what it is.
Claude-SearchBot is the Anthropic crawler that navigates the web to improve search result quality for Claude users.
Identity
- Operator: Anthropic
- User-agent token:
Claude-SearchBot - robots.txt token:
Claude-SearchBot - In Geordy's crawler directory: Indexes pages so Claude search can surface them. See the full list.
What it does, per Anthropic
Anthropic says Claude-SearchBot navigates the web to improve search result quality for users.
Anthropic says that disabling it prevents its system from indexing your content for search optimization, which may reduce your site's visibility and accuracy in search results.
Anthropic documents ClaudeBot (training) and Claude-User (user questions) as separate agents.
robots.txt
Allow Claude-SearchBot
User-agent: Claude-SearchBot
Allow: /Block Claude-SearchBot
User-agent: Claude-SearchBot
Disallow: /Anthropic's page does not print robots.txt lines for Claude-SearchBot, so the ones below are standard robots.txt syntax around the agent's name. The only lines it prints are for ClaudeBot.
Anthropic says its bots honor industry standard directives in robots.txt, and says to add rules for each subdomain you want to cover.
Anthropic does not recommend blocking by IP address, because that may stop its bots reading your robots.txt. It says a request from an address on its published list (claude.com/crawling/bots.json) comes from Anthropic.
What Geordy observes
Share of eligible sites where Claude-SearchBot requested each Geordy-served file at least once, over the 90 days to 12 November 2026.
| Format | Evidence | Sites with a request | Median requests per site per day | Rank |
|---|---|---|---|---|
| YAML | Established pattern | about 5% | 0 | Similar to others |
| RSS Feed | Established pattern | under 5% | 0 | Similar to others |
| Markdown | Established pattern | 0% | 0 | Similar to others |
| Schema JSON (JSON-LD) | Established pattern | 0% | 0 | Similar to others |
| OG.json | Established pattern | 0% | 0 | Similar to others |
| LLMs.txt | Established pattern | 0% | 0 | Similar to others |
| Humans.txt | Established pattern | 0% | 0 | Similar to others |
| Manifest.json | Established pattern | 0% | 0 | Similar to others |
Figures are rounded and cover only sites that served the file for the whole window. Requests are to Geordy-served files, identified by user agent; a request is not evidence of use or citation. Pairs with too few independent sites are not shown. See how we measure.
Check your site
See whether your robots.txt currently allows or blocks Claude-SearchBot, and what else AI crawlers find on your site.
Sources
Last reviewed: 19 October 2026. Every statement about what Claude-SearchBot does comes from the operator's own documentation: