Training crawler
GPTBot: what it is.
GPTBot is OpenAI's crawler for content that may be used to train its generative AI foundation models.
Identity
- Operator: OpenAI
- User-agent token:
GPTBot - robots.txt token:
GPTBot - In Geordy's crawler directory: Collects pages for OpenAI model training. See the full list.
What it does, per OpenAI
OpenAI says GPTBot is used to make its generative AI foundation models more useful and safe.
It crawls content that may be used to train those models.
OpenAI says that disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models.
robots.txt
Allow GPTBot
User-agent: GPTBot
Allow: /Block GPTBot
User-agent: GPTBot
Disallow: /OpenAI names GPTBot as the robots.txt token. Its page does not print example lines, so the ones below are standard robots.txt syntax around that token.
OpenAI notes that the user-agent string carries a version (for example GPTBot/1.4) that may change, so match on the token, not the full string.
Blocking GPTBot does not block OAI-SearchBot or ChatGPT-User. Each has its own token and its own purpose.
Allowing one and blocking the other is covered in GPTBot vs OAI-SearchBot.
What Geordy observes
Share of eligible sites where GPTBot requested each Geordy-served file at least once, over the 90 days to 12 November 2026.
| Format | Evidence | Sites with a request | Median requests per site per day | Rank |
|---|---|---|---|---|
| Markdown | Established pattern | under 5% | 0 | Similar to others |
| Schema JSON (JSON-LD) | Established pattern | under 5% | 0 | Similar to others |
| OG.json | Established pattern | under 5% | 0 | Similar to others |
| YAML | Established pattern | about 5% | 0 | Similar to others |
| RSS Feed | Established pattern | under 5% | 0 | Similar to others |
| LLMs.txt | Established pattern | 0% | 0 | Similar to others |
| Humans.txt | Established pattern | 0% | 0 | Similar to others |
| Manifest.json | Established pattern | 0% | 0 | Similar to others |
Figures are rounded and cover only sites that served the file for the whole window. Requests are to Geordy-served files, identified by user agent; a request is not evidence of use or citation. Pairs with too few independent sites are not shown. See how we measure.
Check your site
See whether your robots.txt currently allows or blocks GPTBot, and what else AI crawlers find on your site.
Sources
Last reviewed: 19 October 2026. Every statement about what GPTBot does comes from the operator's own documentation: