AI bot analytics · live
AI bots are reading your site. You can't see them.
Geordy serves your AI files, so every request to them is counted - including the traffic your analytics has never shown you.
Why your dashboard says zero when reality says thousands.
Standard analytics measures pages with a JavaScript tag. Your AI-readable files - , .md, .json, .yaml - are plain files, so there is nothing for a tag to run in.
The result - requests to those files are never counted. Your dashboard reads 0, so you assume nothing is reading them.
Logs every request to every file Geordy serves.
Named engine, exact file, exact time.
Counted where the files are served. Nothing to install.
The screen you've been missing.
Every AI request to every one of your files - by engine, by format, over time. The whole consumption layer, on one screen.
AI Bot Crawls
Track how AI bots access your structured content
| Bot | Total | YAML | Markdown | LLMs.txt | Schema | RSS | Manifest | Humans.txt | OG |
|---|---|---|---|---|---|---|---|---|---|
| 5,472 | 675 | 493 | 840 | 647 | 587 | 773 | 869 | 588 | |
| 2,297 | 419 | 373 | 481 | 544 | 87 | 130 | 188 | 75 | |
| 2,274 | 177 | 197 | 429 | 150 | 480 | 535 | 194 | 112 | |
| 2,235 | 376 | 294 | 124 | 308 | 295 | 196 | 444 | 198 |
The read layer, finally on the record.
AI engines pull your content constantly and silently - then decide what to cite from what they found. Geordy writes down every visit, so the layer that shapes your AI visibility stops being a black box you have to guess at.
See who's reading - and what they take.
Every hit carries a fingerprint: which engine, which file, which status, how recently. Here is what that visibility makes possible.
Know exactly which AI is on your site.
Every bot that touches a Geordy-served file is matched against its - so GPTBot, ClaudeBot, PerplexityBot, GoogleOther and dozens more are named, not lumped into “other.” New entrants surface the moment they first crawl.
- Matched by user-agent, not guessed from IP
- 27 AI crawlers known out of the box, 39 names in all
- New crawlers appear on their first visit
Which crawlers, by name.
39 crawlers and robots.txt tokens, 27 of them AI. Every name here is either checked against your robots.txt by Geordy's probe, resolved to its operator in your crawl log, or both. The count above is this table's length, not a round number somebody liked.
Reviewed
AI assistants and answer engines
20 AICrawlers run by an AI product. They fetch pages to train on, to index for that product's search, or because a user asked it to open a link.
GPTBotOpenAICollects pages for OpenAI model training.robots.txtnamed in logOAI-SearchBotOpenAIIndexes pages so ChatGPT search can surface them.robots.txtnamed in logChatGPT-UserOpenAIFetches one page because a ChatGPT user followed a link.robots.txtnamed in logClaudeBotAnthropicCollects pages for Anthropic model training.robots.txtnamed in logClaude-SearchBotAnthropicIndexes pages so Claude search can surface them.robots.txtnamed in logClaude-UserAnthropicFetches one page because Claude followed a link for a user.robots.txtnamed in logClaude-WebAnthropicRetired. Still listed in many robots.txt files and still seen in older logs.robots.txtnamed in loganthropic-aiAnthropicOlder Anthropic token. Still the one many robots.txt files name.robots.txtnamed in logPerplexityBotPerplexityIndexes pages so Perplexity can cite them.robots.txtnamed in logPerplexity-UserPerplexityFetches one page because a Perplexity user opened a source.robots.txtnamed in logGoogleOtherGoogleGoogle's fetcher for uses outside Search, which is where its AI products pull from.robots.txtnamed in logApplebotAppleApple's crawler, behind Siri and Spotlight suggestions.robots.txtnamed in log
AmazonbotAmazonAmazon's crawler, behind Alexa answers.robots.txtnamed in logMeta-ExternalAgentMetaCollects pages for Meta's AI products and model training.robots.txtnamed in logMeta-ExternalFetcherMetaFetches one page because a Meta AI product followed a link.robots.txtnamed in logBytespiderByteDanceByteDance's crawler, collecting pages for its AI work.robots.txtnamed in logMistralAI-UserMistral AIFetches one page because a Mistral assistant followed a link.robots.txtnamed in logcohere-aiCohereCohere's crawler for its own models.robots.txtnamed in logDuckAssistBotDuckDuckGoFeeds DuckDuckGo's AI-assisted answers.robots.txtnamed in logYouBotYou.comIndexes pages so You.com can cite them.robots.txtnamed in logTraining and dataset crawlers
7 AICrawlers that collect web data for models in general rather than for one assistant. Common Crawl alone is in the training mix of most large models.
CCBotCommon CrawlBuilds the open Common Crawl archive that most large models are trained on.robots.txtnamed in logcohere-training-data-crawlerCohereCollects training data for Cohere's models.robots.txtnamed in logOmgilibotWebz.ioCollects web data that is sold on as training sets.robots.txtnamed in logDiffbotDiffbotBuilds a commercial knowledge graph that AI products query.robots.txtnamed in logTimpibotTimpiBuilds a distributed web index aimed at AI search.robots.txtnamed in logImagesiftBotImageSiftCollects images and their page context for datasets.robots.txtnamed in logFirecrawlAgentFirecrawlFetches pages on demand for AI agents and retrieval pipelines.robots.txtnamed in logSearch crawlers behind an AI answer surface
8Not AI crawlers. They build a search index, and an AI surface answers from that index, so blocking one blocks the other.
GooglebotGoogleGoogle's search crawler. AI Overviews answer out of this index, so blocking it blocks them.robots.txtnamed in logBingbotMicrosoftBing's search crawler. Copilot answers out of this index.robots.txtnamed in logBingPreviewMicrosoftRenders Bing's page previews.robots.txtnamed in logBaiduspiderBaiduBaidu's search crawler.robots.txtnamed in logBaiduspider-renderBaiduBaidu's fetcher for pages that need JavaScript run first.robots.txtnamed in logYandexBotYandexYandex's search crawler.robots.txtnamed in logYandexRenderResourcesBotYandexYandex's fetcher for pages that need JavaScript run first.robots.txtnamed in logPetalBotHuaweiPetal Search's crawler.robots.txtnamed in logrobots.txt tokens that never fetch
2These send no requests and can never appear in a log. They exist so you can allow or refuse a use of what another crawler already fetched.
Google-ExtendedGoogleA permission token, not a fetcher. It says whether Google may use what Googlebot already fetched for Gemini and Vertex AI grounding. Nothing requests a page under this name.robots.txtnamed in logApplebot-ExtendedAppleA permission token, not a fetcher. It says whether Apple may use what Applebot already fetched to train its models.robots.txtnamed in logSocial and preview fetchers
2Not AI crawlers. Named here because they show up in the same log and get mistaken for one.
FacebookBotMetaMeta's link crawler, separate from its AI agents above.robots.txtnamed in logLinkedInBotLinkedInBuilds the preview card when your page is shared on LinkedIn.robots.txtnamed in logrobots.txt means Geordy checks whether your robots.txt blocks that agent. Named in log means a visit from it arrives in your Crawls view under its operator's name and mark rather than as an unknown string. A crawler nobody has published yet still shows up the first time it visits, under whatever name it sends, which is how names get added to this list.
See which files each engine reads.
ClaudeBot may reach for markdown while GPTBot pulls schema.json and Perplexity leans on llms.txt. Per-engine, per-format counts show where each system looks first - so you know which formats matter for which surface.
- Per-engine × per-format breakdown
- Spot the formats an engine ignores
- Prioritise what actually gets read
Find the reads that never made it.
Filter by status code to surface 4xx and 5xx - the crawls that failed. A soft-404, a bad redirect, a timeout: each is a page an engine came for and could not read. Fix it and the page is back in play.
- Isolate 4xx / 5xx AI crawls in one filter
- See the exact URL and engine that failed
- Every broken read is a page an engine could not reach
Know when each engine last read you.
Publish an update and you're left guessing when AI will notice. Geordy timestamps every crawl - so you can see which engines came back this morning, which have gone quiet for days, and whether your latest content has been picked up yet.
- Last-seen timestamp per engine
- Spot engines that have gone quiet
- Confirm fresh content actually got re-read
Slice it your way.
Resolution by day, week, or month, across any date range. Zoom from a single overnight spike out to a whole season of trends - without leaving the view.
Take it anywhere.
Export the raw numbers as CSV - every bot, every format, every page - and work with them in your own stack.
Stop guessing what AI sees. Watch it happen.
Connect a property in two steps. From then on, every AI visit to your files is counted.