GeordyBot
GeordyBot, Geordy's crawler.
What it fetches, how often, and how to allow it or block it.
What GeordyBot is
GeordyBot reads web pages so Geordy can build AI-readable copies of them: a Markdown copy, schema.org JSON-LD, Open Graph JSON and YAML for each page, and llms.txt, humans.txt, a web app manifest and an RSS feed for each site. It fetches only sites a Geordy customer has added to their account, for the files that account asked for.
Its requests carry this user agent:
GeordyBot/1.0 (+https://geordy.ai/bot)What it requests
- Your robots.txt, once per site in each job, before anything else. See below for how it is applied.
- Each page the customer chose, with a plain GET that asks for HTML.
- Links on your own site that those files mention, at most 20 for one page's files, so that a link answering 404 or 410 is left out. Each is a HEAD request (3 seconds), then a GET (5 seconds) only when the HEAD gets no usable answer. It never checks links to other sites, and a link your robots.txt disallows is not checked at all: it stays in the file.
- For the site's own files: your homepage, your web app manifest and feed when the homepage links them, and one about or team page the homepage links to. When your site has no feed and none of the pages Geordy covers is an article, also up to 4 sitemap files (from robots.txt's Sitemap lines) and at most 15 article pages they list, or up to 2 news or blog pages your homepage links to.
- When a customer imports your sitemap: robots.txt for its Sitemap lines, and the sitemap files. The pages found there are recorded, not fetched.
- To confirm a site belongs to an account: your homepage or /.well-known/geordy-verification.txt, when the account asks, and later to keep that proof current, at most once a day.
Every request gives up after 15 seconds, follows at most 5 redirects, and stops reading after 5 MB. It never connects to a private network address.
How often
Each file is refreshed on the schedule the customer picks: daily, weekly or monthly. When they pick none, it is weekly. A customer can also switch a file's refresh off, or ask for a refresh at any time. A site's own files are refreshed with its homepage.
GeordyBot works on one page of a site at a time: Geordy never runs two jobs for the same site at once.
How it reads robots.txt
GeordyBot obeys robots.txt. When it builds files for your site, it reads your robots.txt once for that job, first, and checks every address against it: each page, your homepage, manifest and feed, the about, sitemap and news pages read for the site's own files, each link it checks, and the browser render. Two requests are not checked against it: confirming who owns a site, which the site's owner asks for, and a sitemap import, which reads only robots.txt and sitemap files.
- It follows the group for User-agent: GeordyBot when your file has one (any case, with or without a version), and otherwise the group for User-agent: *. Only one group applies: its own group replaces the * group, it does not add to it.
- Within that group the longest matching rule decides. When an Allow and a Disallow rule are equally long, Allow wins.
*matches any characters and a$at the end matches the end of the address. The path and the query are both matched. - A robots.txt that answers with a 4xx error (except 429), or that does not exist, allows everything.
- A robots.txt that answers 5xx or 429, or does not answer in time, stops the job: nothing on your site is fetched, and GeordyBot tries again at the page's next refresh.
A page your robots.txt disallows is not fetched and no files are made from it. If a page redirects to an address your robots.txt disallows, what came back is not read either. Geordy records the rule that stopped it, and the customer pays nothing for it.
How it behaves
It says who it is. GeordyBot sends the user agent above, and does not switch to another one when a page refuses it.
It does not get around bot protection. A page that answers with an error, such as 403 or 429, is recorded as unreadable and no files are made from it. A page that answers HTTP 200 with a bot challenge or an interstitial instead of its content is recorded as a refusal, and no files are made from it either. GeordyBot does not retry either one another way, and does not try to solve a challenge. A file that could not be made costs the customer nothing.
It never renders a page to get past a block. Geordy loads a page in a browser only when it answered GeordyBot with HTTP 200 and an empty frame that fills in through JavaScript, never when the page is a bot challenge, and never when your robots.txt disallows it. That browser runs in Cloudflare's Browser Rendering service and identifies itself as GeordyBot: it sends the user agent above. Cloudflare also adds headers of its own that mark the request as coming from its Browser Rendering service.
How to allow it
Geordy does not publish IP addresses for GeordyBot, so rules match its user agent: requests whose user agent contains GeordyBot. Anyone can send that word, so a rule that lets GeordyBot through lets through anyone who copies it.
Cloudflare
Create a WAF custom rule with this expression:
(http.user_agent contains "GeordyBot")Set its action to Skip, choose the protection that blocks GeordyBot (the remaining custom rules, rate limiting rules, managed rules or Super Bot Fight Mode), and place it first. Cloudflare's security events show which one blocked a request.
WordPress security plugins
Security plugins often block crawlers they do not know. Find GeordyBot in the plugin's blocked requests or live traffic log, then add GeordyBot to its allowed user agents, or allow the request from that log.
Any other firewall or CDN
Allow requests whose user agent contains GeordyBot.
robots.txt
No robots.txt line is needed unless your robots.txt disallows the pages for every crawler. If its User-agent: * group disallows them, give GeordyBot a group of its own:
User-agent: GeordyBot
Allow: /How to block it
robots.txt
A Disallow rule for GeordyBot stops it. To keep it off the whole site:
User-agent: GeordyBot
Disallow: /Or only part of it:
User-agent: GeordyBot
Disallow: /private/GeordyBot reads the new rules at the start of its next job on your site. You can also refuse its user agent at your firewall or server.
Cloudflare
A WAF custom rule with the same expression and the action Block:
(http.user_agent contains "GeordyBot")nginx
if ($http_user_agent ~* "GeordyBot") {
return 403;
}Apache (.htaccess)
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} GeordyBot [NC]
RewriteRule .* - [F,L]A blocked page is recorded as unreadable and no files are made from it. To keep a single page out without blocking anything: a page whose robots meta tag or X-Robots-Tag header says noindex is still fetched, but Geordy publishes no files for it.
Contact
Questions about GeordyBot, or a problem with it on your site: use the contact form, or email [email protected].