AI Crawl Budget (ACB)
How much an AI engine is willing to fetch from your site within a window.
Distinct from search-engine crawl budget. AI crawlers prioritize fewer, denser files - llms.txt and schema.json get hit first, often, and quickly. Pages that take long to respond get cut. Sites with massive cardinality (large e-commerce catalogs, deep blog archives) need to be deliberate about which pages get the structured-data treatment first.
AI Crawl Budget is a zero-sum resource that directly determines the recency and completeness of AI knowledge about your domain: The Budget Reality: • Limited Capacity : AI systems crawl billions of pages-each domain gets a tiny fraction • Competitive Allocation : Your crawl budget competes with every other domain • Quality Signals : Sites that respond well get more budget; problematic sites get less • Diminishing Returns : More pages = less budget per page = stale content How Budget Gets Wasted: • Slow pages consume time that could fetch more content • Errors and redirects use requests without gaining knowledge • Duplicate content gets crawled multiple times for same information • Low-value pages consume budget that should go to important content • JavaScript-heavy pages may require multiple requests per page Budget Allocation Factors: • Domain authority and historical crawl success • Page update frequency and content velocity • Technical health: speed, errors, accessibility • Content quality signals from past crawls • Explicit directives (llms.txt, AI-specific sitemaps) Strategic Implications: • 10,000 page site with weekly crawl budget of 100 = most pages stale • High-velocity news site may need daily budget allocation • Product catalogs need efficient structure to maximize coverage • Content pruning can increase budget per remaining page
- Critical Content PrioritizationEnsuring AI crawlers spend their limited budget on pages most important for AI visibility.
- Crawl Efficiency OptimizationReducing barriers that waste crawl budget: slow pages, errors, redirects, duplicate content.
- Freshness ManagementSignaling which content needs frequent recrawling versus stable content that doesn't.
- llms.txt ImplementationProviding AI-specific sitemaps that direct crawlers to priority content.
- Crawl Pattern AnalysisUnderstanding how AI crawlers allocate their budget across your domain.
- Content ConsolidationReducing page count by consolidating thin content, preserving budget for substantive pages.
- Crawl Coverage RatePercentage of site pages crawled by AI systems within a given period.
- Budget Efficiency ScorePercentage of crawl budget spent on successful, valuable page retrievals.
- Average Page FreshnessMean time since last AI crawl across priority pages.
- Crawl Waste RatePercentage of crawl budget lost to errors, redirects, and low-value pages.
- Priority Page Crawl FrequencyHow often critical pages are recrawled by AI systems.
AI systems must balance the desire for comprehensive, current knowledge against the practical constraints of crawling the entire web.
- Crawl Budget Allocation Model
- Crawl Budget Optimization Strategy
- llms.txt for Budget Prioritization
Knowing the term is step one.
Geordy operationalizes every term in this glossary - generating the structured files AI engines actually read.