Crawl budget

Two things set it: how much the engine wants your pages, and how much your server can take. The second is the part most sites can affect and the part that almost never binds — a fast site behind a CDN with a few hundred pages has no crawl-rate problem, and the advice written for sites with millions does not transfer.

It is worth measuring for one narrow reason. A page that has never been fetched cannot be indexed, ranked or cited, so a discovered-and-uncrawled backlog is the one situation where crawl budget is the binding constraint rather than a topic. Server logs answer this and analytics cannot — an automated reader runs no JavaScript and appears in no session.

One day on this site, counted at the edge, 18 August 2026

OAI-SearchBot · OpenAI's index

33 fetches, 31 distinct pages

PerplexityBot

22 fetches, 20 pages

PetalBot · training

16 fetches, 16 pages

Bingbot

9 fetches, 6 pages

Googlebot

4 fetches, 1 page

All declared agents

99 fetches, 10 agents

On this date the answer-engine index took thirty-one distinct pages and the search index took one, four times. Neither figure is a verdict on the site. Both are budget decisions made elsewhere, by parties with different appetites, and the only way to know yours is to count them at your own edge.

Optimising a budget that is not the constraint

Crawl-budget work is sold to sites of a few hundred pages, where the engine is fetching everything it wants and the shortfall is demand rather than capacity. The test is one query against your own logs: if every published page was fetched at least once in the last month, budget is not what is holding the site back, and the money is better spent on the reason nobody is asking for the pages.

Index coverage

The share of the pages you published that a search engine has actually stored — and a gap between the two is ordinary rather than a fault.

---

Assistant fetch

One request for a page by a declared automated client of an AI system — the only unit of attention in answer engines that a site can count on its own server.

---

SEO

Making a site findable by search engines for the questions its buyers actually ask — a structural property of the site far more than a list of tasks performed on it.

---

GEO

Being citable by an assistant that is assembling an answer, rather than merely rankable by an engine that is returning links.

All seventy-eight termsWhat this is regulated by in the UAE

Is this one right in your own numbers?

The definitions are the easy part. Whether the figure on your dashboard was computed this way is a different question, and usually the more expensive one.