Crawl budget

Two things set it: how much the engine wants your pages, and how much your server can take. The second is the part most sites can affect and the part that almost never binds — a fast site behind a CDN with a few hundred pages has no crawl-rate problem, and the advice written for sites with millions does not transfer.

It is worth measuring for one narrow reason. A page that has never been fetched cannot be indexed, ranked or cited, so a discovered-and-uncrawled backlog is the one situation where crawl budget is the binding constraint rather than a topic. Server logs answer this and analytics cannot — an automated reader runs no JavaScript and appears in no session.

One day on this site, counted at the edge, 18 August 2026

OAI-SearchBot · OpenAI's index

33 fetches, 31 distinct pages

PerplexityBot

22 fetches, 20 pages

PetalBot · training

16 fetches, 16 pages

Bingbot

9 fetches, 6 pages

Googlebot

4 fetches, 1 page

All declared agents

99 fetches, 10 agents

On this date the answer-engine index took thirty-one distinct pages and the search index took one, four times. Neither figure is a verdict on the site. Both are budget decisions made elsewhere, by parties with different appetites, and the only way to know yours is to count them at your own edge.

Optimising a budget that is not the constraint

Crawl-budget work is sold to sites of a few hundred pages, where the engine is fetching everything it wants and the shortfall is demand rather than capacity. The test is one query against your own logs: if every published page was fetched at least once in the last month, budget is not what is holding the site back, and the money is better spent on the reason nobody is asking for the pages.

A definition sits still. This does not.

The term is a reading taken at a point in a sequence that is already running. These are the sequences, with the place each of them jams.

From publishing a page to being cited in an answer

A page you wrote, travelling from your own server to a sentence in somebody else's search result or assistant answer. 6 steps, publishing volume against a crawl budget that has not moved is the arrangement that fails while looking reasonable.

Somebody is accountable for this number.

A definition is free. Being answerable for the figure it produces is the part that is bought, and this term is a working part of the engagements below.

Web design and development for Dubai, built search-first

Search architecture is a build decision, not a later engagement.

---

SEO and GEO for Dubai and the UAE

You are indexed as an entity, not as a set of pages.

Index coverage

The share of the pages you published that a search engine has actually stored — and a gap between the two is ordinary rather than a fault.

---

Assistant fetch

One request for a page by a declared automated client of an AI system — the only unit of attention in answer engines that a site can count on its own server.

---

SEO

Making a site findable by search engines for the questions its buyers actually ask — a structural property of the site far more than a list of tasks performed on it.

---

GEO

Being citable by an assistant that is assembling an answer, rather than merely rankable by an engine that is returning links.

---

Internal link graph

Which of your pages link to which others — the structure that decides what a reader and a crawler can reach from the page they are actually on, as opposed to what you have published.

All ninety-six termsWhat this is regulated by in the UAE

Is this one right in your own numbers?

The definitions are the easy part. Whether the figure on your dashboard was computed this way is a different question, and usually the more expensive one.