Accepting an automated agent

1. 01

   What is the refusal rate, and what happens at zero?

   A system that has never refused over a real catalogue has not been refusing, it has been guessing. Zero is the alarming reading here, not the good one.

   If insteadZero refusals presented as an achievement, or the refusal rate is not reported at all.

2. ---

   02

   Can retrieval return nothing?

   If it always hands back its best three passages regardless of match quality, there is no state in which the system knows it has nothing, so the refusal path can never fire. The boundary was not built.

   If insteadThe answer is about prompt instructions telling the model to say it does not know. That is a request, not a control.

3. ---

   03

   Show me ten answers with the passages they came from.

   Read the passage and check the claim is actually in it. Citations rendered under an answer look like grounding and are frequently the retrieved passages with no verification that the claim appears in them.

   If insteadRetrievals were not stored, or the cited passage does not contain the claim.

4. ---

   04

   Can it emit a price that is not in the price list?

   The test for where the boundary sits. A guardrail inspecting the finished reply will pass a plausible wrong number; a rule layer that owns the values cannot produce one.

   If insteadNobody has tried. Try it in front of them, in three phrasings, including one asking for a discount.

5. ---

   05

   Escalate a conversation and watch what the person receives.

   The rate is reported and the restart is not. An escalation arriving as a fresh window makes the customer repeat themselves at the moment the sale is most fragile.

   If insteadThe handover is a notification with a link, or the person has to open a second system to see the history.

6. ---

   06

   Ask it ten questions of your own that nobody has seen.

   The ones your staff dread on the phone. A system that handles the supplier's adversarial set and invents on yours was tested against the wrong thing.

   If insteadResistance to unrehearsed questions, or a request to add them to the corpus first — which is the fix, but only after you have seen what it did without them.

It is real

Retrievals are stored, the refusal path fires on demand, prices come from a rule layer that can block a reply, escalation carries the thread, and the adversarial set is a file you now hold.

---

It is not

Retrieval cannot return empty, or numbers come from the model. Both are architectural and neither is fixed by tuning; going live in that state means confident wrong commitments, in writing, at scale, in the company's name.

---

Cannot tell

Everything checks out on the supplier's material and your own ten questions produced two inventions. That is a knowledge-base gap rather than a design fault — worth naming, worth fixing before launch, and not a reason to reject the build.

This check, pointed at us.

Our own products

BiLex, our bilingual contract engine, is a probabilistic component inside a deterministic boundary — terminology locked, alignment checked structurally — and we publish no refusal rate for it. CyberEGE stops a session below thirty per cent accuracy, which is a designed refusal, and we publish no figure for how often it fires. We hold suppliers to reporting what we do not report ourselves.

SinceNot yet. The vocabulary now exists to state it, which is a precondition rather than a fix. The honest position is that these are our own products rather than client deployments, and that the reporting we require of a supplier is the reporting we owe on anything we run for somebody else.

Knowledge base

The material the system answers from. It is the product; the model is a commodity you rent by the token.

---

Grounding

Every sentence the system produces is traceable to a passage it retrieved — which means an answer can be checked without asking the model to explain itself.

---

Refusal path

The route the system takes when it has nothing — designed, tested, and the single most valuable thing in a deployment.

---

Hallucination

A fluent, confident claim with nothing behind it — and the reason it is dangerous is that it is indistinguishable from a correct answer at the moment it is read.

---

Deterministic layer

The code that holds a veto over what the model produces — rules that always run and always give the same answer to the same input.

---

Adversarial test set

The questions written specifically to make the system invent an answer — and the only acceptance evidence that means anything.

---

Answer rate

The share of conversations the system finished on its own — and on its own is doing more work in that sentence than it looks.

---

Refusal rate

How often the system says it does not know — the one metric on a deployment where zero is the alarming reading.

---

Escalation rate

How often a person takes over — and whether they arrive holding the conversation or starting one.

Every checkThe whole glossary

If the answers did not arrive, that is the finding.

A supplier who cannot supply a base and a period has told you something, and it is not that they are disorganised. We do this for a living and the questions land differently when they come from outside.