Systems that answer
Refusal rate
How often the system says it does not know — the one metric on a deployment where zero is the alarming reading.
Also called Deflection to human · Доля отказов
Formula
Refusal rate = Conversations refused ÷ Conversations
- Refused
- retrieval returned nothing usable and the system said so, rather than producing a reply anyway
- Conversations
- same denominator as everywhere else, same stated window
There is no correct value, and anybody quoting one is selling something. What exists is a floor: a system with a refusal rate of zero over a real catalogue and real questions has not been refusing, it has been guessing, and the guesses are indistinguishable from answers until a customer acts on one.
The reason this metric is unpopular is that it looks like failure to everyone who has not thought about it. A refusal is a conversation the automation did not finish, and the natural instinct of anyone reporting on a deployment is to make that number small.
The instinct is exactly wrong. A refusal is the system correctly identifying the boundary of what it can support, and every refusal that reaches a person with the history attached is a conversation saved. The alternative is a confident wrong answer, which is not a smaller version of the same problem — it is a commitment made in your name to a customer who will hold you to it.
Watching it over time is where it earns its place. A refusal rate rising in one topic is a gap in the knowledge base with an address: those questions, that material, missing. It is the most actionable signal a deployment produces and the one most often switched off for looking bad.
Deployments get tuned until refusals disappear, and the tuning is reported as an improvement. What actually happened is that the boundary moved outward past the knowledge base, and every question that used to reach a person now receives an answer nobody can trace. The failure is now silent, which is worse than the failure it replaced.
Quantities derived from the counts. Each is only as sound as the definitions beneath it.
Cannot be computed without
- Conversation
- One exchange between a person and the system, from their first message to the point it is over — and where that point is decides every rate computed below it.
The term is a reading taken at a point in a sequence that is already running. These are the sequences, with the place each of them jams.
- From a customer’s question to an answer the business can stand behind
- One question arriving at an automated agent, and everything that has to hold for the reply to be something the company would have said itself. 7 steps, buying the model and treating the material as content work is the arrangement that fails while looking reasonable.
Knowing the definition is not the same as being able to check the figure. These are the procedures that do the second thing.
- Accepting an automated agent
- “The agent is trained on your data and ready to go live.” · 45 minutes, 6 questions.
A definition is free. Being answerable for the figure it produces is the part that is bought, and this term is a working part of the engagements below.
- The reply that arrives tonight
- An agent that will not say "I do not know" is not a sales asset. It is a liability with a fast response time.
The definitions are the easy part. Whether the figure on your dashboard was computed this way is a different question, and usually the more expensive one.