Systems that answer
The share of conversations the system finished on its own — and on its own is doing more work in that sentence than it looks.
Also called Containment rate · Доля отвеченного
Formula
Answer rate = Conversations answered from source ÷ Conversations
- Answered from source
- the reply was assembled from retrieved material, and which material is recorded
- Conversations
- the denominator defined in the conversation entry, with its window and its exclusions stated
The qualifier matters more than the ratio. A system that produces a fluent paragraph for every question has an answer rate of one hundred per cent and no relationship to your catalogue, which is the failure this metric is most often used to hide.
Read it against the refusal rate or not at all. The pair describes a system: high answer, sane refusal, low escalation is a knowledge base doing its job. High answer with zero refusal is a model doing its job, which is a different and worse thing.
It also moves for a reason that has nothing to do with quality. A campaign that brings unfamiliar traffic asks unfamiliar questions, so the rate falls in the month the marketing worked. Segment by channel before concluding anything about the system.
- A · conversations
- 1,000
- A · answered, traceable to a source passage
- 620
- A · refused and handed over
- 380
- B · conversations
- 1,000
- B · answered, any fluent reply counted
- 1,000
- B · refused
- 0
A reports 62%, B reports 100%, and B is the one to worry about. A refusal rate of zero on a real catalogue means the system has never met a question it could not answer, which no catalogue permits — so it is answering questions it has no material for. The figures are illustrative; the arithmetic is the point.
It is the easiest metric on this page to improve and the improvement is almost always a loss. Loosening retrieval thresholds, widening what counts as a match, or allowing the model to answer from general knowledge all raise it — and each one converts a refusal that would have gone to a person into an answer that may be wrong. Set a floor for the refusal rate first, then optimise inside it.
Quantities derived from the counts. Each is only as sound as the definitions beneath it.
Cannot be computed without
- Conversation
- One exchange between a person and the system, from their first message to the point it is over — and where that point is decides every rate computed below it.
Knowing the definition is not the same as being able to check the figure. These are the procedures that do the second thing.
- Accepting an automated agent
- “The agent is trained on your data and ready to go live.” · 45 minutes, 6 questions.
The definitions are the easy part. Whether the figure on your dashboard was computed this way is a different question, and usually the more expensive one.