How to Tell Whether Your AI Call Summaries Are Any Good
AI-generated call summaries are convincing. They are fluent, well-structured, appropriately concise, and they read as authoritative whether or not they reflect what was said.
That is the risk. A summary that is 90% accurate and 100% confident gets acted upon as though it were 100% accurate, and the errors are invisible unless someone compares against the source.
Here is how to find out what you actually have, without needing data science.
The failure modes to look for
Summaries fail in specific ways, and knowing the categories makes evaluation much faster:
Omission. The commitment made at the end of the call does not appear. The most common and most consequential — a summary that misses "I'll waive the fee this once" leaves a customer expectation nobody recorded.
Attribution error. Who said what gets swapped. "The customer agreed to the new terms" when the agent proposed them and the customer did not respond.
Confident invention. Details that were never said. Names, dates, amounts that sound plausible. Rare, and serious when it happens.
Hedging lost. "I'll check whether we can do that" becomes "will arrange". Conditional commitments rendered as firm ones is a frequent and expensive pattern.
Wrong emphasis. Technically accurate, but leads with the minor point and buries the thing that mattered.
Negation flips. "Not able to" summarised as "able to". Uncommon but catastrophic when it occurs.
The evaluation
Sample properly
Pull 50 calls, stratified rather than random:
- 15 routine short calls
- 15 longer or more complex calls
- 10 calls with a complaint or escalation
- 5 with poor audio quality
- 5 with accents, background noise, or multiple speakers
Random sampling over-represents the easy cases and tells you the system is excellent. Stratified sampling tells you where it breaks, which is what you need.
Score against a rubric, not an impression
For each call, a reviewer who has listened to the audio marks:
| Dimension | Question |
|---|---|
| Factual accuracy | Does it state anything that was not said? (yes/no) |
| Commitment capture | Was every commitment or next step captured? (all / some / none / n-a) |
| Attribution | Is who-said-what correct? (yes/no) |
| Hedging preserved | Are conditional statements still conditional? (yes/no/n-a) |
| Usable | Could a colleague act on this without the recording? (yes/no) |
Binary where possible. Five-point scales invite the middle and produce averages that hide the failures.
Read the results the right way
The headline is not an average. It is:
- Invention rate. Any call containing a stated fact that was not said. This should be very close to zero. If it is not, the system is not usable for anything consequential
- Commitment capture rate. The number that determines operational value
- Usability rate. The practical bar: would a colleague be able to act on this alone
A system scoring 95% on usability and 85% on commitment capture is useful with a caveat: do not rely on summaries alone where commitments matter.
What to do with what you find
If invention rate is above zero: do not use summaries as a record of fact. They can assist a human who has the recording; they cannot replace it. Document that distinction in your process.
If commitment capture is weak: change the prompt or the configuration to specifically extract commitments and next steps as a separate field. Most platforms allow this, and a dedicated extraction performs much better than hoping the narrative summary includes it.
If quality drops on poor audio: fix the audio. Headset quality, network jitter and packet loss degrade transcription, and everything downstream inherits that. This is frequently the real problem and it is a network and hardware fix rather than an AI one.
If a specific call type fails: scope the feature. Summaries on routine calls, human notes on escalations. Partial deployment that works beats universal deployment that cannot be trusted.
Keep measuring
Models change, often without announcement. A system evaluated once is a system you knew about once.
- Re-run the evaluation quarterly, and after any provider notification of a model update
- Add a one-click "this summary is wrong" control for staff, and review what comes in weekly. This is the cheapest ongoing signal you will get
- Watch for drift in a specific category rather than overall quality
The governance point
If summaries feed anything consequential — a CRM record, a billing decision, a clinical note, a dispute file — then the accuracy rate is not a curiosity. It is a known error rate in a business process, and someone should have decided it is acceptable.
That decision needs a number behind it. Running the evaluation above gives you one, and it takes an afternoon. Deploying without it means the error rate exists regardless; you simply do not know what it is.
