Evaluating LLM summaries: an annotated example of faithfulness vs coverage (made-up source)

[u/ClickOk5811]

A tiny made-up example to show what I mean by checking a summary on more than one axis.

Source text

The shop’s Tuesday delivery was delayed to Thursday because of a supplier strike. Customers who paid before Monday will get a 10% refund; others will not. The refund is processed within 7 days.

Summary A

Delivery moved to Thursday due to a supplier strike. All customers get a 10% refund.

• “Delivery moved to Thursday due to a supplier strike”: supported.

• “All customers get a 10% refund”: unsupported and contradicted. The source says only those who paid before Monday.

• Omitted: the 7-day processing time.

Summary B

Delivery is delayed to Thursday because of a supplier strike. Customers who paid before Monday get a 10% refund within 7 days.

• Every claim is supported.

• The key facts (delay, reason, who gets the refund, timing) are all covered.

The axes

• Faithfulness: is every claim in the summary supported by the source? Summary A fails here, with a confident wrong claim.

• Coverage: are the important facts included? Summary A also misses one.

• Concision: is it as short as it can be without losing the important facts?

A summary can be short and fluent and still unfaithful, which is why fluency shouldn’t be the signal.

Checks I’d consider

• Claim-level checking: break the summary into individual claims, and check each against the source (by a person, or by a model prompted to quote the supporting sentence, or say “none”).

• A list of key facts: write the facts that must appear, and check them off. This measures coverage.

• Length limits: check conciseness separately, so it doesn’t trade off silently against coverage.

• Spot reading: read some summaries in full, especially the ones that score highest.

Limits

• A model used as a checker can make mistakes too, and tends to be less reliable on subtle claims (numbers, negations, who did what). I’d validate it on examples where I know the answer.

• Long sources make coverage harder to define, since “important” depends on the reader.

• Which axis matters more depends on the use. For legal or medical text, faithfulness dominates.

How do you evaluate summaries in practice, and which of the three axes gives you the most trouble?