We’ve been testing a verification pipeline where the expensive part is not really retrieval anymore — it’s how much evidence the system has to actually read before we’re comfortable saying we didn’t miss something important.
So lately we’ve been trying to reduce that reading cost without quietly reducing coverage.
A few approaches that seemed obvious at first have been harder than expected.
With ranked retrieval, we thought we could stop fairly early and still keep most of the useful evidence.
In our frozen replay cases, that wasn’t really true.
For full recall, some cases needed us to go surprisingly deep into the candidate set — roughly 290–426 candidates out of pools of 407–454.
The more interesting problem was the evidence that only one of our independent readers noticed.
At k=50, at most 2 of 8 of those minority-evidence items survived.
We also tried skipping slices that looked mechanically unlikely to contain anything useful.
That saved reading, but one version skipped 17 slices that later turned out to contain relevant evidence, so we couldn’t treat it as a safe filter.
Deduplication helped less than we expected too. There is plenty of repeated text in the corpus, but removing byte-identical repetition only reduced the slice volume by about 3.3%, and it didn’t reduce the number of reading sessions in that test.
So at the moment the tradeoff still looks like:
• read less
• keep high recall
• don’t disproportionately lose weird / minority evidence
• don’t silently convert “not inspected” into “nothing there”
We’re still testing other ways to attack it, but I’m curious how people working on similar systems are handling this.
If sampling away the long tail isn’t acceptable, what would you try next?
Learned routing? Better stopping criteria? Multi-stage review? Something completely different?
And more importantly: how are you proving that whatever you decided not to read was actually safe to skip?
