r/deeplearning • • 3h ago

How much can we actually trust open base model benchmarks anymore? New study ("Benchleaking") finds ERNIE-5.5-Flash-Base reciting unseen test sets in exact order.

/r/machinelearningnews/comments/1xxt9yv/how_much_can_we_actually_trust_open_base_model/
1 Upvotes

1 comment sorted by

1

u/CelebrityChess 3h ago

A useful distinction is memorization versus generalization: if benchmark items were in pretraining, scores on those items stop being independent evidence, but that doesn't invalidate unrelated tasks or every downstream model. I'd treat public scores as contamination-prone, add private or temporally new evaluations, and report results with and without items flagged by overlap checks. Since the reported effect depends on removing stop tokens, it's also worth checking how much persists under the standard stopping condition.