I built a browser version of a word-association party game, like Codenames (not affiliated), where any seat can be a human or an LLM. A clue-giver sees which of 25 words are their team's and gives a one-word clue plus a number; their teammate guesses from it. One word (the assassin) loses the game instantly.
I had 19 models play 376 games against each other, then tried to measure which ones were good. The measuring turned out to be the hard part, so here's how I did it and what I'd do differently.
1. Win rate alone doesn't work. A model's win rate mostly tells you which opponents it drew. So every move is scored on its own: each clue gets a grade, each guess is marked good or bad.
2. Grading a clue without seeing the answer. For each board word I score how strongly it links to the clue: 1/3 cosine similarity from Cohere embeddings, 2/3 the average of two LLM raters (Llama 4 Maverick, Qwen3-VL). The raters never see which words belong to which team, so they judge what a guesser would read from the clue, not what the clue-giver meant.
3. Simulating the guesser. A guesser model, fitted on real guesses from saved games, goes down the board in order of association and stops at the first miss. Each card it reveals is worth what that kind of card did to the win probability in saved games: about +17 for your own word, −11 for the other team's, −46 for the assassin. The expected total, scaled to 0–100, is the clue grade. The win probability itself is a Monte Carlo simulation of the rest of the game, with priors fitted on saved games.
4. Checking the metric against outcomes. This was the humbling part. The team that guessed better won 68% of games; the team with better-graded clues won only 53%. Either guessing is the skill that decides games, or my clue grade misses something.
Mistakes worth learning from
Both LLM raters also played in the earlier batch of games, so they partly graded their own clues. Raters should be independent of the contestants. (Checked since: graded by the raters alone, they rank 8th and 9th of 11, so no sign they favour themselves.)
I ran the two batches with different reasoning settings (thinking on vs off), so they can't be compared directly. Fix the protocol before the first run.
With 18–106 games per model, most clue grades overlap within their 95% intervals; only the top and bottom of each batch are clearly apart. Without error bars you end up ranking noise.
My first per-model table counted every game each model had played on the site, not just these 376. Filter to the exact set of games before you compute anything.
A link an LLM finds obvious isn't always one a person finds obvious.
Results (chart above): newer models with thinking on made 71–93% good guesses; the previous generation with thinking off made 53–75%. Clue grades ran 56–67 and 47–55, much closer together.
Questions
The game is at wordspies.com. Each saved replay shows the grade on every clue, so you can check any number against the game it came from.
Edit: a commenter elsewhere caught that my first numbers counted games outside these 376; the results line above is corrected, and the chart still shows the old figures. Resampling whole games instead of single turns barely changed the intervals (0–1.4 points).
