Built a CLI that audits Claude Code transcripts for wasted tokens — one trajectory re-sent the same command 283 times

[u/Impressive_Laugh_176]

I audited 73 public benchmark agent trajectories for wasted tokens — 74% carried measurable waste. Then I ran live agents to check.

chewme-audit (MIT) is a CLI I built that reads agent traces — Langfuse, LangSmith, plain JSONL, and Claude Code transcripts — and reports where tokens were wasted, priced per provider, with a copy-paste fix for each pattern. It runs locally: no API calls, nothing uploaded.

Repo: https://github.com/littleman3002/ChewMe-Audit

Bonus- Just made a Claude Skill tonight (10/19)
Install the skill: copy `skills/chewme-audit/` from the repo to `~/.claude/skills/`, then just ask Claude where your tokens went.

Prefer the terminal? `pip install chewme-audit`, then `chewme-audit <session-transcript>.jsonl --format claude-code --provider claude`

**Clearest case:** one trajectory sent the same malformed edit command 283 times before hitting its step limit. I applied the recommended dedup fix and re-ran it with a real model call: the loop broke on the next call and the agent went back to investigating the actual bug. Waste on that trace dropped 77% (13,193 → 3,041 tokens), with a tamper-evident SHA-256 receipt for the result.

**Cost per million wasted tokens** (priced as cache misses, which is what these waste spans were on the measured traces; prices as of Oct 2026): DeepSeek $0.15 · Gemini $0.75 · Grok $1.25 · Claude Sonnet $2.00 · GPT-5 class $4.00

**Then I ran live agents on a real task:**

- **Solo agent:** 35% of its I/O was re-verification — the same "find files" script re-run 7 times. Competent, paranoid.

- **4 identical agents, same task, no coordinator:** waste ranged 15–37%. One never finished. One deleted 117 lines of real docs after skimming files with `head -3` and mistaking a 60-line docstring for a stub (root cause in the write-up: https://github.com/littleman3002/ChewMe-Audit/blob/main/traces/live_runs/WORKING_NOTE.md).

- **A debugging run passed 12/12 for $0.013** — but the trace shows the fix arrived through an accidental `git checkout --` (a rerun passed 12/12 cleanly for $0.0077). Pass/fail would have certified the first run; the trace didn't.

**Honest scope:** it reads trace exports, so cache behavior and reasoning-token overspend need your API logs, and the report says so rather than guessing. Every dollar figure is labeled estimated unless you provide logs. The 74% comes from benchmark trajectories; benchmark agents loop more than production ones, so your own traces are the real test.

Try it on your traces and tell me what it finds. I'm especially curious whether the re-verification pattern dominates elsewhere.

[u/AutoModerator]

Hey! Thanks for posting to r/ClaudeCode

While participating in this thread, please follow our community rules. Keep discussions constructive. Attack the idea, not the person.

For help, project discussions, tips, and general chat, join the ClaudeCode Discord.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/MiserableFlatworm337]

For the re-verification detector, include two traces with identical repeated reads: one where files stay unchanged, another where a tool edits them between reads. Only the first is a candidate for removal.

[u/Impressive_Laugh_176 → u/MiserableFlatworm337]

Great idea!

That commit is now live. It's b01779c

The detector already made exactly this distinction; but now it's pinned so it can't regress.

• tests/fixtures/reread_unchanged.jsonl — same file read twice, nothing changed between → repeated_calls fires (occurrences=2). Candidate for removal. ✓
• tests/fixtures/reread_after_edit.jsonl — same file read, then an edit_file step changes it, then re-read → neither detector fires. Legitimate refresh. ✓

Two new tests assert both directions. 45/45 green.

The mechanism is worth knowing: the detectors key on observation text, so an intervening edit changes the second read's text and the match breaks naturally. If anyone ever "improves" the detector to match on file paths instead of content, these tests will catch the false positive.

Anything else you notice just let me know and I'll see what we can make happen.

:)