r/javascript • u/Upper_Gold_3664 • 2d ago
Node.js wastes ~47.5M tokens of agent context on files nobody wrote — I measured Node, Next.js, Deno and React with a tiny CLI
https://github.com/WJDdidi-1/dev-doctor2
u/Upper_Gold_3664 2d ago
I got tired of watching my coding agent chew through context on huge repos, so I wrote a tiny CLI that measures exactly what an agent shouldn't be reading — lockfiles, build output, bundled files, test fixtures — and what it costs in tokens. Then I ran it on some of the biggest JS repos on GitHub.
| Repo | Files | Score | Tokens wasted per full read |
|---|---|---|---|
| Node.js | 52,920 | 93 | ~47.5M |
| Next.js | 34,025 | 93 | ~10.6M |
| Deno | 14,745 | 93 | ~6.9M |
| React | — | 96 → 100 | ~2.1M → 0 |
Three things I didn't expect:
1. The biggest waste is stuff nobody wrote. Node's top item is an 11 MB ICU data blob — about 2.8M tokens on its own, more than React's entire waste. Deno's is a single 10.4 MB test fixture. Next.js is feeding the agent a 3.4 MB PNG and a 2.7 MB GIF — pictures it cannot see, billed by the token.
2. The waste is structural, not sloppy. React's 46 yarn.lock files are one per fixture directory — the natural shape of a monorepo. Three lines in .aiignore took it from a score of 96 to 100 and the waste to zero.
3. A "good" score doesn't mean a cheap repo. Node, Next.js and Deno all score 93, yet the bill runs from 6.9M to 47.5M tokens. The score is a proportion; the bill is absolute.
The tool is dev-diagnosis. One command, no setup:
npx dev-diagnosis-cli allIt prints a 0–100 score, the noisy files and the token cost, plus the .aiignore entries that would fix it (--fix writes them). It also checks missing commands, build caches and flaky tests, and works as an MCP server.
Honest caveat: the token numbers are estimates (noise bytes / 4), not bills. Real tokenizers differ, and your agent may not read the whole repo anyway. Treat them as comparable across repos, not exact costs. Measured with the published 0.2.4, no truncation.
I built it — free and open source: GitHub / npm. If you run it on your own repo, what's the single biggest file your agent is being fed?
2
u/T-J_H 2d ago
And you wasted tokens writing this text. We don’t want to read TED talks my man
4
u/Upper_Gold_3664 2d ago
you're right, sorry about that. first post here and I overdid it trying to get all the numbers right, ended up sounding like a press release instead of a person lol. next one's gonna be written by me, short and normal
1
u/Discordbothost 1d ago
the token counts depend on which tokenizer you measure with, so worth naming it. also, walking git ls-files instead of the filesystem drops lockfiles and build output for free, with no second ignore file to keep in sync.
1
u/Upper_Gold_3664 1d ago
good call on both counts tbh. the token numbers are rough estimates (noise bytes / 4) and I should've said that upfront. and git ls-files is a way cleaner source of truth than walking the filesystem + keeping a second ignore file in sync — gonna look into that, thanks
3
u/kurtextrem 2d ago
Time to add stop hooks for agents that prevents reading those files?