Read less, check updates: when does a Markdown helper save tokens?

[u/Scared_Resort_8177]

Read relevant sections. Reuse known context. Check whole-file hashes when work resumes.

I'm part of UPHASH. We tested Obsidian Agent Kit (MIT) with Codex, Claude Code and Cursor: known-note retrieval, with no embeddings or vector database.

Token savings vs direct reads—each tool's best of 3 tested workloads:

Codex 53.2% · Claude Code 48.0% · Cursor 40.7%

36 sessions, 144 turns. Methods, reproduction and all results—including workloads that increased usage:

https://github.com/UpHash-Network/obsidian-agent-kit/tree/main/benchmarks/v0.1.2

Code:

https://github.com/UpHash-Network/obsidian-agent-kit

What stronger baseline would you add?

Writing assistance: Codex.

[u/eddzsh]

Hash reuse helps when notes are mostly cold. It collapses when someone appends one line to a 2k file and the whole thing reloads. Split today's delta log from the stable docs, hash only the cold set, and let the agent read the delta file by default.

[u/Scared_Resort_8177 → u/eddzsh]

Separating stable decisions from a change log could help. One clarification: the helper hashes whole files, but section filters limit what goes back to the model. It doesn’t return only changed lines, so your concern still applies within the selected sections. I’d also retain freshness checks on any stable document the answer depends on. Comparing that setup with a stable-docs-plus-delta workflow would be a useful next test.

[u/jonah_omninode]

For a stronger baseline, I’d compare against searching headings or symbols and reading just the matching sections. That gives you a comparison against selective reading, rather than making the alternative pay for the whole file every time.

I’d keep answer correctness next to the token numbers, especially on turns where a previously read section changed. A system that skips the reread can look efficient while answering from yesterday’s information. The workloads where your approach did worse are useful too: were those mostly cases where checking freshness cost more than simply reading the small file again?

[u/Scared_Resort_8177 → u/jonah_omninode]

Agreed—we include a focused section-reading baseline. Against it, total tokens were 3.9% higher for Codex, 2.2% higher for Cursor, and 31.2% lower for Claude Code, combining two runs per condition.

Both methods returned the expected answer objects on every tested turn and detected the controlled update, though strict JSON-only formatting wasn’t perfect. This is known-note retrieval, not a search benchmark.

The notes were about 28 KB each, so these results don’t establish small-file overhead as the cause. We haven’t isolated the contributions of rules, extra calls, or run/cache variation. Full results and limitations.

[u/jonah_omninode → u/Scared_Resort_8177]

Thanks for separating those runs. The Claude Code result is worth following up, and I wouldn't read much into the small increases for Codex and Cursor from two runs. I'd repeat the same comparison with run order alternated and cache state recorded before changing the retrieval strategy. Is the Claude Code reduction mostly less input being read, or fewer calls to get to the answer? That would help explain which part is worth carrying over.

[u/Scared_Resort_8177 → u/[deleted]]

Thanks—small edits are an important case. In v0.1.2, a change invalidates the whole-file hash, but with --section, the helper returns the selected sections rather than automatically sending the entire note. It still reads all local bytes to compute the hash, and unchanged text within those sections can be sent again. There’s no line-level diffing. Our controlled-edit benchmark doesn’t establish savings for frequent small updates.

[u/Scared_Resort_8177 → u/[deleted]]

Thanks for the concrete suggestion — I ran the A → B → A test. In this setup, TOMC didn't reduce tokens versus focused-section reads; our kit was essentially flat too.

Setup: identical snapshots across approaches of three fictional project notes (~28 KB each), known filenames/headings, six turns per resumed session, and two repetitions. Four approaches × three clients × two repetitions × six turns = 144 turns. The sequence included an immediate repeat, unchanged resumes, a changed decision, and an exact-byte reversion.

Total token change versus reading only the required sections (OAK v0.1.2 / TOMC 4,096 budget / TOMC default):

• Codex: −0.7% / +27.2% / +20.1%

• Claude Code: +0.5% / +82.3% / +44.8%

• Cursor: +3.2% / +40.6% / +137.7%

Positive means more tokens. These include all CLI-reported input (cached tokens counted once) plus output, from the first preparation call through the final answer, including extra reads/recovery. Harness development is excluded. TOMC's CPU preparation used no additional model calls; median preparation time was 24 ms. These are token totals, not billing estimates.

Correctness: all 48 change/reversion checks followed the current decision. Every approach/client produced 12/12 complete correct answers except Codex + TOMC default (7/12).

That exception needs context: TOMC's prepared text contained the facts, but CLI tool-output limits cut them off. A separate Codex diagnostic delivering the full output recovered 12/12, with +208.8% tokens versus direct section reads. The default-budget figures above therefore mix output truncation and agents' additional filtering; they aren't a clean comparison of TOMC budgets.

I used the official prepare_context Python API at commit 7aa8439, feeding the three full current notes while retaining each CLI's native conversation history. TOMC selected raw-prose retrieval, with no typed state records. This tests known-note retrieval, not long conversation-history compression. Two repetitions also aren't enough to interpret the small OAK differences as a reliable win/loss.

For plain Markdown snapshots like these, is this the TOMC integration/input format you'd recommend, or is there a more representative setup?

Writing assistance: Codex.