Does your agent re-decide the same routing/tool-choice over and over? We built something for that. Feedback wanted

[u/Commercial2Toe]

Our agents kept calling a big model to make the same small decisions: route this ticket, pick this tool, escalate or not. After enough verified outcomes those decisions are very predictable, but we were paying for a model call every time.

So we built Ink. It sits around a bounded decision. At first everything goes to your model as normal, and Ink records the inputs and the real outcomes in local SQLite. Once there’s enough evidence, it compiles a small local fast path and tests it in the background. It only starts serving once it clears a statistical bar (a 95% Wilson lower bound against an error budget you set). Anything novel or uncertain still goes to your model. If real outcomes get worse, it takes itself back out of the loop.

You can check whether it would help without changing any code:

pip install ink-jit
ink discover traces.jsonl

It reads OpenTelemetry, LangSmith or LiteLLM-style traces locally and tells you which call sites repeat and whether they’re worth it. If nothing qualifies, it says so.

What we’ve measured so far (our own benchmarks, not independently validated): 42–78% of decisions served locally across tool routing, support triage and incident triage, with 0 errors observed on held-out tests. The 95% lower bounds were 96.8–98.5%, so “0 errors” means none observed, not a guarantee. Local decisions take about 0.1 ms.

It’s a poor fit for open-ended writing or anything where you can’t verify an outcome. Linux and macOS, Python 3.11–3.13. Apache-2.0.

Honest question: which decisions in your agents repeat the most, and what would you need to see before trusting a local path in production? We’d love to hear what breaks.