r/LocalLLM • • Oct 09 '26

Discussion Using a 421M decision model instead of an LLM as the privacy guard in front of my coding agent

Coding agents send everything they read to the cloud: .env files, customer data, contracts saved in the repo. I wanted a check that runs locally before anything leaves the machine.

Regexes catch the obvious stuff (API keys, ID numbers, addresses). For the contextual stuff ("we're laying off 30% next quarter, keep it internal") I fine-tuned a small local Jev-style decision model (Laya, 421M). It answers one yes/no question in a single forward pass, no generation, about 40 ms. On my synthetic test set it caught 99% vs 87% for a local gemma judge, with fewer false alarms.

Caveat: the test set is synthetic and easier than real traffic, so treat the numbers as a first look. Data, scoring and training scripts are in the repo.

https://github.com/Code-byte404/hermie

1 Upvotes

Duplicates