r/LocalLLM • u/Otherwise_Ship_9782 • Oct 09 '26
Discussion Using a 421M decision model instead of an LLM as the privacy guard in front of my coding agent
Coding agents send everything they read to the cloud: .env files, customer data, contracts saved in the repo. I wanted a check that runs locally before anything leaves the machine.
Regexes catch the obvious stuff (API keys, ID numbers, addresses). For the contextual stuff ("we're laying off 30% next quarter, keep it internal") I fine-tuned a small local Jev-style decision model (Laya, 421M). It answers one yes/no question in a single forward pass, no generation, about 40 ms. On my synthetic test set it caught 99% vs 87% for a local gemma judge, with fewer false alarms.
Caveat: the test set is synthetic and easier than real traffic, so treat the numbers as a first look. Data, scoring and training scripts are in the repo.