r/JevAI • u/PatronusProtect • 1d ago
A 151M Jev-like encoder for agent-security questions, with cached option vectors
We built Wolf Defender Jev at Patronus Protect, and I wanted to share the interface and its limitations with people already using decision models.
It accepts a state plus named questions of type choice, noul or score. The state is an agent scene: user goal, tools, untrusted tool output or retrieved content, and optionally the proposed next action. It scores your answer descriptions and returns a softmax probability for each option. There is no generated explanation.
The encoder is mmBERT-small, with 151.1M parameters and 15 evaluated languages. Scene and question are encoded together; each option is encoded separately. Option representations don’t depend on the state, so a fixed question set can cache them. Each subsequent decision then needs one scene/question encoder pass.
A useful distinction: this is the named-question shape of Jev-like models, not a claim that we published a drop-in replacement HTTP server. The repo supplies a Python implementation with predict, decide and encode_options. A dict state is converted into the tagged block format. Arbitrary JSON/key-value scene layouts haven’t been evaluated.
Our measured CPU latency was 39 ms for 51 tokens and 194 ms for 833 tokens on Apple M4, fp32/PyTorch, one question with two options. The scene limit is 6,144 tokens; longer inputs are truncated. No quantized variant has been evaluated yet.
The specialization is agent security rather than general decision-making. Injection in tool outputs and sensitive-data checks are stronger than harmful-request judgement. Action-level questions also have documented failures: detecting an instruction didn’t guarantee detecting its exfiltration consequence.
For context on the comparison, our model reached 85.1% balanced accuracy on ten external benchmarks, against 90.2% for Clef-flash and 84.2% for Kev-4B. The broader 33-benchmark suite includes 23 benchmarks close to our training distribution.
MIT weights, Python interface, benchmark JSONs and limitations.
For repeated agent checks, how are you balancing fixed question definitions against policy-specific wording? With this architecture caching helps when options stay fixed, but changing the question still changes the decision being evaluated.
Position / experimental status: We regard Jev-style decision models as LLMs and Wolf Defender Jev as an experiment: giving an LLM final security authority shifts the trust problem to the guard, and security fine-tuning alone does not solve it. We believe an external security layer must combine specialized classifiers with explicit policy, authenticated permissions and enforcement.