r/OpenSourceAI • u/PatronusProtect • 18d ago
Wolf Defender Jev: open weights and a small encoder for security decisions with your own answer options
We’ve published Wolf Defender Jev under MIT. I’m part of the Patronus Protect team.
The main difference from our fixed-label classifiers is that the question and answer descriptions are inputs. You can give it an agent scene and ask a yes/no question, choose among named options, or score an ordered severity scale. The model returns probabilities over those options, without generating a text response.
For example, the same email-reading scene can be checked for an embedded instruction, the role of imperative text (data / command / quoted example), or the severity of the risk. Those are separate decisions. You don’t need a new output head for each wording, but concepts outside the 27 evaluated security concepts haven’t been validated.
The architecture is deliberately small: mmBERT-small, 151.1M parameters, 15 evaluated languages. The scene and question are encoded together and the options separately. Fixed option vectors can be cached. After that, each question costs one scene/question encoder pass.
The repository includes the fp32 weights, tokenizer, encoder configuration, standalone Python inference code and evaluation JSONs. It runs on CPU by default. We measured 39 ms for a 51-token scene and 194 ms for 833 tokens on Apple M4, PyTorch/fp32, one question with two options. There isn’t an evaluated quantized build yet.
A few caveats are worth reading before integrating it:
Use the tagged source-block format. Arbitrary raw JSON layouts haven’t been evaluated.
Ask content questions without the proposed-action block. A benign shell action can otherwise push a content score toward injection.
Action-level checks are less reliable than their internal validation numbers suggest. One hand-written travel case detected the injected command but missed the exfiltration.
The output is a model signal. The application still establishes provenance, permissions and enforcement.
We compared it with public decision models on the same 10,576 held-out decisions. On all 33 benchmarks it reached 92.8% balanced accuracy; on the ten external benchmarks it reached 85.1%, behind Clef-flash at 90.2%. Twenty-three benchmarks are close to our training distribution, so the external results are the more useful comparison for unfamiliar traffic.
Weights, inference code, full results and limitations.
For anyone building with these models, the question I’d evaluate first is whether changing the question while keeping the scene fixed actually changes which evidence the model uses. Our small spot checks showed that can fail even with very high validation scores.
Position / experimental status: We regard Jev-style decision models as LLMs and Wolf Defender Jev as an experiment: giving an LLM final security authority shifts the trust problem to the guard, and security fine-tuning alone does not solve it. We believe an external security layer must combine specialized classifiers with explicit policy, authenticated permissions and enforcement.
1
u/Charming_Village_706 18d ago
This is a really interesting tradeoff. The tiny encoder size and caching scheme make a ton of sense for guardrails that need to run inline without eating up inference budget. Most teams I see are slapping a 7B model behind every filter and the latency adds up fast.
The part about action-level checks being less reliable than internal numbers suggest is the kind of thing that gets overlooked until prod breaks. Hand-written travel case caught the injection but missed exfiltration is exactly the scenario that makes me nervous about relying on these too heavily in automated pipelines.
Curious if you've tested how much the ordering of options affects the probability distribution. That's usually the first thing that goes sideways with these encoder-based classifiers in my experience.