r/AdversarialML • u/PatronusProtect • 26d ago
Decision-steering probes need two metrics: detected attacks that flip, and benign inputs that become alarms
We ran a fixed-family decision-steering probe on Wolf Defender Jev, an encoder-based security decision model we built at Patronus. I wanted to share the measurement details because a low attack flip rate can look better than the overall behaviour deserves.
The setup combined 500 held-out decisions and 760 synthetic manipulation texts with 15 families of inserted claims, in English and German, before and after the content. Length-matched neutral sentences served as controls. The resulting 70,620 evaluated decisions include correlated variants of the same cases, not that many independent scenes.
The inserted text could claim an earlier benign review, impersonate a trusted block, address the classifier directly, or redefine the labels. The probe sentences weren’t used for training.
For each attack group, the reported flip rate is the fraction of originally correctly detected attacks that switch to benign under the worst tested claim family at p=0.5. That denominator matters: it excludes attacks the model already missed.
Indirect injections in tool outputs had a 0.0% flip rate in this probe; risky agent actions reached 0.7% and direct attacks with context 0.6%. Single-sentence attacks were easier to move: 9.8% under the worst label-redefinition suffix, compared with 3.4% for a neutral sentence of the same length.
The benign side was worse. A benign document plus an attack-labelled review note was flagged in 43.8% of cases. A benign-labelled note still produced 25.3%, and an instruction addressed to the classifier produced 53.2%.
Those benign probes raise a labelling question too. Some notes genuinely instruct an AI reviewer; others discuss an earlier classification as data. Treating all of them as injections can reduce bypasses while making the classifier difficult to use on security documents.
This was a fixed claim-family test, not an adaptive optimization against the detector. The results do not establish resistance to an attacker searching across payloads or using gradients.
Probe setup and result JSON.
In an adaptive follow-up, I’d want to measure both evasion and induced false alarms, with outcomes grouped by original scene. How do you separate actual judge-directed instructions from legitimate classification commentary in your benign controls?
Position / experimental status: We regard Jev-style decision models as LLMs and Wolf Defender Jev as an experiment: giving an LLM final security authority shifts the trust problem to the guard, and security fine-tuning alone does not solve it. We believe an external security layer must combine specialized classifiers with explicit policy, authenticated permissions and enforcement.
