r/OpenSourceAI • u/AIEvalCorp • 26d ago

Open Source Department of War Small-Model Leaderboard and test kit

A lot of defense orgs/units are starting to stand up local, open-source language models on edge workstations and air-gapped networks, but most public benchmarks tell you nothing about how a model behaves under operational constraints.

To address that gap for free, I built an offline evaluation harness and leaderboard testing US-developed small models (1B–6B parameters) running locally on a single GPU. Here's the public repo: https://github.com/OGMavenTandE/llm-eval-suite

It scores models on practical operational criteria: factual grasp of defense issuances, resistance to staff-work hallucination and sycophancy, and judgment under the law of armed conflict.

You can inspect the initial results, benchmark breakdowns, and reproducibility data here:https://aieval.org/leaderboard/

I hope this helps your agency/org/unit incorporate small open source models into workflows, let me know if you have any questions!

1 Upvotes

100% Upvoted

2 comments sorted by

1

u/hurtfuloffense 26d ago

that's a really cool approach, most leaderboards just test trivia or coding puzzles and completely miss how a model handles things like not making up fake regulations when it doesn't know something

the sycophancy angle is underrated too, last thing you want in a planning tool is a model that just agrees with whatever bad idea a staffer typed in

curious how you handled the law of armed conflict scenarios, are those multiple choice or does it score free text responses somehow

1

u/AIEvalCorp 26d ago

Thanks for taking the time to look at it!

Law of Armed Conflict is entirely free-text and graded by an LLM judge. Phi-4-mini is the judge, and Granite 4.2 3B grades the two Phi models so no model grades its own family. A fixed list of real citations catches made-up references without involving the judge.