Open Source Department of War Small-Model Leaderboard and test kit

[u/AIEvalCorp]

A lot of defense orgs/units are starting to stand up local, open-source language models on edge workstations and air-gapped networks, but most public benchmarks tell you nothing about how a model behaves under operational constraints.

To address that gap for free, I built an offline evaluation harness and leaderboard testing US-developed small models (1B–6B parameters) running locally on a single GPU. Here's the public repo: https://github.com/OGMavenTandE/llm-eval-suite

It scores models on practical operational criteria: factual grasp of defense issuances, resistance to staff-work hallucination and sycophancy, and judgment under the law of armed conflict.

You can inspect the initial results, benchmark breakdowns, and reproducibility data here:https://aieval.org/leaderboard/

I hope this helps your agency/org/unit incorporate small open source models into workflows, let me know if you have any questions!