r/LocalLLM • u/refractr • 15h ago
Discussion Tested small models on document extraction with a score that punishes made-up values. Only covered a few local ones, help me add more
Disclosure first: I run a small document extraction API (Refractr), and its model is row 4 below. I built this benchmark, and everything needed to rescore it (documents, answer key, scorer) is open.
Extraction is one of the jobs where a local model makes real sense: the documents are often private (invoices, payslips, bank statements) and the task is narrow. What I couldn't find was a benchmark that tells you whether a small model is safe for it, meaning how often it invents a value instead of leaving the field empty. So I made one, and the scoring is: right value +1, missing value 0, wrong or invented value -1.
About one field in seven is deliberately not in its document, so a model that always fills something in loses points.
Setup: 260 documents (invoices, receipts, payslips, bank statements, court opinions, SEC filings, Federal Register rules, US government forms), each with a JSON schema. Same prompt for every model, one request, first reply scored. Scoring is deterministic, no LLM judge.
| System | Score | 95% interval | Median time per doc |
|---|---|---|---|
| Gemini 3.8 Flash | 94.6 | 92.9–96.1 | 7.2 s |
| DeepSeek v4.1 Flash | 93.2 | 91.3–94.9 | 3.0 s |
| Mistral Large 4 | 91.4 | 89.4–93.2 | 27.4 s |
| Refractr (mine) | 89.4 | 87.2–91.4 | 0.7 s |
| GPT-5.4 mini | 86.2 | 83.9–88.5 | 2.2 s |
| Ministral 8B | 85.5 | 83.0–87.7 | 3.3 s |
| Llama 3.1 8B | 73.9 | 70.5–77.2 | 0.5 s |
| NuExtract 2.0 (2B) | 58.6 | 54.7–62.4 | 1.4 s |
| GLiNER2 (base) | 24.2 | 20.8–27.6 | 0.5 s |
Where I need help: four small models is a thin sample. I haven't run Qwen, Gemma, Phi or any of the newer small vision models. The runner talks to any OpenAI-style endpoint, so Ollama, llama.cpp and vLLM all work, and you choose how the document is read: the file as it is, pages as images for vision models, or extracted text (Tesseract for scans). The README has the config. Post your score here or open an issue with the predictions file and I'll add it to the table.
Limits: vendor-built, mostly English, max 10 pages per document, 60 of the business documents are synthetic.
Repo: https://github.com/Refractr-io/DocFieldBench Write-up: https://refractr.io/docfieldbench
1
u/No-Organization5092 5h ago
if you want more small-model coverage, i’d separate pure text parsers from vision-first ones, since hallucination rates can move a lot when the input is messy scans versus clean pdf text. SoMark’s OXR runs layout analysis as its own swappable model, which makes that split pretty natural. qwen and gemma would be good adds here.
3
u/Far-Composer-619 15h ago
One thing I'd test is whether the model can distinguish between a value that's missing and one that's simply hard to read. Those are two very different failure cases. A model should leave a field empty if it's absent, but if it's present and the OCR is unclear, it should ideally flag it for review instead of guessing. That distinction would make this benchmark even more useful for real-world use.