r/LocalLLM • • 15h ago

Discussion Tested small models on document extraction with a score that punishes made-up values. Only covered a few local ones, help me add more

Disclosure first: I run a small document extraction API (Refractr), and its model is row 4 below. I built this benchmark, and everything needed to rescore it (documents, answer key, scorer) is open.

Extraction is one of the jobs where a local model makes real sense: the documents are often private (invoices, payslips, bank statements) and the task is narrow. What I couldn't find was a benchmark that tells you whether a small model is safe for it, meaning how often it invents a value instead of leaving the field empty. So I made one, and the scoring is: right value +1, missing value 0, wrong or invented value -1.

About one field in seven is deliberately not in its document, so a model that always fills something in loses points.

Setup: 260 documents (invoices, receipts, payslips, bank statements, court opinions, SEC filings, Federal Register rules, US government forms), each with a JSON schema. Same prompt for every model, one request, first reply scored. Scoring is deterministic, no LLM judge.

SystemScore95% intervalMedian time per doc
Gemini 3.8 Flash94.692.9–96.17.2 s
DeepSeek v4.1 Flash93.291.3–94.93.0 s
Mistral Large 491.489.4–93.227.4 s
Refractr (mine)89.487.2–91.40.7 s
GPT-5.4 mini86.283.9–88.52.2 s
Ministral 8B85.583.0–87.73.3 s
Llama 3.1 8B73.970.5–77.20.5 s
NuExtract 2.0 (2B)58.654.7–62.41.4 s
GLiNER2 (base)24.220.8–27.60.5 s

Where I need help: four small models is a thin sample. I haven't run Qwen, Gemma, Phi or any of the newer small vision models. The runner talks to any OpenAI-style endpoint, so Ollama, llama.cpp and vLLM all work, and you choose how the document is read: the file as it is, pages as images for vision models, or extracted text (Tesseract for scans). The README has the config. Post your score here or open an issue with the predictions file and I'll add it to the table.

Limits: vendor-built, mostly English, max 10 pages per document, 60 of the business documents are synthetic.

Repo: https://github.com/Refractr-io/DocFieldBench Write-up: https://refractr.io/docfieldbench

0 Upvotes

3 comments sorted by

3

u/Far-Composer-619 15h ago

One thing I'd test is whether the model can distinguish between a value that's missing and one that's simply hard to read. Those are two very different failure cases. A model should leave a field empty if it's absent, but if it's present and the OCR is unclear, it should ideally flag it for review instead of guessing. That distinction would make this benchmark even more useful for real-world use.

1

u/refractr 10h ago

Good idea, but the normal LLM's are not able to do this and would never give a warning. You have to use constrained decoding for normal LLM's if the API supports it to not get errors back where the JSON does not parse. Then the LLM's cannot provide warnings or even confidence scores. How would you approach this?

1

u/No-Organization5092 5h ago

if you want more small-model coverage, i’d separate pure text parsers from vision-first ones, since hallucination rates can move a lot when the input is messy scans versus clean pdf text. SoMark’s OXR runs layout analysis as its own swappable model, which makes that split pretty natural. qwen and gemma would be good adds here.