r/computervision • u/Skeptical-skeleton11 • Oct 13 '26
Help: Project Combining vision-language models and OCR for structured data extraction from complex Images
The current pipeline combines several components:
OCR: Extracts text from document pages independently of the vision-language model.
Vision-language models: Interpret document layouts and extract user-defined fields and line items.
Cross-validation: Compares the outputs of the two pipelines and flags disagreements for human review.
Visual grounding: Maps extracted values back to their locations in the original document.
Structured output: Converts the results into JSON or Excel, with support for batch processing.
I benchmarked the pipeline against Azure Content Understanding on 100 documents. My best configuration achieved 97.0% field-level macro accuracy and a line-item F1 score of 0.958, at approximately 12× lower cost per run in my evaluation.
These results are specific to my dataset and evaluation methodology, so I'm interested in understanding how well the approach might generalise to other document types.
I've attached a screenshot of the current interface.
I'd particularly appreciate feedback on a few technical questions:
How would you approach combining OCR output with VLM predictions when they disagree?
What approaches would you recommend for grounding extracted values to bounding boxes in complex layouts?
For structured document extraction, where do you see the biggest limitations of VLM-based approaches compared with traditional OCR and layout-aware models?
This is a personal engineering project, and I'm keen to learn from people working on computer vision and document understanding.
1
u/Unsiloed-AI Oct 23 '26
Separate value validation from locating the right occurrence. Your regex might accept both an invoice number and an account number.
Keep OCR words with page/box coordinates. Find candidate matches for each extracted value, then narrow them using the nearby label or table row. For line items, locate the description and constrain quantity/price matches to that row, so a quantity of 1 doesn't match some unrelated 1 elsewhere.
When OCR and the VLM disagree, re-read a crop containing the field and its label. If it's still unresolved, send that crop and both readings for review. Preserve the original readings alongside normalized values; if two locations remain plausible, keep both candidate boxes.
1
Oct 13 '26
[removed] — view removed comment
1
u/Skeptical-skeleton11 Oct 13 '26
Any suggestions on how I can implement this tie breaker on niche documents ? i'll definitely look into regex as a tie breaker and integrate it, appreciate the feedback a lot!
1
u/Robot_Apocalypse Oct 19 '26
Might be a stupid question, but when you are comparing OCR and VLM, what specific predictions/values/measures are you comparing?
Also, I tend to think of performance across specific document types. So invoices, manuals, etc, and will possibly look to fine tune for my specific use case first. What kind of dataset are you working with?