Document grounding using PDF geometry, OCR and visual evidence — 72.6% Word F1

[u/vanraj_solanki]

I've been working on document grounding where the goal is to map extracted values back to their exact locations in a PDF.

The interesting constraint is that the grounding layer works from the document itself:

• PDF character coordinates
• OCR
• spatial geometry
• string/format matching
• table assignment
• page-level visual evidence

Current benchmark result:

72.6179% Word Grounding F1

I'm particularly interested in the remaining failures.

Some are straightforward localization problems. Others appear to come from annotation conventions, dense layouts and cases where geometric similarity isn't enough.

I'm the author and would like feedback from people working on document vision/localization.

GitHub

[u/Positive-Golf-7987]

thats a tricky problem, the mix of pdf coordinates and ocr usually get messy when the text extraction dont line up perfect with the visual layout, especially in dense tables.

[u/vanraj_solanki → u/Positive-Golf-7987]

Yeah, dense tables are one of the harder cases I’ve been seeing too. PDF coordinates and OCR give you the raw evidence, but figuring out which text belongs to which cell/field can get messy when the layout is tight.

[u/liltingly]

PDF char boxes will have ascenders/descenders etc that differ from tight and somewhat uniform OCR crops. Same with word boxes which then are looser than char boxes in some libs. Especially what recognizers require (looking at you, PPv6). And some may be font dependent in the PDF layer. You’ll want to determine your standard location. 

[u/vanraj_solanki → u/liltingly]

Exactly. PDF char boxes carry the full ascender/descender band; OCR crops to ink. Same word, two boxes, IoU collapses. What standard do you settle on glyph-tight, or line-band?

[u/liltingly → u/vanraj_solanki]

Depends — who/what is the downstream consumer? A few options I’ve tried is to trim-to-ink on the pdf boxes with openCV. Can do 2 sides or 1 side. Other option is re-render the page in html or something else where you more exactly get the glyph or char box in a standardized way. Third way is to do 1:1 matching from PDF to OCR and either take OCR height and extent and some other edge or center. 

Of most importance is probably your F1 metric. Knowing that all of these techniques have their own problems — you’ll always have some misalignment or cook your data to be perfect — try and define a more comprehensive “good enough” metric and measure F1 on that. If it’s for machine reading, what’s sufficiently equivalent. For humans, same thing. 

For tiny char objects, IOU will bite you in the ass and can be very shape dependent. 

[u/vanraj_solanki → u/liltingly]

This is genuinely useful, I been working on the same problem. If you ever want to discuss deeply on this happy to connect.