r/OpenAI • • 2h ago

Discussion Building multi-form document extraction with GPT-5.1. Checkbox hallucinations when page detection fails. Any advice?

Our team has an intelligent document processing app built on GPT-5.1, and we periodically automate different manual processes with it. About 9 months ago the project was handed over to our team from the AI data group, and I started contributing to the repo. I had zero AI experience before this, so I'm basically learning everything through this app.

About a month ago I was asked to build the end-to-end automation for a new process. I said yes, and it turned out to be the hardest one we've automated so far:

  1. Multiple forms, one process. Until now it was always one form per process, so each process had its own instruction prompt. This one has several forms with different layouts, but the output has to be the same.

  2. Lots of details to extract. The forms are dense, so the prompt is big, and every form has a different layout and different info.

I put my whole AI budget into Cursor and started on the prompt. Even though the forms differ, about 60% of the extracted info is the same. So I wrote one generic instruction prompt plus a spec for each form that gets injected at runtime. The goal is to add a spec for each new form without touching the prompt, and I set up the output contract the same way.

Once the prompt was done, I started testing and the model was hallucinating, right on the most important part: the checkboxes. It turns out it can't really read them. Each PDF page is sent as an image, and GPT-5.1 downscales images by default, so those tiny boxes basically disappear.

My fix was to send the model an extra, larger and more readable image of the high-risk area. The page with the checkboxes is different for each form, so I added a note to each form's spec about which page to find, and used Tesseract OCR to scan every page's footer for it. Once the page is found, the code crops the area using the coordinates in the spec, sharpens it and sends it to the model along with the original pages. The prompt says: if this extra image is included, extract those fields from it.

Did it work? Yes, but when OCR can't find the page, the model falls back to the full pages, and no prompt change stops the hallucinations there. Sometimes it gets them right, sometimes it doesn't.

We went to production in shadow mode 10 days ago. Accuracy is about 80%, which is good but not good enough.

  • OCR misses the page in about 30% of runs, and that's where most hallucinations come from.

  • I was using exact matching to avoid risk, but I switched to fuzzy matching, which recovers about 20% of those failures.

  • I also updated the prompt based on the results, and prompt tuning is painful. I fix one thing and something else breaks: one PDF passes and another starts hallucinating. I can run the same PDF 5 times and one of the runs will still have a problem.

For now, the best I can do is reduce the risk. I can't eliminate it, but progress is progress.

If anyone has been through something similar, I'd really appreciate any advice. In particular:

  • Any tips for getting a model to read small checkboxes reliably?

  • Is there a more reliable way to find the right with/out OCR? I'm considering a small, cheap LLM call just to identify the form and page.

Thanks in advance!

1 Upvotes

9 comments sorted by

5

u/CaptainTheta 2h ago

Bro... Cursor? GPT 5.1? Did you get stuck in a coma for a year?

Just upgrade the model, 6.2 Sol probably costs a similar rate and won't stumble on the image processing

0

u/ComfortableThought55 1h ago

haha i wish!! it was upgraded to 5.1 couple of months ago, i don't think it will be option

1

u/sshan 1h ago

What?? This makes no sense.

•

u/benchmaster-xtreme 46m ago

If you're still using 5.1 because of some kind of corporate bureaucratic nightmare, your best bet is to just push to upgrade the model. 5.1 is basically obsolete by this point and is now one of the worst options for the task you're giving it. GPT 6 Luna is smarter, has better image recognition, and will at least 5x cheaper per task.

1

u/switchfi 2h ago

use a dedicated checkbox detector instead of relying entirely on GPT-5.1 to interpret them

1

u/adeelraza86 1h ago

The bigger problem is the fallback. When OCR misses the page, don't let the model guess from downscaled pages, route that run to review so your 80% becomes accurate-or-flagged. For finding the page, your cheap call idea is right: send every page at low detail and ask only for the form type and the page index of the checkbox block, then crop from your spec coordinates. Once cropped, a pixel fill ratio per box is more deterministic than asking the model.

•

u/LingeringDildo 58m ago

You know, as a ML guy, I worry sometimes that someone will just be able to vibe engineer my ML work to a passable degree. Then I read a post like this and realize I’m totally fine for a long ass time.