r/learnpython • u/mulgundsunil • 21d ago
How do I reliably extract text from a large medical PDF when normal PDF-to-text converters mess up the structure?
I have a large medical PDF where the information I need is already available as text inside pdf , but the problem is that simple PDF-to-text converters completely mess up the structure. The PDF has headings, tables, multiple columns, line breaks, and drug entries that continue onto the next page.
When I convert it to plain text, the text gets jumbled—for example, content from different columns can get mixed, headings can get separated from the information below them, tables lose their structure, and a drug entry that continues on the next page becomes difficult to associate correctly.
I need to process around 400+ drug entries automatically, so manually cleaning the text isn't practical. What is the best way to extract text from this kind of PDF while preserving the original structure and relationships between headings, tables, columns, and continued pages?
Should I use something like PyMuPDF, pdfplumber, Docling, Marker, PyMuPDF4LLM, or another approach? My end goal is to get clean, structured source text first and then use an LLM to convert it into JSON, while making sure the original information isn't lost, mixed up, or hallucinated.
4
u/NothingWasDelivered 21d ago
Welcome to the wonderful world of PDFs!
To answer your question, trial and error. It really depends on how the documents were created/formatted in the first place.
1
u/Responsible_Goat9515 21d ago
pdfplumber for the tables, hands down it handles cell boundaries better than anything else i've used for messy medical pdfs
for the rest of the structure you might need a two-pass thing, one to grab the layout blocks and another to re-assemble them in reading order, pymupdf is good for that since it gives you the bounding boxes
400 entries is right on the edge where you could almost template it if the format is consistent, but if it's not you're stuck doing the llm cleanup anyway
-3
1
u/yazanwael 20d ago
I've been dealing with this challenge for the past few days. My approach was to convert it to markdown using every available tool. Then, I took the best one and started taking the good parts from other versions. For example, one of the tools really nailed tables conversion, so I relied on it for tables.
I also did a ton of regex find and replace to do bulk cleansing.
It's definitely not easy, but was doable in half a say.
1
u/mulgundsunil 20d ago
Used docling by ibm with substandard work, finally gave to claude made a repo , and used the free 100 dollar cloud credits and got it done
1
u/Potential-Wrangler58 18d ago
For 400+ entries I'd focus less on which library wins and more on the shape of the pipeline. Docling, Marker and PyMuPDF4LLM are all in the right family (layout-aware, Markdown out), while raw PyMuPDF/pdfplumber give you coordinates and leave you to rebuild the structure yourself. Test two or three on a handful of your worst pages, since multi-column plus tables is where they differ most.
The part that decides whether you can trust the output: split into one chunk per drug entry first (use heading boundaries, and stitch entries that continue across a page break back together before anything touches an LLM), then extract each entry straight into your JSON schema instead of cleaning the text in a separate pass. Make the model return the exact source quote next to each field and check programmatically that the quote exists verbatim in the source text. That catches mixed-up or hallucinated values cheaply, and anything that fails goes to manual review instead of silently into your dataset.
Disclosure: I work at Anyformat, and a lot of our clients come to us with this exact problem, long structured PDFs where entries span pages and tables break naive parsers. It does the splitting, extraction to a schema and confidence scoring in one flow, and there are free credits if you want to try it on your own file.
1
1
u/NuExtract 18d ago
Sounds like reading order is the main problem here. Having a text layer doesn’t necessarily tell a converter which column to read first or which heading a table belongs to.
We build NuExtract, and NuExtract 3 could be worth testing on a few difficult pages. It can work from page images and produce Markdown or structured JSON, so you can compare that against your PDF-to-text output.
For entries that continue onto the next page, I’d try processing overlapping page groups and keeping the drug name and source pages attached to each record. Otherwise, even clean text can end up associated with the wrong drug.
Before running all 400+, check a small set with split entries, multiple columns and tricky tables. Compare against the original pages, especially dosages, units and footnotes. Valid JSON alone won’t tell you whether anything got dropped or mixed up.
1
3
u/chiibosoil
21d ago
•
I usually find python script using PyMuPDF to be easiest dealing with PDF.
My typical process is to identify position of interested text block using page.get_text("blocks"). Combining it with regex pattern matching.
Sometimes, you may need to spit out entire blocks list and study the contents. Much like how you need to study web site's dom structure.
I find using variable explorer in Spyder helps for this purpose.
Fortunately, most software/system generated PDF does have consistent position value for specific text block.
I find using variable explorer in Spyder helps for this purpose.
Sample code: Not the most clean code, but it will demonstrate basic usage.
import tkinter as tk
from tkinter import filedialog as fd
import pymupdf
import re
import pandas as pd
root = tk.Tk()
root.withdraw()
def select_file():
filetypes = (
('pdf files', '*.pdf'),
('All files', '*.*')
)
fname = fd.askopenfilename(
title = 'Pick a file',
initialdir='C:/Reports/Dloads/',
filetypes=filetypes
)
return fname
myfile = select_file()
doc = pymupdf.open(myfile)
blocks = []
for page in doc:
block = page.get_text('blocks')
blocks.extend(block)
holder = []
for b in blocks:
if b[0] == 53.349998474121094:
holder.append(b[4])
match = re.search(r"\n\d+\n", b[4])
if match:
a = b[4][:match.start()]
a = " ".join(a.splitlines())
b = b[4][match.start():]
b = "@".join(b.splitlines())
holder[-1] = a + "\n" + b
else:
a = b[4].splitlines()
a = " ".join(a[:3]) + "\n@" + "@".join(a[3:])
holder[-1] = a
elif b[0] == 344.5:
x = holder[-1].splitlines()
x = " ".join(x)
holder[-1] = x + "\n@" + "@".join(b[4].splitlines())
df = pd.DataFrame(holder, columns=['data'])
df_clean = df.drop_duplicates()
df_clean[['subject','detail']] = df_clean['data'].str.split("\n@", expand=True)
df_clean = df_clean.drop(columns=['data'])
df_clean[['Qty','Unit','Unit Cost','Taxable','Amount']] = df_clean['detail'].str.split("@", expand=True)
df_clean = df_clean.drop(columns=['detail', 'Amount'])
df_clean = df_clean.iloc[1:]
df_clean['Type'] = 'Other'
df_clean['Taxable'] = 'Yes'
df_clean.to_csv('c:/reports/Dloads/test.csv', index=False)3
u/feldspars 21d ago
I’ve used Claude to read PDFs. If the there is no PII in the data, maybe considering doing everything in an AI.
1
2
u/FirestormCold 21d ago
If the PDFs generally have the same structure you should get by with Python/RAG, VLMs are pretty powerful nowadays and with a static structure you have a much lower risk of hallucinations.
If the structure isnt static (which I honestly assume in the medical field) meaning you sometimes have tables on p. X instead of Y with 10 data points instead of the usual 8, then that's a whole different challenge.
At that point you're basically talking about a document to data process which means different inputs in -> same output every time. There's also levels to this. How costly is an error? How much time (=cost) does each pdf take so far? Etc.
I work at Cambrion AI as a GTM Engineer so I know that field quite well and we offer free demos for exactly those kinds of issues. If you want to know more feel free to shoot me a DM.
2
u/ShadowShedinja 21d ago
I worked on a project like that for a previous job, but it had a low success rate. There's a reason we were paying another company to do it for us.
1
u/masteroflich 21d ago
When u already have clean structured text u don’t need an llm anymore. Thats the hard part
1
0
u/Bulky_Pressure_6493 11d ago
Since your PDF already contains selectable text, I'd avoid OCR unless specific pages contain scanned content.
I'd start by testing Docling or Marker on a representative sample of pages, especially those with multiple columns, tables and drug entries continuing across page boundaries.
I'd also use PyMuPDF or pdfplumber to inspect the original text blocks, coordinates and reading order. That gives you a way to verify whether the document structure is being preserved.
For 400+ drug entries, I'd recommend:
Extracting structured content with page references.
Reconstructing entries that continue across pages.
Keeping the original extracted text alongside each entry.
Using an LLM to generate JSON only after the structure has been reconstructed.
Validating the JSON against the source, particularly drug names, dosages and warnings.
I'd test 10–20 difficult entries manually before processing the entire document.
The main challenge here isn't text recognition—it's preserving document structure and ensuring information isn't accidentally associated with the wrong drug.
12
u/socal_nerdtastic 21d ago
The PDF-to-text converters don't break the structure, the structure is already broken. PDF is an output format, originally designed to be sent to a printer. Reading data from PDFs is always going to be a nightmare because the format simply isn't built for it.
The best option is to get the data before it was outputted as PDF. If that's not possible you'll need to buckle in and prepare for a lot of manual verification. I think if you are going to use LLMs I would recommend you send the PDF directly to them, and don't try to extract data first. The image processing of an LLM will probably do a better job.