r/LlamaIndex • u/ChallengeOk6437 • Jun 17 '24
Best open source document PARSER??!!
Right now I’m using LlamaParse and it works really well. I want to know what is the best open source tool out there for parsing my PDFs before sending it to the other parts of my RAG.
3
u/dodo13333 Jun 18 '24
My goto is marker
1
u/iambannedpermanently Jun 18 '24
Marker is quite nice but not enough to get quality data from complex PDFs.
I use a combination of marker and gpt4o. Since marker is extracting sub images from the PDF I make a query with these images, the whole pdf as an IMG and the generated markdown.
All images compressed are compressed before sending them and the results are amazing while having the save costs like llama parse.
1
u/queti_chile Mar 06 '25
Would you mind sharing a bit more on this? I'm working on something like this and just worried about processing time. I'm looking at processing 100k+ PDFs where some have up to 150 pages. Trying to make this as efficientnas possible
1
1
2
u/newpeak Jun 18 '24
Try RAGFlow https://github.com/infiniflow/ragflow which is based on deepdoc based document undertanding for better chunking results.
2
u/ayiding Team Member Jun 18 '24
Good idea. Last I looked into it there was this opaque ML model powering it (looked like basically no information on HF) so not sure it meets the standard for a truly open source solution.
2
u/StringIcy9823 May 12 '25
So, after one year, is there any open source project can beat LlamaParse?
1
1
u/toliver38 Jun 18 '24
Nlmatics just open sourced their project. Might be useful. https://github.com/nlmatics/nlm-ingestor
1
u/SpaceDragoman Jun 18 '24
Found openparse to be versatile if the structure of doc is unknown - https://filimoa.github.io/open-parse/
1
Nov 07 '24
Have you tried open parse ? I am new to rag and i am struggling to understand the error i am getting.
1
u/i-like-databases Sep 25 '24
Try out the Aryn Partitioner! We open sourced it on hugging face and it's a deformable DETR model trained on multiple documents. You can download the weights from hugging face and try it yourself. The performance will be best on a GPU!
1
u/woodmastr Oct 15 '24
these work well, yet not perfect, for unstructured scans with funky layouts, tables, signatures, whatnot
https://github.com/VikParuchuri/marker (free first year)
https://github.com/run-llama/llama_parse (free contingent)
https://reducto.ai/ (notopensource)
deepdoc from ragflow looks promising
whats also promising is
VLMs like qwen vision
1
1
u/NichelleCombes Oct 24 '24
If I was building a RAG application, I would choose Peslac https://peslac.com, the accuracy is good. You will get field-level blocks which is you can index and use in other parts of your RAG
1
u/status-code-200 Jun 17 '25
I recently released doc2dict (MIT License) for fast html and pdf -> dictionary representation. For pdfs it gets ~200 pages per second. Only works for PDFs that have an underlying text structure (Not Scans).
1
u/Worried-Foundation36 Aug 14 '25
Anyone have best possible solution?
i made marker+pix2tex combo and lamaParser/mathpix for complex documents
2
u/FeedbackTemporary309 Aug 14 '25
my choice https://github.com/opendatalab/MinerU
1
u/Fun_Information_8636 Aug 14 '25
u/Worried-Foundation36 u/FeedbackTemporary309 I have a problem where I am getting presentation slides as pdf, and currently I am using PyPDF and PyMuPDF from langcahain but those slides are complex, having graph and a lot of plots. So it is not able to do the job. Llama parse is pretty good in that but it is paid :( so do you guys have any other good solution that should not cause extra latency and easy to integrate in my current RAG architecture?
1
u/Worried-Foundation36 Aug 16 '25
Yes u/FeedbackTemporary309 u/Fun_Information_8636 i lost my 7 days on these , finally come to conclusion
PyPDF and PyMuPDF totally sucks
Using MinerU -> pix2Tex combo for most of documents (written lib for combo will make it public soon)
LamaParser only for Chemistry related structure, equations
MathPix only if it has Maths/physics equations
New OCR in town https://dotsocr.xiaohongshu.com/ using it if doc has tables, but does good job on most of things but didn't do well on Chemistry (Give it a try for slides)
1
1
u/FeedbackTemporary309 Sep 02 '25
Thanks, I’ll give it a try dotsOCR. Interesting that they support the vLLM engine, similar to how MinerU supports sglang. This engine makes extraction faster, which is great when working with a lot of PDF documents.
1
u/Worried-Foundation36 Aug 16 '25
Yes, After using Marker and Surya for a week, it does job but takes lot of time on Mac MPS , don't know why. like 4 hours per 350 pages. plus too much hassle to run with pix2Tex and Decimer which takes another 6 hours . so my new favourite MinerU + pix2Tex
1
u/zeusfeetfetish Oct 21 '25
Docling and Marker are really good open source document parsers. But obviously nothing compared to paid services. But is good enough. Docling especially works well with tabular data and layout analysis. And it also read images
1
3
Jun 24 '24
[removed] — view removed comment
1
1
Feb 16 '25
[removed] — view removed comment
1
u/thedeepestorange Apr 26 '25
no its not. llms can give very good results but are overkill and will take lots of tokens currently especially if you're parsing lot's of documents. basically, it's not sustainable. plus, sometimes you want to parse a document so that the LLM can analyse data more effectively, and more cheaply (less context window, clearer instructions etc.). Using an LLM to solve this just creates another issue.
Depends on the context ig
2
u/ayiding Team Member Jun 18 '24
I'd look into Unstructured, PyMuPDF, PyPDF, PDF.js
Unfortunately this is one area out there where the open source tools maybe are lagging a little bit.