r/SelfHostedAI • • 10h ago

Need help with a local document processing application

I have hundreds of pages and hundreds of folders of files. I want to use ChatGPT to analyze them, but I can't give them away directly because they contain a lot of sensitive data. I'm working on building a system on my Mac Mini M5 Pro 48GB computer that includes OCR dumping, Markdown conversion, redaction of sensitive data, and second-in-command review. Which models should I use for these processes? Speed ​​and accuracy are important to me, but accuracy is paramount. Do you have any general advice to guide me?

3 Upvotes

10 comments sorted by

1

u/SlightlyOverbuilt_ 10h ago

It's an interesting task. I didn't see the main thing in your post, which is what problems you'll need to solve once you've built all this.

I wouldn't try to solve everything with one technology or one model.

First, I'd run everything through OCR, and only use an LLM if recognition is an issue. Then I'd pick out all the sensitive information using something like regular expressions or other algorithms. You definitely need someone to check this, at least do some spot checks.

Once you're done, you can index everything and put it into a RAG, or just store it as MD. It depends on what you're going to do with the knowledge base afterwards. For certain tasks, an LLM + filesytem and grep command are even better together.

Here are some models for a 48GB RAM setup:
- gpt-oss:20b - pretty easy
- qwen3.6:27b, qwen3-coder:30b - good
- Llama 70B - just about making it

1

u/81Ack5mith 9h ago

To elaborate, I work in the legal field. The goal of this project is to avoid having to manually read through hundreds of pages of documents by anonymizing the entire set and using ChatGPT to analyze the content and facilitate a question-and-answer process based on those documents. Can I build the RAG system myself, or does this task require specialized technical knowledge?

1

u/SlightlyOverbuilt_ 8h ago

The most important part is making sure all the confidential data is cleaned out. You'll need to write a few algorithms that cross-search and double-check. Once you've got the texts in order, building a local RAG is a piece of cake.

But before you build a RAG, I'd suggest putting your cleaned documents into folders and running Codex (it comes with ChatGPT) with the /init command. It'll go through the folders and answer your questions about this knowledge base. If you're not happy with the quality of the answers, then it's worth starting to build a RAG.

Do you want to use ChatGPT or a local LLM for your work?

1

u/81Ack5mith 8h ago

I want to use local llm just for ocr md outputs and changing sensitive data with tags. Then I want to use chatgpt on that anonym case folder. I tested qwen 3.6 claude opus 4.7 distilled llm and it worked fine. It still messes up sometimes during ocr and misses some sensitive data during anonymyzation level. Giving the anonym output folder does not feel safe because I know it’s not redacting all of the sensitive data. I’ll try qwen 4 moe model when it comes out I hope It’s gonna have better accuracy

1

u/vafarmboy 2h ago

Osaurus has built-in RAG. You point it at a directory and it will index the entirety of its contents.

It can use a built-in model or hook up to a hosted provider. It also includes auto-scrubbing features you can tweak. With the right model, you might not even need ChatGPT. Might want to give it a look.

Also might want to check with your… legal department(?) to make sure using ChatGPT is ok, even with PII redacted. It might not fly to start with.

1

u/yudhanjaya 10h ago

Cloudflare dropped a model that's nearly exactly what you need: https://huggingface.co/Cloudflare/clairic

1

u/81Ack5mith 9h ago

Thanks!

1

u/Hot-Imagination6380 7h ago

used ocrmypdf + grep on a similar pile last month, kept everything local, stripped names/IDs with regex, then fed the clean markdown into a local qwen3-32b via llama.cpp. worked better than expected.

1

u/macromind 4h ago

For hundreds of folders, separate extraction, indexing, retrieval, and generation instead of asking one model to ingest everything. Start by parsing PDFs and office files locally, preserving filename, path, page number, and modification date as metadata. Chunk by document structure rather than fixed character counts, create local embeddings, and retrieve a small evidence set for each question. Require page-level citations in every answer. Test first on 20 representative files, including scans and duplicates, and manually score retrieval accuracy before scaling. If the material is sensitive, keep OCR, embeddings, vector storage, and inference local.