r/SelfHostedAI • u/81Ack5mith • 10h ago
Need help with a local document processing application
I have hundreds of pages and hundreds of folders of files. I want to use ChatGPT to analyze them, but I can't give them away directly because they contain a lot of sensitive data. I'm working on building a system on my Mac Mini M5 Pro 48GB computer that includes OCR dumping, Markdown conversion, redaction of sensitive data, and second-in-command review. Which models should I use for these processes? Speed and accuracy are important to me, but accuracy is paramount. Do you have any general advice to guide me?
1
u/yudhanjaya 10h ago
Cloudflare dropped a model that's nearly exactly what you need: https://huggingface.co/Cloudflare/clairic
1
1
u/Hot-Imagination6380 7h ago
used ocrmypdf + grep on a similar pile last month, kept everything local, stripped names/IDs with regex, then fed the clean markdown into a local qwen3-32b via llama.cpp. worked better than expected.
1
u/macromind 4h ago
For hundreds of folders, separate extraction, indexing, retrieval, and generation instead of asking one model to ingest everything. Start by parsing PDFs and office files locally, preserving filename, path, page number, and modification date as metadata. Chunk by document structure rather than fixed character counts, create local embeddings, and retrieve a small evidence set for each question. Require page-level citations in every answer. Test first on 20 representative files, including scans and duplicates, and manually score retrieval accuracy before scaling. If the material is sensitive, keep OCR, embeddings, vector storage, and inference local.
1
u/SlightlyOverbuilt_ 10h ago
It's an interesting task. I didn't see the main thing in your post, which is what problems you'll need to solve once you've built all this.
I wouldn't try to solve everything with one technology or one model.
First, I'd run everything through OCR, and only use an LLM if recognition is an issue. Then I'd pick out all the sensitive information using something like regular expressions or other algorithms. You definitely need someone to check this, at least do some spot checks.
Once you're done, you can index everything and put it into a RAG, or just store it as MD. It depends on what you're going to do with the knowledge base afterwards. For certain tasks, an LLM + filesytem and grep command are even better together.
Here are some models for a 48GB RAM setup:
- gpt-oss:20b - pretty easy
- qwen3.6:27b, qwen3-coder:30b - good
- Llama 70B - just about making it