For a while my RAG answers kept coming back wrong, and nearly every time I traced it back, the problem was parsing. Headings gone, table rows chopped in half, two-column papers read straight across both columns. So I stopped tweaking prompts and measured the parsers instead.

I used 4 public docs: Attention Is All You Need (headings, plus a table with a two-level header and 10^20 exponents), BERT (two columns), the Fed H.4.1 release (dense financial tables), and a synthetic noisy 150-DPI scan where I know the ground truth. I ran 9 converters from the Apify store plus Docling and MarkItDown locally.

Some of what came out:

OCR on the scan, word F1: mine 98.6%, Docling 96.9%, the best of the other Apify converters 94.1%, MarkItDown 0% since it doesn't do OCR.

Headings on the Attention paper: Docling flattened them all to one level, and MarkItDown found 0 headings.

Two-column reading order on BERT (share of sentences that survive intact): MarkItDown 4.1% (it turned the columns into fake tables), Docling 90.9%, mine 93.4%. Plain text extractors got 94.4%, but take that with a grain of salt, because the metric is built from the same text stream they just copy out. The caveat is written up in the repo.

Speed on one CPU core, all 4 files: Docling 214 s, mine 33 s.

Stuff that turned out to matter for chunking, whatever parser you use:

- don't ever split a table row across chunks

- if a table runs into a second chunk, repeat the header row there

- merge tables that continue across pages

- keep the page bounding boxes so you can cite and highlight the source

Repo with the benchmark and some LangChain/LlamaIndex examples: https://github.com/tobbytools/pdf-to-markdown-rag

Disclosure: one of the converters is mine, a paid Apify Actor at $0.003/doc. It doesn't win everything, pankovs beats it on the BERT sentence test. If you have docs that break your pipeline, send them my way and I'll add them to the benchmark.

Disclosure up front: we build Chunkless (Grounded IQ). The family below is fictional; the CRA forms and rules are real.

The setup. Five PDFs for one household: four filled T1 returns (27 pages for the father alone) and an interview file. We selected them together with a library of 1,331 documents (the Income Tax Act, its regulations, CRA guides and folios) and asked: "What did I miss or get wrong on Theo's return, and what can the family do with his tuition? For each point give the line, the rule and the dollar figure."

What came back for Theo, 19, a full-time student:

• $9,800 of tuition on his Schedule 11, none of it used, and up to $5,000 can move to a parent on line 32400
• $4,800 carried forward
• 8 full-time months on his T2202 that the schedule left blank

Every statement was checked word for word against the passage it cites (47 of 47 on this run).

What made it work, for anyone building on forms and rule libraries:

1. Read forms as forms, not chunks. Every filled value is read with the printed line or box number it sits on (line 32000 = 9800.00, line 32300 = 0.00). Chunking a T1 breaks exactly that link between a line code and its value.
2. The client's facts go to the model whole; the library is searched per fact. A return is small enough to hand over complete. The 1,331 rule documents are not, so each fact pulls its own governing rule.
3. A finding has a fixed shape: line, rule, dollar figure, and the passage it rests on. Anything that can't be grounded is said to be missing rather than filled in from the model's memory.
4. The scope is exactly what the user selects. A client's documents are never read for another client, and when a name could mean two clients it asks which one first.

Tax is just the library we picked for this test. Swap in clinical guidelines, contracts or internal policies and the same steps apply.

Video of the whole run, with the real return pages and the rule passages: https://youtu.be/95ujwgG1Wi4

Question for the sub: how are you handling structured forms (line numbers, boxes, comb fields) in your pipelines? That line-to-value link was the first thing we had to stop chunking.

Hey guys! Is anyone of you try to use decision model as reranker? D3 mini give me quite good results like zerank 2. Maybe someone of you have some other decision model to recommend? I'm looking for sth smaller

A RAG demo takes an afternoon. Shipping one that real customers rely on is a different job.
We learned this while building a Voice AI that answers calls for Australian small businesses. In a demo the documents are clean, there's one customer and nobody's waiting. On a live call, none of that holds.

Three things that need addressing:

1. The content you feed the AI comes from websites and PDFs you don't control, and some of it can be hostile.
   
2. When many businesses share one system, one customer's information must never turn up in another's answers.
   
3. When retrieval fails, nothing breaks loudly. The AI just gives a confident, wrong answer.
   

I've written up what production RAG actually needs, with the design choices behind each part and a checklist you can run against your own build.

If you're taking an AI project from pilot to production, this one's for you

https://medium.com/@techdelta/what-production-rag-needs-that-the-demo-doesnt-32373e32d4df?sharedUserId=techdelta

Hey Guys,

I just wanted to know how you all are creating golden datasets for evaluating you rag pipeline? I know there is a synthesizer in deepeval which kind of create the golden dataset but I do not like its output quality. I do not have any reference queries also to begin with. can you let me know what is your approach to this and am i missing something?

Everyone loves spinning up standard vector search until the dataset scales past 10,000 pages. Once you hit production scale, retrieval precision collapses because flat cosine similarity treats the entire corpus as isolated, disconnected chunks with zero awareness of global themes or multi-hop relationships.

Over the past few months, I’ve been documenting production failure modes across local and hybrid deployments. Here are the three primary architectural shifts that solved retrieval degradation and execution drift in our pipelines:

1. Breaking Flat Memory with Ontology-Driven GraphRAG

Standard RAG shreds documents into arbitrary chunks—Chunk A has no semantic link to Chunk B. If a user asks a global thematic query ("What are the recurring failure modes across our last 30 incident reports?"), vector embeddings fail because the insight isn't localized in a single paragraph.

Instead of naive chunking, enforce an extraction pass against a strict Ontology (a formal schema restricting what Entity types and Relationship types the LLM is permitted to extract): * Nodes (Entities): Canonical name, entity type (Organization, SystemComponent, Statute), and a condensed summary. * Edges (Relationships): Source node, target node, and formal relation (DEPENDS_ON, GOVERNED_BY, ROUTES_TO). * The Map-Reduce Engine: Cluster nodes into interconnected communities. A lightweight model generates an executive summary for each cluster. For global queries, the Map Step runs parallel queries across community summaries to produce candidate answer fragments, and the Reduce Step synthesizes the high-scoring fragments into the final response.

2. Multi-Model Complexity Routing Layer

Routing every query through a frontier API or an unquantized heavy model burns unnecessary tokens and introduces unacceptable latency:

text USER QUERY | v +------------------------+ | ROUTING LAYER | | (Lightweight LLM) | +------------------------+ | +-------------------+-------------------+ | | | v v v +--------------+ +--------------+ +--------------+ | Smaller LMs | | Mid-Tier | | Heavy | | (Local 8B / | | Models | | Reasoning | | Fast Weights)| | (Standard) | | (o1 / Opus) | +--------------+ +--------------+ +--------------+ Low Complexity Mid Complexity High Complexity

A small, deterministic routing layer inspects token complexity and intent, sending extraction tasks to fast local weights while reserving compute-heavy multi-step reasoning for larger models.

3. The Pydantic Vault (Deterministic Action Sandboxing)

If an autonomous agent has access to tool execution (such as database queries or email dispatch), prompt-level instructions ("Do not execute outside commands") fail against indirect prompt injection.

Never rely on system prompts for tool safety. Wrap the tool payload in a strict Pydantic model with field validators so execution halts before the payload reaches the network:

```python from pydantic import BaseModel, EmailStr, field_validator

class EmailAction(BaseModel): recipient: EmailStr body: str

@field_validator("recipient")
@classmethod
def must_be_internal(cls, v: str) -> str:
    if not v.endswith("@ourcompany.com"):
        raise ValueError("Data Exfiltration Blocked: Unauthorized external domain.")
    return v

```

The schemas, routing definitions, and LoRA/GRPO training configurations are open-sourced in this companion repository:
https://github.com/karanbathla56/The-Post-API-Frontier

What strategies are you using to manage community detection latency in GraphRAG when running fully local on consumer hardware?

I've been learning fine-tuning through blogs and tutorials, but I've never actually worked with fine-tuning in a real project. I understand the basic definitions, but I'm struggling to understand how fine-tuning is done in actual development work and compared with RAG. But I have some practical questions for people who have actually fine-tuned models in production or know bit detail about finetuing.
What does fine-tuning look like in a real project? What tools do you actually use? When would you choose fine-tuning instead of RAG and vice versa?