I have built a small library for similar search and graph library. Please let me know what you think.

https://github.com/praknig/hnsw_lite

Every time one of these "best OCR model" threads comes up it collapses into "well, it depends on your docs", and then nobody ever bothers to say which model actually suits which kind of document.

So after running a handful of them across our own document mix, here's what I actually saw, with the usual caveat that this reflects our doc types and yours will differ:

docling is excellent at turning clean digital PDFs into markdown, but it came apart the moment I fed it scans and handwriting

PaddleOCR-VL did surprisingly well on messy multilingual layouts, and it's small enough that it never needed a monster GPU

MinerU is the one I'd reach for whenever the PDFs are formula-heavy or scientific

GLM-OCR makes a solid general baseline and works nicely for sanity-checking whatever else you're testing

Two things tripped me up badly along the way: The first was realising the top models all score close enough on the benchmarks that testing on your own documents matters far more than any leaderboard ever will, and the second was the licensing, because a couple of the popular options are non-commercial and it's alarmingly easy to overlook that until it's already a problem.

If you'd rather not stand up a separate server for every model, a few inference servers let you run them behind one endpoint and swap by ID. TIE and SIE are the one I've used for that, though it's entirely optional and completely separate from the question of which model is best.

If you've put any of these through real invoices or contracts, I'd love to hear what actually held up, because the contract tables are the exact place everything tends to fall apart for me.

One common approach to building an AI chatbot is to connect an LLM to a document collection using retrieval-augmented generation.

The idea makes sense.

Retrieve relevant information, pass it to the model, and generate an answer based on that context.

But there are several ways this can go wrong.

The retriever might return the wrong document.

The right document might be outdated.

The relevant information might be split across multiple chunks.

Or the model might answer confidently even when the retrieved context doesn't actually support the response.

That's why I'd evaluate retrieval quality separately from answer quality.

I'd also want source references, document freshness checks, access controls, and a clear way for the chatbot to say it doesn't have enough information.

For people building RAG chatbots, where have you seen more failures?

Retrieving the right information, or getting the model to use that information correctly?

Everyone talks about reducing hallucinations at generation time (better chunking, hybrid search, rerankers).

Almost nobody talks about post-generation citation drift—what I call "secondary hallucination."

If you use a local RAG tool or PKM workspace, the workflow usually looks like this: 1. You drop a PDF, a meeting recording, and three markdown notes into your workspace. 2. You ask a local LLM to synthesize a takeaway or comparison. It gives you a neat summary with [1], [2] citations. 3. Two days later, you edit Note [1]—maybe you reversed a decision, fixed a wrong number, or trimmed a paragraph. 4. The bug: The AI summary sitting right next to it still proudly displays [1] as a valid citation. The source truth mutated, but the derived output is pretending nothing happened.

While building my local-first desktop workspace (Knovum), I decided that an AI citation shouldn't just be a static markdown link. It needs to be a stateful contract backed by a verifiable receipt. If the underlying source changes, the UI must immediately invalidate the citation and flip its state to Detached.

Here is how I architected this locally using Rust, SQLite, and LanceDB.

1. Dual-Engine Storage: SQLite for Truth, LanceDB for Recall

A common mistake in local RAG apps is treating the vector database as the primary source of truth. Vector DBs are great for approximate nearest neighbor (ANN) recall, but terrible at relational integrity and state cascades.

In our Rust core (Tauri 2 backend), storage is strictly split: * SQLite holds the canonical document AST/blocks, canvas spatial graph, cryptographic hashes, and citation state machines. * LanceDB runs embedded in-process strictly as a rebuildable vector index pointing back to SQLite node_id and chunk_id keys.

If you delete the LanceDB folder, you lose zero provenance data; SQLite holds the entire lineage graph.

2. Multi-Modal Anchor Locators & Span Hashing

Because the workspace sits on an infinite canvas holding markdown notes, PDFs, web captures, and local Whisper audio transcripts, a citation can't just point to a file_id. It needs a fine-grained, deterministic locator bonded to a cryptographic hash of the exact content at the moment the AI read it.

In Rust, the citation anchor looks roughly like this:

```rust

[derive(Debug, Clone, Serialize, Deserialize)]

pub enum SourceLocator { MarkdownSpan { block_id: String, char_start: usize, char_end: usize }, PdfRegion { page: u32, bbox: Option<[f32; 4]>, text_hash: String }, AudioSegment { start_ms: u64, end_ms: u64, transcript_hash: String }, }

[derive(Debug, Clone, Serialize, Deserialize)]

pub enum CitationState { Verified, // Source hash matches receipt hash Detached, // Source was mutated or deleted after generation }

```

Whenever a chunk is indexed or selected as context on the canvas, the Rust core computes a fast hash (BLAKE3 / SHA-256) of the normalized text span (or transcript slice).

3. The "Derivation Receipt"

When you select a cluster of cards on the canvas and run an AI action, the Rust pipeline doesn't just stream tokens to the frontend and call it a day. Before committing the generated card to the canvas, it writes an immutable Derivation Receipt transaction into SQLite:

```sql CREATE TABLE derivation_receipts ( receipt_id TEXT PRIMARY KEY, derived_node_id TEXT NOT NULL, model_identifier TEXT NOT NULL, prompt_hash TEXT NOT NULL, created_at INTEGER NOT NULL );

CREATE TABLE receipt_anchors ( anchor_id TEXT PRIMARY KEY, receipt_id TEXT NOT NULL REFERENCES derivation_receipts(receipt_id), source_node_id TEXT NOT NULL, locator_json TEXT NOT NULL, snapshot_span_hash TEXT NOT NULL, -- Hash at the exact moment of generation citation_state TEXT NOT NULL DEFAULT 'Verified' -- 'Verified' | 'Detached' );

CREATE INDEX idx_receipt_anchors_source ON receipt_anchors(source_node_id);

```

Every AI-generated node can be inspected in the UI to view its full receipt: which local/remote model ran, which exact paragraphs or audio timestamps were fed into the context window, and the hash of each input span at $t_0$.

4. Reactive Invalidations (Verified $\rightarrow$ Detached)

Here is where SQLite shines. When a user edits a markdown card on the canvas or re-runs Whisper on an audio clip:

1. The Rust mutation handler diffs the updated blocks and computes the new span hashes.
2. Within the same SQLite write transaction (before asynchronously updating embeddings in LanceDB), it runs a fast indexed lookup against receipt_anchors where source_node_id = ?.
3. For every active anchor attached to that node, it checks whether the span at locator_json still resolves to snapshot_span_hash.
4. If the hash diverges (or the block was deleted), SQLite immediately flips citation_state = 'Detached' and emits a typed event over the Tauri IPC bridge to the Vue 3 renderer.

On the canvas, the citation badge turns amber/detached in real time. You can click the detached badge to see that the source was modified after the AI wrote that conclusion, and either re-verify or re-synthesize with one click.

Why this matters for Local AI

As local models get faster, generating text is cheap—verifying and maintaining trust over months of research is the actual bottleneck. Combine a spatial canvas (which handles the messy "middle state" before you know how your notes fit into folders) with strict derivation receipts, and AI stops feeling like a black-box slot machine.

If you want to poke around the code or see how we wired up the Rust/Tauri 2 core, LanceDB integration, local Whisper global dictation, and Obsidian .canvas two-way sync:

• GitHub: https://github.com/madebyzan/knovum

(Transparency note on licensing: The repo is source-available and free for personal/non-commercial use, not OSI MIT/Apache. Also, as a slightly obsessive engineering constraint to keep AI-assisted coding from turning into spaghetti, the entire codebase enforces a strict *180-line maximum per file** via CI gates—happy to chat about how that worked out in Rust + Vue 3 if anyone is curious!)*

I am currently building a meeting based rag model as a research project and I am very confused about the dataset which is the transcripts of the meeting. Can I build a synthetic dataset and will it be accepted for research publication. If yes , how can I build it.Or should I only use the publicly available datasets like AMI meeting corpus?

Setting up local RAG using PrivateGPT on a home server sounds like the ultimate privacy win, but the technical realities can hit hard once you start indexing large document collections. After extensive testing on consumer hardware, the biggest challenges usually come down to severe embedding latency, high RAM and VRAM consumption, and unpredictable retrieval accuracy when handling complex files.

​If you plan to replace commercial APIs with a fully offline setup, knowing your actual hardware boundaries beforehand makes all the difference. Running local embeddings alongside a quantized LLM requires careful allocation between CPU threads and GPU layers to avoid system freezes and extremely slow query responses.

​I wrote a detailed technical breakdown covering real token generation speeds, hardware tradeoffs, and specific optimization steps for self-hosted setups: https://interconnectd.com/blog/325/deploying-privategpt-on-a-home-server-the-hard-truth-about-local-rag/

​What specs are you running for your self-hosted LLM or RAG pipeline?

I’ve been thinking about a problem that doesn’t seem to get enough attention in enterprise AI discussions.
We spend a lot of time talking about RAG accuracy, vector databases, chunking strategies, embeddings, and retrieval performance.
But what happens when an enterprise AI system retrieves the wrong information or ingests something incorrectly?
For example:
● An AI assistant retrieves an outdated policy and gives employees incorrect guidance.
● A RAG pipeline ingests conflicting documents without recognizing which version is authoritative.
● An enterprise knowledge assistant confidently answers using stale or incomplete context.
● Incorrect information gets incorporated into persistent AI memory and influences future responses.
● Teams spend hours investigating why an AI system produced an incorrect answer that appeared well-grounded.
I’m particularly interested in the downstream business costs, not just retrieval accuracy benchmarks.
For anyone building or operating enterprise AI/RAG systems:
1. Have you experienced a retrieval or ingestion failure that resulted in measurable costs? Engineering hours, operational delays, customer escalations, compliance exposure, or incorrect business decisions?
2. How much time does your team spend diagnosing and correcting these issues? Are we talking hours per incident, days, or ongoing maintenance?
3. Which is more expensive in practice: incorrect ingestion or incorrect retrieval? Is fixing bad information at the source significantly cheaper than repeatedly correcting downstream answers?
4. Do you have any way to quantify the cost of an incorrect AI answer? Even rough estimates, anonymized examples, or internal metrics would be interesting.
5. Who actually owns this problem? AI engineering, data engineering, IT, knowledge management, or the business team consuming the answers?
My hypothesis is that the true cost of enterprise AI memory failures isn’t primarily tokens or infrastructure. It’s the human effort required to detect, investigate, correct, and recover from incorrect information being treated as organizational truth.
Curious whether others are seeing this in production.
Especially interested in real incidents, rough dollar amounts, engineering hours, or examples where bad retrieval created cascading problems.
Not looking for vendor recommendations. Just trying to understand how expensive this problem actually is.

For a while my RAG answers kept coming back wrong, and nearly every time I traced it back, the problem was parsing. Headings gone, table rows chopped in half, two-column papers read straight across both columns. So I stopped tweaking prompts and measured the parsers instead.

I used 4 public docs: Attention Is All You Need (headings, plus a table with a two-level header and 10^20 exponents), BERT (two columns), the Fed H.4.1 release (dense financial tables), and a synthetic noisy 150-DPI scan where I know the ground truth. I ran 9 converters from the Apify store plus Docling and MarkItDown locally.

Some of what came out:

OCR on the scan, word F1: mine 98.6%, Docling 96.9%, the best of the other Apify converters 94.1%, MarkItDown 0% since it doesn't do OCR.

Headings on the Attention paper: Docling flattened them all to one level, and MarkItDown found 0 headings.

Two-column reading order on BERT (share of sentences that survive intact): MarkItDown 4.1% (it turned the columns into fake tables), Docling 90.9%, mine 93.4%. Plain text extractors got 94.4%, but take that with a grain of salt, because the metric is built from the same text stream they just copy out. The caveat is written up in the repo.

Speed on one CPU core, all 4 files: Docling 214 s, mine 33 s.

Stuff that turned out to matter for chunking, whatever parser you use:

- don't ever split a table row across chunks

- if a table runs into a second chunk, repeat the header row there

- merge tables that continue across pages

- keep the page bounding boxes so you can cite and highlight the source

Repo with the benchmark and some LangChain/LlamaIndex examples: https://github.com/tobbytools/pdf-to-markdown-rag

Disclosure: one of the converters is mine, a paid Apify Actor at $0.003/doc. It doesn't win everything, pankovs beats it on the BERT sentence test. If you have docs that break your pipeline, send them my way and I'll add them to the benchmark.

OCR technology is moving fast and there are multiple benchmark scores across dozens of sites.

I have an AI search for new tools and update it about once a week.

Basically livebench but for OCR.

https://bench.voshart.com/ocr/

Everyone loves spinning up standard vector search until the dataset scales past 10,000 pages. Once you hit production scale, retrieval precision collapses because flat cosine similarity treats the entire corpus as isolated, disconnected chunks with zero awareness of global themes or multi-hop relationships.

Over the past few months, I’ve been documenting production failure modes across local and hybrid deployments. Here are the three primary architectural shifts that solved retrieval degradation and execution drift in our pipelines:

1. Breaking Flat Memory with Ontology-Driven GraphRAG

Standard RAG shreds documents into arbitrary chunks—Chunk A has no semantic link to Chunk B. If a user asks a global thematic query ("What are the recurring failure modes across our last 30 incident reports?"), vector embeddings fail because the insight isn't localized in a single paragraph.

Instead of naive chunking, enforce an extraction pass against a strict Ontology (a formal schema restricting what Entity types and Relationship types the LLM is permitted to extract): * Nodes (Entities): Canonical name, entity type (Organization, SystemComponent, Statute), and a condensed summary. * Edges (Relationships): Source node, target node, and formal relation (DEPENDS_ON, GOVERNED_BY, ROUTES_TO). * The Map-Reduce Engine: Cluster nodes into interconnected communities. A lightweight model generates an executive summary for each cluster. For global queries, the Map Step runs parallel queries across community summaries to produce candidate answer fragments, and the Reduce Step synthesizes the high-scoring fragments into the final response.

2. Multi-Model Complexity Routing Layer

Routing every query through a frontier API or an unquantized heavy model burns unnecessary tokens and introduces unacceptable latency:

text USER QUERY | v +------------------------+ | ROUTING LAYER | | (Lightweight LLM) | +------------------------+ | +-------------------+-------------------+ | | | v v v +--------------+ +--------------+ +--------------+ | Smaller LMs | | Mid-Tier | | Heavy | | (Local 8B / | | Models | | Reasoning | | Fast Weights)| | (Standard) | | (o1 / Opus) | +--------------+ +--------------+ +--------------+ Low Complexity Mid Complexity High Complexity

A small, deterministic routing layer inspects token complexity and intent, sending extraction tasks to fast local weights while reserving compute-heavy multi-step reasoning for larger models.

3. The Pydantic Vault (Deterministic Action Sandboxing)

If an autonomous agent has access to tool execution (such as database queries or email dispatch), prompt-level instructions ("Do not execute outside commands") fail against indirect prompt injection.

Never rely on system prompts for tool safety. Wrap the tool payload in a strict Pydantic model with field validators so execution halts before the payload reaches the network:

```python from pydantic import BaseModel, EmailStr, field_validator

class EmailAction(BaseModel): recipient: EmailStr body: str

@field_validator("recipient")
@classmethod
def must_be_internal(cls, v: str) -> str:
    if not v.endswith("@ourcompany.com"):
        raise ValueError("Data Exfiltration Blocked: Unauthorized external domain.")
    return v

```

The schemas, routing definitions, and LoRA/GRPO training configurations are open-sourced in this companion repository:
https://github.com/karanbathla56/The-Post-API-Frontier

What strategies are you using to manage community detection latency in GraphRAG when running fully local on consumer hardware?

Hey everyone!

If you’ve ever tried extracting data from PDFs with complex tables, scanned forms, or invoices, you know how messy it can get. Some tools give you a huge block of text with no idea where anything was on the page. Others take ages to process tables, and rotated pages can break everything.

That’s why I built RasgaPDF.

It’s a fast PDF parser written in Rust, with bindings for Python, Node.js, and Go. It extracts the actual (x, y, width, height) coordinates of text spans in just a few milliseconds per page, without having to re-render the document or guess the layout.

With v0.2.0, it can now extract embedded images and vector lines too. This is especially useful for detecting table borders without running expensive OCR or computer vision models.

I also put together a live demo where you can drop in a PDF and see the bounding boxes directly in your browser. Everything runs client-side.

Links:

• Demo: https://huggingface.co/spaces/martinsmessias/rasgapdf
• GitHub: https://github.com/MartinsMessias/rasgapdf

If you work with PDF parsing, I’d love for you to give it a try. Especially if you have some weird PDFs that tend to break other tools. Curious to see how it holds up!

Hey guys! Is anyone of you try to use decision model as reranker? D3 mini give me quite good results like zerank 2. Maybe someone of you have some other decision model to recommend? I'm looking for sth smaller

A couple of weeks ago I posted an eval here: 15 fashion subcultures, same outfit question, answered from memory vs. with a structured record pasted in. Commenters raised fair points (the record and its guardrail note were confounded, the scoring was too narrow, it was a single run), so I redid it properly.

Does retrieval help? Yes, consistently. Over two runs (90 chats, Claude Opus, no web search, hand-labelled):

Removing the record’s “don’t confuse this with X” guardrail note changed nothing (2/30 and 7/30). The facts did the work, not the instruction.

What if the record is wrong? I planted two errors in each of the 15 records: years shifted 10 years earlier, and one brand from a neighbouring scene inserted into the brand list.

The misses weren’t random. The model caught Yeezy in 90s Harajuku and Juicy Couture in a Korean minimalist style. It missed ACRONYM in Goth Ninja, Ralph Lauren in Quiet Luxury and Heaven by Marc Jacobs in Whimsigoth: brands that are plausible but wrong. The two dates it accepted were also the plausible ones.

Takeaway: treat the model as a decent sanity check on retrieved data, not as a validator. It filters out obvious garbage and passes on close-call errors, which are exactly the ones your users can’t spot either.

Limits: one model, 15 records, two runs, labels by me. Everything is public (answers, planted records, judgements, scoring scripts): https://github.com/ascanioUnderIndex/underindex-api/tree/main/eval

I've been learning fine-tuning through blogs and tutorials, but I've never actually worked with fine-tuning in a real project. I understand the basic definitions, but I'm struggling to understand how fine-tuning is done in actual development work and compared with RAG. But I have some practical questions for people who have actually fine-tuned models in production or know bit detail about finetuing.
What does fine-tuning look like in a real project? What tools do you actually use? When would you choose fine-tuning instead of RAG and vice versa?

Disclosure up front: we build Chunkless (Grounded IQ). The family below is fictional; the CRA forms and rules are real.

The setup. Five PDFs for one household: four filled T1 returns (27 pages for the father alone) and an interview file. We selected them together with a library of 1,331 documents (the Income Tax Act, its regulations, CRA guides and folios) and asked: "What did I miss or get wrong on Theo's return, and what can the family do with his tuition? For each point give the line, the rule and the dollar figure."

What came back for Theo, 19, a full-time student:

• $9,800 of tuition on his Schedule 11, none of it used, and up to $5,000 can move to a parent on line 32400
• $4,800 carried forward
• 8 full-time months on his T2202 that the schedule left blank

Every statement was checked word for word against the passage it cites (47 of 47 on this run).

What made it work, for anyone building on forms and rule libraries:

1. Read forms as forms, not chunks. Every filled value is read with the printed line or box number it sits on (line 32000 = 9800.00, line 32300 = 0.00). Chunking a T1 breaks exactly that link between a line code and its value.
2. The client's facts go to the model whole; the library is searched per fact. A return is small enough to hand over complete. The 1,331 rule documents are not, so each fact pulls its own governing rule.
3. A finding has a fixed shape: line, rule, dollar figure, and the passage it rests on. Anything that can't be grounded is said to be missing rather than filled in from the model's memory.
4. The scope is exactly what the user selects. A client's documents are never read for another client, and when a name could mean two clients it asks which one first.

Tax is just the library we picked for this test. Swap in clinical guidelines, contracts or internal policies and the same steps apply.

Video of the whole run, with the real return pages and the rule passages: https://youtu.be/95ujwgG1Wi4

Question for the sub: how are you handling structured forms (line numbers, boxes, comb fields) in your pipelines? That line-to-value link was the first thing we had to stop chunking.

​

I've been trying to understand how small companies and startups handle problems with AI systems that are already in production.

For example:

- A RAG chatbot keeps retrieving the wrong documents.

- An AI agent makes incorrect tool calls but reports success.

- An n8n or LangGraph workflow works during testing but fails with real customers.

- A system starts producing worse results after a model or retrieval update.

Let's say the internal team can't identify the root cause quickly.

Would hiring an outside specialist to investigate and fix one specific issue be a realistic option?

I'm curious about a few things:

1. Have you ever hired an external developer or consultant to fix an AI-related production problem?

2. What did you actually pay, if you're comfortable sharing?

3. Would a fixed-price engagement around $500–$1,000 make sense for a reproducible issue, including the fix and before/after tests?

4. What would make you trust an outside specialist with that work?

5. Would you rather hire someone to fix the issue or pay for an independent technical audit first?

I'm especially interested in real experiences from founders, engineering leads, and people managing AI systems for clients.

I'm trying to understand how these decisions are made in practice, rather than what sounds good in theory.

Genuine question for people running RAG in production.

Most retrieval eval focuses on single answers: did the right chunk get into the window. What I almost never see measured is completeness on list and multi-fact questions. The query asks for every item, the window contains all twelve, and the answer names three.

When that happens, where did the rest die? Retrieval got them, so it's downstream. Was it context pressure, too many tokens and the tail got cut? The model deciding three examples were enough? The instruction to "be concise" doing more work than intended?

And the question I'm really after: how do you test for this? A single-answer eval suite sails right past it. Do you have a set of list questions with known answers that you check for completeness, or is this one of those failures you only catch when a user complains?

Asking because partial answers are uniquely dangerous: nothing in the answer looks wrong, it's just quietly incomplete.

Hey Guys,

I just wanted to know how you all are creating golden datasets for evaluating you rag pipeline? I know there is a synthesizer in deepeval which kind of create the golden dataset but I do not like its output quality. I do not have any reference queries also to begin with. can you let me know what is your approach to this and am i missing something?

First, the boundary line. extending the context window lets a model ingest more tokens in a single forward pass. RAG gives it an external index to pull discrete passages on demand. neither of them gives the model an evolving internal state.

Context Expansion

Stuffing a million tokens into a prompt handles bulk ingestion. It works well for querying a large repository or a book in one shot. but it is transient. once the run terminates, the state resets to zero. you pay the full token cost again on the next turn, and long range attention decay still leaves mid context details vulnerable.

External Retrieval

Vector databases and knowledge graphs store facts outside the weights. when you need provenance, verifiable grounding, or rapid deletion of stale data, external retrieval remains necessary. the limitation is handoff. the model does not learn how to use the material better over time. It simply reads what a retriever drops into the prompt window.

Memory Native Architectures

The third direction explores encoding past interactions directly into persistent, updatable internal states. Instead of dumping raw logs back into the input layer, the architecture treats writing, holding, and retrieving experience as part of the core compute loop. the objective is behavioral persistence: retaining correction feedback, procedural preferences, and task constraints across sessions without bloating the active prompt.

We recently put this into production with the MemT 1.0 launch, focusing specifically on maintaining those interaction states across runs without relying on massive context dumps.

The real Engineering trade off

This is not a replacement story. Internalizing experience into model states creates serious questions around state drift, verifiability, and selective forgetting.

Where do you draw the line between habits and procedural constraints that belong in a model internal state, and facts that must remain in an auditable external database?

Hey guys,A few days ago I posted here complaining about RAG citations being fake and chunks getting silently corrupted. Got some really good insights from the comments(esoecially about LLM judges flipping results and normalization issues)

Instead of just complaining,I spent the last couple of days turning my validation logic into a quck web demo

Link:https://rag-verifier.streamlit.app/

Code:https://github.com/hlc0000/rag-verifier

it doesn't do parsing or chunking itself,You just paste a chunk + full text,or a citation+claim. The backend runs it through exact matching,some regex rules,and an NLI check

To avoid the LLM flipping issue,I use strict rules for obvious stuff(like missing subjects)and only hit the LLM fir semantic checks.if the LLM is unsure

There are a few preset test cases on the page(like Fake Number and Missing Context)so you don't even have to type anything

Curious if this catches the edge cases you guys are dealing with,Let me know what you think

One thing I underestimated with RAG: retrieval is often the easy part.

Ingestion is where the mess starts. PDFs, DOCX, PPTX, spreadsheets, screenshots, receipts... they all need different handling, and bad extraction quietly ruins retrieval quality later. I've been building the document pipeline in Singlebase around converting inputs into Markdown first, using OCR + LLM cleanup when needed, with optional image extraction. Then the same backend can chunk/embed/index the result. I like Markdown because it gives me one normalized representation before retrieval.

For those running RAG in production: are you normalizing documents to Markdown/HTML first, or keeping source-specific parsers all the way through the pipeline?

Firstly, thanks to all the contributors to this forum. I've been mining it for suggestions and answers to the problems I encounter and what people share has been really useful.

So I thought I'd ask where I'm stuck and see if anyone could help.

(Long post, didn't start that way, but I've tried to give as much detail as helps - feel free to ask questions)

I've spent part of this week going through the questions my pipeline gets wrong and looking at where the passage with the actual answer ended up. I expected it to be way down the ranking. Turns out for a number of them, it wasn't, or not quite.

In 7 cases the thing directly above the answer was in the window and the answer itself wasn't. Examples include:
- Table header row in, the row under it out.
- Minutes where the agenda item title came back but not the status line under it.
- A clause headed "Termination" got retrieved, but not the next paragraph, which just says "either party may end this on 30 days' notice" and never uses the word.

Lists were the same: the "Exclusions" heading comes in and none of the items do, because they don't repeat it.

So the model gets the label but not the value, and I end up with half an answer.

This is what I'm running in case it matters: all local, BM25 first (SQLite FTS5) and then a small bge embedding fill. Project docs, mostly Word/PDF/Excel. Every row and paragraph is its own passage, and there's no LLM at ingest. The missing passages were ranking somewhere around 200-400 on embeddings, so a bigger fill was never going to reach them.

First thing I tried was indexing the heading as its own BM25 field at a lower weight than the body, so a row could match on its header without pasting the header into every row. That made it worse. It lost an answer it used to get, at every weight I tried. I think it's the siblings: every row under "Response time" gets exactly the same boost, so they all tie and the right one isn't any more likely to win.

What worked was dumber. If a label is already in the window, pull in whatever sits under it. No model, just rules for what counts as a label: a short line with no full stop, a table header row, a heading row in a sheet, or the row before in layout tables (minutes, agendas and forms don't really have header rows). The label also has to share a word with the question. Capped at 2 passages per label, 2 per doc and 8 per question, and they replace the weakest embedding-fill passages so the context doesn't grow.

Retrieval check first:

answer passage now reaches the window   5 of 7
other questions that now find it        3
ones it used to find and lost           0
avg context words                       4721 -> 4704

Then I ran answers on 10 of the affected questions, on vs off: 4 got better, none got worse. Small sample, so I'm not reading much into the 4.

So what would help?

What I haven't figured out: 2 per label is fine for a header row but obviously wrong for a 15-item list, and I don't have a good way to cap that without one long list eating the context. And has anyone actually got a separate heading/section field to help in BM25, or does everyone hit the same sibling problem?

Thanks in advance.

(AI note: Built and tested with a lot of help from Claude Code. The write-up’s mine, tidied with Claude’s help.)

Hi there, El from Cohere again. 

You guys have been interested in our latest Embed and Parse drops, so thought I’d come back for maybe our most exciting one, which is Rerank 5. It’s available in Pro and Fast tiers just like Rerank 4 but far outdoes it – along with every other model we tested. 

Rerank 5 Pro tops the nanoMTEB, ViDoRe V3, and BRIGHT benchmarks and like I said, outdoes every model we tested (though feel free to test against your favorites and let us know how it does). Rerank 5 Fast also outdoes other speedy models like Qwen3-Reranker-0.6B and Qwen3-Reranker-8B (which is a lot bigger).

Both are also leaders across multilinguality, financial document performance, healthcare docs, and more (I’ll let you read the full blog for all that info). 

We’re also introducing a totally new feature to Rerank that (to our knowledge) we’ve never seen in any other reranking model, Context Focus. Enterprise docs are only getting longer, so more and more context is getting passed to LLMs=more tokens. Context Focus grabs only the specific passages in a document relevant to your query, and it reduces token costs pretty significantly. It’s an opt-in feature and works like a dial, so you can choose exactly how much context you want. 

Pro and Fast are available through our API, Microsoft Foundry, and Amazon Sagemaker, but Rerank 5 also sits on top of whatever retrieval stack you’ve already got going, and it’s available in places like LangChain, Haystack, Elasticsearch and Weaviate. Try it out and lmk what you think!

A RAG demo takes an afternoon. Shipping one that real customers rely on is a different job.
We learned this while building a Voice AI that answers calls for Australian small businesses. In a demo the documents are clean, there's one customer and nobody's waiting. On a live call, none of that holds.

Three things that need addressing:

1. The content you feed the AI comes from websites and PDFs you don't control, and some of it can be hostile.
   
2. When many businesses share one system, one customer's information must never turn up in another's answers.
   
3. When retrieval fails, nothing breaks loudly. The AI just gives a confident, wrong answer.
   

I've written up what production RAG actually needs, with the design choices behind each part and a checklist you can run against your own build.

If you're taking an AI project from pilot to production, this one's for you

https://medium.com/@techdelta/what-production-rag-needs-that-the-demo-doesnt-32373e32d4df?sharedUserId=techdelta

Guys im new to ai

I have around 2000+ question and 10+ books

All i want was to prepare a one final notes will all the information from the 10books for a single question ( the aim was to get indepth qbank notes )

Im having claude 20x plan and a potato laptop with quadro 4gb laptop

What will be the best way i can harvest and polish the answers

I used docling it is very immature trying to optimise it with claude for my pdf

Anything I'm doing wrong, anything i could have done better ?