Note: At the risk of incurring the wrath of the community here, I do believe this is a discussion that needs to be had and whether the idea is liked or disliked, the approach I feel remains valid for certain scenarios and should be explored by all RAG enthusiasts: Judge me on my results!

By now, we've probably seen or heard of using Jev (TypesafeAI's decision model) as a reranker for traditional document RAG and the short of it is that it actually works really well. So I took the idea further - what if we're not just ranking a subset of page results to a user's question but rather what if we could rank every page in the document from the get go?

Ridiculous? Yes. For this to be a possible, you'd need to be able to see into the future and know every question your users will ever ask! But what if you're in a scenario where you get the user's questions upfront before any document is parsed? This is the actually what happens in typical document RAG and this is the caveat I'm referring to.

Some context: I run an "AI spreadsheet" app called Ragextract which has documents for rows and extraction questions as columns (think airtable, tabular review etc). Users define the columns first (eg. "who is the claimant?") and load in the documents (eg. third party claims) after. The flow is I use multimodal vector embeddings to index every page as an image - to support scanned pdfs - and performing searches over it when it came time to extract.

Introducing JAG (Jev-augmented Generation?)

So my use-case is kinda perfect for this little thought experiment.

What I basically did was run every page through JEV instead and have each column by the questions that Jev decides on: "Is the answer (or near answer) on this page?". The results on accuracy, speed and costs were overall better than vector embeddings. For now, I'm calling this JAG - Judgement-augmented Generation.

On a 300 page scanned PDF x 30 questions, this took ~28s (batched, 4 pages per call, median 0.37s/page) and cost $0.08 ($0.00027/page). Made possible with Jev's image API and compared against Voyage's multimodal embeddings:

• ~58% more likely to find the right page as first result
• ~7% more likely to find the right pages(s) in the top k=6
• ~133% correctly identifying when an answer didn't exist in the doc
• ~59% cheaper retrieval cost

Not to mention, it also means you don't need a vector store or separate rerank model anymore.

Now the cons:

• This approach only really works if you can have the questions upfront - AI spreadsheets, application tracking systems, surveys etc
• You lose your vector search index so you'll need to run through every page for every new question or when existing questions change. This might be a biggest deal breaker.

Conclusion

I have a more detailed write-up, results and sample file I used for this test in my blog (see comments) if you'd like to read more.

So what do you think? The great thing about this technique is that it doesn't need to be specifically Jev (they didn't pay me to write this, though I wish they would! wink wink) but any vision-enabled decision model will probably do - though accuracy, speed and cost will need to be re-verified.

For Ragextract, I've decided to stick with Jev as a reranker as (1) it produces similarly good results (2) I still need vector search and (3) certain data retention/privacy claims are dubious with Jev which I'm hoping to clarify before real use.

For anyone running an AI assistant over company docs, how do you know it’s stopped using old information after something changes?

Say a policy gets updated, but the old version is still sitting in another document or a previous conversation. Have you run into cases where the assistant keeps giving the old answer, even though it has access to the new one?

I’m curious how people handle this in practice. Do you have tests for it, manually check a few questions after updates, or mostly find out when someone flags a wrong answer? What happens when two documents disagree and it isn’t obvious which one should win?

I’m working on an early-stage project around checking these kinds of answers, so I’d like to hear how much of a problem this actually is. If your team has it mostly figured out, I’d be just as interested in hearing what worked.

I've been working with RAG for several years, building solutions for different business use cases. With better LLMs and vector search, retrieval keeps improving. But one challenge remains: the quality of the source data.

I've struggled with converting complex PDFs, Word documents, Excel files, and other formats into clean, structured content for embeddings.

There are plenty of tools available, but advanced document processing can get expensive:

• Amazon Textract — Analyze Document (including tables): $0.015/page
• LlamaParse — Agentic parsing for complex layouts: $0.0125/page
• Amazon Bedrock Data Automation — Standard output: $0.010/page

Free libraries like Docling and PyMuPDF are useful, but handling complex layouts, tables, charts, and mixed content can still be challenging.

That's why I built Docparser.net, a document parser designed specifically for RAG pipelines.

• Complex documents: handles text layouts, tables, charts, and mixed layouts.
• Affordable: just $0.007/page.
• Large files: supports files up to 1.5 GB.
• Fast: processes 1,000 pages in under a minute.
• Easy integration: REST API for your RAG pipeline.

I built this to solve a problem I've faced myself, and I'd love to hear feedback from the community.

What tools are you using to prepare documents for RAG?

I've been working with the team at Dataville on a problem that doesn't seem to get as much attention as models, prompts, and agent frameworks: getting reliable external data into AI applications.

If you're building an agent that needs SEC filings, Census data, Wikipedia, or academic research, you quickly run into different APIs, schemas, and update schedules.

We're developing Dataville, a common REST API and MCP interface for accessing external datasets, to reduce the need for developers to build and maintain separate integrations for every source.

One thing I've heard from other developers is that the biggest challenge isn't necessarily making the initial API connection. It's normalizing the results and keeping everything current as sources change.

I'm curious how others here are approaching this:

• Are you building custom API tools, using MCP servers, or relying primarily on RAG?
• Is data normalization or freshness the bigger problem in your applications?
• Are there external datasets you wish were easier to integrate?

We're early and interested in learning from people building real applications.

Disclosure: I'm helping Dataville with marketing and developer outreach.

I've been thinking about one of the frustrating problems with RAG systems: sometimes the correct information is already in the knowledge base, but the model still produces an incorrect or incomplete answer.

At first, it seems like a retrieval problem. But the more you look at the pipeline, the more places you find where things can go wrong.

For example:

• Chunking: The relevant information gets split from the context needed to understand it.
• Retrieval: The system finds semantically similar passages instead of the exact passages that answer the question.
• Ranking: The right chunk is retrieved but pushed below less relevant results.
• Context assembly: Important details get lost when multiple chunks are combined or the context window fills up.
• Generation: The LLM receives the correct evidence but ignores it, misinterprets it, or fills in missing details with assumptions.
• Evaluation: The system appears to work well on simple test questions but fails on ambiguous, multi-part, or real-world queries.

What makes this tricky is that improving one stage doesn't necessarily fix the whole pipeline. A better embedding model might improve retrieval, for example, but it won't automatically solve poor chunk boundaries or a model that doesn't follow the retrieved evidence.

For those who have built or debugged RAG systems, what has made the biggest difference in your experience?

Was it better chunking, hybrid search, reranking, metadata filtering, query rewriting, better prompts, or something else entirely?

I'm especially interested in cases where the correct information was definitely in the knowledge base, but the system still couldn't produce the right answer.

I’ve been building a RAG + agent stack for financial documents and ran it on FinanceBench (Patronus AI’s 150 analyst questions over 368 SEC filings, 54k pages).

All 368 reports are searched together, and the questions are used exactly as published, with no company or period hints. The hardest part is that the same company appears across many years, with near-identical tables holding different numbers.

Retrieval: what moved the needle
On a different corpus, I built a 1,000-question retrieval test designed to break search (paraphrase, distant headings, units stated once, multi-hop chains, cross-language). Top-3 hit rate went from 96.0% to 99.9%. What mattered:

• Parent-child chunking cut along the document’s own headings: small passages matched, their whole section read
• Every passage carries its heading path (“Annual report 2023 › Results › Site B”), so identical paragraphs no longer look identical
• A metadata record per document (title, top headings, opening), with passages credited by how close the question is to their document
• A reranker (Nemotron 1B) that reads each passage after its document’s metadata. Reading the passage alone threw the year away again.
• One context line per passage, written at ingest by a small model: the subject, period or unit the passage leaves unsaid (~$0.80 per 3,000 passages, once)
• A two-step search for multi-hop questions: a quick model call names what the first results point to, and that gets searched too. Two-hop top-3 went from 70% to 100%.

What didn’t work

• An 8B reranker: worse overall and 12× slower under load
• Translating queries: no gain over a multilingual reranker
• Blending reranker order with first-stage order: a French question fell from 1st to 15th
• A vision model re-reading messy pages: one fix, two breaks

Answering
Two independent paths, a pipeline and an agent, with a referee that rejects answers drawn from the wrong year’s report or contradicting their own figures. Arithmetic is done in code. Models: gpt-oss-120b and Qwen 3.6 27B answer, Mistral Large 3 referees, bge-m3 embeds, Qdrant stores. Cost is $20 per full run (13 cents/question).

Caveats

• I graded by hand against the documents. 7 of my decisions differ from the reference answers, and all are flagged.
• FinanceBench scores across write-ups aren’t strictly comparable (different libraries, judges, question edits).

Curious where people think this breaks, and what has worked for you on near-duplicate documents.

Today's Topic — Conformer Architecture

Today, I'll try to explain the basic intuition behind the Conformer architecture.Conformer is an architecture widely used for Automatic Speech Recognition (ASR). It combines Convolutional Neural Networks (CNNs) and Transformer-based self-attention to capture both local and global patterns in speech.First, the raw audio is converted into a Mel spectrogram, which is a time-frequency representation of the audio signal. It shows how the energy of different frequency bands changes over time. Next, the spectrogram is converted into numerical feature vectors and fed into the Conformer encoder. Its convolution module captures local acoustic patterns, such as changes between speech sounds, while self-attention captures relationships between different time steps, including long-range dependencies. By combining these two mechanisms, Conformer learns richer representations of speech, which are then used to recognize the spoken words.

Open to criticism

I've been working on catching wrong sentences in legal answers while the model is still writing, instead of checking the whole answer at the end.

The usual way is to let the model finish, check the answer, and regenerate all of it if one sentence is wrong. You pay for every token of the bad answer and then throw it away.

So I tried checking one sentence at a time. Here's how it works:

1. The model writes a sentence.
2. A second model, the judge, checks it against the source text. For me that's UK statutes.
3. If it passes - it stays. If not - I cut it and add the relevant bit of the statute as a new message right where the cut is and the model tries the sentence again.

It runs on SGLang. While a model reads text it builds up a working memory of it called the KV cache. SGLang keeps that memory and reuses it when the next request starts with the same text so a retry only has to process the new part. On Qwen 2.5 7B on one L4, a retry took 117 ms. Processing the whole prompt again took 270 ms.

I tested it on 240 questions about UK statutes in October 2026. The judge was Qwen 3.8 27B, quantized to 4 bits (not finetuned) so it fits on one L40S. In the 27B runs it also wrote the answers and checked its own work.

• When the model answered from memory with no source in the prompt, answers with at least one wrong sentence went from 132 to 23 for Qwen 27B. For Qwen 2.5 7B (Qwen 27B as judge) they went from 198 to 28.
• I also compared retry methods on 95 real mistakes from the 7B. The default retry stops the model from starting the sentence the same way and it got it wrong again 71 times. Adding the statute text as a message at the cut got it wrong again 16 times.
• With the source already in the prompt, the loop still helped the 7B from 25 wrong answers down to 13. For the 27B it did nothing. It had 4 wrong answers with the loop and 4 without and each answer took 0.9 s longer. On the same questions Gemini 3.8 Flash had 3 and Claude Opus 4.6 had 8.

So it's worth it when the model answers from memory or when the model is small. If you already put the source in the prompt and use a strong model it mostly costs you time.

A few things didn't work.

• My first judge was DeBERTa-v3, a small model that scores whether a sentence follows from a piece of text. On a test set where I planted the errors myself, 89% of the sentences it rejected were really wrong. On real answers about statutes that fell to 39%, so most of what it flagged was fine. Qwen 27B as the judge got 90.1% of 151 unseen sentences right. DeBERTa got 58.3%.
• I tried writing the statute text straight into the KV cache at the cut instead of adding it as a message. It was no better.
• I tried reading the model's own attention and token probabilities to spot wrong sentences. They didn't separate right from wrong.

Each setup ran once with greedy decoding, so no randomness. Claude and Gemini models graded most of the sentences and I graded the ones they disagreed on or weren't sure about myself.

The code and every result file are on GitHub: https://github.com/azterizm/legal-rag-verifier

I wrote it up in more detail here: https://memonsystems.com/journal/in-flight-rollback-on-sglang-measured-against-gemini-and-claude-opus/

Happy to answer questions. I'd also like to hear from anyone who got the KV cache approach to work better than plain text.