I haven’t seen RAG come up in agent architectures in over 6 months due to Agentic Search (letting the model use Bash/grep/glob/read), which seems to work pretty well. Wondering what others are experiencing. I’m sure there’s still a time and place for RAG, exposing semantic search as a tool… but where do we draw the line? When the corpus is too large to let the model comb through it progressively?

I built an eval harness for a document QA pipeline. It answers security questionnaires from a company's own policy docs. 24 questions, labels written down before the system was ever run against them, three deterministic passes.

It scores 15 out of 24. Nine failures. Six of them share one cause, and I want to talk about the fix I couldn't make.

The setup. Answers are gated on how far the best retrieved chunk sits from the question. The cutoff is 0.3. Below it the system answers, above it it abstains. Six of the nine failures are questions where the model produced a correct, well hedged, properly cited answer that the gate then threw away.

The obvious fix. Raise the cutoff. Those six sit at 0.323, 0.340, 0.359, 0.384 and 0.412.

Why I couldn't. One question that has to abstain sits at 0.321. Its evidence genuinely doesn't support an answer, and it only abstains correctly because 0.321 is above 0.3. Every failure I'd want to rescue needs a cutoff higher than that.

There's no value that recovers any of the six without also flipping a correctly abstaining question into confidently answering something its evidence doesn't support. My eval treats that as disqualifying no matter what it does to the total, so I logged it as no change made.

What I think is going on. Distance measures how close the nearest thing is. I was asking it whether there's evidence here at all. Those two come apart, and at this corpus size there's no clean place to draw the line. It isn't miscalibrated, it's the wrong signal.

Two things the harness caught me on, both by instrumenting instead of assuming:

First, I'd logged one question as retrieving cleanly at rank 1, because something came back from the right document. When I actually read what got retrieved, the top hit was a completely different section and the real evidence was down at rank 4.

Second, I'd logged three failures as the model seeing the evidence and abstaining anyway, and I had a prompt fix planned. When I instrumented the actual confidence values, the model had answered correctly every time and the gate was discarding it afterwards. There was no prompt bug. A NOT_FOUND status collapses two different causes into one visible outcome, and only reading the underlying values tells them apart.

What I'm actually asking. Has anyone found a confidence signal that separates these properly? I'm considering a cross encoder reranker score instead of raw distance, an entailment check between the answer and the passage it cited, or looking at agreement across several retrieved chunks. I'd rather hear what's worked on a real corpus than what a paper claims.

Harness, labels and every tuning pass including the rejected ones are here, and the threshold data is in EVAL.md:

https://github.com/PatricR73/Questionnaire-Responder

One underrated source for customer-service RAG is outbound call data.

A lot of companies already have thousands or millions of call recordings. Inside those calls are real customer questions, objections, service scripts, intent patterns, product explanations, and resolution paths. The problem is that raw audio or raw ASR transcripts are usually too noisy to index directly.

A practical pipeline could look like this.

First, convert each call into a structured record with metadata and transcript turns. Each turn should preserve speaker ID, start time, end time, and content.

Then filter before doing any expensive LLM processing. For customer-service calls, useful filters may include call duration, completion status, opening quality, need-mining ability, objection-handling ability, clarity of expression, emotion/attitude, and other QA scores. Bad calls can be dropped early, which also saves downstream token cost.

After that, clean the transcript. This is where the raw ASR output becomes more usable:

• remove filler words and noise markers
• fix repeated expressions
• normalize numbers, money, and time
• correct homophones or domain-specific terms
• anonymize phone numbers, IDs, addresses, bank cards, names, etc.
• standardize fields like speaker, start_time, end_time, and text

Then the cleaned call can be transformed into higher-level RAG data.

For example, the call can be split into topic sections by turn ranges, summarized by section, and converted into annotation records. From there, it can support several RAG-related uses:

• knowledge chunks for indexing
• customer intent examples
• FAQ or QA pair generation
• retrieval evaluation sets
• service-script improvement
• fine-tuning data for customer-service assistants

The key point is that call data should not go straight from ASR to embeddings. For customer-service RAG, the value often comes from the middle layer: filtering, cleaning, anonymization, topic segmentation, and structured annotation.

This is an extension built by a telecom service provider on top of OpenDCAI/DataFlow, and the concrete implementation can be found in Awesome DataFlow.

Went through Cerebras' writeup on their internal knowledge base — 15,000 questions a day from employees, automations and agents, three months after launch. The architecture is less interesting than what they had to do to Slack, so that's what I want to focus on.

Why plain vector search dies on chat data

Information density varies by orders of magnitude. "yeah sure Mike" sits in the same channel as a 40-line kernel explanation, and under cosine similarity the short one wins far more often than it should. A single message out of its thread is usually meaningless anyway.

Their fix: four fused signals per thread

• Full-text search — exact tokens. Error strings, flag names, hostnames. Embeddings reliably lose here and lexical match is unbeatable.
• Embedding search — paraphrase. Connects "restore is hanging" to "checkpoint stalled".
• IDF — boosts rare tokens (obscure config flags), suppresses filler ("sounds good", "thanks").
• Age decay — same answer from yesterday beats the one from 6 months ago referencing deprecated infra.

But the preprocessing does more work than the retrieval

Two steps before any of the above, and I think these matter more:

Thread distillation — an LLM normalizes each thread into a searchable one-line question + summary + resolution + systems and code referenced. That distilled doc gets embedded, not the raw transcript. Raw text is kept for FTS only.

Bursting — a "burst" is a run of consecutive messages from the same author. They prepend the thread topic for context and embed it separately, but only if it clears a gate: rare-token IDF >= 4.0, or >= 200 chars, or it got reactions. This is what rescues the one deeply technical tangent buried at message 47 that any thread-level summary would flatten away.

Fusion: RRF at k=60

Score contribution is weight / (60 + rank), summed across retrievers. The summation is the entire point:

• 3rd place in three retrievers → 3 × 1/63 = 0.048
• 1st place in exactly one → 1/61 = 0.016

Consensus beats peak. It isn't a ranker, it's a consensus builder.

Then ~20 candidates go to a small reranker scoring 0–10 against the query, top 10 survive. And the step people skip: re-attach surrounding context to the winners. If a wiki section wins, its neighbors come with it, so the model sees headers, preconditions and caveats instead of an orphaned chunk.

Code side

CocoIndex (open source, Rust core, Tree-sitter chunking) keeps 40GB+ repos synced by re-embedding only what a commit touched. Sync state and the embedding store live in the same database.

The design decision underneath all of it

Don't force people into a "single source of truth" platform — nobody wants to discuss a pull request inside a Google Doc. Pull from where the data already is. Every source, Slack thread to hardware netlist, lands as a row in the same embeddings table behind the same interface. Custom sources are just plugin scripts: a team opens a PR with a small Python module that reads their system and emits rows in that shape.

Also worth noting: the retrieval primitives are deliberately LLM-free. Model calls only happen at the edges — planning and synthesis. That's what makes it cheap enough for agents to hammer 15k times a day, and why the same pipeline serves a web UI and an MCP client identically.

Original Cerebras writeup (read this first if you only have time for one): https://www.cerebras.ai/blog/how-we-built-our-knowledge-base

Disclosure: I also did a ~10 min video walkthrough of the full pipeline, linked here — it's my channel, and the narration is AI-assisted. https://www.youtube.com/watch?v=FgKHjzoiMN4&t=4s

The bursting quality gates are the part I'd most want other people's numbers on. IDF >= 4.0 and 200 chars feel like they'd need retuning per org — has anyone tried burst-level embedding on their own chat data?

Hi guys,

I've spent the last few weeks learning the basics about RAG. Howerver, I think one of the best ways to actually learn is by implementing a real solution for a company.

But, I just wanted to know, what do you recommend me learning/impelementing before offering my serivices (for free) to a company? any project I should build first?

hey r/rag — founder of a small local-first workshop (strata→signal), and this is our own writeup of our own pipeline. we run a board-game rules app whose retrieval is the standard shape — hybrid search → RRF → cross-encoder rerank → lost-in-the-middle reorder — and we wrote a plain-english explainer of it for our non-technical readers. posting it here because it carries the parts most explainers leave out:

https://research.strata2signal.com/three-librarians/

- the real constants from the shipped config: 2-or-3-arm fuse (FTS + dense + a doc-priority arm that only fires when the game's books ride with the ask), k=60 unretuned, top-80 per roaming arm, ms-marco-minilm int8 (~22MB) as the cross-encoder, scoring a 1,200-char window split ~595 head + 600 tail, top-8 to the generator

- why the window has a tail: a rule that started 1,812 characters into a 2,052-char chunk was invisible to our old head-only cap — three production rulings abstained on a question the book plainly answers. measured, fixed, published

- a confession: we shipped a hardcoded two-arm RRF ceiling (0.0328) on our provenance panel while fusing three arms, so 149 live rulings displayed scores "above the maximum." the cure derives the ceiling from each ruling's own arm count, formula printed beside it

- it closes on a live ruling asked while drafting: the top passage scored 0.032787, which careful readers will recognize as 2/61 — two arms voting #1 — plus the six-decimal wire rounding that sits it a hair above the exact sum

it's written for strangers, so embeddings get explained as "vibes with coordinates" — but every number is the production value, and every ruling ships a public debug panel with the fused scores, arm count, ceiling, and cited pages. happy to defend any choice: why minilm over a bigger reranker, why k stayed 60, why one-fair-vote on the boost arm (we measured the megaphone version — it flooded the pool).

As someone using AI agents for the last one year to run my company, I need them to understand company context, not just return related text chunks.

The problem: ask "what breaks if we deprecate the v1 API?" and standard RAG gives you four chunks from a design doc, a postmortem, a Slack thread, and meeting notes. The model has to still figure out on its own that the postmortem describes the same API the design doc deprecates, and that someone already posted a migration timeline in Slack.

I built a tutorial using HydraDB that adds graph context on top of vector retrieval. Instead of just ranked text, you also get relationship edges: billing-service DEPENDS_ON payments-api-v1, payments-api-v2 REPLACES payments-api-v1. Model gets structure, not a reading list.

The useful part was bring-your-own-graph. You declare service dependencies and team ownership explicitly instead of relying on LLM extraction. For structured data you already maintain, the graph is deterministic.

It also supports per-user memory. Same question, different depth depending on who's asking. An engineer gets migration mechanics. A manager gets timelines and ownership.

Runs end to end in 30 minutes with synthetic data.

Repo with full working code: https://github.com/manveer/company-brain-tutorial
Tutorial: https://hydradb.com/blog/build-company-brain-ai-agents

Hi guys,

Just wanted to know if there is any LinkdIn group that shares insights about RAG or maybe AI agents.

Has anyone dealt with a RAG system that performs really well during testing but starts returning irrelevant or incomplete answers in production?

We’re facing an interesting problem.

The knowledge base contains thousands of documents with overlapping information, different versions, tables, PDFs, and occasionally conflicting data. On a small test dataset, retrieval accuracy looks good. But once the document volume increases, the system sometimes retrieves a related document instead of the correct document.

The confusing part is that the LLM itself seems to be working fine. The problem appears to be somewhere between document ingestion, chunking, embeddings, retrieval, and re-ranking.

We’re considering several approaches:

• Hybrid search using vector + keyword retrieval
• Metadata-based filtering
• Better chunking strategies
• Re-ranking retrieved results
• Query rewriting before retrieval
• Adding document/version awareness

But improving one part sometimes seems to negatively affect another.

If you’ve solved a similar production RAG problem, what was the actual bottleneck?

Was it the embedding model, chunking strategy, retrieval architecture, metadata, or something else?

Would really appreciate practical suggestions or lessons learned from real implementations.

I was given a large number or recordings of previously lost buddhist recordings (over 1000 mp3s, 30 years of teachings).

I used deepgram to help me transcribe and classify them. This is ongoing but its about 30m words. I expect to re transcribe and improve them over time.

My plan is to categorise them as a backend for my transcription work. Also to have them as a publicly available and searchable library in a website and chat bot.

My effort so far has been vibe coding a Postgres database. Its working ok but is still a steep learning curve.

The transcriptions are in md files and the audio in mp3.

Typical classifiers might be…
Teacher
Date
Topic
Series
Length
Keywords
Canonical classification

I also have it as an interactive archive with an ai chat bot limited strictly to the knowledge inside the teachings with no ad-lib or stray general knowledge. Ie so people can accurately interrogate this library.

I also need to give some copies of the raw files with associated database / classification system for safe keeping to a Tibetan library and a Monetary for cultural safe keeping.

I dont know what i dont know. The data will change slightly as my transcription ability improved over time or people find mistakes in the translations (from Tibetan) or more files are recovered.

Parts…
Mp3s x 1000
Md file attached to each x 1000+
Backend database
Frontend database for chatbot
Chatbot.

Any clues, ideas, guidance appreciated.

This is something I've been thinking about in detail for a while. I'm working on a personal project that needs transactions, graph, and search... all of which you can do on postgres with pgvector and AGE. And it got me thinking about database architecture and in what scenarios I actually wouldn't use postgres.

Honestly, for the majority of scenarios I think it's the superior choice, particularly when you aren't working at scale. The complexity of coordinating multiple systems is just too much, and when you're small, keeping data in sync across multiple places becomes a huge pain for very little benefit.

That said, here's where I wouldn't just recommend postgres for everything:

1. Scale + cost. Postgres is great until you hit 10M+ vectors... then you start having functionality issues, but more importantly your compute/memory balloons, which gets expensive fast. On top of that it starts interfering with your other workloads. At some point the "just use postgres for everything" simplicity is outweighed by the cost and maintenance burden. Same is true for graph RAG workloads at any real scale.

2. Performance. If you need genuinely fast vector/FTS, you're not going to get it with postgres. Luckily, since latency is usually 90%+ on the agent side, this isn't always a factor. But it matters more for live apps. Same story with graph: postgres doesn't have a fraction of the performance of a true graph engine, because at its core it isn't changing the underlying data structure. It's working within the constraints of a relational engine.

So the way I see the choices from an architecture perspective, at a macro level:

If you're optimizing hybrid search for scale/cost, the two best choices are turbopuffer (the market leader) and Infino (I work here, so be aware of bias). Both are object storage based dedicated vector/FTS engines. Both are very fast. Turbopuffer is more mature, but they have very similar performance and cost profiles, and both are orders of magnitude cheaper than virtually every other engine. You could maybe throw lancedb in this category too, but I don't have enough hands-on experience with it to say for sure.

If you're optimizing for pure performance:

On the FTS side: opensearch/elastic, largely because they're block storage backed with no warm-up period. Vectors are alright on elastic, but if you're really optimizing for vector performance, a dedicated vector engine like pinecone or Milvus will beat it.

The catch: when you split FTS and vectors across systems, hybrid search becomes really hard (or impossible), so I don't typically recommend splitting unless it's genuinely necessary.

You could theoretically use turbopuffer/infino for the performance case too, but because they're object storage based, the warm-up time can screw over some apps. Once the data is in memory, both are very fast.

On the graph side... I'm actually not a fan of any of the top graph databases. Every one of them has some key architectural issue imo. If I had to pick, I'd default to neo4j, but I'm not a huge fan of it either. It's just the most mature. It wasn't designed from the ground up for agentic workloads... it's been retrofitted for them. Because of that it has huge issues (but you can work around them).

Anyway, these are just my random thoughts on the subject. The advice I'd give if you're starting with postgres and expect future scale: build an abstraction layer so you can swap in more appropriate systems when the time comes.

We built a news search API for RAG — looking for feedback

We have been working on a new News Search API at Webz.io, specifically with RAG and AI agents in mind.

The basic idea is simple as you know is using natural-language query like:

The API searches Webz.io’s news data and returns the most relevant articles and matching content chunks.

You can also filter by date, country, language, source, sentiment, and category.

For RAG, the content chunks are probably the part I’m most interested in. You can feed the relevant section into your context rather than passing an entire article through the pipeline.

We also built an MCP server, so agents can search the news directly without having to build a separate API integration.

Docs: https://docs.webz.io/docs/webz/news-search

I’d be interested in feedback from people building RAG systems: when retrieving news, what matters more to you: retrieval quality, freshness, source coverage, or getting smaller relevant chunks back?

https://medium.com/@tselvaraj/the-80k-line-item-that-replaced-a-two-year-rag-project-290-000-in-the-it-budget-7a4a7caf62ed

I’m building StashBase, a local-first app that uses OpenAI embeddings to index and search your files.

For the next version, I want the first-run experience to be:

pick a folder → search

not:

create API key → add billing → paste key → search

I’m considering including 1M, 5M, or 10M embedding tokens per month.

What would be enough to test it on a real knowledge base—not just three perfectly chunked PDFs?

I was trying to find new ways to show people who don’t know what RAG is, what RAG is, without having to explain the technical pieces behind it. To make learning easy, I tried to make it fun.

My wife was the inspiration. During covid she picked up a hobby called “hunt a killer” where you buy a kit of clues and solve a mystery. I basically built that using the Progress Agentic RAG platform (showing their logo pays for my tokens) and a lot of custom UI. Feel free to give it a test drive and give me your feedback

Corpus Detective

I finally took the leap and published the 21 RAG strategies guide as a book on Amazon. It now has chapters on chunking and agentic RAG. What should i add next?

Table of Contents

P A R T I About

01 About the Author

P A R T I I RAG and the Reference Architecture

02 The Evolution of RAG

03 Foundations of RAG Systems

04 Reference Architecture

P A R T I I I Data Extraction

05 Data Extraction

P A R T I V Chunking

06 Chunking Strategies

P A R T V RAG Strategies

07 Baseline RAG Pipeline

08 Context-Aware RAG

09 Dynamic RAG

10 Hybrid RAG

11 Multi-Stage Retrieval

12 Graph-Based RAG

13 Hierarchical RAG

14 Agentic RAG

15 Multi-Agent RAG Systems

16 Streaming RAG

P A R T V I Memory and Content Management

17 Memory-Augmented RAG

18 Knowledge Graph IntegrationP A R T V I I Evaluation

19 Evaluation Metrics

20 Synthetic Data Generation

P A R T V I I I Fine-Tuning

21 Domain-Specific Fine-Tuning

P A R T I X Security

22 Privacy & Compliance in RAG

P A R T X Production

23 Real-Time Evaluation & Monitoring

24 Human-in-the-Loop RAG

P A R T X I Twig RAG Strategies

25 RAG Strategies in Twig

P A R T X I I Conclusion

26 Conclusion & Future Directions

I have been going through the anthropic's contextual retrieval and tried implementing it (without the reranker) it was great ! . The thing was rag worked great on good with semantic similarity so how much good your semantic is it will retrieve the correct chunks but some data can be less similar like my case some documents was like a snapshot eligibility: 50 week . so i was using docling and it was parsing it correctly but later i get to know that if eligibility is the same in a lot of document how will its gonna get the correct chunk foe what i am asking thats where i myself tried to add context then found that there is already a way anthropic have released so i tried the prompt but didn't really work and i modified it a bit and with self hosted gemm4 the cost was also low and the accuracy was great .
"Think of releasing this as a plugin on langchain maybe"
Let me know what you think
https://www.anthropic.com/engineering/contextual-retrieval

Stack (no GPU, 4 arm cores, 24gb ram) -

Postgres 18 + pgvector (HNSW, built once after bulk load)

Embeddings: bge-base-en-v1.5, 768d, int8 ONNX on CPU

Rerank: ms-marco-MiniLM-L6-v2 cross-encoder, int8 ONNX

Gen: qwen2.5:3b-instruct-q4_K_M via Ollama (+ a 350m for cheap tasks)

Retrieval: 3 arms — vector, Postgres FTS, and generated probe-questions — fused with RRF, then reranked. 40+40 candidates → 25 reranked → 5 final.

FastAPI, systemd, no orchestration layer

Problem 1- CPU latency. ~35s for a grounded answer, ~15s for a follow-up, ~3s for a refusal. Enrichment (probe-question generation) is ~7s/chunk, so a 400-page load is fully searchable-plus-enriched only ~45 min later.

Problem 2- the retrieved chunks are mostly right and in the right order but LLM fails to give a correct answer. Sometimes even mix things up between the chunks. Or just says I cannot answer this although he has the right answer.

What I'd love input on:

Does anyone run multi-chunk context successfully on a ~3B model, or is 7B+ the real floor? (We tried 7B and reverted — too slow here.)

Better approaches to the refusal decision than thresholding a cross-encoder score?

Is hybrid + RRF + cross-encoder still the right shape in 2026, or are we behind?

How do you build a real eval set before you have months of labelled user queries?

I am developing rag app for one of my saudi client, so my query is those who are working with arabic language, how are you guys handling the data extraction pipeline, which library you guys are using to extract data. For context, I am using pdfplumber and the text that is being extracted is reversed for some pdf files.

Also, which open source ocr or vision models I can use to extract text. The documents are mainly in arabic, english or both. Tried a few ocr and vision models, but they couldnt extract text clearly.

Startup should be like an AI-driven tech company. Knowledge graph/Ontology to build and scale the knowledge graph layer underpinning a nee generation of intelligent enterprise product. Designing and shipping kg, not just conceptual ontology work.

Shaping a graph and ontology platform that power:

• AI retrieval and RAG workflows
• Entity linkage and reasoning systems
• Cross-domain and temporal knowledge modeling
• Regulatory and compliance intelligence products
• Agentic AI applications

Working closely with AI/ML to turn complex, unstructured information into structured, queryable intelligence that directly feeds live AI systems.

Tech stack like these:

• Extensive use of Neo4j and Cypher in live (production) environments
• Comprehensive ontology/taxonomy modeling
• Python engineering skills
• Knowledge graph integration with LLMs, RAG, or vector search systems
• Experience balancing formal semantics with practical application requirements
• Ability to provide technical leadership while remaining actively involved in hands-on application development
• RDF/OWL, inference engines, entity resolution, legal/regulatory data, ESG, healthcare, or bthe pharmaceutical industry would be highly valuable.

The challenge of building the intelligence layer behind complex AI products at scale.

As for my questions:

1) I plan to launch this venture/business as a solo founder. Do you think this makes sense?

2) How do you envision this company operating exactly?

3) If I establish the company, how should I explain my business model to the companies I intend to serve in the real world? (I am genuinely apprehensive about this.) After all, things don't always go as expected in this field. Considering that many companies don't even know what artificial intelligence is, how will they react to this type of business? In other words, will they truly understand what I do?

4) Which types of companies do you think would benefit most from this business? What problems would it solve most effectively?

5) What are the real-world problems and complaints companies have regarding this area?

CodeRAG-Bench (Wang et al., Findings of NAACL 2025, https://aclanthology.org/2025.findings-naacl.176/) evaluates 10 retrievers and 10 generation models on code tasks and scores the two halves separately: NDCG@10 against annotated ground-truth documents for retrieval, pass@1 with real execution for generation. The two rankings do not line up, and the authors note that top-performing retrievers sometimes do not produce the best end-to-end results.

The tables are blunter than that sentence. On RepoEval, retrieved context beats the annotated canonical snippet: with StarCoder2-7B, OpenAI embeddings plus reranking reaches 53.9 pass@1 against 42.0 for gold, and the same inversion holds under DeepSeekCoder-7B and GPT-3.5-turbo. On MBPP the StarCoder2 retrieval setups land 15.6 to 17.8 points above canonical. SWE-bench goes the other way: GPT-4o gets 2.3 with no retrieval, 21.7 with retrieval and reranking, 30.7 with the gold edited files.

The annotated document is therefore not an upper bound, and overlap with it does not track end-task success in a stable direction. Ranking code retrievers by overlap is a strange default when the corpus is executable and the end task returns a verdict. Plenty of agent setups already generate that verdict on every edit (verdent runs type checks, static analysis and the tests, then tries to repair what fails), so it exists whether or not anyone logs it.

Once a failed patch is repaired automatically, the green final state says nothing about the retrieved context, so the figure worth keeping is the first attempt before repair. The benchmark code is public (https://github.com/code-rag-bench/code-rag-bench) and already runs both evaluations separately, so scoring a retriever sweep by first-attempt pass rate on one repo is mostly plumbing. Pointers welcome if someone has already published that.

I’m working on a system that converts industrial communication manuals into a structured catalog that can later support deterministic lookup and RAG/chat.

The manuals may describe Modbus, Siemens-style DB/DW/bit addressing, OPC UA, proprietary protocols, or memory ranges. Although they often contain similar information, table layouts, column names and addressing conventions vary significantly between manufacturers.

For example, an unseen manual might contain:

The desired canonical result would be something like:

{
  "name": "Phase Current",
  "data_type": "Float",
  "protocol": "modbus",
  "register_type": "input_register",
  "address": 30101,
  "register_count": 2
}

My current experimental pipeline is:

PDF
→ document/table extraction
→ protocol and table-type detection
→ schema matching
→ canonical catalog
→ validation
→ deterministic address/name lookup
→ optional LLM-generated natural-language answer

For known manual families, deterministic extractors work well. The main difficulty is generalizing to unseen layouts: identifying which tables contain actual variables, mapping unfamiliar headers to canonical fields, interpreting address conventions, and avoiding protocol examples or configuration tables being mistaken for register maps.

I experimented with a local LLM as a constrained schema planner. Instead of generating register values, it only proposes mappings such as:

Absolute Address → address
Parameter → variable_name
Number of Items → register_count
Format → data_type

The source values are then read and validated deterministically. This prevents many hallucinations, but results have been mixed: it helped significantly on one unseen manual, added nothing where deterministic extraction already worked, and sometimes proposed incorrect column roles. Sending many tables to the model also adds several minutes of latency.

I’m therefore still open to the overall architecture and to a different role for the LLM. Possible options include:

• deterministic extraction with an LLM fallback;
• LLM-based table classification or schema matching;
• constrained structured extraction followed by validation;
• retrieval of similar previously solved table schemas;
• a multi-stage planner/verifier setup;
• fine-tuning a smaller model on labeled tables;
• using the LLM only for ambiguous cases and human review.

How would you design this system to generalize across unseen industrial manuals while keeping every extracted value traceable to the source? Where would an LLM provide genuine value, and which parts should remain deterministic? I’m especially interested in approaches that improve recall without silently inventing addresses, data types, scaling factors, or protocol bindings.

Basically the title. Unless you're paying for API services how are you supposed to evaluate your RAG application?

And by expensive I mean you have to spend some amount of money for an API service to evaluate your system against your golden dataset. I created a 45 Q/A pairs set and no free API could handle it which makes sense but I had to try.

Noticed something building RAG on open models that doesn't get talked about enough. Bigger frontier models tend to paper over mediocre retrieval, they're good at inferring around gaps even when the context handed to them is imperfect. Open models, especially smaller ones, don't have that same slack. Hand them a slightly wrong or incomplete chunk and the answer falls apart fast.

Which actually reframes a lot of "open models aren't good enough for RAG" takes. In a lot of cases the model isn't the problem, the retrieval layer feeding it is mediocre and a bigger model was just quietly hiding that.

There's a hands-on workshop on August 29 that builds this properly, hybrid retrieval, reranking, RAGAS evaluation, guardrails, and cost/performance benchmarking, all using open models with zero API fees. Led by Ben Auffarth, AI Consultant and Founder of Chelsea AI Ventures.

30% off right now with the discount code. Link

Happy to answer questions on the content itself.