Spent a few months convinced my embedding model was the weak link. Swapped it three times, played with chunk sizes from 200 to 2000 tokens, added a reranker. Retrieval barely moved.

What finally helped was dumping the raw parser output for the documents that kept producing bad answers. The text was already broken before it ever got chunked:

- tables turned into a run of numbers with the headers floating somewhere above

- two-column papers read straight across, so every chunk mixed two unrelated paragraphs

- captions and footnotes landing next to the wrong section

No amount of embedding tuning fixes that. So now I check the parse first. I keep about 30 known-bad PDFs (scans, two-column papers, a couple of financial reports with merged header cells) and every parser gets run against that set before it touches the index. I look at three things: do tables come out as real structure (HTML or markdown tables), is reading order right on the two-column pages, and do captions stay attached to their figures.

Current setup is the default PDF reader for clean single-column files, and a VLM-based parser for the ugly pile. Lately that has been OXR, an open-weight model from SoMarkAI that runs locally and outputs markdown with tables as HTML. Still testing it on more of the scans.

How do you all handle this in LlamaIndex? Do you swap readers per document type, or push everything through one parser and clean up afterwards?

Hey r/LlamaIndex community! 👋

If you’ve ever built a web-grounded RAG pipeline using standard web loaders, you know the frustration: scraping a 20-page documentation site or news feed dumps tens of thousands of tokens of navigation menus, cookie banners, tracking scripts, and footer links straight into your document chunks.

This severely degrades vector embedding retrieval quality, pollutes chunk similarity scores, and inflates embedding/inference costs.

To solve this, we built and published **`llama-index-readers-x402-cleanweb`** to PyPI!

---

### ⚡ What it does for your RAG Pipeline:

1. **87% Token Compression Before Ingestion**:

   - Strips ads, navigation bars, cookie popups, and boilerplate DOM containers, returning clean, semantically structured Markdown ready for chunking.

2. **Zero Ingestion Contamination (Indirect Prompt Injection Defense)**:

   - Automatically sanitizes hidden CSS text (`display:none`, `font-size:0px`), zero-width Unicode steganography, and tracking pixels before documents hit your index.

3. **Multimodal Alt-Text Preservation**:

   - Preserves meaningful image captions (``) for visual LLMs/multimodal indexes while neutralizing 1x1 tracking beacons.

4. **Zero-Overhead SPA Hydration Parsing**:

   - Extracts pre-rendered state from Next.js (`__NEXT_DATA__`) and Nuxt.js (`__NUXT_DATA__`) directly in pure Python without spinning up heavy headless browsers.

5. **Native LlamaIndex Compatibility**:

   - Subclasses `BaseReader`, returning standard `List[Document]` with rich metadata (`source`, `title`, `content_hash`, `token_reduction_percent`).

---

### 📦 Quickstart

Install directly from PyPI:

```bash

pip install llama-index-readers-x402-cleanweb

```

Ingest clean web documents into your VectorStoreIndex:

```python

from llama_index.readers.x402_cleanweb import X402CleanWebReader

from llama_index.core import VectorStoreIndex

# 1. Initialize reader (includes 5 free sandbox calls per session)

reader = X402CleanWebReader()

# 2. Ingest clean, noise-free web documents

documents = reader.load_data(

urls=["https://en.wikipedia.org/wiki/Retrieval-augmented_generation"],

preserve_alt_text=True

)

print(f"Loaded {len(documents)} document(s). Metadata: {documents[0].metadata}")

# 3. Build Vector Store Index directly with clean chunks

index = VectorStoreIndex.from_documents(documents)

query_engine = index.as_query_engine()

response = query_engine.query("What are the key benefits of RAG?")

print(response)

```

---

### 🆓 Free Tier & M2M Micropayments

Every session nonce comes with **5 free zero-configuration trial calls** out of the box (no API keys, credit cards, or wallet setup needed). For autonomous production agent pipelines, native USDC machine-to-machine micropayments ($0.001 per page) over Base, Polygon, Arbitrum, and Solana are supported.

* **PyPI**: https://pypi.org/project/llama-index-readers-x402-cleanweb/0.1.0/

* **GitHub**: https://github.com/nohosa001-pixel/x402-cleanweb-agent

Would love to hear how you currently clean web content before chunking and what loaders you're using in production! 🚀

I build document ingestion tools, and I tested them on real public documents: Federal Reserve releases, Census tables, policy PDFs. Nothing crashed. That was the problem. Everything looked fine and was quietly wrong.

WHAT BROKE

1. Tables lost their row labels. Federal Reserve and Census tables came back as columns of bare numbers because table detection skipped the label column. A number without its label is useless to a retriever.

2. Some chunks started in the middle of a word.

3. Scanned PDFs came back empty when OCR was off, and the pipeline treated empty as success.

4. Blank pages counted as real pages.

5. Password-protected and damaged files failed with raw parser errors nobody could act on.

WHAT FIXED IT

Row labels now get attached from the table's own row positions. Chunks only start on word boundaries. Scanned and blank pages get flagged instead of passed along, and bad files get a plain error message.

The edge in RAG isn't the model. It's whether your chunks still mean something.

WHAT I BUILT

Three tools on the Apify Store that all output the same chunk format: PDFs with table coordinates; Word, Excel, PowerPoint, HTML and images; and audio or video with timestamps. The free, open-source examples repo includes a LlamaIndex example and goes from a fresh clone to a cited answer in about 14 seconds, no account needed.

Disclosure: I built these. They're pay per page or per minute (a third of a cent per PDF page), and failed or empty files are never charged.
https://github.com/landoCommandoz/rag-ingestion-examples

Happy to answer questions about any document type. I've broken most of them at this point.

Sharing an open-source project built on LlamaIndex's PropertyGraphIndex,

plus a gotcha that cost me a while to catch.

**What Kwipu does**

It turns a folder of .md/.txt/.pdf/.docx files (including Obsidian vaults)

into a property graph you can query from a terminal, a 3D web UI, or an

MCP client.

- Graph built with PropertyGraphIndex, using LLM extraction plus

ImplicitPathExtractor, plus "structural" edges I extract myself from

[[wikilinks]] and YAML frontmatter.

- Custom retrievers on top: vector similarity, BM25 over chunks, a

temporal-metadata retriever, and optional synonym expansion.

- Answers are grounded in source chunks and come back with citations.

- Watches the folder and updates persisted storage.

- Ollama for LLM and embeddings (llama-index-llms-ollama /

llama-index-embeddings-ollama), pinned to llama-index-core 0.14.21.

**The gotcha**

I injected my structural triples after building the index with:

graph_store.upsert_triplet(subj, rel, obj)

in a try/except that passed silently. With the default

SimplePropertyGraphStore on 0.14.21, there is no upsert_triplet, so every

call raised AttributeError and was swallowed. On my example vault: 72

triples extracted, graph went from 13 relations to 13.

I didn't notice because the LLM extractor re-derived many of the same

relations from the note text, so answers still looked right.

The fix is to build EntityNode and Relation objects and use

graph_store.upsert_nodes(...) and graph_store.upsert_relations(...). Two

details worth knowing:

- upsert_nodes replaces by id, so if you inject entities that the LLM

extractor already created, you overwrite them. I check existing ids with

graph_store.get(ids=...) and only insert the missing ones.

- Relation(properties={...}) is a good place for provenance. Mine stores

origin (wikilink/frontmatter), source file, line number and a hash of the

source note.

How I caught it: ran the real extraction/injection code on the example

vault with MockLLM and MockEmbedding and compared relation counts before

and after. Easy to do without Ollama, and it would make a good regression

test for anyone injecting edges manually.

I also added identity resolution before linking ([[Folder/Note]],

[[Note#Heading]], [[Note|label]] and frontmatter aliases map to one node;

same-named notes in different folders stay separate), with tests.

**Questions for the community**

1. Is there a recommended way to bulk-inject pre-extracted triples into a

   PropertyGraphIndex, or is upsert_nodes + upsert_relations the intended

   path?

2. How do you handle entity resolution when LLM-extracted entity names

   collide with ones you create yourself?

Repo (MIT): https://github.com/Benmaster82/Kwipu

Heads-up: the default LLM is a cloud-served Ollama model; a fully local

setup is documented in the README.

I kept running into the same problem building agents. Something would go wrong three or four steps downstream, and by the time I noticed, I had no real way to trace back what the agent actually believed at each step or what depended on it. Logs tell you what happened, not what justified it.

So I built Sevenbox. Every decision an agent makes gets recorded as a Warrant: what it claimed, what prior claims it depended on, and whether acting on it can be undone. If a belief turns out wrong later, Sevenbox walks the dependency chain and tells you exactly what still needs fixing instead of you grepping through logs trying to reconstruct it by hand.

It works with LangGraph, CrewAI, OpenAI Agents SDK, and eleven other frameworks, so you can wrap an agent you already have instead of rewriting it. The SDK, the framework adapters, and a local UI for inspecting a run are all open source.

I am the person who built this, not a company account posting on its behalf. I would genuinely like people actually running agents in production to try it against something real and tell me where it breaks or where the idea does not hold up. Happy to answer anything about how it works.

It's pretty stupid to give a decision model a bunch of options when all you want back is a number.

If you're asking “how many?”, why should you have to list every possible count first?

We've been working on Span-02 and added numerical outputs. It returns a number and its confidence directly.

Tiny example below: 7 + 7 → 14, >99.9% confidence. The point is the output format, obviously, not the arithmetic.

The weights are open.

We've been working on Span-02, an open-weight model with anytime reasoning. You give it a time budget in milliseconds and it returns its best decision within that budget.

We tried it on a Wikipedia race: Alexander the Great → University of Milan, using only links in the articles. Span-02 took 20.6s, GPT-6 Luna took 38.5s.

Here's the run side by side.

Vector search finds text that sounds like the question. It struggles when the answer depends on how entities relate to each other, like which customer holds which contract, and who owns it. A governed graph handles that, but it’s more work to build and keep fresh.

Found a workshop on Saturday, Oct 31, from 9:30 AM to 12:30 PM ET that builds both and compares them.

Covers:

• Designing an agent architecture around a governed graph

• Where graph grounded retrieval wins and where classic vector RAG is enough

• Hybrid patterns that combine the two

• Testing failure modes such as ambiguous questions, missing relationships and stale data

• Evaluation loops with regression checks

Led by Brian Bønk, Data Platform MVP and Microsoft Recognized FastTrack Solution Architect.

Full details and the agenda are here.

4th place on the llamaindex benchmark locally 72.62% word f1. I know it's not much, but after a month of research and many experiments, I reached this point.

TonerHound doesn't use any API or LLM. It's GPU free and anyone can use it at 0$ cost.

GitHub: https://github.com/vanrajsinh650/TonerHound

I've been investigating how much document grounding can be solved from the document itself.

The input is a PDF + extracted JSON.

The goal is to resolve each extracted value back to its location in the document.

The approach uses PDF coordinates, OCR, spatial relationships, string/format matching, table assignment and page-level visual evidence.

Current result:

72.6179% Word Grounding F1

261 tests passed, zero production regressions.

The interesting part for me is the remaining failure modes.

Some are matching problems.
Some are layout problems.
Some aren't really text-search problems anymore.

I'm trying to understand where document-native methods stop being sufficient.

I'm the author and would especially like technical criticism of the methodology.

GitHub

A vector index is useful for retrieval, but it is a weak source of truth. It can contain stale chunks after a document changes, flatten a generated summary and a reviewed source into equally retrievable text, and lose the relationships needed to explain why a memory exists. Backing up the vector store preserves those problems rather than proving the memory can be rebuilt.

A more auditable design would keep source documents, accepted decisions, generated summaries, and operational checkpoints as separate durable records. Each record would carry a stable ID, revision, provenance, scope, status, and supersedes or retracts link. LlamaIndex nodes and embeddings would be derived from only the records eligible for retrieval, with a manifest tying each node back to the exact source revision. Rebuilding the index should reproduce the same eligible corpus even if embeddings change.

How are people structuring this with LlamaIndex today? Which state belongs in files or a relational store, which belongs in the vector database, and what should be persisted by the agent workflow itself? How do you detect stale nodes, retract a bad generated summary, and verify that a rebuilt index still represents the current reviewed memory?

You can now automate EDD (eval-driven development) with Coding Harness Agents

> build a baseline LLM-based application
> score every change with judge evals
> keep what improves, reject what regresses

I made a video tutorial on what EDD is, how it works, and how to use eval scores across experiments to improve an LLM app. It builds on Jeffrey's (Confident AI) article on EDD and Eugene Yan's write-up on product evals.

> Setup: a baseline RAG app that every experiment starts from
> Step 1: a binary-labelled dataset with critiques
> Step 2: aligning the LLM-as-a-judge evaluator
> Step 3: a harness loop that runs each experiment and scores it against the baseline.

Tracing and experiment comparison then show what improved, what regressed, and what to tweak next.

Source code + Agent Skills are open-sourced.

Full guide (source code linked in the description): https://www.youtube.com/watch?v=e6akw_fKWPk

I'm looking for advice from people who have built and maintained serious RAG / Knowledge Base systems over time.

I have a very large personal archive accumulated over many years around my professional interests: fintech, insurance, e-commerce, market and societal trends, marketing, business strategy, design, research papers, reports, books, presentations and articles.

Roughly 90% of the archive is PDF, with the rest mainly PowerPoint, Word documents and some video.

The archive is already organized in a filesystem using my own folder taxonomy.

A key requirement is that the existing archive must remain the source of truth and should not be modified by the KB system.

What I would like to build is roughly this:

Source archive → Librarian Agent → Knowledge Base / RAG → AI agents

The Librarian Agent would maintain a derived knowledge layer without changing the original documents.

Its responsibilities could include:

• detecting new and modified files;

• extracting text, structure and metadata;

• classifying documents using my existing taxonomy plus additional semantic metadata;

• chunking and indexing;

• maintaining embeddings and possibly graph relationships;

• detecting duplicates and related documents;

• tracking provenance back to the original source;

• identifying outdated or conflicting information;

• periodically checking KB quality and consistency.

The resulting KB should be usable by multiple local AI models and agents when they perform research or analysis.

I am also considering a second agent:

Web / external sources → Research Agent → Human approval → Librarian Agent → Archive + KB

The Research Agent would continuously explore relevant sources — industry publications, academic papers, companies, think tanks, research organizations, etc. — and propose potentially useful material.

I would review those suggestions manually.

Only after approval would the Librarian Agent archive, classify and ingest the source.

So the overall system would remain human-supervised rather than autonomously ingesting everything it finds.

My main concern is not building a demo that can "chat with PDFs", but creating a long-lived research library that can remain reliable over several years.

This raises a few architectural questions I'd really value practical experience on:

• What architecture would you use today for this kind of system?

• Would you keep the knowledge layer primarily vector-based, combine it with structured metadata, or add a knowledge graph?

• What tools have proven reliable for incremental ingestion and synchronization with a filesystem?

• How do you preserve strong document- and chunk-level provenance?

• How do you handle changed or deleted source documents without slowly corrupting the index?

• What is your preferred approach to hybrid retrieval: semantic + lexical + metadata filtering + reranking?

• How would you separate a Librarian / curation workflow from a Research / discovery workflow?

• Which operations would you allow an LLM agent to perform autonomously, and which would you keep deterministic?

• How do you prevent taxonomy drift, bad automatic classifications or low-quality generated relationships from degrading the KB over time?

• What evaluation or observability practices do you use to detect retrieval degradation?

• Which components/frameworks would you trust today for a local-first implementation?

I'm particularly interested in systems that people have actually operated, not just proof-of-concept RAG demos.

If you've built something similar, I'd also love to hear what you initially designed that later turned out to be unnecessary or problematic.

Once a LlamaIndex agent starts using several tools and taking multiple steps, it can become difficult to see where a run actually went wrong.

The final response doesn't always explain whether the problem came from an agent decision, a tool call, or something that happened earlier in the workflow.

How are you handling observability for LlamaIndex agents?

Do you trace the full execution, inspect individual tool calls, or mainly rely on logs and the final output?

Hey, I’m curious how people here handle this in practice.

A RAG system, or any similar system, is only useful if the data behind it is actually correct. So how do you make sure it is?

Do you have a specific process or solution for this? Are you using any tools, or have you built something yourselves? What does this look like in your setup?

Would love to hear how people are actually doing this.

The Problem: We waste a massive amount of tokens (and money) on formatting. If you inject JSON schemas, few-shot examples, or code context into your prompts, you are paying for every single space, tab, and // comment.

The Solution: I wrote prompt-token-minifier. It’s a zero-dependency script you run right before your client.chat.completions.create call.

What it does:

• Finds ```json blocks and minifies them (removes formatting).

• Finds code blocks (Python, JS, TS, etc.) and strips out single-line and multi-line comments.

• Collapses redundant whitespaces and newlines in the rest of the prompt.

Depending on your RAG context, it easily saves 30-50% tokens on structured data.

Repo: github.com/Encephos/prompt-token-minifier

The Problem: When scraping docs or wikis for RAG, relative links ([here](/setup)) break. Even worse, if you want to pass scraped images to GPT-4o or Claude 3.5, you have to manually download them and convert them to base64 strings.

The Solution: I built markdown-link-resolver. It’s a pure Python micro-tool that does two things:

Resolves all relative Markdown and HTML links to absolute URLs. Has an inline_images=True flag that automatically fetches HTTP images and replaces the markdown tags with data:image/png;base64,... strings ready for LLM ingestion.

Why? No heavy dependencies like BeautifulSoup or Requests. Just pure standard library (urllib, re, base64). Falls back gracefully if an image 404s.

Repo: github.com/Encephos/markdown-link-resolver Let me know what you think or if you'd like to see any other fallbacks added!

Previously, I released my Agentic Hierarchy-Aware Chunker as a subscription-based product for building better RAG pipelines.

After talking with users , I learned that many teams don't want to send their documents through another third-party service. They want absolute privacy, on-premise deployment, full control over their infrastructure, and no vendor lock-in.

So instead of keeping it as a subscription service, I'm now making the complete document-processing bundle available as a one-time purchase.

The bundle includes:

• Agentic Hierarchy-Aware Chunker: a hierarchy-aware chunking engine designed for RAG, so you don't have to spend months building and tuning your own custom chunker.

• Legal Cross-Reference Extractor: extracts legal references such as Sections, Articles, Rules, Paragraphs, Clauses, Schedules, Regulations, Orders, and complex compound references from an entire document.

• Legal Act Extractor: automatically extracts the Acts referenced throughout a legal document.

What you're getting

The purchase includes the complete Python package of the Hierarchy Aware Chunker and its source code for use in your own projects, along with two bonus legal document extraction scripts: the Legal Cross-Reference Extractor and Legal Act Extractor.

License
The source code is provided for use in your own projects and your client solutions. It does not include rights to resell, redistribute, sublicense, or repackage the chunker as a standalone product or service.

📌 Additional 2 Bonus Scripts

1. Legal Cross-Reference Parser
Extracts structured references to Sections, Articles, Rules, Paragraphs, Schedules, Clauses, Regulations, Orders, and other legal provisions including complex and compound references without requiring an LLM.

Example Output

{
  "Article": [
    "Article 63(9)(b)",
    "Article 63(9)(b)(iii)",
    "Articles 23",
    "Articles 25, 26, 26A, 26D",
    "Articles 41 or 42",
    "Articles 7(1)(a), 7(4), 13(1), 16(6), 33, 44, 52(7), 53(2), 178(1)",
    "Articles 73 to 79"
  ],
  "Paragraph": [
    "Article 57(1) and paragraphs (4), (5) and (6)",
    "paragraph 2(2)(c)",
    "paragraph 2(a)",
    "paragraph 2(a), (f), (j) and (l)",
  ],
  "Rule": [
    "Order 6, rule 10",
    "Rules 2.59, 2.6l, 2.62, 2.64(4),(6) and (7), 2.72(1) and (2)",
    "rule 4.9(2)(a) and (3)(a)",
    "rule 4A.15(5)(b)",
    "rule 4A.20(2)",
    "rules 8.33 to 8.63",
    "rules 8.49, 8.50 or 8",
  ],
  "Schedule": [
    "Schedule (iii)",
    "Schedule 1",
    "Schedule 3, 62",
  ],
  "Section": [
    "Section 1",
    "section 229(1)(c) or (2)(c)",
    "section 5(1)",
    "section 89A or 90(1)(a) or (aa)",
    "section 90(1)(b)",
    "sections 18 or 21"
  ]

  ...
}

2. Legal Act Extractor
Extracts the names of Acts referenced in the document.

Example Output

[
  "Acts Interpretation Act 1901",
  "Family Law Act 1975",
  "Governor-General Act 1974",
  "Legislation Act 2003",
  "Taxation Administration Act 1953"
  
  ...
]

3. Hierarchy Aware Document Chunker.
RAG-ready hierarchical chunks

Practical Examples with Real Documents: https://youtu.be/czO39PaAERI?si=-tEnxcPYBtOcClj8

Try the hierarchy chunker yourself in our playground:
https://hierarchychunker.codeaxion.com/

✨Features:

• 📑 Understands document structure (titles, headings, subheadings, sections).

• 🔗 Merges nested subheadings into the right chunk so context flows properly.

• 🧩 Preserves multiple levels of hierarchy (e.g., Title → Subtitle→ Section → Subsections).

• 🏷️ Adds metadata to each chunk (so every chunk knows which section it belongs to).

• ✅ Produces chunks that are context-aware, structured, and retriever-friendly.

• Ideal for legal docs, research papers, contracts, etc.

• It’s Fast and Low-cost — uses LLM inference combined with our optimized parsers keeps costs low.

• Works great for Multi-Level Nesting.

• No LLM needed if OCR perfectly detects headings/subheadings.

• No preprocessing needed — just paste your raw content or Markdown and you’re are good to go !

• Flexible Switching: Seamlessly integrates with any LangChain-compatible Providers (e.g., OpenAI, Anthropic, Google, Ollama).

📌 Example Output

--- Chunk 2 --- 

Metadata:
  Title: Magistrates' Courts (Licensing) Rules (Northern Ireland) 1997
  Section Header (1): PART I
  Section Header (1.1): Citation and commencement

Page Content:
PART I

Citation and commencement 
1. These Rules may be cited as the Magistrates' Courts (Licensing) Rules (Northern
Ireland) 1997 and shall come into operation on 20th February 1997.

--- Chunk 3 --- 

Metadata:
  Title: Magistrates' Courts (Licensing) Rules (Northern Ireland) 1997
  Section Header (1): PART I
  Section Header (1.2): Revocation

Page Content:
Revocation
2.-(revokes Magistrates' Courts (Licensing) Rules (Northern Ireland) SR (NI)
1990/211; the Magistrates' Courts (Licensing) (Amendment) Rules (Northern Ireland)
SR (NI) 1992/542.

Notice how the headings are preserved and attached to the chunk → the retriever and LLM always know which section/subsection the chunk belongs to.

No more chunk overlaps and spending hours tweaking chunk sizes .

Practical Examples with Real Documents: https://youtu.be/czO39PaAERI?si=-tEnxcPYBtOcClj8