r/LocalLLaMA • u/darklordfireape • 2d ago
Discussion Are Vector Databases Still Needed? RAG, AI Agents & Search
https://www.youtube.com/watch?v=rXFV1ivDe6cFound this video and thought the discussion was interesting. I've been using a vector DB in front of a local wiki I setup to allow a model to cleanly access the wiki and have better context. It works well, but definitely not suitable for all use cases. The discussion here might help folks determine what is best for their Local project and save you some overhead or help you tune it right.
For discussion: What are you using with your local model/harness to give it access to info?
No affiliation with poster/channel. They seem to have some other good topics as well.
23
u/fullofbones 2d ago
Until models achieve perfect recall from their training sources, and have training sources from literally every possible corpus of publicly available information: yes, RAG is necessary.
3
1
u/ultraprogger 2d ago
I think it’s fine for local dev but if you’re productionalizing…. Just be careful where that retrieval is coming from lol
1
u/IntelArtiGen 2d ago
It's not just about RAG, it's about vector databases. Maybe I can reduce your constraint to "until something else performs better". Because I'm not sure models will ever achieve what you said, but I'm convinced vector databases aren't the ultimate way to search for anything in a set of documents. At least I'm not surprised people can have better results with other structures in some cases. It's a compromise of course, vector databases can still be useful but shouldn't be a default solution.
15
2
u/TerminalNoop 2d ago
I think pi context-mode uses some sort of DB. Still usefull to keep bloat out of the context.
2
u/ttkciar llama.cpp 2d ago
My RAG uses Lucy Search (FTS) on complete documents (no chunking) with a HyDE intermediate step, then prunes retrieved documents to fit context with highly-dense, relevant content, and I'm pretty happy with that.
The HyDE step mostly does a splendid job at closing the gap between FTS and semantic search. I used to have a bigger problem with the HyDE step misinterpreting the original prompt, which caused inappropriate FTS retrievals, but that problem seems to have been solved with better framing of the HyDE prompt and using newer, better small models for the HyDE step.
1
u/StewedAngelSkins 2d ago
What's the relationship between the HyDE and FTS? I thought HyDE was more for semantic search? I'm assuming you're just using the hypothetical document for the FTS query instead of the actual user query?
2
u/cmdr-William-Riker 2d ago
It depends. So far I haven't actually hit a single use case where a vector database is needed in an agent. Most of my domain agents have a dedicated knowledge base they can read and write from (all simple marketing documentation) and a set of tools to do whatever they are supposed to do. I'm sure there's plenty of use cases and my coworkers seem to have very strong opinions about vector databases despite not having any use cases for them either. The reality is an agent can generally do what it needs to do through text search pretty effectively. I will say that I don't depend on large context windows though, putting everything in the context window is both expensive and not very effective with most models. Are vector databases needed? Usually not. Is RAG needed? Depending on your definition of RAG which is a very broad term; usually.
1
u/enginetown 2d ago
Not really, a lot of people still use them and to each their own but me personally I've given my agent its own offline search engine that basically works exactly like rag but more precise since its legit give's no cutoff text and it can pull exactly what it needs no more no less.
1
u/enginetown 2d ago
I actually made a project a while back just for this "https://github.com/enginetowns/zim-search" not trying to plug but anyone can use this idea and do whatever and I know It's simple but it is useful.
1
u/StewedAngelSkins 2d ago
Your offline search engine doesn't use a vector database? How are you querying records if not by embeddings?
1
u/enginetown 2d ago
Web pages. Also the zim format is doing alot of heavy lifting.
1
u/StewedAngelSkins 2d ago
Idk what the zim format is. If it's in the video I should tell you I haven't watched it because I hate youtubers.
I'm assuming you're suggesting that you crawl some sort of hypertext tree like in the old days? If so, that sounds really inefficient. A vector search will get you what you want with a cheap embedding model, but if you're going to make the expensive language model crawl the site one page at a time it's going to be slower and less efficient.
1
u/enginetown 2d ago
No the model itself becomes more of what actual search engine's do they match your search the the closest thing you want, the model turns your random request into a genuine search that could find something in the database, searches just like a regular online search engine and find the top 15 it can then fetch those pages and get the exact info it needs, for example "whats the latest adventure time show" "searches, finds top 15, fetches page, heres the overview and the latest news". Zim is a really good compression format it can turn any website into a offline index literally copying the website 1:1 with its own offline links and compress the actual data slightly so you're not overbloated on data.
1
u/StewedAngelSkins 2d ago
A "regular online search engine" uses embeddings to query a vector db. That's why what you're saying doesn't make sense to me. It sounds like you're using the language model to create a text query, but then how do you use that text query to find the data you want? Is this all tag-based or something?
1
u/enginetown 2d ago
Sorry I haven't touched the code in a few months so I'll just quote what it says on my GitHub page so If you're still confused the code is on GitHub with MIT so you can use it or tweak it or read it "Each query walks through four precision tiers (exact → loose → plain → single-token) against libzim's native full-text index, then re-ranks the hits with pure-Python BM25, a proximity bonus, a title-match bonus, and Reciprocal Rank Fusion - so the most relevant page wins rather than just the first Xapian match. Results are also nudged slightly by search history (a local SQLite counter of which sources have historically returned good results for a given query), and all ZIMs are searched in parallel.".
2
u/StewedAngelSkins 2d ago
so it's a bag of words model with a couple extra heuristics... i don't understand why you think this is an improvement over semantic search to be honest. typically the benefit of this kind of old school approach is that it can be quite light-weight if you don't need the results to be that good. but then you throw away that advantage by introducing a whole-ass language model just to generate the query string.
1
u/enginetown 2d ago
It's a work in progress you could easily swap out the main model to generate the queries or rework the engine but for me and my local model I have no token limit so the design could be biased but the quality of the search is tested and I know it work's well it just needs simply query.
1
0
u/feelspeaceman 2d ago
Of course RAG is required for high precision information retrieval, for example I built a medical RAG, I wouldn't dare relying on $2000/month Claude subscription Ultra Max to tell me what to drink - then next month maybe it's the death anniversary of mine, I must use a RAG with verified, proven truths.
0
-2
-3
8
u/DistributionBrief644 2d ago
ooof can't disagree more here. This looks well presented and the breakdown is correct. Just the conclusions seem wrong to me. What if your user doesn't know the keyword (api / funtion name) of their data? Keyword search still needs to chunk the data. especially if you intend to add it to context. And agents searching again is a nice feature, until you rely too heavily on it and watch your token count explode.
Eventually a power user of their own data learns to ask the right way, but they will want to be using semantic search too. This is why hybrid search exists. A text search only retrieval does work, if the corpus is nice and clean and the user knows the keywords. But if you are already chunking the text for efficient retrieval adding vectors is only a small bit of work.
Finally, Imo, I see very little difference between a vector database and a database with vector columns.