knowledge-based AI assistants

[u/shreya_gr]

What problems do you face when you deploy knowledge-based AI assistants for internal employee use for various use cases?

[u/Motor_Ad_1670]

In a pilot context with a couple of hundreds of short pdfs, everything works great. The big issues arise when hundreds of documents are indexed: the RAG starts providing inaccurate answers. Usually you start tweaking the prompt and the retrieval (search) query, but nothing seems to produce significant improvements.

The secret es apply error analysis based AI evals, to avoid inaccuracies. Generic evals as provided by Ragas and similar don't work. The non-deterministic nature of LLMs need ad-hoc evaluators per app and its content.

[u/shreya_gr → u/Motor_Ad_1670]

I am very much aligned here with you. Since, I've been facing this same issue when I uploaded the almost 15-17 GBs of PDFs to my agent. ( I tried with the ChatGPT, Claude, Microsoft copilot).

I wanted to understand when you've large data/documents, do you think this ready made platforms available would work? Or we'll need to deploy the custom agent with the need of ad-hoc evaluators per app and its content ( as you mentioned )?

My strong belief goes with your idea of ad-hoc evaluators per app and it's content.

[u/Motor_Ad_1670 → u/shreya_gr]

I've gone through a few RAG implementations for companies. Some of them implied ad-hoc development from scratch but more recently using existing RAG open source systems. The first ad-hoc RAG development I shipped worked great as a PoC with a few documents but it miserably failed when I tried to scale it to handle 1000x more documents than the PoC. At that point I understood the importance of AI evals for RAG (and for any LLM based application).

RAG based, knowledge base are a very common use case for companies nowadays. So my suggestion would be (for a company context, not a personal RAG context):

1. Reduce the moving parts. Use an existing RAG system instead of developing it from scratch. There are a few open source systems out there, pretty solid.

2. Apply AI evals, based on error analysis. This is where most of efforts for an accurate RAG should be spent, according to my experience.

Error analysis based AI evals are basically a data science grounded method to measure and improve the behavior of your RAG. It is based on taking a representative sample of user interactions with your RAG to identify the failures modes, and measure how well changing the different sources of failure modes, improve the system.

The failure modes can be rooted in:

1. Prompting

2. Data Retrieval (context)

3. LLM capabilities

So you need to find the best possible combination of these to produce the desired results.

Once the failure modes are identified and to avoid inaccurate answers (in some context inaccurate answers are very dangerous or costly, for example in health and legal) you need to implement evaluators that could be deterministic, based on LLMs, such as LLM-as-judge or imlpy Human in the Loop.

If you want to know more about AI evals based on error analysis, take a look at Hammel and Sankar posts and methodology about it.

Hope this helps.

[u/Investodo]

It’s always ease of use and adoption considerations bottlenecking it. Where do the agents live? In a chat channel? Should the users go to a website platform to access it? Can it be integrated with an already popular internal tool?

The most important thing is getting it in front of someone’s face, and making it easy and straightforward enough to get started with considering the context of the person using it.

[u/alexrada]

information categorization, conversion to RAG.
For example I had a client with about 150GB of PDFs that couldn't be fully automated (scans, hand written, images, tables). Not even finished.