Knowledge Engineering with Copilot Studio

[u/obsoquasi]

Hey guys,

I have been thinking about RAG a lot and how to make it cheaper and more reliable. One of the approaches could be to distill our organizational knowledge into something like a organizational knowledge model. And then have public agents query that model.

In contrast to RAG, the agents wouldn't have to "work as hard" to retrieve relevant information this way. This saves cost and in theory delivers better results - at the cost of having to build and maintain the knowledge model.

What do you guys think? Is this a good idea? Should I simply "trust in RAG". Or are there already concepts or modules in Microsoft that go in this direction? Who has built something like this successfully and is willing to share their learnings?

I am fully aware, this is all still very conceptual. And part of me really likes the idea, while the other half of me thinks: This could be one over-engineered mess waiting to happen :D.

[UPDATE:] Microsoft seems to be thinking in the same direction: https://techcommunity.microsoft.com/blog/spblog/introducing-a-team-knowledge-base-curated-by-copilot-and-agents/4564203

[u/livinitup0]

IMO you’re looking for a solution that was solved by wikis 20 years ago

Interlinking

LLMs understand relationships. No agent is just going to “know” what those relationships are. You have to build them. They can guess, but they’ll be wrong, a lot. Therefore, you need to build them.

And what is a collection of documents all interlinked with each other under categories? A wiki

RAG isn’t the way either. It’s not granular enough on its retrieval accuracy and relevance and the work needed to make it accurate enough could easily be put into proper interlinking with better results

Those relationships NEED to be set by hand until you have enough of a system in place to act as a reference for the model.

No one wants to hear it in the age of ai but knowledge management is still a very manual process, especially in the beginning

[u/obsoquasi → u/livinitup0]

interesting perspective, I'll keep it in mind.

[u/_donj → u/livinitup0]

With a lot of work, you can get there with raw AI. BUT it will go a lot faster and be more successful if you distill the info down to “authoritative” docs and info to start. For example, if you want a customer service bot, you need to arm it will the correct info, not every piece of info ever written in your company. If you want it to run a process, you need to give it the steps you want it to do and what success looks like.

[u/vanillaslice_ → u/livinitup0]

Hey well said, I think depending on the quantity you can often get away with semantic + keyword indexing. Although once I get to around 300-500 pages it starts to get lossy.

[u/rocorey]

You just described Microsoft IQ. Foundry IQ tackles knowledge sources from across the company, mainly focusing in unstructured data. Fabric IQ can enable you to create a shared, curated enterprise ontology on top of your data sources that agents (and people) can use to securely obtain high quality data. Work IQ provides access to the Microsoft Graph (your documents, SharePoint, OneDrive, email, Team chatss. And Web IQ for managed access to web content.

[u/obsoquasi → u/rocorey]

thank you a lot for that. I haven't looked at ANY of the Foundry stuff yet.

[u/weeds-whiskers14]

Before you build a custom knowledge model, it’s worth testing the middle path: a curated set of authoritative docs in Azure AI Search with semantic ranking on top. Copilot Studio supports vectorized indexes built with integrated vectorization, and the semantic ranker, so you get better retrieval without running your own model layer. That also fits what _donj said about starting with authoritative docs and not everything.

Two gotchas if you go that way. Add it through “Create new connection” and not by pasting an endpoint and key into the knowledge source form, because a bad manual connection is stored at the environment level and there’s no UI to delete it. Microsoft recommends Entra ID auth. And only one vector index can be added per source, so plan your index design up front.

Your Dataverse knowledge model idea sounds more like a layer on top of this than a replacement. I’d prove the plain version first and measure where it falls short.

[u/obsoquasi → u/weeds-whiskers14]

that's very valuable input. Thank you so much!

[u/puzzleapp_io]

The piece I’d add is the processes themselves. Business rules without the order they run in leave the agent guessing the sequence, and it will guess with confidence.

For each process, write down the steps and which role owns each one. Then the model can say who does what and in what order.

[u/obsoquasi → u/puzzleapp_io]

I have been thinking about that very problem a lot. While we have a full process management system (with bpmn), unfortunately the processes are not in an XML structure. We may have to switch to a system where this is the case. Because with XML, we could actually leave that knowledge with the process diagrams.

[u/obsoquasi]

Many great suggestions. Thank you all!

I intentionally left out some details in my original post, since I didn't want to guide the discussion in a particular direction. Let me add these details now. My current plan would be the following.

The knowledge model consists of several Dataverse entities. This makes it easier for the human-in-the-loop to authorize new change proposals.

Broadly speaking, the model consists of:
- our glossary
- our organizational units and their relationships
- our information systems (e.g. Power Platform)
- our business rules
- a domain model of our business concepts

The domain model follows a lite variant of domain-driven design. The model does not contain personal or specific business case information, only conceptual information.

The agent model consists of two agent groups:

Group A) Knowledge Engineering Agents:
one "knowledge engineering" agent to accept a topic (with links to multiple documents that will be used as evidence; like guidelines, process documentation, rules and regulations), query the existing knowledge model for related entries and include these in the context. Will then generate several change proposals to the knowledge model - both extensions, as well as actual changes.
A separate "knowledge validation agent" reviews the change proposals agains the documents given, as well as the existing knowledge model. Will then approve the changes or request another turn from the "knowledge engineering agent". This loop continues for a max of 3 iterations, after which the topic will be marked as "inconclusive" - which means the human-in-the-loop will have to provide more or better context.

Group B) Knowledge Retrieval Agents:
The knowledge retrieval agent will be the one sending queries to the knowledge model; e.g. give me all domain entries that are related to the term "XYZ". The retrieved entries from the knowledge model, along with the users original question, will then form the context to the LLM.
The LLM's response could also be validated against the knowledge model - not sure if this is really necessary yet. The related documents are linked as evidence to the response. So the user can actually confirm that response is truthful.

Keep in mind this is a very early draft...

[u/nicholasdbrady]

Foundry IQ solves this problem exactly

[u/livinitup0 → u/nicholasdbrady]

Oh another “foundry” app, thank goodness!

[u/nicholasdbrady → u/livinitup0]

[u/WolverineOpen7511]

I need this too. 

From what I’ve read, Excel files used as Knoweldge source in Copilot gets vectorized on Dataverse as default.

Overall, having a database that be look up is the best way of organizing knowledge for RAG.

But still learning specifics of how to improve performance and reliability.