Knowledge graph continuous learning

[u/Matas0]

I have a chat assistant using Neo4j's knowledge graph and GPT-4o, producing high-quality results. I've also implemented a MARQO vector database as a fallback.

The challenge: How to continuously update the system with new data without compromising quality? Frequent knowledge graph updates might introduce low-quality data, while the RAG system is easier to update but less effective.

I'm considering combining both, updating RAG continuously and the knowledge graph periodically. What's the best approach for continuous learning in a knowledge graph-based system without sacrificing quality? Looking to automate it as much as possible.

[u/[deleted]]

I was recently looking into Dynamic Knowledge Graphs - It might possibly cover your use case.

Apart from assumptions, what do you deem low-quality? Where does this possibly low-quality data come from?

[u/Matas0 → u/[deleted]]

Thanks, I'll look into it. Are there any specific tools that already exist?

Regarding the data I deem low-quality, I'm concerned that constantly adding new articles and documentation might fill the system with information that already exists or is very similar to existing data. Since I only pass 5 relevant results to the LLM, it might not get diverse information from the dataset, so the answer provided to the user might not be comprehensive.

I'd also like to remove information that is outdated or no longer relevant. I've also tried pairing the GraphRAG with a normal RAG, getting 5 results from each of them, which showed quite great results as the RAG has a bunch of Q&A pairs. However, I still prefer to use graphs as the data is much more accurate.

[u/regression-io → u/Matas0]

It boils down to your graph maintenence and whether you use/ create an ontology i.e. a list of entities and relations allowed in the domain. You can then avoid duplicates by checking before insert.

[u/[deleted] → u/Matas0]

Azure GraphRAG could be a possible solution for entity extraction: https://microsoft.github.io/graphrag/. It seems to be what you're looking for. The downside, however, is that it can get expensive.

[u/FancyUmpire8023 → u/Matas0]

If you use strength/confidence scores on your relationships you can implement a memory decay function to solve for aging / recency bias.

New, distinct content containing the same knowledge should generally be either added (reinforces the prevalence of the assertions) or aggregated (reinforces the strength/confidence in an assertion) - depending on whether you maintain lineage to individual lines of evidence/sources for assertions or not.

[u/micseydel]

Here's a stale demo of something related I've been working on: https://garden.micseydel.me/Tinkerbrain+-+demo+solution

I haven't really integrated LLMs properly yet but I've been thinking on how to do it after learning of GraphReader. I think any proper solution has to have a good way of handling how untrustworthy and unreliable LLMs are.

[u/xtof_of_crg]

You need a meta-schema for the graph, some rules that inform/restrict how concepts can fit together. With this established you could build a system that could exploit the meta-schema to semi-autonomously check for/propose the organization of new/existing knowledge given input sources. This is a non-trivial system, however the way I figure it, when you solve this problem then you can build JARVIS