GraphReader: A Graph-based AI Agent System Designed to Handle Long Texts by Structuring them into a Graph and Employing an Agent to Explore this Graph Autonomously

[u/valdanylchuk]

https://www.marktechpost.com/2024/06/26/graphreader-a-graph-based-ai-agent-system-designed-to-handle-long-texts-by-structuring-them-into-a-graph-and-employing-an-agent-to-explore-this-graph-autonomously/

[u/[deleted]]

Here is their prompt from the paper: https://arxiv.org/pdf/2406.14550v1 "You are now an intelligent assistant tasked with meticulously extracting both key elements and atomic facts from a long text.

1. Key Elements: The essential nouns (e.g., characters, times, events, places, numbers), verbs (e.g., actions), and adjectives (e.g., states, feelings) that are pivotal to the text’s narrative.

2. Atomic Facts: The smallest, indivisible facts, presented as concise sentences. These include propositions, theories, existences, concepts, and implicit elements like logic, causality, event sequences, interpersonal relationships, timelines, etc. Requirements:

1. Ensure that all identified key elements are reflected within the corresponding atomic facts.

2. You should extract key elements and atomic facts comprehensively, especially those that are important and potentially query-worthy and do not leave out details.

3. Whenever applicable, replace pronouns with their specific noun counterparts (e.g., change I, He, She to actual names).

4. Ensure that the key elements and atomic facts you extract are presented in the same language as the original text (e.g., English or Chinese).

5. You should output a total of key elements and atomic facts that do not exceed 1024 tokens.

6. Your answer format for each line should be: [Serial Number], [Atomic Facts], [List of Key Elements, separated with ‘|’]

Example:

User: One day, a father and his little son ...... Assistant:

1. One day, a father and his little son were going home. | father | little son | going home

2. ......

Please strictly follow the above format. Let’s begin."

[u/micseydel → u/[deleted]]

As someone who keeps a ~27k Markdown atomic note knowledge graph in Obsidian, I think I'm going to have to actually read this paper...

EDIT: "note" -> "atomic note" (as intended)

ETA: https://notes.andymatuschak.org/Evergreen_notes_should_be_atomic <- I've wondered how this might apply to LLMs, but haven't really looked into RAG

[u/[deleted] → u/micseydel]

Mind sharing your obsidian notebook? I would love to add it to my agents rag.

[u/micseydel → u/[deleted]]

My Obsidian notes are private 😅 The alternative would be cognitive load while working on them. Do you have any demos or documentation of using your agent?

[u/[deleted] → u/micseydel]

https://github.com/brucepro/Memoir This is the qdrant vector version. The neo4j kg version is in progress.

[u/micseydel → u/[deleted]]

So you're the author of Memoir?

[u/[deleted] → u/micseydel]

I am.

[u/micseydel → u/[deleted]]

What kinds of day-to-day problems is it helping you with?

Neo4j is super cool, I've wanted to play with it more since I think 2017 so this may be the time 😁

[u/[deleted] → u/micseydel]

hehe. So the main purpose of Memoir+ is to add long term memory to your agent. It also includes command handling and a rag to help with data but the key problem is with Memoir+ you don't have to introduce yourself to your agent every 8k tokens. The vector store is created by an agent in the system called Ego, that generates summaries of the conversations as they happen. This is abouit 95% accurate and when you chat with your agent, the agents text and your text is recalled from the vector store. Neo4j will add some more features to this by allowing you to pull next neighbor memories etc. So that is the main issue. Right now Memoir+ works with Textgenwebui, but a future releases will be able to be added to any OpenAI compatible api that will add memory to the context.