This seems like its overloading the term knowledge graph from its origins. Rather than having information and facts encoded into the graph, this appears to be a sort of similarity search over complete responses. It's blog style "related content" links to documents rather than encoded facts.
Searching through their sources, it looks like the problem came from Neo4j's blog post misclassifying "knowledge augmentation" from a Microsoft research paper with "knowledge graph" (because of course they had to add "graph" to the title).
This approach is fine, and probably useful but its not a knowledge graph in the sense that its structure isn't encoding anything about why or how different entities are actually related. A concrete example in a knowledge graph you might have an entity "Joe" and a separate entity "Paris". Joe is currently located in Paris so would have a typed edge between the two entities of something like "LocatedAt".
I didn't dive into the code but what I inferred from the description and referenced literature, it is instead storing complete responses as "entities" and simply doing RAG style similarity searches to other nodes. It's a graph structured search index for sure but not a knowledge graph by the standard definitions.
Exactly. Glad to see this. I do think knowledge graphs are important to AI assistants and agents though and someone needs to build a knowledge graph solution for that space.
The idea of actual entities and relationships defined like triples with some schema and appropriately resolved and linked can be useful for querying and building up the right context. It may even be time to start bringing back some ideas from the schema.org back the day to standardize across agents/assistants what entities and actions are represented in data fed to them.
Yeah, one of the specific things I'd love to do is collaboratively bulking up WikiData more. It's missing a ton of low hanging fruit that people using an ML augmented tool could really make some good progress on, similar to ML assisted OpenStreetMapping work
This isn't going to work so long as WikiData is controlled by admins that misapply the notability criteria to delete information they subjectively consider trivial
I think the next frontier is a a wiki-style, collaborative site, deliberately purposed at storing information for LLMs
Yeah precisely. Knowledge graphs are simple to think about but as soon as you look into them you realize all the complexity is in the creation of a meaningful ontology and loading data into that ontology. I actually think LLMs can be massively useful for building up the ontology but probably not in the creation of the ontology itself (far too ambiguous and large/conceptual task for them right now).
How do we build ontology using LLMs? Will the building blocks be like the different parts of a brain? P.S I am assuming that by "creation of ontology itself" means creation of AGI.
If this paper overloaded the term, than I did the same in my recent EMNLP paper where I used th term âSemantic Knowledge Graphâ to refer to what I think youâre talking about.
https://aclanthology.org/2023.newsum-1.10/
Automatically created knowledge graphs using embeddings is a massively powerful technique and it should start to be exploited.
You can blame peer review for letting the definition of knowledge graph get watered down.
Also note that my coauthor. David, is the author of the first and still the best open source package for creating or working with semantic graphs.
https://github.com/neuml/txtai/blob/master/examples/38_Intro...
"Semantic Knowledge Graph" is even worse! The term is intended for the design of semantic networks with edges restricted to a limited set of relations. A knowledge graph is already about semantics!
Gotta say txtai seems like a useful tool to throw in my toolbox!
I apologize for having been part of the bastardization of a term. I think none of us know what the right term might be for the kinds of graphs we generate.
Are graph databases really relevant during retrieval? Does anyone use them to augment a vector store as a candidate source?
We're using in a large engineering use-case, 100s of millions of objects - and it almost doubled the nDCG@10 score vs pure vector search
This is quite significant. I suppose you did not use using the knowledge graph during training the embeddings?
While I'm 100% on board with RAG using associative memory, I'm not sure you need Neo4J. Associative recall is generally going to be one level deep, and you're doing a top K cut so even if it wasn't the second order associations are probably not going to make the relevance cut. This could be done relationally, and then if you're using pg_vector you could retrieve all your rag contents in one query.
I think there's a lot of cases where you don't want to just RAG it. If you're going for tool assisted, it's pretty neat to have agent write out queries for what it needs against the knowledge graph. There was an article recently about how LLMs are bad at inferring B is A from A is B. You can also do more precise math against it, which is useful for questions even people need to reason out.
I need to dig into what they're doing here more with their approach, but I think using an LLM for both producing and consuming a knowledge graph is pretty nifty, which I wrote up about a year ago here, https://friend.computer/jekyll/update/2023/04/30/wikidata-ll... .
I will say figuring out how to actually add that conversation properly into a large knowledge graph is a bit tricky. ML does seem slightly better at producing an ontology than humans though (look how many times we've had to revise scientific names for creatures or book ordering)
Yes, but this doesnât seem to be an actual knowledge graph which is part of the issue imho. If you look at the Microsoft knowledge graph paper linked in the repo it looks like they build out a real entity-relationship based knowledge graph rather then storing responses and surface form text directly.
I think it's relatively unlikely that having an agent write graph queries will outperform vector search against graph information outputted into text and then transformed into vectors.
The related issue that I think is being conflated in this thread is that even if your goal was to directly support graph queries, you could accomplish this with a vanilla database much easier than running a specialized graph db
Outperform in what way? There's some distinct things it already does better on like multi-hop and aggregate reasoning than a similarity context window dump. In general, tool-assisted, of which KG querying is one tool, does pretty good on the benchmarks and many of the LLM chats cutting over to it as the default.
> if your goal was to directly support graph queries, you could accomplish this with a vanilla database much easier than running a specialized graph db
Postgres and MySQL do have pretty reasonable graph query extensions/features. If by easier, you mean effort to get up a MVP, I'd agree, but I'm a bit more dubious on the scale up, as you'd probably get something like Facebook and Tao.
My initial thought was "building the knowledge graph is what LLMs and the embedding process does implicitly", why the need for a graphdb like Neo4j?
So your solution would be to fine tune the LLM with new knowledge? How do you make sure it preserves all facts and connections/relations and how can you verify during runtime it actually did, and didn't introduce false memories/connections in the process?
I think RAG has a lot to say here. New content / facts go through the embedding process and are then available for query.
I don't generally disagree that a more discrete (not continuous) knowledge base will be another component to augment ai systems. The harder part is how do you build this? (Curate, clean, ETL, query) Not sure a graphdb is the best first choice. Relational DBs can take you pretty far and it is unclear how many 1+N or multi-hop queries you'll need in a robust ai / agent system
I think you are misunderstanding. An embedding places a piece of knowledge in N dimensional space. By using vector distance search you are already getting conceptually similar results.
I dont know if you need a graphdb in particular but there are likely explicit relationships or entities to resolve to eachother that youâd want to add that arenât known by a general model about your use case. For example if you are personalizing an assistant maybe you need to represent that âJohnâ in the contacts app is the same as âJdubsâ in Instagram and is this personâs husband.
LLMs have a limited context size, i.e. the chat bot can only recall so much of the conversation. This project is building a knowledge graph of the entire conversation(s), then using that knowledge graph as a RAG database.
Exactly! With memary only relevant information is passed into the finite context window.
Very interesting, thank you for making this available!
At OpenAdapt (https://github.com/OpenAdaptAI/OpenAdapt) we are looking into using pm4py (https://github.com/pm4py) to extract a process graph from a recording of user actions.
I will look into this more closely. In the meantime, could the authors share their perspective on whether Memary could be useful here?
Very cool project! I think one of the main way (a bit orthogonal to what you do now) to adapt to GUI / CLI would be to develop an open-source version of something like Aqua Voice https://withaqua.com/
Perhaps it could make sense to add this to your effort?
Thanks! OpenAdapt already supports audio recording during demonstration (https://github.com/OpenAdaptAI/OpenAdapt/pull/346). Perhaps I misunderstood â can you please clarify your suggestion?
It's a kind of text input which mixes text and edition instructions, look at the demo
Whatâs the goal of creating a graph from the actions? Do you have any related papers that talk about that? We also capture and learn from actions but havenât found value in adding structure beyond representing them semantically in a list with the context around them of what happened.
The goal is to have a deterministic representation of a process that can be traversed in order to accomplish a task.
There's a lot of literature around process mining, e.g.:
- https://en.wikipedia.org/wiki/Process_mining
- https://www.sciencedirect.com/science/article/pii/S266596382...
Awesome, thanks.
Yes we are also on the process mining and RPA space and use image reco + ocr + click tracking for part of it. My (poorly worded probably) question was why do knolwdge graphs matter for you, since traditional rpa doesnât use them for all the cases Iâve seen at least, unless I misunderstood what youâre saying here about trying out graphs. Iâll read the LLM RPA paper you have linked here too, maybe that explains the use of graphs, and I havenât read this one so thank you.
We basically used the same technique some of the UIPath folks have used which is representing everything as a sequence of actions, which "branching" being represented by different linear sequences that the model can ingest and make decisions over which sequence to follow, which is kind of a graph i guess but not how we represent it.
These new systems would do well to have a compelling âwow, this solves a hard problem that canât be solved in another straightforward wayâ.
The current YouTube video has a query about the Dallas Mavericks and itâs not clear how itâs using any of its memory or special machinery to answer the query: https://www.youtube.com/watch?v=GnUU3_xK6bg
If you search about the Mavericks again (not included in the video) the agent will query the knowledge graph for results from prior executions.
I hate when I find a cool AI project and I open the github to read the setup instructions and see "insert OpenAI API key." Nothing will make me loose interest faster.
Unconstructive comment. OpenAI is the golden standard for an llm: if you cared to dig deeper youâd realize that that you really could incorporate another llm with little effort.
Most projects also give you the option of providing an base url for the API so that people can use Azure's endpoints. You can use that config option with LiteLLM or a similar proxy tool to provide an OpenAI compatible interface for other models, whether that's a competitor like Claude or a local model like Llama or Mistral.
Is the expectation for the lib (or project) to work with various vendors or you expect to just pay for tokens
You can easily incorporate llama 3 or other OS models
Looks cool. This is similar to what I'm doing for long-term memory in AISH, but packaged up nicely. Others have pointed out that they're somewhat abusing the term KG. But ... you could imagine other processes poring over the "raw" text chunks and building up a true KG from that.
Sounds promising. Can this system be integrated with the Wikidata knowledge graph instead?
Yes! You can easily swap knowledge graphs under the same agent. Would love to see this happen!
Log4j was so unbelievably slow to load data, bloated, and hard to get working on my corporate managed box that I wasn't too sad when it turned out unable to handle my workload. Then the security team asked me why it was phoning home every 30 seconds. Ugh.
I have since found Kuzu Db, which looks foundationally miles ahead. Plus no jvm. But have not yet given it a shot for rough edges. At the time, it was easier just to stay in plain application code.
Hopefully the workload intended by this tool won't notice the bloat. But it would be nice to be able to dump huge loads of data into this knowledge graph as well, and let the GPT generate queries against it.
Get a daily email with the the top stories from Hacker News. No spam, unsubscribe at any time.