Personal Knowledge Vault: AI Research Agent
AI Agent Augmentation For Helping Someone with Functional Memory Challenges
Trying to find useful knowledge across a myriad of repositories that people use to store their knowledge in files and documents is a challenge at the best of times.
Now imagine how frustrating it would be trying to find files relevant to a topic for someone with their functional memory impaired.
Imagine if they had an agent that you could give a single topic to, and have AI agents do the work of scanning, collating and summarizing baseline knowledge from a private knowledge vault combined with knowledge from Wikipedia for that topic, making it possible for them to focus on writing.
Thanks to work from the team at Langchain who created the Langgraph library for building stateful multi-actor applications, and work from Shao et al (STORM: Synthesis of Topic Outlines through Retrieval and Multi-perspective Question Asking) it was possible for me to develop an AI based topic research agent that generates Wikipedia-style articles using the Personal Knowledge Vault that I am working on to help a person I know with functional memory challenges as a source.
The Multi-Agent Systems (MAS) Pattern
When we were developing the Menome (https://www.menome.com) dataLink system, we used a Multi-Agent System (MAS) design pattern. This pattern involves developing a set of agents each with a specific capability that can then be linked into a process to accomplish an objective.
For Menome we created a framework called the ‘BotFramework’ that provided a pattern for connecting agents together into workflows using JSON messages to carry data from one agent to the next using RabbitMQ as the broker. Results were stored into a knowledge graph.
The pattern was very useful, flexible and quite a powerful method of addressing complex processing challenges, but it had to be manually configured for each use case. The agents were also not capable of adapting to different requests or conditions dynamically.
With the advent of LLMs, Agents have taken on a whole new set of very powerful capabilities that enables them to not only perform much more sophisticated tasks, but also to generate steps out required to accomplish an objective, derive plans to accomplish the objective, and execute the plan. While there is still manual configuration required, the AI based agent pattern is flexible within the overall process context it is designed to be used for.
There are several frameworks available that can be used to do this, autoGEN is one of the more prominent libraries, Llamaindex, another excellent AI framework, has DataAgents and Langchain has Langgraph. The new AutoGEN studio looks very promising, and I am planning to investigate it in the near future.
As an initial target though for the Personal Knowledge Vault, one of the key use cases I have is to be able to generate a basic literature review with researched references from a single objective leveraging a referenced public source, in this case Wikipedia and the articles and documents stored in the Personal Knowledge Vault.
As luck would have it, and the fact these days that whenever a thought arises code seems to manifest itself for said thought. No sooner did I think the thought that a paper ‘Assisting in writing wikipedia-like articles from scratch with large language models’ (STORM: Synthesis of Topic Outlines through Retrieval and Multi-perspective Question Asking) seemed to magically appear. The Langchain team on March 14 then posted a youtube video and jupyter notebook based code example for STORM implemented using Langgraph.
The Langgraph Storm agent, as designed by Shao et al., is a research assistant tool that extends the concept of “outline-driven Retrieve-And-Generate (RAG)” for producing richer article content. The Storm agent is particularly aimed at generating Wikipedia-style articles based on a user-provided topic. To achieve this, it utilizes two main insights which help in crafting more organized and comprehensive articles. The Storm model includes conversation simulation, which contributes to increasing the reference count and information density in the article, enriching the content.
I have modified the original jupyter notebook code to work in context of an endpoint inside Personal Knowledge Vault application. This means that the sources used are Wikipedia on the public research side, and my own highly curated Knowledge Vault on the private side in lieu of FreshWiki or duckduckgo or equivalent.
The result is the ability to have agents generate a literature review derived from knowledge I have written or collected in papers or articles I have read, identify additional reference topics for further exploration and research by authors using public sources, and collate this into a nicely formatted literature review style summary with referenced links to sources.
For example, if I want to do a literature review for a session I am doing for corporate executives next week on the subject of AI Risks in the Enterprise, I can now ask my Personal Knowledge Vault AI agent to do this:
It is pretty amazing watching the system work as it generates an outline, list of topics to research, a set of ‘editors’ with different perspectives to explore the topic provided from a number of different viewpoints.
[AIMessage(content="Yes, that's correct. I focus on the ethical considerations surrounding AI implementation in enterprises,
particularly highlighting the risks associated with biased algorithms, privacy concerns, and societal impacts.
Is there a specific aspect of AI risks in the enterprise that you would like to know more about?",
response_metadata={'token_usage': {'completion_tokens': 54, 'prompt_tokens': 216, 'total_tokens': 270
}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': 'fp_b28b39ffa8', 'finish_reason': 'stop', 'logprobs': None
}, name='Dr. AI Ethics')
]
INFO:root:Messages: [AIMessage(content='Thank you for sharing that insightful information on biased algorithms in AI systems. It is crucial to address biases in AI,
especially in enterprise settings where the decisions made by these systems can have significant impacts on individuals and society as a whole.
In your experience, have you encountered any specific case studies or examples where biased algorithms in enterprise AI systems have led to negative consequences or ethical dilemmas?',
response_metadata={'token_usage': {'completion_tokens': 76, 'prompt_tokens': 528, 'total_tokens': 604
}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': 'fp_b28b39ffa8', 'finish_reason': 'stop', 'logprobs': None
}, name='Dr. AI Ethics')
]
INFO:httpx:HTTP Request: POST https: //api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
[AIMessage(content='Thank you for sharing those examples of how biased algorithms in enterprise AI systems have led to negative consequences and ethical dilemmas across various sectors.
It is clear that addressing biases in AI systems is crucial to ensure fairness, equity, and accountability in decision-making processes.
In your research or experience, have you come across any best practices or strategies that organizations can implement to mitigate biases in their AI systems and promote ethical AI development in enterprise settings?',
response_metadata={'token_usage': {'completion_tokens': 86, 'prompt_tokens': 866, 'total_tokens': 952
}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': 'fp_b28b39ffa8', 'finish_reason': 'stop', 'logprobs': None
}, name='Dr. AI Ethics')
]
[AIMessage(content='Thank you for outlining those comprehensive best practices and strategies for mitigating biases in AI systems and promoting ethical AI development in enterprise settings.
These steps are crucial for organizations to ensure that their AI systems operate fairly and responsibly. I appreciate your insights and the valuable resources you have shared. Thank you so much for your help!',
response_metadata={'token_usage': {'completion_tokens': 63, 'prompt_tokens': 1517, 'total_tokens': 1580
}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': 'fp_b28b39ffa8', 'finish_reason': 'stop', 'logprobs': None
}, name='Dr. AI Ethics')
]
[AIMessage(content="You're welcome! I'm glad I could help. If you have any more questions or need further assistance in the future, feel free to reach out.
Good luck with your article on ethical considerations in AI implementation!\n\nCitations:\n\n", name='Subject Matter Expert')
]These are framed into a set of interview graphs used to explore the topics using the search source provided as the basis for the topic.
The results of the interview process are used to refine the original outline. This process is done with the larger context LLM — GPT4 in this case. Output of that process is then used to render an article.
I have to admit it is pretty amazing to watch the console output of the persona agents ‘discussing’ the research with each other. The system though is very slow due to the number of interactions with the LLM.
I’ll therefore be moving this over to the python Celery Async pattern now that I have it working synchronously.
The input from the Personal Knowledge Vault knowledge needs tuning as well. Due to the fact I am using a neo4j Knowledge Graph, I have a fair amount of latitude in terms of determining which aspect of the relevant knowledge identified by the agents to use as input to the article generation process.
I can use the baseline chunks, the page containing the chunk or the entire article. Using the entire article would increase the cost substantially as then I would be processing a lot more tokens through the OpenAI API, and its not likely the resulting summarized output would be better than using a more targeted page or chunk.
I am also summarizing the pages as part of the baseline graph document processing pattern, so I think using the page level summary associated with the identified chunk will be the optimal approach. In any event, I’ll setup a flag on the endpoint and on the UX, so the user can choose how much detail use for article processing.
Example:
For my example topic AI risks in the enterprise, converting the URLs from the references into a cypher query illustrates how the articles referenced did indeed come from my Personal Knowledge Vault. I can also verify this in the code sequence, but it is useful to validate using the actual markdown output.
This image also shows the graph document structure being used currently with the Document Node, Page Nodes, and chunk nodes. The articles and documents in the graph currently were decomposed using naive chunking. I have added in the Langchain Semantic Chunking pattern to assess if it does a more accurate job of breaking documents into a graph document structure that has more semantic relevance rather than using a fixed chunk delimiter (Semantic Chunking | 🦜️🔗 Langchain).
Next on the list of things to do for Agents:
Once I get the new endpoint wired into the Personal Knowledge Vault UX, I have some additional exploration to do on the agent side for features I have in mind for the Personal Knowledge Vault.
One of the reasons I am interested in exploring Langgraph is the fact that it uses a graph approach to the multi-agent pattern from the ground up. This means that I can project the graphs produced by Langgraph directly into neo4j as part of the Knowledge Set.
I’ll store the topic linked to the user as a node, the interview graph linked to the topic, the agents generated to explore the topic as nodes linked to the topic and the outline graph.
The ability to store the full agent research interaction is one of the benefits of using Neo4j over other basic vector databases. Not only can I can capture the request the user is making, but I can also capture the graph of the question and answer pattern the agents are generating.
These ‘associative trails’ that the user and agents are generating are a useful resource for further analysis and refinement. I’ll also link the articles used from the Personal Knowledge Vault to the topic, providing valuable insight for tracking the frequency of use of knowledge from the graph. These patterns can conceptually be used for analysis, ranking and potentially for recommenders as it will become possible to see which chunks are being hit and used in context of users and topics.
I can also capture the markdown output generated by the agents.
If the user then uses the article as a basis to write a new one, I’ll be able to detect when the new article comes back into the system, and link it back to its original source.
Storing the Topic, associated Agent interaction graphs and results will allow me to explore an additional hypothesis that I have: generating multiple paths towards an objective and finding the optimal path from several potential paths to said objective is similar to solving for spatial routes. Seeing if its possible to setup an algorithm like A* but for knowledge objectives is something I think would be useful to try, and neo4j is the perfect environment to do this.
Having the agents do the work of scanning, collating and summarizing baseline knowledge will enable them to keep using their own work stored into the vault’s knowledge graph from a curated sharepoint site.
To make this even easier, I am going to wire this into a GPT next. If it works, the person should be able to ask the GPT to research a given topic in natural language on their mobile device, and have it return a markdown formatted response. One challenge is the process is very long running, so I’ll have to figure out how to handle that in the context of a GPT.
I just got notification though that my Rabbit-OS device shipped today, so perhaps that will hold an even easier and more effective solution by connecting the Large Action Model to the Personal Knowledge Vault API…
- Source code for Storm: langgraph/examples/storm/storm.ipynb at main · langchain-ai/langgraph (github.com)
- Paper: [2402.14207] Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models (arxiv.org)
- Youtube video explaining the approach: Building STORM from scratch with LangGraph — YouTube
- LangGraph: Multi-Agent Workflows (langchain.dev)
Example generated literature review using Personal Knowledge Vault for topic AI Risks in the Enterprise follows:
AI Risks in the Enterprise
Introduction
The integration of Artificial Intelligence (AI) technologies within the enterprise has introduced a transformative shift in operational dynamics, fostering innovation, efficiency, and a competitive edge. However, this integration is accompanied by a spectrum of risks ranging from data security and privacy to ethical concerns such as bias and fairness. The implications of these risks are profound, impacting not just the enterprises but also customers, employees, and society at large. It is vital for businesses to adopt a proactive stance in identifying, understanding, and managing these risks, ensuring their AI applications are ethical, equitable, and aligned with societal values. This article delves into the various risks associated with AI in the enterprise, offering a comprehensive framework for AI risk management.
Comprehensive Classification of AI Risks
As AI technologies become increasingly embedded in enterprise operations, understanding and mitigating associated risks is critical. Risks can be broadly categorized into model risks, application risks, malicious use risks, psychological risks, and operational risks, each presenting unique challenges that require specific management strategies.
Model Risks
Issues related to the construction and utilization of AI models, including data curation, testing, design, and documentation. Continuous evolution in AI technology necessitates vigilance against emerging risks.
Application Risks
The consequences of transitioning decision-making processes from humans to machines, emphasizing the need for careful management of automation processes to avoid unintended outcomes.
Malicious Use Risks
The exploitation of AI for illicit purposes, including criminal activities, propagation of biases, misinformation, and potential job losses, which call for comprehensive security measures.
Psychological Risks
The impact of AI on human psychology and societal norms, including the illusion of objectivity that may lead users to overlook inherent biases and limitations.
Operational Risks
A range of issues that can arise during the deployment and use of AI technologies, including system failures, data quality concerns, and the fear of losing control over AI systems.
Cybersecurity Challenges
With the rise of AI in enterprise systems, significant cybersecurity challenges have emerged, necessitating proactive and dynamic strategies to protect enterprise assets and data. These challenges include data security risks, operational risks, and compliance and legal risks.
Data Security Risks
The necessity for robust data protection measures to safeguard against data breaches, unauthorized access, and leaks of sensitive information.
Operational Risks
Issues such as system failures and unexpected outcomes that can compromise security, highlighting the importance of continuous testing, monitoring, and updating of AI systems.
Compliance and Legal Risks
The dynamic regulatory landscape requires diligent oversight to ensure AI applications comply with data protection laws and ethical standards, avoiding legal liabilities and maintaining public trust.
Mitigation Strategies and Best Practices
To effectively mitigate AI risks, enterprises can adopt strategies across data management, diverse data sets, rigorous testing, legal compliance, and operational flexibility. These best practices are crucial for reducing adverse impacts and ensuring the secure, fair, and compliant use of AI technologies.
Data Management and Governance
Strong data management and governance frameworks are fundamental for maintaining data integrity and quality, reducing biases, errors, and security vulnerabilities.
Diverse Data Sets and Awareness of Labeler Bias
Using diverse data sets and addressing labeler bias are critical for achieving fair and unbiased AI outcomes.
Rigorous Testing and Monitoring
Comprehensive testing and continuous monitoring are essential for identifying and addressing potential errors and biases in AI applications.
Adherence to Legal Frameworks and Ethical Guidelines
Compliance with legal and ethical standards is crucial for avoiding legal liabilities and maintaining public trust.
Operational Flexibility
The ability to adapt to emerging errors, biases, regulatory requirements, or ethical considerations is key for the effective use of AI technologies.
Real-World Examples and Case Studies
Real-world examples and case studies illustrate how enterprises mitigate AI risks through strategies such as visibility, monitoring, high-quality test datasets, industry-specific AI applications, and stress testing. These examples highlight the importance of thorough testing, validation, and tailored approaches to addressing the unique challenges and risks of each industry.
References
- VentureBeat — The quest for nirvana: Applying AI at scale
- 3 Quarks Daily — The hazards of AI: Operational risks
- Fortune — How do CEOs, executives really feel about AI? Davos WEF
- Databricks Blog — Efficient Fine-Tuning: LoRA Guide LLMS
- H2O.ai Blog — Effortless Fine-Tuning of Large Language Models with Open Source H2O LLM Studio
- ZDNet — 75% of businesses are implementing or considering bans on ChatGPT
- AWS Big Data Blog — Data governance in the age of generative AI
Note: URLs in the footnotes provide further reading and sources for the information presented in this article.
