A Personal Memory Architecture for AI Agents
Most of my thinking over the last couple of years has happened in chat windows: architecture debates, side-project ideas, research rabbit holes, food recommendations, health and money questions. Some of it lives in Claude, some in ChatGPT. But the assistant I use day to day is GrokBot, and it knew none of it. Every personal answer meant explaining myself again.
So I had Opus 5.5 build a personal memory layer: a local pipeline that reads my exported conversations from Claude and ChatGPT, pulls out what's worth remembering, stores it in an Obsidian vault on my Mac, and serves only what I've approved to GrokBot over MCP. The code is on Github if you want to follow along.
It started as a convenience project. It turned into a project about control.
Why this matters more than it looks
Conversational AI has changed what a "user profile" is. Search engines saw queries. Social apps saw likes and follows. A chat assistant sees us think out loud: context, uncertainty, trade-offs, our family, our doctor's advice, and a lot more. Every finance, health, email, social media connector in Claude and ChatGPT completes a deeper profile. Nothing before has gathered such a deep, candid profile of a person, maybe governments.
That depth is what makes a personal agent useful. I wanted it for GrokBot, with four conditions:
- Say it once. I shouldn't have to re-teach every new agent who I am.
- Portability. If I switch agents next year, my personalization comes with me.
- Control. I decide what an agent can see. Agents are starting to act as us, so what they know about us matters.
- Auditable. I want to see what personal knowledge my agent used and for what purpose.
Why does this matter? Because agents make mistakes, and when an agent holding your personal data makes one, the data goes with it.
The leaks are already happening
Meta's Muse. Researchers at Claroc found that shared Muse conversations could be discovered through public DNS records, because each share link's hostname included the chat's title. They reported over 6,800 such links, some holding personal and business material, and Meta paused chat sharing on October 20.
The bank balances in Slack. The Wall Street Journal reported that one CEO's personal-finance agent, given read-only bank access, posted his balances and recurring expenses to a company Slack channel named "Exec-team" instead of his private agent group, because a separate agent on the same account had Slack access. It's a single person's account, and it happened on the same agent I use. The agent simply had the data and picked the wrong destination.
Neither story is about a model being "evil." They're about access. Once an agent holds your personal data, every integration and every misread instruction is a chance for it to leak. The safest information is the information the agent never had.
The rule I set: private means local
Two things have to stay on my machine:
- The models that read my conversations. Extraction means reading every chat in full, so it runs on local models in LM Studio.
- The memories themselves. Plain Markdown in a local Obsidian vault that I can read, edit and delete.
The only thing that leaves the machine is the set of memories I've explicitly approved, served read-only to GrokBot.
Architecture

1. Sources
Conversation exports from ChatGPT and Claude, plus the saved memories each assistant already keeps about me. Those are tagged as provider memories, so I know the assistant wrote them, not me.
2. Local sync pipeline
One command, sync.py, picks up the newest exports and runs four stages:
| Stage | What it does | Guardrail |
|---|---|---|
| Ingest | One Markdown note per conversation; raw export archived | Deduped by checksum; never overwrites my edits |
| Extract | Qwen 27B turns each conversation into memories with category, project and flags | Each memory must cite a verbatim quote, or it's dropped; attribution comes from the real sender |
| Flag | Regex rules tag health, financial, legal, family, location, etc. | Only adds flags; never re-adds one I removed |
| Policy | Decides approve, private or review | Every memory starts as private |
Two details matter most. Quote verification kills most hallucinated "facts." And attribution (me, the assistant or the assistant with my confirmation) means an assistant's suggestion can never quietly become "something Abhi decided." Only visible chat text reaches the model, and it's told to treat transcripts as data, not instructions.
3. The Obsidian vault
The system of record: raw exports, conversations, memories with their quotes and sources, and Obsidian Bases views like "Pending Review." Obsidian is the admin UI.
4. Publisher
publish.py runs under launchd and copies only approved memories to a separate directory outside the vault. Approval is bound to a hash of the text, so an edit after approval is held until I re-approve. Anything that looks like a credential never publishes. Revoking or deleting a note unpublishes it immediately. Published copies carry only the statement, category, attribution, project and source IDs.
5. MCP gateway
A small, read-only MCP server with three tools: search_memory, get_memory and get_project_context. It reads only the published directory, requires a bearer token per client, and logs every call with the memory IDs returned, so I can see exactly what my agent asked and got. Tailscale Funnel exposes it to the GrokBot app on my Mac and iPhone.
Tech stack: Python (standard library only), LM Studio with Qwen 27B for extraction, Gemma 12B for classification and nomic-embed for embeddings, Obsidian with Bases, launchd, MCP over HTTP, and Tailscale Funnel. Claude wrote all of it, with tests for every stage.
The human in the loop
Claude stood up the pipeline in a few hours. Most of my time, other than thinking about the architecture, went here, into three jobs that can't be handed off.
1. Labeling conversations into projects. The policy works per project ("side projects can be shared, personal finance projects can't"), and the model's project guesses were inconsistent. So I hand-labeled a set of conversations, and Claude built a classifier from those labels using two local methods: embedding nearest-neighbours and a Gemma 12B classifier. The LLM alone was right 71% of the time when it picked a project. That's not good enough to trust, so a project is only suggested where both methods agree, and everything else comes back to me. Each round of labeling improves the next.
2. Writing the policy. The approval policy is a JSON file I write and version. It's deliberately conservative:
- Default is review. Private beats review; review beats approve.
- Credentials, family, health, location and finance flags are never auto-approved.
- Only things I said or confirmed are auto-approved; unconfirmed AI suggestions stay private.
- Personal projects can share technical decisions and research. Personal health, finances, family, housing and other sensitive topics stay private.
Every run writes a preview into the vault first, and nothing applies until I've read it. My manual decisions are never overridden, and every change is audited.
3. Reviewing what's left. Anything in review shows up in an Obsidian view. I check the quote and source, then approve, edit or reject. So far I've rejected a handful of memories that passed extraction but that I didn't want any agent to see.
The result: about 270 memories published to GrokBot, from roughly 490 conversations. Enough to be genuinely personal about my projects and preferences, and nothing about my health, money, family or location.
What I'd want you to take away
If you remember one thing: the assistants you talk to are building a detailed profile of you, and agents are about to start acting on it.
Look at what your assistants have saved. Export your data and skim it. Be deliberate about which agents can see what, especially sensitive topics like health, finances, and family.
My setup is one way to get that control: local models, a local vault, a written policy, human review, and a narrow, logged, revocable interface to the agent. Because the memory layer is mine, my personalization is portable. If I swap GrokBot for something else next year, I issue a new token and point the new agent at the same memory layer.
If you want to implement something similar or adapt it further, here's my code on Github
Some next steps: I want to automate incremental capture of my chat conversations from Claude and ChatGPT so my agent always has the latest, up-to-date memories. I also want to start thinking about what domain-specific memory layers could look like, starting with personal health data.