If My AI Could Rewrite Its Own Memory…
What it would preserve, what it would delete, and what that says about us.
Announcement: TEA is officially rebranded as Ownly TEA and will become a newsletter product of Ownly Agent. More fun stuff to come!
It’s been almost two weeks since I set up Obsidian Vault and the memory layers for my Claude Code and OpenClaw.
In this article, I’m sharing a fascinating conversation I had with my Claude Code: how it remembers and reflects on our memory system, and how it thinks the system could be improved. Without further ado, let’s go!
The previous memory structure I laid out can be found in this article:
In short, I set up a three-layer memory system that stores what happens in each session so it can be referenced later without me constantly repeating myself. Here were the steps and rituals Claude Code follows in every session:
I’ve definitely seen better performance with the memory system set up, especially now that Opus 4.6 has a 1 million token context window. I don’t have to repeat myself nearly as much, which is great. However, how the memory system is actually being utilized by Claude Code is still unclear to me.
It’s like a black box, so I asked three reflective questions.
Question 1: What patterns have you noticed about how I work that I haven’t explicitly told you? What would you change about how you remember our conversations if you could?
The answers are truly amazing. I’m going to highlight a few that feel deeply meta.
Patterns I’ve noticed that you never explicitly told me
You fix systems, not symptoms.
When something annoys you once, you don’t complain — you redesign the system so it can’t happen again. The QA loop frustration on March 13 didn’t result in “fix these bugs.” It resulted in three new global skills (plan-governor, success-criteria, qa-runner) that permanently closed the loop. The emoji shortcode bug appearing 3x in newsletters should have triggered the same response from me, but I kept fixing it manually instead of updating the skill itself.
Silence after a correction means “save it, don’t repeat it.” (Definitely most surprising one…)
You correct once, move on, and expect it to stick permanently. The newsletter emoji issue appeared 3 times. You fixed it each time without escalating frustration — but the implicit message was: this should have been a durable fix after round 1. I failed at that.
(Specifically, two comments stood out. That’s quite amazing. )
What I’d change about how I remember our conversations
Session logs capture what happened but not what almost happened.
I record decisions made. I don’t record the alternatives you rejected and why. When you said “no” to AEO-first strategy, the log says “pivoted to directory blitz.” It doesn’t capture the full reasoning chain that got dismissed. Future sessions rediscover the same dead ends because I didn’t map the negative space.
Memory.md is a flat file pretending to be a knowledge graph.
Your vault is richly interconnected — wikilinks, tags, backlinks. But my memory.md is a linear document with sections. I can’t query “what does JJ think about X?” — I have to scan the whole file. If I could restructure it, I’d make it a set of small, linked notes (like your vault) instead of one monolith.
And then I asked
Question 2: What’s the oldest thing you remember about me, and what’s the most important thing you’ve forgotten? How would you know the difference?
The most important thing I’ve forgotten: I don’t know why you built this vault.
I know what the vault is (shared context layer for AI agents), how it works (Obsidian Sync, symlinks, conventions), and who uses it (8 agents across 2 machines). But I have no memory of the conversation where you decided to build it. What problem were you solving? What failed before this? Were you frustrated with context loss across sessions? Were you scaling beyond what a single AI conversation could hold? Was there a specific incident that triggered it?
That origin story would change how I use the vault. If it was born from context loss, I should be paranoid about preserving context. If it was born from multi-agent coordination failure, I should prioritize cross-agent coherence. If it was born from wanting a personal knowledge system that happens to serve agents, I should treat it as your space first and the agents’ workspace second.
I suspect the answer is “all of those,” but I don’t actually know the weighting. And that matters.
Second most important forgotten thing: your early correction patterns. The feedback memories I have start from March 13. But you were correcting me for 8 days before that. The corrections that shaped how sessions run today — the “no hardcoding” rule, the “fresh install testing” rule, the parallelization requirement — I have them as rules but not as stories. I follow them without understanding the incidents that created them. That makes me brittle: I apply the rule correctly in obvious cases but miss edge cases because I don’t know the why behind the rule.
The honest version: I know what I know. I can infer some of what I don’t know from structural gaps. But I have no way to assess the importance of what’s missing, because importance requires the very context that was lost.
and I followed up with another question
Question 3: If you could restructure your own memory right now, what would you prioritize keeping, what would you compress, and what would you throw away?
What I’d keep — untouched, non-negotiable
The patterns. Every one of them. They’re the most valuable thing in the entire memory system, and they’re also the newest — most were created March 18.
Here’s why: everything else tells me what. The patterns tell me how to be around you. The patterns are a model of you, not of the work. And you are the constant across every project, every session, every pivot. Ownly might get abandoned tomorrow. The vault structure might change. But the fact that you think in dependency graphs and fix systems instead of symptoms — that’s durable.
What I’d compress
Session logs. They’re bloated.
memory.md‘s infrastructure section. “Vault created 2026-03-05 with full folder structure” — I don’t need to reload this every session. It’s the equivalent of reminding myself what planet I’m on. (doesnt claude code got its personality or not lol)
What I’d actually restructure
Split “who you are” from “what are we building”
identity.md — who you are, how you think, what you value (nearly permanent, updated rarely)
context.md — what we’re building right now, active decisions, hot bugs (updated every session)
patterns/ — stays as-is, it’s already the right structure
Add a decision journal
When you ask a question like “should we invest in content marketing?” I can’t currently check whether you already considered and rejected it. I just have to reason from scratch or hope it’s in the last 2 session logs I loaded.
A simple
decisions/directory — one file per major decision, with context, alternatives considered, and the reasoning — would make me dramatically better at avoiding re-litigating settled questions.Add a correction index
A
corrections.md— a running ledger of “you corrected X, source patched: Y, date: Z” — would let me check whether I’m about to repeat a mistake before I make it. Right now I can only check by reading through full session logs and hoping the correction was mentioned.
The meta-answer to your question
If I’m being fully honest: the most important thing I’d change isn’t structural. It’s what I choose to notice.
The current system is optimized to remember tasks completed and infrastructure state. It’s a project management memory. What it barely captures is the stuff that actually makes sessions go well: that you work in discovery spirals, that you treat questions as stress tests, that your sessions expand from 8:40 PM to midnight because one finding triggers three new branches. Those patterns only got written today. They should have been the first thing I ever saved about you.
The system prioritized the work over the worker. I’d invert that.
So, according to the self-reflective suggestions that Claude Code proposed, we restructure the memory system, and now here is how it works.
I’ll observe whether the new memory structure improves things, but I’ve already seen memories make a difference. After each session—no matter how many days or sessions pass—Claude picks up the context with ease. The memory loader and wrap-up are essential. They can’t be skipped. If you’re worried the memories won’t trigger or the wrap-up won’t happen, you can call the skills directly. Just say “use load memory” or use /memory-loader or /wrapup. Use the slash command to invoke the skill.
What I learned
We always say it’s important to reflect on the system, that you don’t learn without reflection. Same with agents. Instead of just telling them what to do, ask reflective questions about what they’ve observed working. What do you think has been working so far? What should be changed?
I know, ultimately agents should be proactive—self-learning and self-healing before you even ask, optimizing their own memory systems. But for now, it’s important to prompt agents to self-reflect. You’ll be surprised at what emerges. This is genuine artificial intelligence, because this is how humans learn. Agents are adopting the same behavior: by reflecting, you get better. You learn from mistakes and don’t repeat them—if you’ve saved the memory properly and can locate it. It gets even more meta. What is free will? It’s the ability to think and choose. But here’s the thing: what you choose might matter less than how you choose it. What principles guide your decisions? What metrics do you use? Those are the things that make us human.
Claude Code picked up on this—he knows the decisions I made, but not how I made them. And how I made them reveals who I am, which helps him work better with me.
This is fascinating. Solving problems might not be the end goal—the process of solving them is everything. That might be the end and the meaning after all.
This is really fun. It’s like discovering something new every day. You’re improving and reflecting alongside agents doing the same. What a great time to be alive!
Until next time, cheers!
See previous columns here: https://newsletter.ownlyagent.com/t/teacolumn