There was a lot of talk about Andrej Karpathy’s second brain system. At that time I was busy with some other automations, so I couldn’t pay attention.
But as you start working more with AI, you’ll pretty soon realize the need to build a context library for your AI systems.
You see, each session in Claude starts with limited memory of what it did previously. What if you could give Claude more memory from earlier sessions, and allow it to build and extend that memory itself?
When I looked around, a lot of folks were using different approaches to build this. I went with the simplest way, so that I could understand what I was building and explain it clearly to the AI and to myself.
I started with a Claude session. I had hand-drawn a basic system and asked Claude to brainstorm with me.
Next came the building part. I started by giving Claude more details about myself: the work I do, my job description, my career trajectory, and more. Then I added examples of my work from the pre-AI era. Then I built the style and voice structure so Claude could write like me (it still cannot, so let me know if anyone has any tips on that).
The result is a basic file system with folders and .md files. It only has four folders, and each folder has fewer than 10 .md files (even after three months of regular work). The idea is to keep the second brain itself a smart, lean system and convert other things to skills.
The CLAUDE.md file automatically loads the index in each session, and Claude instantly has the knowledge needed to make decisions like me.
Obviously, it didn’t do that when we first started. But it kept getting better as I provided it with more information and context.
With Fable 5, Claude even started suggesting what to record. I had several aha moments when it recommended recording something in the second brain that we had come across while working on regular tasks at work.
While this system is simpler and gives me a lot more control, I do want to experiment with persistent memory systems like Hermes.
I have a Linux machine that I can use with Hermes Agent. But before that, I need to decide on two things.
First, whether I want to build a system deeply dependent on Claude, or a system that works regardless of which AI platform I am using. Currently, my employer has provided me with a Claude account with generous limits.
However, Claude is way too expensive to run for personal projects that are not related to my job and/or don’t make money.
Second, I need to decide whether I would like to experiment with a clean slate or with my existing second brain.
I believe using my existing second brain will give Hermes Agent a ton of useful context. However, then I wouldn’t know if the brilliance behind Hermes Agent comes from its own system or from the system I already had working.