FIELD NOTE · AGENT MEMORY · 5 MIN READ
Why our agent memory stopped waiting for a language model
Our memory server for AI coding agents first asked a local 8-billion-parameter model to write every answer. On a CPU that took one to two minutes, far too slow for an agent in the middle of a task. We took the model out of the query path. The memory then got used every day, from March to May 2026.
Founder, Bodham LabsPublished · Updated
THE SHORT ANSWER
- A local 8B model wrote about 3.58 tokens per second on CPU. Writing one answer took roughly 60–120 seconds.
- We removed answer-writing from the query path. The server returns raw passages; the calling agent, which already runs a strong model, writes the answer.
- After that, the memory was used daily. Its log from 9 March to 21 May 2026 shows 505 starts, 111 queries and 82 ingests.
- We replaced a small language model with GLiNER for entities, because the language model invented them.
- Latency percentiles will be published after a proper re-run.
What does this memory server do?
It gives AI coding agents a long-term memory. When an agent starts work, it can ask what was decided last week, which file holds a function, or why a library was dropped. When it finishes, it can save what it learned.
The server speaks the Model Context Protocol (MCP), an open standard for connecting AI assistants to tools and data. It runs locally. Our first version, in August 2025, used cloud services (Neon and OpenAI). By March 2026 it was fully local.
| TRANSPORT | MCP over stdio, on the same machine as the coding tool |
|---|---|
| VECTORS | ChromaDB, with hot, warm and cold tiers and time decay |
| GRAPH | Neo4j, for entities and how they relate |
| EMBEDDINGS | all-MiniLM, a small sentence-embedding model |
| ENTITIES | GLiNER, which picks names straight out of the text |
| CONSOLIDATION | Qwen3 8B, offline only, never while an agent waits |
| ROUTING | Plain rules, no model |
Why was a language model in the search path at all?
Because that is the textbook shape of retrieval-augmented generation: find the relevant passages, then ask a model to write an answer from them. When a person asks, that second step helps. People want an answer, not five excerpts.
Our caller was not a person. It was a coding agent that already runs a strong model. Asking a smaller, slower local model to summarise passages for a bigger model was duplicated work. It was also slow.
We measured a local 8B model at about 3.58 tokens per second on CPU. At that speed, an answer of 200–400 tokens takes about one to two minutes. An agent in the middle of a task cannot wait two minutes for a memory.
What changed when we took it out?
The server now returns the most relevant raw passages, with their sources. The calling agent reads them and writes the answer itself. In the words of our design notes:
Synthesis is removed from the query hot path.
After that, the memory was used every day. The server log covers 9 March to 21 May 2026.
| MEASURE | VALUE |
|---|---|
| Local 8B model, generation speed on CPU | ≈ 3.58 tokens/s |
| Time for the model to write one answer | ≈ 60–120 s |
| Server starts in the log | 505 |
memory_query calls | 111 |
memory_ingest calls | 82 |
| Hot → warm → cold tier moves | Seen in the log |
| Query latency today | Not published yet |
Why did we swap a small language model for GLiNER?
The memory keeps a graph of entities: people, projects, files, libraries. Our first extractor was a language model. It went wrong in a specific way:
We used a 1.7B parameter language model (SmolLM2) for entity extraction… it hallucinated entities.
A made-up entity in a memory graph is worse than a missing one. It links unrelated notes and sends later searches the wrong way. We moved to GLiNER, a small model built for named-entity recognition. It can only select substrings that exist in the input text
. It can miss a name. It cannot invent one.
The general rule: when the job is to point at something in the text, use a model that can only point.
Why stdio instead of SSE?
Our first transport was server-sent events (SSE) through a Node bridge. It failed. We moved to stdio. With stdio, the coding tool starts the server as a child process and talks to it over standard input and output. There is no port to open and no bridge process to keep alive.
How does the memory forget?
Every memory has a score that fades with time since it was last used: score = exp(−λ · days), with λ = 0.05. The score never drops below a floor of 0.3, so old decisions fade but stay findable.
| DAYS SINCE LAST ACCESS | SCORE |
|---|---|
| 0 | 1.00 |
| 7 | 0.70 |
| 14 | 0.50 |
| 24 or more | 0.30 (floor) |
Two more rules keep it honest:
- A surprise flag. A new chunk that is very similar to an old one but says something different gets flagged. This
prevents silent overwrite of decisions
. - No model for routing. Deciding whether a request is a query or an ingest is done by plain rules,
because our operation space is small and deterministic
.
What this does not prove
- Use is not quality. 111 queries shows the memory was used. It does not show that it found the right passage each time. We have not measured recall yet.
- No latency figures here. They wait for a proper re-run.
- Internal use, one machine. The log is from our own coding work, not a client team.
- One CPU measurement. A GPU or a smaller model would write faster. The design choice rests on a different point: the caller already has a strong model.
What we decided
No language model on the query path. Retrieval returns passages; the agent that asked writes the answer. Models still do two jobs: GLiNER points at entities, and Qwen3 8B consolidates memories offline.
Next: a recall test on real questions, and the latency re-run.
Questions
Should a memory server for AI agents write the answer itself?
Not when the caller is an AI agent with its own strong model. In our case a local 8B model wrote about 3.58 tokens per second on CPU, so each answer took 60–120 seconds. Returning raw passages and letting the agent write the answer made the memory usable every day.
Why use GLiNER instead of a small language model for entity extraction?
A 1.7B language model (SmolLM2) invented entities that were not in the text. GLiNER can only select substrings that exist in the input, so it cannot invent a name.
How does the memory decide what to forget?
Each memory's score is exp(−0.05 × days since it was last accessed), with a floor of 0.3. Memories also move between hot, warm and cold tiers.
RAW DATA
The logs and notes behind every number on this page. Available on request.
CITE THIS NOTE
Chandra, P. (2026). Why our agent memory stopped waiting for a language model. Bodham Labs Notes.
Founder of Bodham Labs. BTech in Electrical Engineering, MCA and B.Ed; teaches AI and builds the systems in our work ledger.