Long-term memory for AI agents
THE QUESTION
Can a small, fully local server give AI coding agents a memory that lasts between sessions?
Founder, Bodham LabsUpdated
What did we build?
- Nexus RAG: a local MCP server, over stdio, that gives AI coding agents long-term memory.
- Vectors in ChromaDB with HOT, WARM and COLD tiers and time decay. An entity graph in Neo4j.
- Small models for each job: all-MiniLM for embeddings, GLiNER for entity extraction, and Qwen3:8b only for offline consolidation.
- Decay score = exp(−λ · days since last access), with λ = 0.05 and a floor of 0.3.
- A surprise flag: a new chunk that is very similar to an old one but says something different is flagged. This prevents silent overwrite of decisions.
- A router with no ML model in it, because the set of operations is small and deterministic.
What does the evidence show?
| WHAT | VALUE | SOURCE |
|---|---|---|
| Bootstrap starts | 505 | MCP server log, 9 Mar – 21 May 2026 |
| memory_query calls | 111 | MCP server log, 9 Mar – 21 May 2026 |
| memory_ingest calls | 82 | MCP server log, 9 Mar – 21 May 2026 |
| Tier moves | Real hot → warm → cold moves recorded | MCP server log, 9 Mar – 21 May 2026 |
| Local 8B model, generation speed on CPU | about 3.58 tokens/s | CPU benchmark, cited in our internal research paper |
| Time to write one answer with that model | roughly 60–120 s | internal research paper |
| Query latency | Not published yet | we'll publish percentiles after a proper re-run |
Where has it led us?
The biggest change was removing synthesis from the query hot path. A local 8B model took roughly 60–120 s to write an answer. Now the server returns raw passages, and the calling agent writes the answer.
A 1.7B model (SmolLM2) hallucinated entities during extraction. We switched to GLiNER, because it can only select substrings that exist in the input text.
An SSE transport through a Node bridge failed. We moved to stdio.
The project moved from a first cloud version (Neon and OpenAI, Aug 2025) to fully local (Mar 2026). It was then used daily until May 2026.
What is still open?
- What are the real latency percentiles over a large, repeatable set of queries?
- How much memory does the full stack need when measured rather than estimated?
- Does the surprise flag catch the contradictions that matter, and how often does it flag by mistake?
NEXT STEP · NOT DONE YET
What would we do next?
Re-run the latency benchmark with a proper query set, measure RAM directly, and publish the percentiles.
Have a problem that needs this kind of research?
Tell us about it