Where state space models help, and where they don't

What is the idea, in one picture?

Picture two note-takers in a long meeting. The first writes down every word and, before each new sentence, rereads the whole notebook. Nothing is lost, but the notebook gets thicker every minute, and so does the work of rereading it. That is a transformer. Its attention keeps every past token, so the memory it needs grows with the length of the text.

The second note-taker has one index card. After each sentence they rewrite the card: keep what matters, squeeze the rest, cross out what no longer seems useful. The card never gets bigger. But a figure mentioned an hour ago may be gone by now. That is a state space model (SSM). It compresses the past into a fixed-size hidden state: constant memory, fading detail.

Mamba (Gu and Dao, 2023) is the best-known modern SSM. It lets the input decide what the card keeps and what it drops. RWKV (Peng et al., 2023) is a modern recurrent network that works the same way in practice: one fixed-size state, updated step by step. Jamba (AI21, 2024) mixes Mamba layers with a few attention layers, so a model gets both the card and some of the notebook.

What did seven months of work look like?

The arc runs from an always-on agent design in early 2026 to a phone app in October 2026 that decided not to use SSMs at all. Most of it is design and code. Only one SSM ever ran on real data, and that run failed.

The hybrid design, 10 March

RAMU was meant to run continuously, reading its surroundings and keeping a rolling internal state. Our evaluation argued that a transformer would keep piling up tokens until it hit a limit, while an SSM could keep going on local hardware. So the always-on loop would be an SSM. When it met something unusual or a decision that mattered, it would pause, fetch what it needed from storage, make one isolated call to a large transformer, and resume.

The same document was candid about the cost:

The policy network, same day

RAMUPolicyNet takes 14 input features, projects them to a width of 128 (d_model), runs them through one Mamba block with a hidden state of size 64 (d_state), and feeds the result to an actor-critic head that picks one of a few actions. Its smoke test pushes random numbers through it and prints the output shapes. The function for live, step-by-step use, the part that would make it an always-on loop, is an empty stub. We found no training run and no results. It is code, nothing more.

The GPU run, 11 March

This is the only SSM that met real data. It failed for a dull reason: raw values around 69,000 went in unscaled, and the hidden state grew to about ±10¹⁵ over 1,024 steps. We wrote it up in full in our note on the exploding hidden state, including the earlier “SSM” that turned out to be a GRU.

The staged design, late March

Mamba-4 is a set of nine design documents. The phases include data plumbing, a “Micro-Jamba” model (pure Mamba first, attention layers added later), Hopfield-network recall, an offline champion-versus-challenger tournament, OPLoRA “sleep” consolidation of new learning into the weights, optional BitNet compression, an optional mixture-of-experts overlay, and production safety from day one. The research page lists 24–36 million parameters as planned. The blueprint calls itself a “Planning Document”, and the roadmap's status line reads “PRE-DEVELOPMENT (Phase 0)”.

Two details show how the thinking moved. The blueprint aims to keep all memory inside the model, with no external database: the opposite of the SQLite answer two weeks earlier. And it adds attention layers, at a 7:1 ratio, to patch what it calls Mamba's “>33-minute amnesia”. That figure is a planning assumption, not something we measured. But the fix it reaches for, attention, is an admission that the card alone forgets.

What does a fixed-size state buy?

On a long, never-ending input, three things. These are the reasons in our March evaluation, and they are sound.

• Memory that does not grow with time. The state is the same size after ten steps or ten million. A loop that runs for weeks never hits a context limit and never needs a restart to clear one.
• Steady cost per step. Each new input updates the state; nothing is reread. So the work per step does not climb as the history gets longer.
• A cheap layer underneath an expensive one. A small model can watch the stream all the time and call a large transformer only when it matters. That is the same idea as our cost-tiered agent design: spend on the big model only on escalation.

What does it cost?

• Details fade. The card is a summary. An exact string read early on may not come back word for word much later. Our March notes gave the example of an API key read on day 1 and not recalled on day 100. Anything that must survive exactly needs a separate store.
• The weights still take memory. A fixed-size state saves only the memory a transformer would spend on keeping past tokens. The model itself, its weights, costs the same either way. On a short input there is little history to save, so the saving is close to nothing.
• The state has to be watched. It is a running total. Feed it unscaled numbers and it grows without limit, as ours did to about ±10¹⁵. Let it freeze and it carries nothing, as our GRU's did at 1.3850. Neither failure shows up as an error.
• A thinner toolkit. Our evaluation also noted that forcing strict structured output, such as JSON, was harder than with transformers, and that the SSMs it compared were weaker at multi-step reasoning. That is why the hard decisions went to a transformer.

Why a shipped app said no

Mementos records short voice memos and searches them on the phone. Memory on a phone is tight, so a model family famous for constant memory looked worth a try. The changelog entry for 12 October 2026 records why it was dropped:

That one line holds the whole lesson. The fixed-size state is a saving on length. Where there is no length, there is no saving.

What would we test next?

These are open questions. None of them has an answer yet.

• With normalised inputs, does a small Mamba model beat a simple baseline on the same live stream?
• How fast do details actually fade? If we plant a fact early in a stream, how many steps later can the model still use it?
• At what input length does the saved memory outweigh the cost of the weights on our own hardware?
• Does an SSM plus an external store recover exact facts as reliably as a transformer that keeps them in its window?
• Is a Jamba-style hybrid worth the extra complexity at 24–36 million parameters?

The full research question, evidence and next step are on the state space models research page.

What this does not prove

• Nothing about how good SSMs are. We never had a clean training run, so we have no quality result either way.
• The designs are designs. RAMUPolicyNet was never trained, and Mamba-4 was never built.
• The Mementos call was reasoning, not a benchmark. We did not measure an SSM on the phone; the memos are short, so there was nothing for the state to save.

Questions