FIELD NOTE · STATE SPACE MODELS · 5 MIN READ
Our Mamba model's hidden state reached 10¹⁵. Normalise your inputs.
In March 2026 we ran a Mamba model on a GPU against live market-data streams, in paper mode, as research. An audit found its hidden state had reached about ±10¹⁵ over 1,024 steps. The architecture was not the problem. The inputs had never been normalised.
Founder, Bodham LabsPublished · Updated
THE SHORT ANSWER
- Our inputs went in raw: moving averages around 69,000 and one “z-score” of 86.74, where a real z-score rarely leaves ±3.
- Over 1,024 steps the hidden state grew to about ±10¹⁵ and drowned 14 other features by about 10¹³. Three features were always zero.
- An earlier model's hidden-state norm stayed at exactly 1.3850 from step 100 to step 96,900. It was stuck, not stable.
- Standardise every feature, drop dead ones, log the hidden-state norm, and test on synthetic data first.
What is a state space model, briefly?
A state space model (SSM) reads a sequence one step at a time and carries a fixed-size summary of everything so far, called the hidden state. At each step the new state is a mix of the old state and the new input. The output is read from the state.
Mamba (Gu and Dao, 2023) makes that mix depend on the input, so the model can choose what to keep and what to forget. Its cost grows linearly with sequence length, and its memory stays fixed. That makes it attractive for long, never-ending streams. Jamba (AI21, 2024) mixes Mamba layers with Transformer attention layers.
One consequence matters here. The hidden state is, roughly, a weighted running total of past inputs. Feed it huge numbers and it holds huge numbers.
What went wrong?
On 11 March 2026 a Mamba (S6) context model ran on a GPU in a container, taking 396 MiB. Its inputs were features computed from live market-data streams, in paper mode, for research. A data-integrity audit then found the inputs had never been normalised.
| FINDING | VALUE | WHAT IT SHOULD BE |
|---|---|---|
| Moving-average (EMA) inputs | ≈ 69,000 | Around 0, spread about 1 |
| Largest “z-score” input | 86.74 | Rarely beyond ±3 |
| Hidden state after 1,024 steps | ≈ ±10¹⁵ | Bounded, moving with input |
| How far it drowned 14 other features | ≈ 10¹³ × | Comparable scales |
| Features that were always zero | 3 | 0 |
A real z-score is a value minus its mean, divided by its standard deviation. For most data it sits within about ±3. A “z-score” of 86.74 means the scaling step was wrong or missing.
When one feature is 10¹³ times larger than the rest, the other 14 are numerically invisible. If the model runs in 32-bit floats, numbers near 10¹⁵ are spaced tens of millions apart, so smaller contributions are simply lost. In effect, one feature drowned out the rest.
Why did an earlier model look healthy?
Before the Mamba test, a component we had labelled “SSM” was in fact a GRU, a classic recurrent network, used as a PPO actor-critic in reinforcement learning and exported to OpenVINO. We ran a three-way tournament of configurations: 2×256, 1×512 and 3×128.
Its log showed a hidden-state norm of h_norm = 1.3850 at step 100, and the same 1.3850 at step 96,900. Steady numbers feel reassuring. This one was the opposite. A recurrent state that does not move when its input moves is stuck. It carries no information about the sequence.
Two lessons from one log line. Name components for what they are. And treat a metric that never changes as a bug report, not as stability.
What should you check before trusting a sequence model?
- Scale every featureStandardise each feature with a mean and standard deviation from the training data only. Then check the result really sits within a few units of zero.
- Find dead featuresA feature that is always zero, or always the same, carries nothing. Drop it or fix the code that computes it.
- Log the hidden-state normIt should move with the input and stay bounded. Growing without limit points to scale; frozen points to a stuck state.
- Test on synthetic data firstFeed a sine wave or another pattern with a known answer before any real stream. If the model cannot learn that, it will not learn the real thing.
- Name the model honestlyA GRU called an SSM hides what you are actually testing.
What comes next?
The next design is a small hybrid in the Jamba style, with 24–36 million parameters planned. It is documented but not built. Its status line reads: “PRE-DEVELOPMENT (Phase 0)”.
It is built around shadow mode. The model runs alongside the system and its outputs are only recorded, never acted on. A gate decides what happens next: if its shadow evaluation stays below a set threshold for two weeks, we revisit the architecture, and the model does not leave shadow mode.
What this does not prove
- Nothing about Mamba itself. We never had a clean run, so we cannot say whether an SSM suits this data.
- One setup, one audit. The numbers describe our pipeline in March 2026.
- The next design is not built. Its size and gates are plans.
- Why the GRU state froze is open. The log shows that it froze, not why.
What we decided
No new sequence model gets real data until it passes the checklist above on synthetic data, with its hidden-state norm logged. Model sophistication comes after clean inputs, not before.
Questions
Why would a Mamba model's hidden state explode?
In our case, because the inputs were not normalised. Moving averages around 69,000 went in raw, and over 1,024 steps the hidden state grew to about ±10¹⁵. Standardise each feature before it reaches the model.
Is a constant hidden-state norm a good sign?
No. Our earlier GRU model logged h_norm = 1.3850 from step 100 to step 96,900. A healthy recurrent state changes as its input changes; one that never moves is stuck.
RAW DATA
The logs and notes behind every number on this page. Available on request.
CITE THIS NOTE
Chandra, P. (2026). Our Mamba model's hidden state reached 10¹⁵. Normalise your inputs.. Bodham Labs Notes.
Founder of Bodham Labs. BTech in Electrical Engineering, MCA and B.Ed; teaches AI and builds the systems in our work ledger.