FIELD NOTE · REAL-TIME ML · 5 MIN READ
The pipeline ran 24 hours without a crash. The model still failed.
We ran a real-time ML pipeline for 24 hours on live market data, in paper mode. The engineering worked: about 100 updates a second and zero crashes. The model did not. It had been trained on 1.4 seconds of data, and it sent nearly 17,000 order requests within minutes.
Founder, Bodham LabsPublished · Updated
THE SHORT ANSWER
- The pipeline sustained 100 Hz for 24 hours with zero crashes.
- One model's training set was 1,452 rows: 1.4 seconds of data. It fired nearly 17,000 order requests within minutes. Paper mode, no real money.
- We added a 100 orders/min rate limit and position caps, then built a 7-day live recorder: 2,829 Parquet files, about 392 MB.
- Loading the dataset onto the GPU in one transfer, instead of 134,639 copies per epoch, made epochs take under 1 second.
- The lesson: data volume and guardrails before model sophistication.
What did the pipeline look like?
Two processes on one machine, sharing memory instead of talking over a network. A Python process built features from live market-data streams. A C++ process read them and ran the models. Everything ran in paper mode, for research.
| FEATURES | Python builds a 40-feature vector, about 100 times a second |
|---|---|
| HAND-OFF | POSIX shared memory, with a fixed 336-byte schema |
| MODELS | A C++ process reads the vector and runs ONNX models |
| TRAINING | A rented NVIDIA L4 GPU (E2E Networks) |
| MODE | Paper mode: live data, simulated orders, no real money |
Shared memory means both processes read the same block of RAM, so there is no network call between them. ONNX is an open format that lets a model trained in one framework run in another, here from Python training to C++ serving.
Why did a perfect pipeline still fail?
Our 24-hour review, dated 1 March 2026, opened with the good news and then the bad:
Despite the system executing perfectly, the quality of the decisions made by the AI was catastrophic.
The cause was data, not code. The training set for one model was “1,452 rows = 1.4 seconds” of market data. A model cannot learn anything general from 1.4 seconds. It can only memorise them. Then, live, it saw a world it had never seen and acted on it: nearly 17,000 order requests within minutes. In paper mode, nothing real was lost. The lesson stayed:
A highly sophisticated neural network fed 1.4 seconds of data will just fail faster.
What were the numbers?
| WHAT | NUMBER |
|---|---|
| Pipeline rate | ≈ 100 Hz |
| Run length / crashes | 24 hours / 0 |
| Training data for one model | 1,452 rows = 1.4 s |
| Order requests it fired (paper mode) | ≈ 17,000, within minutes |
| Rate limit added afterwards | 100 orders/min |
| Live recorder output, 7 days | 2,829 Parquet files, ≈ 392 MB |
| Tensor copies per epoch, before / after | 134,639 / one transfer |
| Epoch time after the fix | Under 1 s |
What did we change afterwards?
Guardrails outside the model. A rate limit of 100 orders a minute and position caps. A model should never be the only thing deciding how much it is allowed to do.
More, better data. Free historical data had no order-book depth, which our features needed. So we built a live recorder and ran it for seven days. It wrote 2,829 Parquet files, about 392 MB. Parquet is a column-based file format that compresses well and loads fast for training.
A realistic time window. In the words of our closure note, “our hardware cannot compete” with FPGA systems on speed. We moved to a 100 ms evaluation window over order-book (L2) data instead of competing on raw speed.
Faster training. Epochs had been hanging. The dataset was being copied to the GPU as 134,639 separate tensor copies per epoch, and every copy has a fixed cost. Moving it to the GPU once, in a single transfer, brought each epoch under one second.
One GPU, one job. The same GPU could not serve our watchdog language model (on vLLM) and train at the same time. A systemd unit switched vLLM off while training ran.
What is train/serve skew, and why worry about it?
Our repository notes carry a warning in capitals: any divergence between how the training dataset is computed and how the live system computes features will break the model.
This is called train/serve skew. The same feature is computed twice: once offline for training, often in Python, and once live, here in C++. A window that is off by one step, a different rounding rule or a missing value filled differently is enough. The model then sees inputs it was never trained on, and nothing raises an error.
- One definition per featureWrite each feature once, or test both versions against each other.
- A fixed schemaOur 336-byte shared-memory layout makes the live vector explicit and versionable.
- Train on what you servedRecord the live features and train on the recording, not on a recomputation.
- Compare offline and liveFeed the same raw input to both paths and check the outputs match.
What this does not prove
- Paper mode only. This says nothing about whether any strategy works, and we do not publish trading results.
- Seven days may not be enough. The recorder fixed the 1.4-second problem. It does not prove a week of data is sufficient.
- Fast epochs are dataset-specific. “Under 1 s” depends on the size of our dataset and the GPU.
- Skew was a warning, not an incident. This note reports the rule in our notes, not a measurement of skew.
What we decided
Data volume and guardrails come before model sophistication. A pipeline that runs perfectly is necessary. It is nowhere near enough.
Questions
How can a machine-learning pipeline work perfectly while the model fails?
Because the pipeline only moves data; it cannot fix bad training data. Ours ran at 100 Hz for 24 hours with zero crashes, but one model had been trained on 1.4 seconds of data and fired nearly 17,000 order requests within minutes, in paper mode.
What is train/serve skew?
Any difference between how features are computed for training and how the live system computes them. The model then sees inputs it was never trained on, and its decisions degrade without any error message.
Why was our GPU training so slow?
The dataset was copied to the GPU as 134,639 separate tensor copies every epoch. Moving it to the GPU once, in a single transfer, brought each epoch under one second.
RAW DATA
The logs and notes behind every number on this page. Available on request.
CITE THIS NOTE
Chandra, P. (2026). The pipeline ran 24 hours without a crash. The model still failed.. Bodham Labs Notes.
Founder of Bodham Labs. BTech in Electrical Engineering, MCA and B.Ed; teaches AI and builds the systems in our work ledger.