Real-time ML pipelines on modest hardware
THE QUESTION
Can a Python-to-C++ pipeline on a laptop and one rented GPU run ML models on a live data stream at 100 Hz without falling over?
Founder, Bodham LabsUpdated
What did we build?
- A Python feature builder that writes a 40-feature vector (a 336-byte schema) into POSIX shared memory at about 100 Hz.
- A C++ process that reads the vector and runs ONNX models.
- Training on a rented NVIDIA L4 GPU (E2E Networks).
- A 7-day live recorder, because free historical data did not include order-book depth.
- The test bed was live market-data streams, in paper mode only, with no real money.
What does the evidence show?
| WHAT | VALUE | SOURCE |
|---|---|---|
| 24-hour paper run, pipeline | sustained 100 Hz with zero crashes | performance review, 1 Mar 2026 |
| Training data behind one model | 1,452 rows = 1.4 seconds | performance review, 1 Mar 2026 |
| Simulated order requests fired | nearly 17,000, within minutes | performance review, 1 Mar 2026 |
| Live recorder output | 2,829 Parquet files (~392 MB) over 7 days | recorder output folder |
| GPU copies per epoch, before the fix | 134,639 individual tensor copies | training phase closure notes |
| Epoch time, after one bulk transfer | under 1 second (it used to hang) | training phase closure notes |
| Latency spikes from synchronous disk writes | 2075 ms | engineering log |
| After async writes | max 142 ms, avg 92 ms; later max 111 ms, avg 100 ms | engineering log |
| Live feed rate held | 68 updates/s; synthetic bursts of 200 and 500 Hz handled | engineering log |
Where has it led us?
The pipeline worked and the model failed. In a 24-hour paper run, the system sustained 100 Hz with zero crashes. But in our own review: "Despite the system executing perfectly, the quality of the decisions made by the AI was catastrophic."
The cause was data, not code. One model was trained on 1,452 rows, which is 1.4 seconds of data. "A highly sophisticated neural network fed 1.4 seconds of data will just fail faster."
That model fired nearly 17,000 simulated order requests within minutes. Afterwards we added rate limits of 100 orders a minute and position caps.
Synchronous disk writes caused 2075 ms spikes in the GPU loop. Moving to async writes brought the max down to 142 ms, and later to 111 ms.
Training hung because the dataset went to the GPU in 134,639 small copies per epoch. One bulk transfer made each epoch take under a second.
One GPU could not serve the watchdog LLM (vLLM) and train at the same time. So systemd switched vLLM off during training.
WSL networking was a bottleneck on an i5-1235U laptop. We now take the data on the Windows side and send it to WSL over UDP.
Our hardware cannot compete with FPGA systems on latency, so we moved to a 100 ms evaluation window over order-book (L2) data.
Our own notes warn that any difference between how the training data is computed and how live features are computed will break the model. This is train/serve skew.
What is still open?
- How much recorded data does a model need before its decisions are worth evaluating?
- Can one shared feature-builder serve both training and live use, so train/serve skew cannot happen?
- Where is the latency floor of this design on the same laptop?
NEXT STEP · NOT DONE YET
What would we do next?
Train only on the full 7-day recording, compute training and live features with the same code, and repeat the 24-hour paper run with the guardrails on.
Have a problem that needs this kind of research?
Tell us about it