2-second stalls from one synchronous disk write

What was the setup?

A research loop on a GPU, fed by live market-data streams, in paper mode. The loop also wrote records to disk as it went, and it had to keep up with the feed.

Why can one disk write stall a loop for two seconds?

Our log said it plainly: “2075ms spikes detected in GPU loop. Sync Disk I/O identified as cause.”

A synchronous write means the loop stops and waits until the operating system says the write is done. Most of the time that is quick. Now and then the system is busy flushing buffers to the disk, and the same call takes far longer. The loop cannot do anything else while it waits.

Think of a shop cashier who walks every receipt to the back office before serving the next customer. On a quiet day nobody notices. When the office has a queue, the whole shop stops.

What did the fix look like?

We moved the writes off the hot loop. The loop hands each record to a background writer and carries on; the writer deals with the disk at its own pace. Here is a minimal version of that pattern in Python. It is an illustration, not our production code.

import queue
import threading
from typing import BinaryIO


class BackgroundWriter:
    """Hands records to a writer thread so the hot loop never waits on disk."""

    def __init__(self, f: BinaryIO, maxsize: int = 10_000) -> None:
        self._q: queue.Queue[bytes | None] = queue.Queue(maxsize=maxsize)
        self._f = f
        self.dropped = 0
        self._t = threading.Thread(target=self._run, daemon=True)
        self._t.start()

    def write(self, record: bytes) -> None:
        try:
            self._q.put_nowait(record)  # never blocks the loop
        except queue.Full:
            self.dropped += 1           # count it, don't stall

    def close(self) -> None:
        self._q.put(None)
        self._t.join()

    def _run(self) -> None:
        while (rec := self._q.get()) is not None:
            self._f.write(rec)
        self._f.flush()

Two details matter. The queue has a maximum size, so a slow disk cannot eat all the memory. And when the queue is full, the loop counts a dropped record instead of waiting. For a real-time loop, a counted drop is better than a silent stall.

How much did it help?

The average went up a little between the two later runs, from 92 to 100 ms, while the worst case went down. For a loop that has to keep up with a live feed, the worst case is the number to watch.

What was the WSL networking bottleneck?

A second bottleneck had nothing to do with the disk. It was the laptop's network path. On an i5-1235U laptop, the code ran in Linux under WSL, the Windows Subsystem for Linux. WSL 2 runs Linux in a lightweight virtual machine with its own virtual network, so data from the outside world takes an extra hop to reach it.

The fix was to take the market data on the Windows side and send it into WSL over UDP. UDP has no handshake and no retransmission. A lost packet is simply gone instead of holding up the ones behind it, which suits data where only the latest value matters.

In this setup the live feed held 68 updates per second, and synthetic bursts at 200 Hz and 500 Hz were handled.

What this does not prove

• Max and average only. We logged the worst case and the mean, not percentiles. A p99 would say more.
• One laptop. Disk and network behaviour differ on other machines and on servers.
• Bursts were synthetic. 200 Hz and 500 Hz were generated loads, not a real feed at that rate.
• Engineering, not results. These numbers say how fast the loop ran, nothing about how good the model was.

Questions