Receiving a Market-Data Feed

Notes from the Packet Path

1. The assignment

Our broker recently offered a new generation of CFFEX market data — full-depth futures and options quotes — delivered as raw multicast to a colo box, with a closed reference library to read it. The library works, but it lands quotes in its own process, in its own format, on its own clock. We wanted the packets inside our market-data daemon instead: one normalized quote stream, one timestamp discipline, one process to monitor. Between the two sat a body of work this note records — the wire, the cipher, two threading bugs, and one race at startup — roughly one commit per problem, each found the honest way: in production.

Two rules governed the whole job, and they are the two I would keep next time:

• Validate standalone, then port the shell, never the logic. The decoder ran first as an isolated program, verified against the official shared-memory pipeline for a full session: zero differences, tick for tick, same-millisecond latency. When it moved into the daemon, the protocol, decryption, and merge logic were not touched at all — only the outer shell changed (receive path, logging, symbol filtering). Port and rewrite are never allowed in the same change.
• Everything measurable gets a counter. Every drop, skip, retry, and queue depth peak is counted and logged. Half the story below is only knowable because the counts existed.

2. What the wire looks like

No public specification comes with the feed, so the format was recovered from observation and from the behavior of the reference library. In outline: each UDP frame carries a small header, then a list of tag-length-value fields; heartbeats arrive once per second per source; incremental quote updates merge into per-instrument snapshots. Three findings are worth recording because each cost a day and none was guessable from the outside:

• Prices are deltas, not absolutes. A price field is an integer scaled from a per-instrument base, per a per-instrument tick. Getting the base wrong by one tick looks like plausible, slightly wrong prices — the worst kind of bug. The only empty-slot sentinel is a specific all-ones 32-bit pattern; a decoded −1 is a real, tradeable price one tick below base. Confusing the two corrupts volume and turnover accounting.
• The exchange timestamp counts seconds since midnight, not a decimal HHMMSS. Anything that parses it as a clock time produces a beautiful, wrong curve at 10:00 and 13:00, when the two interpretations diverge.
• A single sequence counter covers the whole market and increments by exactly one per frame. This is a gift: gap detection, duplicate detection, and end-of-day accounting come free, and any loss anywhere in your pipeline is visible as a hole in the count. We checked it at every stage we could.

3. The day cipher

Since this autumn, the feed is encrypted in full. The interesting part is the key management: the cipher changes daily, and the day’s cipher is not a key value — it is delivered, inside the pre-market login response, as executable code. The receiving program maps that code segment executable and calls it, field by field, over the packet. Nothing about any single day’s algorithm is worth reverse-engineering, because hard-coding today’s algorithm simply fails tomorrow morning when the exchange rotates it; treating the delivered code as opaque and calling it is the only durable design. The lesson generalizes: when the protocol’s own design says “this changes every day,” build an interpreter, not a memory.

That login response, then, is the day’s root of trust — one file, produced before the open. Remember this for §6.

4. Day one: 4,984 frames

The port went live on a Tuesday. The receiver runs on a kernel-bypass path (EfVi): the NIC delivers frames into a receive ring of 1,024 slots in user space, no kernel, no copies. The first session lost 4,984 frames — all of them inside the two open-burst windows at 09:30 and 13:00, with 31,522 starved-descriptor events alongside.

The forensics wrote themselves, because the loss pattern named the culprit: not the network, not the box — the burst. At the open, a futures-and-options feed delivers a wall of updates at once, and our receive loop was doing decrypt + decode + snapshot-merge + publish inside the receive callback. Every millisecond spent publishing was a millisecond the ring went un-drained; during the avalanche the ring wrapped, and frames were overwritten before they were ever seen. Quiet hours: perfect. Open: holes in the sequence count.

4.1 The structure: three arrays, two threads

The fix does not tune the receive loop — it removes the loop’s reason to be slow. The process gains one more thread and three arrays:

• a slot pool — 32,768 fixed slots of 2,048 bytes each (64 MiB total); every slot holds a 2-byte length plus one frame’s payload;
• a free ring — a circular array of slot indices, counting the slots that are safe to write;
• a ready ring — the same, counting slots that hold a frame awaiting work.

Each ring carries two 64-bit counters, head and tail, with one invariant doing all the work: head is written only by the consumer, tail only by the producer. With a single writer per side, no lock is ever needed — synchronization is a handful of atomic loads and stores on cache-resident counters, and the producer’s entire critical path is copy-in, two index bumps:

producer  (receive thread)                 consumer  (worker thread)
------------------------------------------------------------------
h = free.head                             h = ready.head
if h == free.tail:                        if h == ready.tail: keep spinning
    drop++, return        // full         idx    = ready.buf[h & MASK]
idx = free.buf[h & MASK]                 ready.head = h + 1
free.head = h + 1                         payload = slots[idx]
slots[idx].len = len                      handler(payload)   // decrypt+decode
memcpy(slots[idx].body, frame)            //        +merge+publish
ready.buf[ready.tail & MASK] = idx        free.buf[free.tail & MASK] = idx
ready.tail = ready.tail + 1              free.tail  = free.tail + 1

No locks, no allocation, no system calls on the receive path — about a hundred nanoseconds from packet to queue, then straight back to polling the ring. “Consumer too slow” as a loss mechanism is not mitigated; it is structurally eliminated.

4.2 Sizing: drops become a never-observed event

The avalanche peak on this feed is on the order of ten thousand back-to-back packets; 32,768 slots is several multiples of that, deliberately. The sizing argument is the design: a burst that fits in the queue is delayed, not lost — the worker accumulates backlog through the open and drains it in the seconds after, while quotes keep publishing at full depth. The drop counter at the queue remains as a tripwire for a burst that exceeds all sizing assumptions, visible as one attributable number — never again as a silent ring overrun whose absence of frames is only discovered at reconciliation.

Order survives too: one queue, FIFO, so frames reach the decoder in wire order and the market-wide sequence counter keeps its dedup and gap semantics unchanged. A parallelized decode — several workers pulling frames — would have been faster on paper and wrong in fact: reordered frames would force the merge logic to re-sort, and a dropped frame would no longer be a clean hole in one count.

4.3 The worker: pinned, spinning, instrumented

Three smaller decisions complete the pipeline, each paid for by a bug it prevents:

• Affinity is set inside the thread body. The pinning call binds whichever thread executes it; setting it in the parent and then spawning quietly binds nothing — a bug that presents as excellent benchmark numbers on the wrong core. A pinned worker that cannot get its core fails loudly at start-up instead of floating.
• The worker never sleeps — this was §5’s fix, and it belongs to the same loss story: the sleep’s wake-up latency was a component of drain time during bursts.
• The queue is instrumented, not averaged. Every ten seconds the worker records the queue’s high-water depth; a peak above 64 slots, or any drop in the window, raises one warning line. That number is the dashboard which decides whether the next optimization (a cheaper merge structure) is worth building. Without it, “the queue feels fine” is a vibe.

Even shutdown follows the same discipline: on stop, the worker exits only when the ready ring is empty — the pipeline may wind down, but it does not swallow its tail.

5. The twenty-microsecond nap

With the pipeline split, the worker idled between frames. The polite implementation sleeps a little when the queue is empty — ours slept 20 microseconds, a nice compromise that caps latency and “doesn’t burn the dedicated core.” Measurements said otherwise. Side by side against the standalone reference, the daemon’s path was slower by 4.4 ms on average, with 11% of updates over 20 ms — on identical packets, identical decode. The sleep was the whole story, twice over:

1. Waking up is not free. A sleeping thread rejoins the run queue, and that re-queue wait is precisely the millisecond-scale tail we were measuring.
2. An idle core invites squatters. The core was budgeted to this component when the box was partitioned. Sleep, and the scheduler sees an idle core and floats unrelated work onto it; the partition convention binds registered components, not strangers. Sleeping saves no quota — the core was paid for at registration — it only donates it.

The worker now busy-spins with a pause instruction (which yields the pipeline to a hyperthread sibling). Latency matched the reference from that morning. On a dedicated core, sleep is not thrift — it is abandonment.

6. The login-response race

The last bug was not in the packet path at all. Recall from §3: the day’s cipher arrives in a pre-market login response, captured to a file by a separate script in the last hour before the open — on its own schedule, not ours. The daemon, meanwhile, is started even earlier by operations. One morning the daemon read the file before the capture script had written it, and exited — the failure surfacing only when someone looked.

The fix has three layers, and each is a general rule:

• A deadline, not a hope. The daemon now waits until a configured wall-clock time for the file to appear; if the moment is already past (a mid-day restart, say), it skips the wait — the file should long since exist.
• Retries with a fatal end. Missing, empty, or short file means the writer may still be writing — retry every minute, three times. Three failures is not bad luck; it is a dead upstream, and the process exits loudly rather than running headless with no cipher.
• Serve before you’re ready. The daemon’s listener sockets now open before the wait, so downstream consumers can connect immediately and simply receive nothing yet — the same “connectable but quiet” contract the shared-memory consumers have always had. Dependencies resolve in the background; the door is open from minute one.

The same commit moved all fatal config validation ahead of any waiting: an invalid value (we support one or five book levels, nothing else) kills the process at start-up — instantly, in plain text — instead of at 08:55 with the market thirty-one minutes away. Fail at boot for what you control; fail at the deadline for what you don’t; never fail silently.

7. The checklist

1. Validate the decoder standalone against a trusted source, then port only its shell. Zero-diff first, refactor never at the same time.
2. The receive thread receives. Everything slower than a hundred nanoseconds lives downstream of a lock-free queue.
3. The queue never backpressures the receiver; it drops loudly and counts.
4. Instrument high-water marks, not averages — the peak is the design input for the next optimization.
5. Set thread affinity inside the thread body.
6. Never sleep a dedicated core; spin with pause. The quota was paid at registration.
7. When a protocol rotates by design, treat its payloads as data to interpret — never hard-code a snapshot of it.
8. Startup order: open your doors first, wait for day-inputs against a wall-clock deadline, retry briefly, then fail fatally and visibly.

A feed like this is never “done” — the exchange will rotate something else eventually. But the packet path is now boring, which in market data is the highest compliment a system can receive.