The Shader That Repeated Itself ended with a 2013 Kepler that could finally compile a model in a hurry. But one Kepler is one Kepler. A three-regime mixture posterior — the model that decides whether a market is trending, mean-reverting, or merely volatile — takes between 37 and 108 seconds to sample on a GT 750M. A fifteen-minute trading cycle across a fleet of instruments needs far more posteriors than one GPU can chew through in sequence. The problem was never the arithmetic. It was the scheduling.
The serial wall
A single GPU samples one posterior at a time. At roughly a minute each, one Kepler tops out near ten or twelve instruments per cycle; past that, the queue grows faster than it drains. There is a name for what happens next, because it happened: one run took in 258 sampling jobs, completed 3, let 141 go stale, and left 7 workers wedged. That is not a slow GPU. That is a serial resource asked to do parallel work — a scheduling wall wearing a performance problem’s clothes. And the whole time, two other correctness-verified GPUs sat in the same building doing nothing.
Offload the computation, not the trader
The obvious move is to shard: give each host a slice of the instruments and let it run its own trader. It’s the wrong move. Sharding scatters trading logic — and, worse, the ability to place orders — onto every host, to save a coordination cost the heavy model never actually pays. The right move is the opposite: keep all coordination on one node and move only the one expensive thing. A coordinator hands each posterior to whichever GPU in the cluster is free; the result comes home; the coordinator alone decides and places the order.
This buys a safety property for free. The worker nodes are given no broker and no account keys — so they cannot place an order, not by policy but by absence. The Keplers compute; only the coordinator trades. What used to be “paper-only if we configure it right” becomes “inference-only by construction.”
The model travels as a description, not a shader
Here is the part that surprised me. When the coordinator hands a job to a
remote Kepler, what crosses the wire is not a shader. It is the model’s
description — an Nx IR: the priors, the mixture
likelihood, the observations. The worker builds the shader itself. And it does
not decode that description at run time the way a virtual machine
interprets bytecode. It transpiles it, once:
extract_components(ir) // the free RVs and their densities
-> render to GLSL // the log-density and gradient become
// literal GPU arithmetic, spliced into a
// fixed leapfrog-integrator template
-> glslangValidator // GLSL -> SPIR-V
-> content-addressed cache // keyed by a hash of the GLSL
The synthesizer walks the log-density expression and emits it as GLSL, dropping
it into the holes of a fixed leapfrog kernel — the softmax denominator
log(1 + exp(min(w1,10)) + exp(min(w2,10))) becomes literal shader
text (and this is exactly the text that
common-subexpression
elimination shrinks). The model’s structure is compiled into
a bespoke SPIR-V binary. Its prior hyperparameters are baked in as constants.
Only the parts that change per sample — the position q, the
observation buffer, the step size, the observation count — are pushed to
the shader at dispatch, through a 24-byte push-constant block and a handful of
storage buffers. The GPU never parses a model. The model became the
GPU program.
Which means the first posterior of a given shape on a given worker pays a
quarter-second of glslang; every posterior after that is a warm
cache hit. The description arrives; the shader is minted once; the fleet runs
warm thereafter.
One portability detail earns its place in the code. An IR built on a GPU backend holds device-resident tensors — handles to memory on the wrong machine, meaningless once they cross a node boundary. So a mesh-bound IR is built with its tensors pinned to the plain host backend, as ordinary bytes, and the worker moves them onto its own GPU. Build it portable; let the destination localise it.
The measurement
First, a single real posterior across the wire. A coordinator on a Linux box
built a genuine three-regime IR and dispatched it to a worker on a FreeBSD
Kepler. The Kepler synthesised the shader, sampled on Vulkan with zero
divergences, and cast the full posterior — mu_trend and all
eight parameters — back across the node boundary. Seventy seconds for 40
warmup and 40 samples on the thirteen-year-old GT 750M.
Then two Keplers, four posteriors:
4 posteriors, 1 worker : 184,672 ms (+47s, +93s, +138s, +185s — strictly serial)
4 posteriors, 2 workers: 96,468 ms (two run at once)
------------
1.91x — near-linear
On one worker the jobs finish forty-six seconds apart, one at a time — the serial wall, drawn in timestamps. On two, they finish in pairs. Add a GPU, and the wall recedes by almost exactly the GPU you added.
The honest part
Routing a posterior to a 2013 Kepler is not faster, per posterior, than computing it on the coordinator’s own modern card. If the coordinator isn’t the bottleneck, offloading to slower silicon adds latency, not throughput. The mesh does not make any single posterior quick. It makes many posteriors finish sooner — when there are more of them than one GPU can serially chew, N GPUs chew N at a time. And because the model compiles once per worker, the effort spent shrinking that compile compounds across the whole fleet: each worker pays the small cost once, then runs warm forever.
What makes it worth doing isn’t peak FLOPS. It’s that the fleet is heterogeneous and mostly idle — a modern Ampere card and two thirteen-year-old Keplers, each verified against the same reference draws, sharing the load. The Vulkan backend that let a posterior run on any GPU is what lets it now run on all of them at once. Portability was never a convenience feature. It was the precondition for the fleet.
The model ships as a description and becomes a shader wherever it lands. Once a posterior can run on any GPU, every idle GPU in the building can run one at the same time — and a rack of machines nobody was using becomes a single inference engine.