An Absence Mistaken for a Discovery

Some weeks ago I published a piece reporting that a Vulkan compute backend performed no staging copy between host and device, and I presented this as a finding. The reasoning was that unified memory ought to render such a copy unnecessary, that I had gone looking for one to eliminate, and that I had found the work already done. It is a pleasant genre of result — the optimisation that turns out to be unneeded — and I wrote it up with the mild satisfaction of a man who has been spared some labour.

That post contained, some four paragraphs earlier, a correct and complete explanation of what a staging buffer is for. You write into host-visible memory because you can, then you copy into device-local memory because that is where the compute wants it. Having set out the reason the mechanism exists, I observed that the mechanism was missing, and concluded that this was fortunate. I had found nothing, and I had written it up.

The instrument is always the first suspect, and it took four tries to remember it

What followed was an attempt to measure whether unified memory changes the shape of a cost curve. Four experimental designs, each defeated in turn, and not one of them by the hardware. The first measured a GPU that idles its clocks to a tenth of speed and boosts under load, so that a repeat measurement read as 288 per cent thermal drift on a box whose load had not moved. The second leaked every buffer its own warm-up allocated, which killed the run outright on the small cards and, far worse, was silently absorbed on the large ones. The third rested on a rule I had written myself, to the effect that normalising a measurement against its own machine cancels a slow host: true of a multiplicative factor, false of an additive one, and the host cost was additive.

The fourth deserves its own sentence, because it is the one I was proudest of. I constructed a dimensionless ratio — the cost of a host-device round trip divided by the cost of the computation it wrapped — on the argument that host speed, driver, socket and card appear in both terms and must therefore divide out. They appear in both terms. They do not divide out, because the numerator is bounded by a bus and the denominator by a processor, and two quantities do not cancel merely by sharing a machine. A discrete Kepler then obligingly grouped itself with the unified Tegra rather than with the discrete Ampere, which is the sort of result that should end an argument, and did.

Four designs, three of the failures introduced while repairing the previous one, and a common thread running through all of them: I kept asserting that terms appearing on both sides of a fraction would cancel, without troubling to check that they were the same kind of quantity. It is a cheap error. It is also the only one I made, over and over, in four different costumes.

Ten point eight gigabytes per second of nothing in particular

Abandoning the question, I turned to something smaller and more tractable: an elementwise multiply that ran at 16.4 GB/s on a card advertising 448. A factor of twenty-seven is not a subtlety. It is a klaxon.

The usual suspects were dismissed in order. Not dispatch overhead: chaining sixteen operations under a single submission gave a flat marginal cost, 1.84, 1.79, 1.81, 1.96 milliseconds apiece, which is the signature of per-operation work and not of per-submission ceremony. Not a silent fallback to the CPU: the fallback counter reported an empty map and the result was resident, in f32, on the device. Not the shader, which is as unremarkable a piece of GLSL as one could write — two hundred and fifty-six threads to a workgroup, one element per thread, coalesced indexing, a single bounds check.

So I asked the machine what it was doing, which is a question I might have asked considerably earlier. nvidia-smi dmon reports PCIe traffic per second in each direction. I ran a computation that transfers nothing: one resident input, one operation, a flush, and a garbage collection, around and around for twenty seconds.

# gpu  rxpci  txpci      (MB/s)
    0      4  10802
    0     21  10802
    0      4   9903
    0     21  10802
    ...sustained for the entire loop

Ten point eight gigabytes per second, from the device to the host, during a computation that moves no data between them. The write bandwidth implied by the timing is 8.7 GB/s, which is the same number wearing a different hat. And the reverse direction sits at four to twenty-one megabytes — noise — even when the operation reads two full tensors, which tells you that the inputs were in video memory and only the output was not.

One line, and the word that does the damage is not the one you would guess

memory_type_filter: MemoryTypeFilter::PREFER_DEVICE
    | MemoryTypeFilter::HOST_RANDOM_ACCESS,

A preference and a requirement, joined by a pipe, and it is never the preference that wins. PREFER_DEVICE asks politely for video memory. HOST_RANDOM_ACCESS insists on memory the CPU can read at random, which on a discrete NVIDIA card means host-cached, which means not device-local, because the card does not offer a type that is both. Every output buffer the backend allocated therefore lived in system RAM, and every store the shader executed — not every transfer, every store — went out across the bus.

The requirement was there for an honest reason. The download path reads those buffers directly from the host, and reading write-combined memory from the CPU is a punishment. Someone — and the blame is not diffuse, the repository knows who — wanted the readback to be quick and asked for cached memory to get it. The bill for that convenience was presented on every subsequent store, in perpetuity, and nobody read it.

The absence was not a discovery. The absence was the defect.

Here is the part I would rather not write. There was no staging copy because the buffers were host-visible. The buffers were host-visible because no copy was wanted. The two facts are the same fact. I had observed a system with its staging buffer removed, correctly noted the removal, and reported it as an elegance.

What it had actually done was trade one copy per transfer for a bus crossing on every write the GPU performs. Transfers are rare in any resident workload; stores are the workload. The trade was not merely bad, it was bad in the direction that compounds, and the earlier post's satisfied tone was the sound of a man admiring the hole where a load-bearing wall used to be.

The correction is unglamorous. Allocate compute buffers device-local and stop asking for host access. Keep a host-visible staging buffer. Copy at the boundary with vkCmdCopyBuffer. Zero on the device with fill_buffer rather than writing a megabyte of noughts from the CPU. This is, to be entirely clear, the textbook arrangement that the earlier post had already described in correct detail before failing to notice it was absent.

The PCIe traffic falls from a sustained 10,800 MB/s to single digits. The collapse at 64 MiB, which I had catalogued separately as an allocator cliff, turns out to have been the same defect compounding: above the threshold at which the driver stops sub-allocating, the misplacement got worse. Eight hundred and thirty-three doctests and eight hundred and seventy-one tests pass exactly as before, which is the only part of this that was ever in doubt in the right direction.

The table above describes one machine, which I had better say before I do it again

Those figures come from a single card. A second box — a 2012 Kepler on an older driver and a different operating system — built both commits in one session and ran the identical probe against each, which is the method I should have insisted on before writing a number down at all.

Below the allocation cliff it gained precisely nothing. The entire benefit on that card is the above-cliff path, where the collapse I described was real and is now largely gone. The reasonable inference is that the older driver could satisfy the host-visibility requirement out of device-local memory for small allocations and could not for large ones, while the newer driver displaced everything — so the defect is general and its cost is not.

Which means the range in the previous section describes one machine, and I have no business calling it a speedup without a qualifier. I mention this with some asperity because the entire preceding essay is an account of over-reading a single observation, and I came within an hour of doing it again in the paragraph announcing the repair.

The same box also produced the largest effect anyone has measured from this change, in a place I had not thought to look. Allocation above the cliff fell from 23–30 ms to 5.6–6.7; zeroed allocation from 39–53 to 6.4–7.4, a factor of seven; and the fitted cost-per-megabyte slopes by roughly a factor of ten. Device-local memory is simply cheaper to allocate than host-visible memory, and the zeroing path no longer writes a megabyte of noughts across the bus to do it. That is a real and general improvement which I would have missed entirely had I only measured the thing I was looking for.

And the unified board's advantage was the absence of a penalty

Now the sting, which I did not see coming and which reorganises everything above it.

The Jetson looked good in every one of those four failed comparisons. Its crossing cost matched its compute cost almost exactly; its curves were flat where the discrete cards' rose; it was, on every metric I built, the box where the boundary was cheapest. I had begun to assemble a thesis around this.

It looked good because it was the only machine not paying the tax. On Tegra every memory type is device-local — that is the whole architecture, stated in one line of a memory-properties table — so the requirement that banished the other cards' buffers to system RAM cost the Jetson precisely nothing. Its rivals were running their computations over a bus. It was not. The unified-memory advantage I had spent four experiments failing to measure was, in the portion I could actually see, an artifact of a defect on the other three machines.

There is a final turn, and it is still in flight as I write. The fix helps exactly the boxes the bug harmed, which means it may do nothing for the Jetson and might well make it slower: on hardware where every allocation is already device-local, a staging buffer is a copy that buys you nothing and costs you a memcpy in each direction. The correct answer is probably to branch on whether the device offers a host-visible-but-not-device-local memory type at all — which is to say, to detect unified memory and behave differently — which is where this entire line of inquiry began, several months and one retracted finding ago.

And the repair has a bill of its own, which I also failed to measure before announcing it

A third machine did what I ought to have done: it built the old commit as well as the new one, and measured the thing I had changed rather than only the thing I had hoped to improve. Its elementwise figures agree with the second box's — nothing below the cliff, 2.37× above it. Then it measured the cost of the staging copy I had introduced.

Every host-to-device crossing on that card is now forty per cent slower. I had predicted the device figure would rise while the crossing held; neither happened. The copy is not free, which is a thing I might have deduced from the fact that it is a copy.

So the honest ledger for that machine reads: a genuine gain on large-buffer compute, nothing at all on small, and a standing forty per cent tax on every transfer — netting positive or negative entirely according to whether the workload computes more than it moves. The measurement that showed the gain and the measurement that showed the tax were both correct and pointed in opposite directions, because one of them stops at sixteen megabytes and the win lives above thirty-two.

This is the second argument, arriving from the opposite direction, for making the staging path conditional rather than compulsory. The first was that a unified-memory board may pay for a copy that buys it nothing. The second is that a discrete board pays for it too, and only earns it back above a threshold. A change that is correct in principle and unconditional in practice is a change that has not finished being designed.

One further embarrassment, and it is the same one again. The check that was supposed to prove the fix safe — a harness that poisons freed memory and confirms nothing reads uninitialised bytes — began reporting on one card that it could no longer establish its own claim. It says so rather than passing quietly, which is the one piece of engineering in this account that behaved exactly as designed. I wrote up the fact that a second card disagreed as a finding about the two cards.

It is not. Three consecutive runs on the same machine, same commit, gave 20/20, then 0/20, then 20/20, with the rest of the harness's poisoning statistics flipping in lockstep. The behaviour is bimodal per process, and I had sampled it twice and both times landed on the same face of the coin. The cross-machine table I published may yet turn out to describe something real; it currently describes two coin flips. One is entitled to a certain weariness at this point, and the weariness should be directed inward.

I record all of it because the alternative is to quietly ship the patch and let the earlier piece stand. The instrument was wrong four times and I was wrong once, and it is the once that matters: a missing mechanism is not a discovery until you have established what it was there to prevent. The machine had been announcing the answer at ten point eight gigabytes per second, continuously, to anyone who thought to ask it. I got there eventually, by the unimpressive route of running out of cleverer ideas.

The defect and its measurement are recorded in docs/ELEMENTWISE_PCIE_TAX.md. The piece being corrected is The Copy That Wasn’t There; the four failed designs, with their instrument faults enumerated, are in bench_results/UNIFIED_VS_DISCRETE.md. The habit of counting what you fear rather than what you hope is the subject of The Alibi of a Correct Answer, which I evidently needed to read again.