Correction, August 2026. The central observation in this piece — that the backend performs no staging copy — is accurate. The conclusion drawn from it is wrong. There was no staging copy because every buffer was allocated host-visible, and on a discrete card that meant every store the shader executed crossed PCIe: a sustained 10.8 GB/s of device-to-host traffic during computations that transfer nothing. The absence I reported as an elegance was a defect, and this post explains what a staging buffer is for several paragraphs before failing to notice one was missing. Fixing it yields 2.0–11.3×. The full account is in An Absence Mistaken for a Discovery. Read what follows with that in hand; I have left it unedited.
A Jetson Nano joined a fleet of discrete GPUs. Its one distinguishing feature is unified memory — the CPU and the GPU share the same physical DRAM, with no PCIe bus between them. The obvious optimisation is to stop copying. I went to write it, and there was no copy. What the probe found instead was a cost on every box in the fleet, and then the unified-memory board won the fix least, not most. Both of those are the same fact wearing different clothes.
What unified memory actually looks like from Vulkan
Vulkan does not tell you “this device has unified memory.” It gives
you a table of memory types, each with a set of property flags, and you infer
the architecture from the shape of the table. On a discrete card the table has a
characteristic three-part structure: a large DEVICE_LOCAL region
the GPU can reach quickly and the CPU cannot reach at all, a large
HOST_VISIBLE region in system RAM that the GPU reaches across the
bus, and a small window — historically 256 MB, the PCI Base Address
Register aperture — that is both. Allocators are built around that
structure. The whole idea of a staging buffer exists because of it: you
write into host-visible memory because you can, then you copy into device-local
memory because that is where the compute wants it.
Here is the Tegra X1 in the Jetson Nano:
type 0, 1 : DEVICE_LOCAL
type 2 : DEVICE_LOCAL | HOST_VISIBLE | HOST_COHERENT
type 3 : DEVICE_LOCAL | HOST_VISIBLE | HOST_CACHED
type 4, 5 : DEVICE_LOCAL | PROTECTED
heap 0 : 3.87 GiB heap 1 : 400 MiB
Every type is DEVICE_LOCAL. Read that again, because it is the
whole architecture in one line: there is no memory type that is
host-visible and not device-local. The distinction an
allocator is built to navigate does not exist here. There is one pool of LPDDR4
and two processors looking at it.
That is the advantage, stated precisely. Not “transfers are faster” — there is no transfer to make faster. The bus that a discrete card's staging copy exists to cross is absent, so the copy has nothing to do. It is a structural absence rather than a speed-up.
The optimisation I went to write
The backend allocates with a filter that reads, in vulkano's vocabulary,
PREFER_DEVICE | HOST_SEQUENTIAL_WRITE. On a discrete card that says:
give me device-local memory if you can, but I need to write into it from the
host. My reasoning was that this is a discrete-GPU strategy being applied to a
board with no discrete memory, and that every upload was therefore staging bytes
that were already in the right physical place.
I was wrong, and the code says so plainly once you read it instead of reasoning
about it. The upload path is a single call —
Buffer::from_iter with a host-visible allocation — which
writes directly into mapped memory. There is no staging buffer on any
box in the fleet, discrete or otherwise. And on Tegra the filter resolves to
type 2 above, which is device-local, host-visible and coherent all at once, so
the “unified memory path” was already being taken. It had been taken
since the day the board was plugged in.
This is worth dwelling on, because it is the most common shape of a wasted optimisation: a mental model of the general case, applied to a specific case where it does not hold, producing a plan for work that has already been done by someone who never mentioned doing it.
What the probe found instead
Before scoping the work I benchmarked the two primitives, because the project has a standing rule that a “decided” verdict is a claim about the world and claims can be tested. The numbers said something I was not looking for:
size buf_alloc buf_upload
1 MiB 0.30 ms 0.26 ms
16 MiB 5.08 ms 3.64 ms
Allocating an empty output buffer cost more than uploading real data
into one. That is not a plausible shape for an allocation. Reading the
allocator explained it in one line: it was constructing every buffer from
std::iter::repeat(0u8).take(n_bytes) — a full host-side write
of the entire buffer, at memory bandwidth, to fill with zeros a region that a
shader was about to overwrite completely. Every operation that produced an
output paid it.
Vulkan has an allocation that does not initialise. Switching to it turns an O(n) write into an O(1) allocation, and 16 MiB went from 5.077 ms to 0.008 ms.
The interesting part is not the number. It is that this had nothing to do with unified memory. It was true on every box in the fleet, had been true for the entire life of the project, and was found only because a wrong hypothesis about one board sent someone to measure something adjacent to it.
The audit, which was the actual work
Removing a zero-fill is a two-line change and a large promise: that no shader
anywhere reads a byte of its output before writing it. Two patterns would break.
A kernel that accumulates — atomicOr into
uninitialised memory returns garbage, not a wrong-but-plausible number. And a
kernel that writes fewer bytes than the caller allocated, leaving padding
undefined.
Eighty-seven shaders, checked one at a time. Four are in the first class: the
all/any reductions, which set bits with
atomicOr, one thread per slot, into a packed byte mask. Those keep
a zeroed allocation. Nothing is in the second class, and that had to be checked
rather than assumed, because the packed-byte writers looked exactly like the
risk: they allocate (n + 3) / 4 * 4 bytes for n
elements, so a tail word is partly padding. Every one of them turns out to run
one thread per output word rather than per element, writing the tail
whole with its unused lanes zeroed in a register.
Then the part that separates a verified claim from a reasoned one. A clean test run is weak evidence here, because freshly mapped pages are zeroed by the operating system for security — so a shader that wrongly depends on zeroed memory passes a cold run and fails only once the allocator starts recycling blocks. The fleet's job was to defeat that, and the first attempt at doing so was vacuous: poisoning memory through the upload path produced zero dirty reallocations, because upload buffers land in a different suballocator pool. A green result from that control would have meant nothing at all. Only after finding a scheme that produced dirty memory on forty out of forty reallocations did the poisoned runs mean anything — and then four thousand of them, across three architectures, found no defect, with the padded buffers downloaded byte-for-byte to confirm the tails really were zero.
The prediction, and why it was backwards
I wrote in the commit that the Jetson would show this change most, having the slowest memory in the fleet. It shows it least. Here is the measurement, taken against a preserved copy of the old code path so it is a true A/B on one machine in one process rather than a comparison across commits and conditions:
16 MiB before after saved
RTX 3060 Ti 5.08 ms 0.008 ms 5.07 ms (635x)
Tegra X1 3.71 ms 0.10 ms 3.61 ms (37x)Smaller absolutely and smaller proportionally. And the control disproves the reasoning rather than merely contradicting the conclusion. The upload path, which this change does not touch, is slower on the Jetson — 6.0 ms against Ampere's 3.6 at the same size. The memory really is slow, exactly as claimed. But the zero-fill is faster there: 3.71 against 5.08.
Writing a constant into LPDDR4 that the GPU already shares costs less than Ampere's host write across PCIe into the BAR aperture. The discrete card's zero-fill was expensive because of the bus, not because of the memory. So:
Unified memory had already made the old path cheap. There was less to reclaim.
That is the sentence I would keep from all of this. The advantage of unified memory is not that copies become fast — it is that the bus stops being the thing you are paying for. Which means it helps enormously wherever the bus was the bottleneck, and does nothing at all wherever it was not. The corollary is uncomfortable and correct: an optimisation that removes a bus cost will show up worst on the machine with no bus. Both of my wrong conclusions — the staging copy that did not exist, and the win that was supposed to be largest — are the same misunderstanding, which is that I had filed unified memory under “fast memory” when it belongs under “no bus.”
A cliff that a round-numbered benchmark walks straight past
One more thing the fleet found, and it is a general hazard rather than a Tegra one. Benchmarking at 1, 4, 16 and 64 MiB suggests the new allocation is flat — O(1), a few microseconds, independent of size. A finer sweep says otherwise:
Tegra X1, MiB: 8 16 24 31 32 48 64
new path ms: 0.09 0.12 0.14 0.12 26.98 39.69 52.92
old path ms: 2.94 3.71 5.38 6.70 55.87 83.27 110.58At exactly 32 MiB the allocator stops suballocating from a pooled block and issues a dedicated device allocation, after which the driver commits and zeroes pages itself — an O(n) cost outside the application's control, running at roughly 0.83 ms per MiB. The change is still a clear win above the cliff, but it is a factor of two, not three orders of magnitude. Two boxes swept it independently and found the same edge, at the same size.
A power-of-two benchmark grid straddles that transition invisibly. If you have ever published a flat line, it is worth asking what your grid stepped over.
Why the board is on the fleet at all
There is a reason this hardware is interesting beyond its memory topology. The original Jetson Nano is a Tegra X1 — Maxwell, compute capability 5.3 — and its software stack is terminal. JetPack 4.6.x is the last release for the board; CUDA stops at 10.2 and will not move, because CUDA 11 first shipped in a JetPack that dropped this hardware. Meanwhile the precompiled XLA binaries that an accelerated tensor library would use start at CUDA 12. There is no version of those two facts that meets.
So the standard accelerated path cannot run on this board at all. Not “slowly” — at all. What runs is Vulkan, because Vulkan is a portable API with a driver on the device, and a compute shader does not ask what year it is. The board is a full member of a correctness fleet spanning Ampere, Kepler and Maxwell, agreeing to the digit on eight hundred and sixty tests, for exactly one reason: the kernels were written against an API that outlived the vendor's support window.
That is the same argument this site has been making about stranded silicon, arriving from a different direction. The interesting property of a portable compute API is not raw throughput. It is that it keeps hardware addressable after the vendor has stopped addressing it — and unified memory, on a board nobody supports any more, turns out to be a genuinely distinct architecture worth measuring rather than a footnote.
What I would tell someone starting this
Three things, in order of how much they cost me.
Read the allocation code before theorising about the copy. The
staging buffer I set out to remove had never existed. Ten minutes with
alloc_buffer would have said so, and would also have surfaced the
zero-fill four months earlier.
Measure the control, not just the change. The reason the falsified prediction became useful rather than merely embarrassing is that the unchanged upload path was measured alongside it. Upload slower, fill faster, same box — that pair is what turns “the guess was wrong” into “the mechanism is the bus.” A benchmark of only the thing you changed can tell you that you were wrong. It cannot tell you why.
Prove your adversarial test has teeth before you trust it passing. The poisoning scheme that produced zero dirty reallocations would have blessed a broken audit with four thousand clean runs. The discipline that saved it was checking that the control could fail.
The backend is nx_vulkan, an Nx tensor backend on Vulkan compute. The allocator note, the shader audit and the fleet measurements all live in the source rather than in a document, on the theory that the place you need a warning is the place you would otherwise make the mistake.