AI-hardware prices are rising in the places that matter. That should be a tailwind for the sixty-dollar decade-old GPU. It half is — and the other half is a wall no benchmark can talk its way past. This is a memo about which half is which, and where a Vulkan tensor backend lives in the gap.
Here is a hypothesis worth taking seriously, because a lot of homelab folklore leans on it: the price of everything involved in training and serving models is going up, so the discarded hardware of five and ten years ago — the Keplers and Pascals piling up on the secondary market — should be getting a second life, if only the software were good enough to wring the most out of it. Cheap silicon, clever kernels, second chances.
I went looking for whether it holds. The honest answer is that it holds in a
shape different from the one the folklore assumes, and the difference is the
whole point. So this is the argument, in three claims and a correction, and then
where nx_vulkan — the Vulkan tensor backend this site has been
documenting for a season — actually sits in it. The full sourced version,
with confidence tags on every number, lives in the repo; this is the readable
one.
Claim one: the prices really are going up — in the right places
The first thing to get straight is that “AI hardware is getting more expensive” and “compute is getting cheaper” are both true, and they are not in contradiction. They are two views of the same machine.
The costs that are rising are the ones you cannot design around. High-bandwidth memory — the HBM stacks that feed a modern accelerator — is sold out for all of 2026 under fixed-price contracts, with double-digit price hikes booked and a shortage the vendors themselves project past 2027. It has gone from a fifth of a datacenter GPU’s bill of materials to more than half. Grid power is the other one: datacenter electricity demand jumped seventeen percent in a single year, and one US capacity auction cleared at roughly eleven times its prior price. New accelerators are allocated, not sold — Blackwell is spoken for through the middle of the year, with a backlog measured in millions of units. The cost of a single frontier training run is growing something like two-and-a-half times a year and is headed past a billion dollars.
The cost that is falling is the one the headlines love: cost per useful FLOP, dropping thirty to forty percent a year; inference measured per token, down more than two orders of magnitude in eighteen months. The reconciliation is an old one — Jevons’ paradox, which Satya Nadella name-checked in January. Make the effective unit cheaper and demand outruns supply, so the unit price falls while the aggregate bill and the scarcity both climb. Both things are true. The hypothesis leans on the climbing half, and the climbing half is real.
But note what the falling half is doing while we’re not looking. The same force that makes compute cheaper per FLOP is the force that makes the newest hardware the most attractive place to buy that FLOP. That is the counter-pressure the second-chance thesis has to survive, and it is the reason the next claim needs a correction.
Claim two: old hardware gets a second chance — but a narrower one than advertised
The supply is unquestionably there. Hyperscalers stretched their depreciation schedules to five and six years, which is the accounting that quietly floods the secondary market with three-year-old gear; Amazon then reversed course in early 2025, shortening the life it assigns its servers and taking a near-billion-dollar charge for early retirements, explicitly blaming the pace of AI. Old datacenter GPUs do not get scrapped so much as cascade down the tiers — training, then inference, then dev boxes, then eBay.
And on eBay the prices are almost funny. A Tesla P100 with sixteen gigabytes of real HBM2 goes for around eighty dollars. A P40 with twenty-four gigabytes runs a couple hundred. A K80 can be had for sixty. What is striking is what the market prices: not speed, but capacity. The P40 commands its premium over a compute-identical M40 for exactly one reason — twenty-four gigabytes in a single usable pool, which is the number the local-LLM crowd is bidding on. And it works: a P100 will do fifty tokens a second on a 7B model; a four-P40 rig assembled for the price of a laptop will serve a 120-billion-parameter mixture-of-experts model at twenty-eight. These are real, reproducible numbers, not vibes.
So where is the correction? Two places.
The first is a hard, dated line: in 2025, CUDA 13 dropped offline compilation and library support for Maxwell, Pascal, and Volta. The minimum architecture the modern NVIDIA toolchain will target is now Turing. PyTorch followed; the fast attention kernels require Ampere; bfloat16 requires Ampere. Everything older than about 2018 fell off the CUDA cliff in a single release. That is precisely why the K80 costs sixty dollars — not because it stopped computing, but because the software world it was born into evicted it.
The second correction is what the people actually doing this buy. It is not the ten-year-old card. The consensus “best value” is a used RTX 3090 — a 2020 Ampere part, twenty-four gigabytes, six to nine hundred dollars, with real tensor cores and bfloat16 and the fast-attention path intact, four or five times a P40. The genuinely ancient silicon is cheap because it is stranded. So the defensible version of the thesis is not “five to ten years old.” It is roughly four to eight, and the upper, older end of that band is viable on one condition: that you leave CUDA behind. Which is the whole reason the third claim is the interesting one.
Claim three: software unlocks it — up to a wall it cannot move
This is the load-bearing claim, and it is the best-supported of the three, because it rests on a model rather than a hope.
The model is the Roofline — Williams, Waterman and Patterson, 2009. It says a kernel’s attainable performance is the minimum of two ceilings: what the arithmetic units can do, and what the memory system can feed them, the second scaled by arithmetic intensity, the FLOPs you perform per byte you move. Two numbers you can look up for any device — peak compute, peak bandwidth — fix a crossover point that tells you, for any given kernel, which ceiling you are actually under. This is what “the well-researched boundaries of a specific hardware combination” means, concretely. It is not a metaphor. It is a chart you can put a kernel on and read its fate off the axis.
And it delivers a genuinely non-obvious result for exactly the workload in question. Language-model inference splits in two: the prefill is compute-bound, but generating each token is memory-bandwidth-bound, arithmetic intensity down near one. That is why quantization — the trick of moving fewer bytes per weight — is the lever that rescues weak hardware, and why the thing that limits a decade-old card is almost never its arithmetic. It is the memory subsystem. Knowing which boundary you are under tells you which knob matters.
The second half of the mechanism is the escape hatch from the CUDA cliff. Vulkan compute — and its portable intermediate form, SPIR-V — runs against the graphics driver, not the CUDA toolchain. So it keeps working on a card CUDA has formally abandoned. The proof is on a public scoreboard: a llama.cpp Vulkan build gets a decade-old AMD RX 470 doing two hundred tokens a second on a quantized 7B. One binary spans NVIDIA, AMD, Intel, Apple through MoltenVK, Android. The vendor stops supporting the silicon; the vendor-neutral API does not care.
Which brings us to the wall. The same roofline that tells you how to reach a device’s ceiling also tells you the ceiling is real and names it: memory bandwidth an old subsystem cannot exceed no matter how good the kernel; VRAM capacity that a 70-billion-parameter model simply does not fit inside; hardware features that were never fabricated — no tensor cores before Volta, half-precision on consumer Pascal running at one sixty-fourth of single. And there is a price for the escape hatch itself: Vulkan runs about twenty to thirty percent behind CUDA on the very same NVIDIA card. Above all there is perf-per-watt, where an H100 does roughly three times the inference per joule of an A100, and the gap only widens as you go back in time. On a grid meter that never stops, the cheap old card is the expensive choice.
The reframing
Put the three claims together and the naive form of the hypothesis — that old hardware is secretly as good as new if only the software tries hard enough — does not survive. The perf-per-watt evidence and the CUDA cutoff kill it. Software does not repeal the wall.
But the useful form does survive, and it is worth saying precisely, because the precise version is the one that is actually true:
Software doesn’t move the wall. It moves the line of what’s worth doing on your side of it.
Rising new-silicon cost and structural scarcity widen the band of workloads for which slower, cheaper, already-owned, correctly-utilized old hardware is the rational choice. Vendor-neutral, roofline-disciplined software is the thing that converts a stranded card from e-waste back into a compute asset — not by making it fast, but by making it usable for the work that fits under its ceilings. The hardware you already own, driven to the edge of its known boundaries, instead of the hardware a vendor would like to allocate you.
Where the backend sits
I have been describing the general case, but I chose it because the specific case has been the subject of this site for a season, and it turns out to be an unusually clean instance of the argument.
nx_vulkan is a Vulkan tensor backend for Elixir’s Nx. It is
not a language-model runtime — its slice of the thesis is scientific and
numerical compute in double and single precision, the kind an Axon training loop
or a NUTS sampler needs. But it lands exactly on the seam the three claims
describe, and it is validated on precisely the stranded hardware in question.
The test fleet is a 2012 GeForce GT 650M — a fourteen-year-old mobile Kepler, on FreeBSD, a platform CUDA never served at all — alongside its sibling GT 750M and a 2021 RTX 3060 Ti as the modern control. The whole suite, eight hundred and sixty-three doctests and three hundred and sixty-one tests, runs green on all three, in f32 and f64, byte-identical against the reference backend. That is the thesis running in a lab: the vendor-neutral API keeping a card alive a decade past the point its vendor stopped caring, and being the only GPU path on an operating system CUDA never shipped for.
And the engineering method is the roofline discipline, done empirically rather than assumed. The backend does not guess where the boundaries are; it races them on the actual cards. Sixteen-by-sixteen tiling to fix a matmul cliff. An accumulator policy that buys nearly two to three times on the compute-bound path. A register-blocking scheme that helps the Ampere card and regresses both Keplers, so it stays switched off. And — the sharpest example, its own post last week — a common-subexpression elimination pass that every compiler textbook endorses, built, measured, and shipped disabled, because on a GPU the recompute it eliminates is cheaper than the memory round trip it costs. Every one of those decisions is the same move: find the specific device’s boundary by measurement, then do the thing that boundary rewards.
Placed honestly, the backend inherits every ceiling in claim three — the Vulkan tax, the bandwidth limit, the perf-per-watt penalty that makes a Kepler on a metered grid a poor idea. Its real home is exactly where the second-chance case is strongest: intermittent, capacity-bound, correctness-first, capex-sensitive work, and the CUDA-orphaned platforms where it is not the cheap option but the only one. That is not a small niche. It is the niche the whole thesis is about.
The wall is real, and no amount of clever code moves it. But the line on your side of it — the boundary between “worth doing on the card in the drawer” and “needs the card you have to be allocated” — that line moves, and it has been moving in the stranded card’s favor. Software is what redraws it. A tensor backend that knows the boundaries of a fourteen-year-old GPU well enough to run correct f64 on it, on an OS its maker forgot, is what a second chance actually looks like.
The full argument with confidence-tagged sources — hardware pricing,
the used-GPU market, the roofline mechanism, the CUDA-13 cutoff, and where the
backend fits — is the research memo in the repo:
docs/SECOND_CHANCES_THESIS.md.
The CSE race that the boundary-discipline paragraph refers to is
Compute It Twice; its raw data is
bench_results/CSE_SOFTMAX_RACE.md.