Why this plant runs on a graphics driver
The vendor compute stack reports six cards and uses one. The failure is silent, and it decided the architecture of everything we have built since.
A machine here holds six AMD RX 580 cards. The obvious way to use them is the vendor compute stack, which is what every tutorial assumes and what every benchmark is written against. We tried it first and it appeared to work. The tools enumerated six cards. Jobs ran. The numbers were wrong by a factor of about six, and it took a while to accept that the machine was only ever using one card.
The cause is an instruction, not a bug in our code. The compute runtime wants atomic operations against host memory. This generation of card does not implement them. Rather than refuse the second card, the runtime quietly keeps the work on the first one.
What we changed
Nothing in the compute stack. We left it where it was and moved the whole plant onto the graphics driver instead. The Vulkan path has no such requirement, addresses every card in the machine, and is maintained by people who still care about hardware of this age.
That decision has a cost. Vulkan gives no libraries: no tuned matrix multiplication, no attention kernel, no ready quantisation. Everything the compute stack would have handed us had to be built, and the build has to be compiled from source against a specific Mesa, a specific driver and a processor with no advanced vector instructions.
What it bought
Six cards per machine instead of one. Across the plant that is nine hundred and sixty cards instead of one hundred and sixty. There is no version of this company that works the other way.
The lesson we keep repeating
A silent fallback is worse than a refusal. The runtime had every piece of information needed to say this card cannot do what you are asking, and instead it produced a plausible number. Now, before any machine here is accepted into the fleet, it has to report what it actually used, card by card, and the report is compared against what was asked for.