The tuning was there. The card was filed as unknown.
One attention kernel went from three thousand and twenty-two milliseconds to one hundred and fifty-two. Nothing was optimised. A test was corrected.
For months the plant ran a kernel that should have been fast and was not. The measurement was three thousand and twenty-two milliseconds for one flash attention pass. Every profile pointed at the same code, and the code looked correct.
The code was correct. It was never being used.
The test
The inference server chooses its tuned path by asking the driver which architecture the card belongs to. On this platform the question is asked through a driver extension. Our cards do not publish that extension, so the question returned nothing, and nothing was filed as other. Every tuned path for this family of hardware was therefore skipped, on every card, on every node, for every job.
Once the detection took a second route the tuned path came alive. The same kernel, on the same card, with no change to the kernel itself:
- before: 3022 ms
- after: 152 ms
Why we write this down
The interesting part is not the number. It is that the whole cluster ran for weeks in a state where a capability check failed open into the slow path and nothing anywhere said so. There was no error, no warning, and no line in a log. The system was working, at one twentieth of its speed.
Our rule since: a capability test that fails must say which test failed and what it fell back to, at the moment it happens, with the name of the card in the line. A quiet fallback costs more than a crash, because a crash gets fixed on the day.