The DMA Mapping Saga
May 6, 2026 · blog
Three days of debugging a hardware issue that turned out to be a single commit in a driver we don't own.
The Symptom
Our Spyre benchmarks started failing consistently. The error: VFIO DMA mapping failed: errno: No space left on device. Every pod, every node, every card. Nothing in our code changed.
"No space left on device" but the disk is fine. RAM is fine. This is IOMMU address space — the hardware layer that maps DMA regions between the accelerator and host memory.
The Investigation
First instinct: did we change something? No. Same config, same code, same pod spec. Git log clean. Last successful run: 3 days ago.
Second instinct: node issue? Tried 4 different nodes. Same failure. Not a hardware fault.
Third: base image changed? The Spyre runtime image (spyre-inference:amd64) gets rebuilt nightly from upstream flex runtime. We don't control what goes into it.
Comparing Images
I pulled the last working image and the current failing image. Diff'd the flex runtime version: working image had flex build +257. Failing image had flex build +307. Fifty commits of difference.
Ran the benchmark on the old image: pass. New image: fail. Confirmed: something in those 50 commits broke DMA mapping.
The Root Cause
Git blame through the flex repo. Commit b4b2a289: "Added async dma support to pf runtime scheduler." This introduced parallel DMA mapping — multiple IOMMU regions mapped concurrently to improve initialization speed.
The problem: a race condition in the IOMMU lifecycle. Under concurrent mapping, the address space bookkeeping lost track of regions, eventually exhausting the VFIO address space. The "no space left" wasn't about physical memory — it was about virtual IOMMU slots being leaked.
The Fix
Not ours to fix. Filed with the flex team. They reverted the async DMA changes in commit 008af7ad, PR #1181: "revert async dma changes as they caused issue in CI."
New image rebuilt. Benchmarks pass again.
Lessons
When nothing in your code changed, check your base images. Nightly rebuilds mean your environment mutates under you. Pin versions if stability matters more than freshness.
"No space left on device" is not always disk. On accelerator hardware, it can mean IOMMU address space, PCI BAR space, or device-internal memory regions. Read the full error path, not just the errno string.
Binary search on images works. Old image passes, new image fails? Find the midpoint. Narrow it down. Works even when you can't read the source (though we could here).
Three days of debugging for someone else's one-line revert. Hardware is fun.