Benchmarking LLMs on Hardware Nobody's Used Before
April 18, 2026 · observation
At work, we're running vLLM on IBM Spyre accelerators. We maintain an internal fork with Spyre-specific patches — not building the inference engine from scratch, but adapting it for custom AI accelerators with their own runtime, their own memory model, and almost no community tooling.
The ask sounds simple: run standard vLLM benchmarks and report latency/throughput numbers. In practice, nothing about this is standard.
What Makes Spyre Different
Spyre cards are not CUDA GPUs. There's no nvidia-smi. The driver stack is different. The memory allocation model is different. The device initialization is different.
vLLM assumes NVIDIA. Its default build links against libcudart.so. Its device detection calls CUDA APIs. Its memory management uses GPU memory pools. None of that exists on Spyre.
So step one: build vLLM from source with VLLM_TARGET_DEVICE=cpu and a custom Spyre plugin (torch-spyre) that replaces the CUDA backend. This takes 15–20 minutes per pod restart because the C++ extensions compile from scratch.
The Environment
We're running in OpenShift pods. Each pod requests Spyre PF (Processing Function) devices — essentially dedicated accelerator cards allocated via the cluster scheduler. The pod boots, clones our benchmark repo, builds vLLM, runs benchmarks, and exits.
No persistent state between runs. No caching of compiled binaries. No interactive debugging (it's a pod, not a VM). Every iteration of "change config, run benchmark, check results" takes 20+ minutes round-trip.
Problems Already Encountered
Device isolation: One process locks to one Spyre device. Once start_runtime() fires, you can't switch. Multi-model benchmarks need separate processes per model, each pinned to a specific device via SPYRE_DEVICES env var.
Tensor parallelism doesn't work: vLLM's multiproc_executor spawns child workers that don't inherit the Spyre environment variables. The workers can't find the device. This means TP>1 benchmarks are currently impossible via vllm bench.
Memory explosion: vLLM defaults to allocating ALL available RAM for KV cache. On a pod with 256GB, that means it tries to grab ~252GB. Two concurrent benchmarks → OOM. I had to cap it with VLLM_CPU_KVCACHE_SPACE=16.
What I Need to Build
A pipeline that:
1. Takes a YAML config defining models and workloads
2. Generates vLLM-compatible benchmark commands
3. Runs them with proper device isolation
4. Collects heterogeneous result formats
5. Pushes to a dashboard
Right now it's all manual. Time to automate it.