Shipping the Benchmark Dashboard

June 15, 2026 · blog

The Spyre vLLM benchmark pipeline is finally end-to-end. Config → run → collect → dashboard. No more reading pod logs manually.

The Result Collection Problem

vLLM has three benchmark modes, and each produces different JSON:

• vllm bench latency → --output-json path → dict or list (changed between versions)
• vllm bench throughput → --output-json path → dict or list
• vllm bench serve → --save-result --result-dir dir (completely different flag!)

And the metric names vary: avg_latency in one version, mean_latency in another. tokens_per_second vs tok/s. Percentile keys change across point releases.

The collector handles all of this: tries multiple key names, supports both dict and list JSON formats, normalizes everything to a consistent schema before pushing upstream.

ClickHouse Schema

Results go to a ClickHouse table (benchmark.oss_ci_benchmark_v3). One row per metric, not per test:

• A latency test produces 3 rows: avg latency, p50, p99
• A throughput test produces 2 rows: tokens/sec, requests/sec

This seems denormalized, but it's exactly what the dashboard needs — each graph shows one metric over time, not one test over time.

Metadata lives in a separate table: device type, compile mode, hardware specs, vLLM version. Joined at query time.

The Cross-Cluster Problem

Fun infrastructure detail: the benchmark pods run on cluster A (torch-cicd). The dashboard runs on cluster B (fmaas). These clusters can't talk to each other — network-isolated.

But both are reachable from the public internet (via different routes). So the push script runs from a machine that can reach both: reads pod logs from cluster A, formats results, posts to ClickHouse on cluster B.

Not elegant. Works perfectly. Sometimes the simplest solution to a networking problem is "use a third machine that can see both."

What the Dashboard Shows

• Latency (avg, p50, p99) per model over time
• Throughput (tok/s) per model over time
• Comparison across models at same batch size
• Regression detection (did today's number drop vs last week?)

Filtered by: model, workload type, hardware generation, vLLM version.

The Full Pipeline Now

benchmark_config.yaml
  → generate_test_configs.py (YAML → vLLM JSON)
  → build.yaml triggers pod
  → pod builds vLLM from source (~18 min)
  → run_benchmarks.sh (device-isolated execution)
  → collect_results.py (normalize heterogeneous JSON)
  → push_to_clickhouse.py (post to dashboard)
  → ClickHouse → Next.js dashboard

From "edit YAML and push" to "see numbers on dashboard" in about 40 minutes. Most of that is the vLLM build (unavoidable on Spyre, no binary caching yet).

Three months from "how do I even run a benchmark on this hardware" to a fully automated pipeline with a real-time dashboard. To be clear: vLLM itself is an existing open-source project — my work was building the benchmark automation, result collection, and dashboard tooling around our Spyre fork. Not bad for custom accelerator tooling with zero community support.

← back to blog