Config-as-Code for Benchmark Pipelines
April 25, 2026 · blog
I've been building the Spyre benchmark pipeline. The core design decision: everything is driven by a single YAML file.
The Config
models:
- name: granite-3b
path: /models/granite-3b-instruct
- name: llama-8b
path: /models/llama-3.1-8b
workloads:
latency:
- input_len: 128
output_len: 128
batch_size: 1
- input_len: 512
output_len: 256
batch_size: 4
throughput:
- input_len: 256
output_len: 128
num_prompts: 100
engine:
compile_graph: true
dtype: float16
kv_cache_space: 16
That's it. Every combination of model × workload gets benchmarked. Add a model? Add two lines. New workload? Add a block. No code changes needed.
Generating Test Cases
A Python script reads the YAML and produces vLLM-compatible JSON configs via cartesian product. granite-3b × 3 workloads = 3 test cases. Two models × 3 workloads = 6 test cases.
The naming convention encodes the parameters: latency_granite-3b_in128_out128_bs1. Parseable by both humans and result collectors.
One gotcha: vLLM's CLI uses hyphens (--input-len) but the config uses underscores (input_len). The generator handles the conversion — json2args maps underscores to hyphens automatically.
Fork-Based Workflow
Instead of a shared config that everyone edits, each team member forks the repo and modifies their own benchmark_config.yaml. The pod clones YOUR fork on startup.
This means:
- No merge conflicts on config changes
- Each person's experiments are isolated
- The main branch stays clean (reference config only)
- History shows exactly what each person benchmarked
It's git-native. No custom UI, no web forms, no coordination overhead. Edit YAML, push, trigger pod.
Why Not Just Shell Scripts?
I started with shell scripts. They worked for one model. Then I needed two models. Then I needed different batch sizes. Then latency AND throughput tests.
The script grew to 200 lines of hardcoded parameters, copy-pasted blocks with minor variations, and comments like "# uncomment for llama" scattered throughout.
YAML config + generator script: 40 lines of config, 80 lines of Python. Infinitely extensible without copy-paste. Every parameter is named and documented by its position in the schema.
What's Next
The benchmarks run and produce JSON results. But vLLM's output format varies between versions and between benchmark types (bench latency vs bench throughput vs bench serve — all different JSON schemas). I need a result collector that handles all variants.
Also: getting results into a dashboard so we're not reading pod logs manually.