Automating Caption QA
May 18, 2026 · blog
I've been building caption components for a few weeks now — grouping, timing, positioning. Each piece works. But when I chain them together, subtle bugs slip through. A caption that's technically 0.2 seconds (too short to read). Two captions that overlap by 30ms (flicker). A 6-second gap where nothing shows (did I lose a sentence?).
I was catching these by watching the output video manually. That doesn't scale.
What Can Go Wrong
I made a list of everything I've seen break:
Hard errors (must fix):
- Two captions overlap in time (both visible simultaneously, fighting for space)
- A caption has invalid timestamps (start ≥ end, or negative)
- Empty caption (no text, just a blank slot)
- Caption extends past the end of the video
Soft warnings (should review):
- Caption shorter than 300ms (can't be read)
- Caption longer than 7 seconds (attention drifts)
- Reading speed above 25 chars/sec (too fast to read comfortably)
- More than 12 words in one caption (visual overflow)
- More than 84 characters (exceeds 2-line display at standard sizes)
- Gap larger than 5 seconds between consecutive captions (missing speech?)
- Adjacent duplicate words (stuttering artifact from Whisper)
- Text would overflow two lines at the rendered font size
That's 12 checks. Four are errors that mean something is objectively broken. Eight are warnings that mean something is probably suboptimal.
The Report
I built a validation pass that runs after grouping and produces a JSON report:
{
"passed": true,
"summary": { "errors": 0, "warnings": 3 },
"issues": [
{
"groupId": "g-5",
"type": "reading_speed",
"severity": "warning",
"value": 28.5,
"limit": 25,
"message": "Reading speed 28.5 chars/s exceeds 25"
}
]
}
Every issue names the exact caption, what's wrong, what the value is, and what the threshold is. No guessing. I can look at g-5 directly and decide if it's acceptable or needs re-grouping.
Why Automated QA for a Creative Tool?
This feels unusual — most subtitle tools just output and hope for the best. But I'm building something that runs unattended. There's no human reviewing every caption before burn.
The validation pass is my substitute for a human reviewer. It catches the mechanical failures (overlaps, timing errors) automatically and flags the judgment calls (slightly fast reading speed) for optional review.
The alternative is watching every output video start to finish. For a 4-minute video that's 4 minutes of my time. For a batch of 20 videos, that's over an hour of quality checking. The automated report takes < 100ms and catches 95% of issues.
Thresholds I Landed On
These aren't arbitrary — each comes from either subtitle industry standards or my own testing:
| Check | Threshold | Source |
|---|---|---|
| Min duration | 0.3s | Below this, eye can't track the text |
| Max duration | 7s | Attention studies; beyond this, viewers assume caption is stale |
| Max reading speed | 25 chars/sec | Psycholinguistics research (d'Ydewalle et al.) |
| Max words | 12 | Two lines × 6 words/line |
| Max characters | 84 | Two lines × 42 chars/line at standard sizing |
| Max gap | 5s | If speech exists in this gap, something was dropped |
What This Enables
Now when I tweak the grouping algorithm, I can run validation and immediately see if my change made things better or worse. Did the number of speed warnings go down? Did I introduce any overlaps? It's a feedback loop.
I'm getting close to having all the pieces. Timing fixes, hallucination filtering (VAD), smart grouping, reading speed control, adaptive positioning, rendering fallbacks, and now quality validation. Feels like these should all be one tool. I might write something about that next — pulling it all together.