The Problem with Auto-Generated Captions
July 8, 2026 · observation
I was building a video pipeline and needed one simple thing: take a video, burn in captions, no cloud API, no manual work. Whisper exists, FFmpeg exists — how hard can it be?
Turns out, very.
Timing is Off
Whisper gives you word-level timestamps via DTW, which sounds great until you realize they're consistently 50–150ms off. Doesn't sound like much, but a 0.8-second caption appearing 100ms late and vanishing 100ms early feels broken. It flickers.
And sometimes Whisper just gives up on timing entirely — I kept seeing clusters of 3–4 words all stamped at the exact same millisecond. The renderer shows them for one frame and they're gone.
Whisper Hallucinates. A Lot.
I fed a video with a 10-second guitar intro through Whisper and got back "Thank you for watching, please subscribe." Confident. Timestamped. Completely fabricated.
This happens on every video with background music, ambient noise, or a quiet pause. Whisper is trained to always produce tokens — it doesn't know how to say "there's nothing here."
Grouping is Braindead
Even if transcription were perfect, you still need to decide which words show up together on screen. Every tool I found does the same thing: fixed chunks of N words. Every 5 words, new caption.
This falls apart immediately:
- Sentences get split mid-thought across two captions
- Fast speakers hit 30+ characters per second and viewers can't keep up
- A dramatic pause gets crammed together with the next phrase
The comfortable reading speed is around 20–25 characters per second. Above that, people start skipping. But fixed chunking doesn't know or care about pace.
Captions Are Invisible Half the Time
White text, bottom center, black outline. That's what every tool gives you. Works great until there's a bright sky, a white shirt, beach sand, or a well-lit room in the lower third of the frame.
Every video is different. Some need captions at the top. Some need a dark background behind the text. Some are fine as-is. No tool I found adapts to any of this.
So Now What
I'm staring at four distinct problems:
- Timing — Whisper's drift needs correcting, zero-duration clusters need interpolating
- Hallucinations — phantom speech during music/silence needs filtering
- Grouping — text needs to break at natural boundaries, not arbitrary word counts
- Positioning — captions need to be readable against whatever's actually on screen
Every existing tool solves zero or one of these. Most just pipe Whisper output into an SRT and ship it.
I'm going to look at what's actually out there and see if anything comes close.