Surveying the Auto-Captioning Landscape
July 10, 2026 · blog
Following up on my last post about the four captioning problems — I spent the past few days trying every approach I could find. Here's what exists and why none of it quite works.
Raw Whisper → SRT
The obvious starting point. whisper --output_format srt, burn with FFmpeg, done in two commands.
It works if you squint. The grouping is arbitrary, music sections produce hallucinated text, timing drifts accumulate, captions overlap each other, and everything is hardcoded white-on-bottom. Fine for a rough draft. Not something I'd ship.
Cloud APIs (AssemblyAI, Deepgram, Rev)
Better transcription accuracy (bigger models, more post-processing), speaker labels, proper punctuation. But they still give you fixed-size groups. No per-video positioning. Your audio leaves your machine. Costs scale with minutes.
I was processing batches of videos. The API costs added up fast, and I still needed to solve grouping and rendering myself on top of whatever they returned.
Subtitle Editors (Aegisub, Subtitle Edit)
The gold standard for quality — because it's a human doing everything manually. Adjusting every timestamp, splitting every group, checking every frame.
Also 4–10x realtime effort. A 5-minute video takes half an hour to subtitle properly. I needed this to be automatic.
CapCut / Premiere Auto-Captions
One-click, decent results, nice styling. But completely locked into their ecosystem. No CLI, no API, no way to script it into a pipeline. I needed something I could run in a build step, not something I click through in a GUI.
React-Based Video (Remotion)
Total styling control — you're rendering React components frame by frame. Animations, pixel-perfect placement, whatever you want.
But Chrome renders each frame individually. A 4-minute video takes 4–10 minutes. That's insane for text-on-video. And it doesn't solve transcription or grouping at all — you still need a source for the words and timestamps.
VAD + Whisper
Running Voice Activity Detection first, then only feeding speech segments to Whisper. This actually kills the hallucination problem — Silero VAD is fast, small, accurate, and runs offline.
But it only solves one of the four problems. Essential piece. Not complete.
LLM Post-Processing
I tried feeding raw transcripts to an LLM asking it to "group these naturally." The text quality was better, but it destroyed all timestamp information. It also can't reason about reading speed without knowing durations. And it's non-deterministic — same input, different grouping every run.
The Gap
| Need | Whisper | Cloud | Manual | Editors | React | VAD | LLM |
|---|---|---|---|---|---|---|---|
| Good transcription | yes | yes | yes | yes | — | — | — |
| No hallucinations | no | kinda | yes | no | — | yes | no |
| Natural grouping | no | no | yes | no | — | no | kinda |
| Reading speed control | no | no | yes | no | yes | no | no |
| Adaptive positioning | no | no | yes | no | yes | no | no |
| Timing correction | no | kinda | yes | no | — | no | no |
| Offline | yes | no | yes | kinda | yes | yes | no |
| Scriptable (CLI) | yes | yes | no | no | yes | yes | yes |
| Fast (< 1 min) | yes | kinda | no | no | no | yes | kinda |
Nothing fills every row.
What I Think Needs to Exist
A pipeline that chains: VAD → Whisper with DTW → timestamp normalization → intelligent grouping → quality validation → adaptive positioning → render. Each stage deterministic. Each stage cacheable. All offline. All under a minute.
Nothing I found does this. I think I'm going to have to build it myself.