Introducing CaptionKit
July 12, 2026 · blog
I built it. The pipeline I described in my last two posts — VAD, timestamp fixing, smart grouping, adaptive positioning — it's real now. I'm calling it CaptionKit: an offline CLI that takes a video and produces burned-in captions in under 60 seconds. No cloud, no manual editing, no GUI.
Here's how it works under the hood.
The Pipeline
I structured it as 7 sequential stages, each writing a JSON artifact to a project directory. This way I can re-run grouping without re-transcribing, change styles without re-detecting speech, and inspect any intermediate result when something looks off.
Video → VAD → Transcription → Grouping → Overlap Fix → Positioning → ASS Generation → FFmpeg Burn
Caching is hash-keyed: SHA256(model + language + vocabulary). Change the font? Skips straight to rendering. Change max-words? Re-groups but reuses the transcript. Only changing the Whisper model triggers full re-transcription.
Killing Hallucinations
I use Silero VAD — a 2MB ONNX model that processes audio in 32ms windows and outputs a speech probability per frame.
My config after a bunch of tuning:
- Speech threshold: 0.5
- Min speech duration: 250ms (ignore anything shorter)
- Min silence to end a segment: 500ms
- Padding: ±300ms (extend segments to catch the edges)
After Whisper runs, I drop any word whose timestamp falls outside a VAD segment (with 200ms tolerance). If VAD crashes for some reason, the pipeline keeps going — it's not fatal, just means you might get a few hallucinated lines.
For videos with long music intros, I added a second filter based on audio energy: compute RMS in 0.5s windows, find the median, set threshold at median × 0.6, wait for 1.5 seconds of sustained speech above that threshold. Everything before that onset gets dropped. This catches the case where Whisper transcribes a guitar riff as "Thanks for watching."
Fixing Timestamps
Whisper's DTW timestamps need cleanup before they're usable.
Token merging: Whisper splits contractions ("don" + "'t"), fragments words ("C" + "aught"), and leaves standalone punctuation. I re-merge all of these — contractions back together, fragments back together, punctuation attached to the previous word.
Zero-duration clusters: When Whisper stamps 3 words at identical times, I find the span between the previous word's end and next word's start, and distribute evenly. Three words at [5.0, 5.0, 5.0] between 4.5 and 6.0 become [4.5–5.0], [5.0–5.5], [5.5–6.0].
Simple interpolation, but it eliminates the single-frame flash problem completely.
The Grouping Algorithm
This is the part I'm most proud of. Seven rules in priority order:
| # | Rule | When it fires |
|---|---|---|
| 1 | Long pause | Gap ≥ 500ms between words |
| 2 | Sentence end | Word ends with . ? ! |
| 3 | Comma + pause | Comma AND gap ≥ 250ms AND already 3+ words |
| 4 | Character limit | 42 chars (20 for CJK — wider glyphs) |
| 5 | Word limit | 7 words in current group |
| 6 | Duration limit | Group spans ≥ 2.5 seconds |
| 7 | Clause boundary | Next word is a conjunction/subordinator AND gap ≥ 100ms |
The key insight: natural speech has rhythm. A 500ms pause always means a break — the speaker did it on purpose. A sentence ending always deserves its own line. But "and" only deserves a break if the speaker actually paused there — otherwise "peanut butter and jelly" stays together.
Post-processing — merge short groups: Anything under 300ms gets absorbed into its neighbor. Captions that short just flicker.
Post-processing — split fast groups: If reading speed exceeds 25 chars/sec, I split at the best point. "Best" is scored: punctuation gets +3/+4, proximity to middle gets +2, creating a one-word orphan gets -5. So a 68-character caption spanning 1.1 seconds gets split at the nearest comma, not just chopped arbitrarily.
Anti-orphan rule: Even when a rule triggers, I won't create a group with a single word unless it's a hard break. No dangling "the" or "a" on their own line.
Overlap Prevention
A quick pass after grouping ensures sanity:
- Every group has positive duration
- No caption's end exceeds the next caption's start
- Minimum 50ms gap between captions (prevents flicker from back-to-back rendering)
If overlap exists, I trim the previous caption: prev.out = max(prev.in + 0.1, curr.in - 0.05). Always preserves at least 100ms of visibility.
Adaptive Positioning
This one was fun to figure out. For each unique group timestamp (deduplicated within 500ms windows to save FFmpeg calls), I probe two regions of the frame:
- Top 20% of pixels
- Bottom 20% of pixels
I extract the mean grayscale luminance (0–255) from each.
Thresholds are per-video, not hardcoded:
mean = average of all bottom luminances
stddev = standard deviation
bright = clamp(mean + stddev, [140, 220])
dark = clamp(mean - stddev, [30, 100])
A dark moody video and a bright outdoor vlog get completely different definitions of "too bright." Then the logic is straightforward:
- Bottom is bright, top is dark → move captions to top
- Bottom is bright, top isn't dark → keep bottom but add a 60% black background
- Bottom is medium → white text + light background
- Bottom is dark → white text, no background needed
Rendering
I generate ASS (Advanced SubStation Alpha) subtitles, which support per-caption positioning, colors, and one trick I really like: \fad(200, 150) — a 200ms fade-in and 150ms fade-out on every caption. This hides Whisper's 50–150ms timing inaccuracy. Instead of a caption snapping in slightly late, it fades in and the offset becomes imperceptible.
Three rendering fallbacks depending on FFmpeg's compile flags:
1. ASS filter (best, ~22s for 4 minutes)
2. Subtitles filter (~25–30s)
3. Drawtext filter (universal fallback, slowest, batched 20 at a time)
Performance
End-to-end on a 4-minute video:
| Stage | Time |
|---|---|
| VAD | ~5s |
| Transcription | ~7–10s |
| Grouping + validation | < 100ms |
| Positioning probes | ~20s |
| FFmpeg burn | ~22s |
| Total | ~55–60s |
Compare: React/Remotion takes 4–10 minutes. Manual subtitling takes 20–50 minutes.
What I Didn't Build
- No web UI — it's a CLI that outputs JSON, designed to plug into other tools
- No speaker diarization — Whisper doesn't do it reliably, and I haven't needed multi-speaker yet
- No word-by-word animation by default — simple mode covers 90% of cases
- No custom cache eviction — hash-keyed entries are tiny and harmless if stale
Everything runs offline, produces deterministic output (same input → same captions), and the whole thing is one npx captionkit video.mp4 away from done.