WhisperX

Last reviewed Aug 26, 2026 · 755 words

WhisperX's README claims about 70 times realtime on large-v2 with a GPU, and on a 12 GB card that is roughly what you get: an hour of meeting audio transcribed, word-aligned, and labelled by speaker in about a minute. Plain Whisper gives you sentences with timestamps that drift by a second or two; WhisperX runs a voice-activity detector first, batches the speech segments through faster-whisper, then aligns every word against a phoneme model and, if asked, runs pyannote diarisation to say who said it. It is BSD-3-Clause Python, 23,747 stars, rated Medium, and wants 4 GB of RAM, which is the CPU-side requirement; the GPU side is where the work happens.

Install, and the version trap

pip install whisperx
whisperx meeting.mp3 --model large-v3 --language en --compute_type float16 --output_format srt

Pass --language whenever you know it: without it the first 30 seconds are used to guess, and the alignment model is chosen from that guess. The trap is not WhisperX, it is the CUDA and cuDNN versions that PyTorch and CTranslate2 each expect. A mismatch produces an error about a missing libcudnn library, sometimes only when the alignment step starts. The reliable fix is a fresh virtual environment, install PyTorch from its own index for your CUDA version first, then pip install whisperx, and pin both. Save the working pip freeze; the next upgrade of either side is the next time you will need it. If you would rather not fight it, community Docker images bundle a known-good set, and the VRAM math post covers which model fits which card: large-v3 in float16 needs under 8 GB with batching, and int8 roughly halves that.

Diarisation needs a Hugging Face token, and a click

Speaker labels come from pyannote's diarisation pipeline, which is gated: you create a free Hugging Face token, accept the terms on the pyannote/speaker-diarization-3.1 and pyannote/segmentation-3.0 model pages while logged in, then pass the token:

whisperx meeting.mp3 --model large-v3 --diarize --hf_token hf_xxx --min_speakers 2 --max_speakers 4

Skipping the accept step gives a 401 or a confusing "could not download" error, and it is the con the catalogue puts first for a reason. Once the models are cached, the token is only needed for the download and the pipeline runs offline. Give --min_speakers and --max_speakers whenever you know them; unconstrained diarisation on a two-person call will sometimes find a third speaker in the coughing.

CPU-only works, at a price

On a CPU, use --compute_type int8 and a smaller model:

whisperx podcast.mp3 --model medium --compute_type int8 --device cpu --batch_size 4

Expect around realtime or a little faster on a modern 8-core desktop for medium, several times slower for large-v3, and alignment adds a further chunk on top. Lower --batch_size if memory is tight; it trades speed for RAM and nothing else. Diarisation on CPU is slow enough that I would only do it for short clips. If your machine has no GPU and you only need timestamps at sentence level, plain faster-whisper is the better fit; WhisperX earns its extra steps when you need word timing for subtitles or captions, or speaker labels for a meeting.

Where it fits in a self-hosted stack

WhisperX is a library and CLI, not a server, so it sits behind something. The common shapes: a watch-folder script that turns dropped recordings into SRT files; a cron job that transcribes a podcast feed into searchable text; a subtitle generator feeding Bazarr and Jellyfin with word-accurate captions; or a small FastAPI wrapper so an n8n flow can call it. For a service other apps expect to speak the OpenAI transcription API, faster-whisper based servers already exist; WhisperX is what you reach for when their output is not precise enough.

What I'd do

One pinned virtual environment on the GPU box, a transcribe.sh that runs large-v3 with diarisation and writes SRT, JSON, and TXT side by side, and a watch folder on the NAS that feeds it. Accept the pyannote terms on day one so the models are cached before you need them. Keep the raw JSON, which holds per-word timing and speaker; everything else can be regenerated from it. On a machine without a GPU, use medium with int8, skip diarisation, and be pleasantly surprised how usable the word timing still is.