Guide
Speech recognition: from wav2vec to Whisper
Two opposite routes to robust speech recognition: self-supervised pretraining on unlabeled audio (wav2vec 2.0) and weak supervision at web scale (Whisper).
The short answer
The bottleneck in speech recognition is labeled data. wav2vec 2.0 learns from unlabeled audio and then needs only a little transcribed speech. Whisper instead trains on 680,000 hours of imperfect transcripts from the internet and works well zero-shot. Most modern speech systems descend from one of the two ideas.
Transcription tools such as the ones in our speech and audio category sit on top of a research story with two main routes. Knowing which route a system follows tells you where it is likely to be strong and where you should test it.
Why is labeled data the bottleneck in speech recognition?
Transcribed audio is scarce compared with raw audio. Two papers attack that bottleneck from opposite directions. wav2vec 2.0 learns from audio alone and then needs only a little transcribed speech. Whisper skips self-supervision and trains on a very large, imperfect pile of transcripts collected from the internet.
How does wav2vec 2.0 learn from audio alone?
Baevski et al. show that learning representations from speech audio alone, then fine-tuning on transcribed speech, can outperform the best semi-supervised methods while being conceptually simpler 1. The mechanism combines two ideas:
- Masking in latent space. The model encodes raw audio into latent speech representations and masks parts of that latent sequence 1.
- A contrastive task over learned quantized targets. The model must identify the right quantized latent for each masked position, with the quantization learned jointly 1.
The headline results are about label efficiency. With all labeled Librispeech data it reaches 1.8 / 3.3 word error rate (WER) on the clean / other test sets. With one hour of labels it beats the previous state of the art on the 100-hour subset while using 100 times less labeled data. With just ten minutes of labels, after pretraining on 53k hours of unlabeled audio, it still reaches 4.8 / 8.2 WER 1. Implication: for a low-resource language or domain, pretraining on unlabeled audio is the lever.
How does Whisper scale supervision instead?
Radford et al. study systems trained simply to predict transcripts of audio found on the internet. Scaled to 680,000 hours of multilingual and multitask supervision, the models generalize well to standard benchmarks and are often competitive with prior fully supervised results in a zero-shot setting, with no dataset-specific fine-tuning. Compared with humans, the models approach human accuracy and robustness 2.
- No self-supervision or self-training. The results come without techniques that had been mainstream in large-scale speech recognition 2.
- Broader scope. Training is multilingual and multitask, not English-only recognition 2.
- Plain architecture. The released models are Transformer encoder-decoder models that take log-Mel spectrograms as input. This detail comes from the Hugging Face Whisper model documentation, not the paper abstract.
- Released for reuse. The authors released models and inference code as a foundation for further work 2.
Caveat on the data: Whisper's training set is weakly supervised, meaning its transcripts are noisy. The paper reports finding many subpar transcripts in the raw dataset and filtering for them 2.
Which approach fits your situation?
| Question | wav2vec 2.0 | Whisper |
|---|---|---|
| What does it learn from? | Unlabeled audio, then a small labeled set | A large noisy labeled corpus, 680,000 hours |
| Needs fine-tuning for your data? | Yes, that is the design | Not required to be competitive zero-shot |
| Best fit | Low-resource languages and domains | General-purpose, multilingual transcription |
| Main risk | You still need some labels and a fine-tuning pipeline | Transcript noise in training data; behavior outside the training distribution |
What changed in practice?
Whisper-style zero-shot transcription removed the fine-tuning step for most applications, while self-supervised speech models remain the standard backbone when you need to adapt to a specific language or acoustic setting. Claims about the current best model are time-sensitive: check leaderboards and vendor reports before relying on them.
How do you apply this when choosing a transcription tool?
Do not rely on a paper's benchmark number. Measure word error rate on your own recordings, separately for accents, background noise and specialist vocabulary, as the speech and audio category describes, and read the vendor's retention terms. For how to read quoted numbers, see reading benchmark claims. Terms such as word error rate, self-supervised learning and zero-shot are in the glossary.
Evidence in this article Third-party research Work published by researchers or institutions other than us or the vendor. We link it; we have not reproduced it.
Frequently asked questions
What is the difference between wav2vec 2.0 and Whisper?
wav2vec 2.0 pretrains on unlabeled audio and is then fine-tuned on a small labeled set. Whisper trains directly on a large, noisy labeled corpus of 680,000 hours and is competitive zero-shot, without dataset-specific fine-tuning.
How much labeled data does wav2vec 2.0 need?
The authors report 4.8 / 8.2 word error rate on Librispeech clean / other with just ten minutes of labels, after pretraining on 53,000 hours of unlabeled audio.
What does zero-shot mean for Whisper?
The model is used on a benchmark it was not fine-tuned for. Radford et al. report that Whisper is often competitive with prior fully supervised results in this setting.
Is Whisper the best speech recognition model today?
This guide does not say. Claims about the current best model are time-sensitive, so check recent leaderboards and vendor reports, and test on your own recordings.
Sources
- wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations Baevski, Zhou, Mohamed, Auli. NeurIPS 2020. Third-party research paper, as supplied. The figures quoted here are from its abstract and were not re-checked by the site build.
- Robust Speech Recognition via Large-Scale Weak Supervision Radford, Kim, Xu, Brockman, McLeavey, Sutskever. 2022. Third-party research paper, as supplied. The figures quoted here are from its abstract and were not re-checked by the site build.