Whisper

Whisper is a general-purpose speech recognition model. It is trained on a large dataset of diverse audio and is also a multitasking model that can perform multilingual speech recognition, speech translation, and language identification.

A Transformer sequence-to-sequence model is trained on various speech processing tasks, including multilingual speech recognition, speech translation, spoken language identification, and voice activity detection. These tasks are jointly represented as a sequence of tokens to be predicted by the decoder, allowing a single model to replace many stages of a traditional speech-processing pipeline.

The codebase also depends on a few Python packages, most notably OpenAI's tiktoken for their fast tokenizer implementation. You can download and install the latest release of Whisper with the following command: pip install -U openai-whisper

The .en models for English-only applications tend to perform better, especially for the tiny.en and base.en models. The turbo model is an optimized version of large-v3 that offers faster transcription speed with a minimal degradation in accuracy.

Use cases

• Transcribe multilingual speech and translate to English