# Standard Options To transcribe or translate an audio file, you can either copy an URL from a website (all websites supported by YT-DLP will work, including YouTube). Otherwise, upload an audio file (choose "All Files (.)" in the file selector to select any file type, including video files) or use the microphone.

For longer audio files (>10 minutes), it is recommended that you select Silero VAD (Voice Activity Detector) in the VAD option, especially if you are using the large-v1 model. Note that large-v2 is a lot more forgiving, but you may still want to use a VAD with a slightly higher "VAD - Max Merge Size (s)" (60 seconds or more).

Model

Select the model that Whisper will use to transcribe the audio:

Size	Parameters	English-only model	Multilingual model	Required VRAM	Relative speed
tiny	39 M	tiny.en	tiny	~1 GB	~32x
base	74 M	base.en	base	~1 GB	~16x
small	244 M	small.en	small	~2 GB	~6x
medium	769 M	medium.en	medium	~5 GB	~2x
large	1550 M	N/A	large	~10 GB	1x
large-v2	1550 M	N/A	large	~10 GB	1x
large-v3	1550 M	N/A	large	~10 GB	1x

Language

Select the language, or leave it empty for Whisper to automatically detect it.

Note that if the selected language and the language in the audio differs, Whisper may start to translate the audio to the selected language. For instance, if the audio is in English but you select Japaneese, the model may translate the audio to Japanese.

Inputs

The options "URL (YouTube, etc.)", "Upload Files" or "Micriphone Input" allows you to send an audio input to the model.

Multiple Files

Note that the UI will only process either the given URL or the upload files (including microphone) - not both.

But you can upload multiple files either through the "Upload files" option, or as a playlist on YouTube. Each audio file will then be processed in turn, and the resulting SRT/VTT/Transcript will be made available in the "Download" section. When more than one file is processed, the UI will also generate a "All_Output" zip file containing all the text output files.

Task

Select the task - either "transcribe" to transcribe the audio to text, or "translate" to translate it to English.

Vad

Using a VAD will improve the timing accuracy of each transcribed line, as well as prevent Whisper getting into an infinite loop detecting the same sentence over and over again. The downside is that this may be at a cost to text accuracy, especially with regards to unique words or names that appear in the audio. You can compensate for this by increasing the prompt window.

Note that English is very well handled by Whisper, and it's less susceptible to issues surrounding bad timings and infinite loops. So you may only need to use a VAD for other languages, such as Japanese, or when the audio is very long.

none
- Run whisper on the entire audio input
silero-vad
- Use Silero VAD to detect sections that contain speech, and run Whisper on independently on each section. Whisper is also run on the gaps between each speech section, by either expanding the section up to the max merge size, or running Whisper independently on the non-speech section.
silero-vad-expand-into-gaps
- Use Silero VAD to detect sections that contain speech, and run Whisper on independently on each section. Each spech section will be expanded such that they cover any adjacent non-speech sections. For instance, if an audio file of one minute contains the speech sections 00:00 - 00:10 (A) and 00:30 - 00:40 (B), the first section (A) will be expanded to 00:00 - 00:30, and (B) will be expanded to 00:30 - 00:60.
silero-vad-skip-gaps
- As above, but sections that doesn't contain speech according to Silero will be skipped. This will be slightly faster, but may cause dialogue to be skipped.
periodic-vad
- Create sections of speech every 'VAD - Max Merge Size' seconds. This is very fast and simple, but will potentially break a sentence or word in two.

VAD - Merge Window

If set, any adjacent speech sections that are at most this number of seconds apart will be automatically merged.

VAD - Max Merge Size (s)

Disables merging of adjacent speech sections if they are this number of seconds long.

VAD - Process Timeout (s)

This configures the number of seconds until a process is killed due to inactivity, freeing RAM and video memory. The default value is 30 minutes.

VAD - Padding (s)

The number of seconds (floating point) to add to the beginning and end of each speech section. Setting this to a number larger than zero ensures that Whisper is more likely to correctly transcribe a sentence in the beginning of a speech section. However, this also increases the probability of Whisper assigning the wrong timestamp to each transcribed line. The default value is 1 second.

VAD - Prompt Window (s)

The text of a detected line will be included as a prompt to the next speech section, if the speech section starts at most this number of seconds after the line has finished. For instance, if a line ends at 10:00, and the next speech section starts at 10:04, the line's text will be included if the prompt window is 4 seconds or more (10:04 - 10:00 = 4 seconds).

Note that detected lines in gaps between speech sections will not be included in the prompt (if silero-vad or silero-vad-expand-into-gaps) is used.

Diarization

If checked, Pyannote will be used to detect speakers in the audio, and label them as (SPEAKER 00), (SPEAKER 01), etc.

This requires a HuggingFace API key to function, which can be supplied with the --auth_token command line option for the CLI, set in the config.json5 file for the GUI, or provided via the HF_ACCESS_TOKEN environment variable.

Diarization - Speakers

The number of speakers to detect. If set to 0, Pyannote will attempt to detect the number of speakers automatically.

Command Line Options

Both app.py and cli.py also accept command line options, such as the ability to enable parallel execution on multiple CPU/GPU cores, the default model name/VAD and so on. Consult the README in the root folder for more information.

Additional Options

In addition to the above, there's also a "Full" options interface that allows you to set all the options available in the Whisper model. The options are as follows:

Initial Prompt

Optional text to provide as a prompt for the first 30 seconds window. Whisper will attempt to use this as a starting point for the transcription, but you can also get creative and specify a style or format for the output of the transcription.

For instance, if you use the prompt "hello how is it going always use lowercase no punctuation goodbye one two three start stop i you me they", Whisper will be biased to output lower capital letters and no punctuation, and may also be biased to output the words in the prompt more often.

Temperature

The temperature to use when sampling. Default is 0 (zero). A higher temperature will result in more random output, while a lower temperature will be more deterministic.

Best Of - Non-zero temperature

The number of candidates to sample from when sampling with non-zero temperature. Default is 5.

Beam Size - Zero temperature

The number of beams to use in beam search when sampling with zero temperature. Default is 5.

Patience - Zero temperature

The patience value to use in beam search when sampling with zero temperature. As in https://arxiv.org/abs/2204.05424, the default (1.0) is equivalent to conventional beam search.

Length Penalty - Any temperature

The token length penalty coefficient (alpha) to use when sampling with any temperature. As in https://arxiv.org/abs/1609.08144, uses simple length normalization by default.

Suppress Tokens - Comma-separated list of token IDs

A comma-separated list of token IDs to suppress during sampling. The default value of "-1" will suppress most special characters except common punctuations.

Condition on previous text

If True, provide the previous output of the model as a prompt for the next window. Disabling this may make the text inconsistent across windows, but the model becomes less prone to getting stuck in a failure loop.

FP16

Whether to perform inference in fp16. True by default.

Temperature increment on fallback

The temperature to increase when falling back when the decoding fails to meet either of the thresholds below. Default is 0.2.

Compression ratio threshold

If the gzip compression ratio is higher than this value, treat the decoding as failed. Default is 2.4.

Logprob threshold

If the average log probability is lower than this value, treat the decoding as failed. Default is -1.0.

No speech threshold

If the probability of the <|nospeech|> token is higher than this value AND the decoding has failed due to logprob_threshold, consider the segment as silence. Default is 0.6.

Diarization - Min Speakers

The minimum number of speakers for Pyannote to detect.

Diarization - Max Speakers

The maximum number of speakers for Pyannote to detect.

Repetition Penalty

ctranslate2: repetition_penalty
This parameter only takes effect in faster-whisper (ctranslate2). Penalty applied to the score of previously generated tokens (set > 1 to penalize).

No Repeat Ngram Size

ctranslate2: no_repeat_ngram_size
This parameter only takes effect in faster-whisper (ctranslate2). Prevent repetitions of ngrams with this size (set 0 to disable).

Whisper Filter options

This is an experimental feature and may potentially filter out correct transcription results.

when enabled, can effectively improve the whisper hallucination, especially for the large-v3 version of the whisper model.

Observations for transcriptions:

duration: calculated by subtracting start from end, it might indicate hallucinated results when inversely proportional to text length.
segment_last: the last result for each segment during VAD transcription has a certain probability of being a hallucinated result.
avg_logprob: average log probability, ranging from logprob_threshold (default: -1) to 0, is better when a larger value. A value lower than -0.9 might suggest a poor result.
compression_ratio: gzip compression ratio, ranging from 0 to compression_ratio_threshold (default: 2.4), a higher positive value is preferable. If it is lower than 0.9, it might indicate suboptimal results.
no_speech_prob: no_speech(<|nospeech|> token) probability, ranging from 0 to no_speech_threshold (default: 0.6), a smaller positive value is preferable. If it exceeds 0.1, it might suggest suboptimal results.

Four sets of filtering conditions have now been established, utilizing text length, duration length, as well as the avg_logprob, compression_ratio, and no_speech_prob parameters returned by Whisper.

avg_logprob < -0.9
(durationLen < 1.5 || segment_last), textLen > 5, avg_logprob < -0.4, no_speech_prob > 0.5
(durationLen < 1.5 || segment_last), textLen > 5, avg_logprob < -0.4, no_speech_prob > 0.07, compression_ratio < 0.9
(durationLen < 1.5 || segment_last), compression_ratio < 0.9, no_speech_prob > 0.1

Translation - Batch Size

transformers: batch_size
When the pipeline will use DataLoader (when passing a dataset, on GPU for a Pytorch model), the size of the batch to use, for inference this is not always beneficial.
ctranslate2: max_batch_size
The maximum batch size.

Translation - No Repeat Ngram Size

transformers: no_repeat_ngram_size
Value that will be used by default in the generate method of the model for no_repeat_ngram_size. If set to int > 0, all ngrams of that size can only occur once.
ctranslate2: no_repeat_ngram_size
Prevent repetitions of ngrams with this size (set 0 to disable).

Translation - Num Beams

transformers: num_beams
Number of beams for beam search that will be used by default in the generate method of the model. 1 means no beam search.
ctranslate2: beam_size
Beam size (1 for greedy search).

Translation - Torch Dtype float16

transformers: torch_dtype=torch.float16
Load the float32 translation model with float16 when the system supports GPU (reducing VRAM usage, not applicable to quantized models such as Ctranslate2, GPTQ, GGUF)