

TITLE
Wav2Vec2ForPreTraining doc example has None loss

COMMENTS
3

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.15.0
- Platform: Linux-5.11.0-46-generic-x86_64-with-glibc2.31
- Python version: 3.9.7
- PyTorch version (GPU?): 1.10.1 (True)
- Tensorflow version (GPU?): 2.7.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


### Who can help
Models:
- Wav2Vec2, HuBERT, SpeechEncoderDecoder, UniSpeech, UniSpeechSAT, SEW, SEW-D, Speech2Text: @patrickvonplaten, @anton-l

Documentation: @sgugger


## Information

I'm trying to additionally pretrain Wav2Vec2.0 model on my dataset. [In the docs](https://huggingface.co/docs/transformers/model_doc/wav2vec2#transformers.Wav2Vec2ForPreTraining) you have an example for running the `Wav2Vec2ForPreTraining`:

```python
import torch
from transformers import Wav2Vec2FeatureExtractor, Wav2Vec2ForPreTraining
from transformers.models.wav2vec2.modeling_wav2vec2 import _compute_mask_indices
from datasets import load_dataset
import soundfile as sf

feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained("patrickvonplaten/wav2vec2-base")
model = Wav2Vec2ForPreTraining.from_pretrained("patrickvonplaten/wav2vec2-base")


def map_to_array(batch):
    speech, _ = sf.read(batch["file"])
    batch["speech"] = speech
    return batch


ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
ds = ds.map(map_to_array)

input_values = feature_extractor(ds["speech"][0], return_tensors="pt").input_values  # Batch size 1

# compute masked indices
batch_size, raw_sequence_length = input_values.shape
sequence_length = model._get_feat_extract_output_lengths(raw_sequence_length)
mask_time_indices = _compute_mask_indices((batch_size, sequence_length), mask_prob=0.2, mask_length=2)

with torch.no_grad():
    outputs = model(input_values, mask_time_indices=mask_time_indices)

# compute cosine similarity between predicted (=projected_states) and target (=projected_quantized_states)
cosine_sim = torch.cosine_similarity(
    outputs.projected_states, outputs.projected_quantized_states, dim=-1
)

# show that cosine similarity is much higher than random
assert cosine_sim[mask_time_indices].mean() > 0.5

# for contrastive loss training model should be put into train mode
model.train()
loss = model(input_values, mask_time_indices=mask_time_indices).loss
```

If you print the `loss` that you get in the end, you will get `None`. This happens because [in the definition](https://github.com/huggingface/transformers/blob/f4b7420dfe419fe653908f091976517635a119e6/src/transformers/models/wav2vec2/modeling_wav2vec2.py#L1511) you have to pass `sampled_negative_indices` in order to get not `None` loss.


## To reproduce

Steps to reproduce the behavior:

1. Run the above code
2. `print(loss)` in the end

## Expected behavior

Expected to have some example on how to get the actual loss and train the model.
