

TITLE
Your example code for WNUT NER produces array indexing ValueError

COMMENTS
16

REACTIONS
+1: 11
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 2

BODY
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.4
- Platform: Google Colab
- Python version: 3.6.9
- PyTorch version (GPU?): 1.6.0
- Tensorflow version (GPU?):
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 
 albert, bert, GPT2, XLM: @LysandreJik 
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao 
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao 
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->
@stefan-it, @sgugger
## Information

Model I am using (Bert, XLNet ...): DistilBERT

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

I'm trying to run the example code Advanced Guides --> Fine-tuning with custom datasets --> [Token Classification with W-NUT Emerging Entities](https://huggingface.co/transformers/custom_datasets.html#token-classification-with-w-nut-emerging-entities).

Steps to reproduce the behavior:

1. I already have a [Google CoLab notebook with your code](https://colab.research.google.com/drive/1i5N7Xc-i91bqXmcp9hamt5q3a_5a-ZnZ?usp=sharing).
2. I use the `tokenizer` with `max_length=64`, which is typically my "best practice" choice. Note that if I set `max_length=None`, everything runs successfully.
```python
max_length = 64
encodings = tokenizer(texts, is_split_into_words=True, max_length=max_length, return_offsets_mapping=True, padding=True, truncation=True)
```
3. When I run `encode_tags()` on the WNUT data, I get a ValueError
```python
labels = encode_tags(tags, encodings)
     11         # set labels whose first offset position is 0 and the second is not 0
---> 12         doc_enc_labels[(arr_offset[:,0] == 0) & (arr_offset[:,1] != 0)] = doc_labels
     13         encoded_labels.append(doc_enc_labels.tolist())
     14 

ValueError: NumPy boolean array indexing assignment cannot assign 29 input values to the 24 output values where the mask is true
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
I expect that `encode_tags()` should return the correct IOB tag labels when I run your `Tokenizer` with a `max_length=64`.