

TITLE
`run_squad_trainer` doesn't actually use a Rust tokenizer + errors in `squad_convert_example_to_features` when using a Rust tokenizer

COMMENTS
4

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.3.1
- Platform: Linux-4.14.35-1902.303.4.1.el7uek.x86_64-x86_64-with-oracle-7.8
- Python version: 3.6.8
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

and

- `transformers` version: 3.3.1
- Platform: macOS-10.15.6-x86_64-i386-64bit
- Python version: 3.8.5
- PyTorch version (GPU?): 1.6.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help
@mfuntowicz
@LysandreJik
@patil-suraj 

## Information

Model I am using (Bert, XLNet ...): bert-base-uncased

The problem arises when using:
* [x] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

Firstly, in `run_squad_trainer.py`, I noticed that the "use_fast" arg doesn't get propagated into the tokenizer instantiation: https://github.com/huggingface/transformers/blob/0acd1ffa09a06084efa7cfa0e4e9d97cffdda5f9/examples/question-answering/run_squad_trainer.py#L107  Probably should be 
```
    tokenizer = AutoTokenizer.from_pretrained(
        model_args.tokenizer_name if model_args.tokenizer_name else model_args.model_name_or_path,
        cache_dir=model_args.cache_dir,
        use_fast=model_args.use_fast
    )
```
However, when I make that change, the script hangs at the call to `squad_convert_examples_to_features` in SquadProcessor. 

So, I did a little digging. The error is in `squad_convert_example_to_features` and seems to be due to inconsistencies in the behavior of `tokenizer.encode_plus` between the Python and Rust tokenizers, detailed below. I've also provided a [gist](https://gist.github.com/k8si/a143346dfa875c28d98e95cba1f82f1b) that hopefully elucidates & will help reproduce each of these points. I tested both BertTokenizer/BertTokenizerFast and GPT2Tokenizer/GPT2TokenizerFast.

1) Python tokenizers handle negative values for `stride`, Rust tokenizers throw an exception (`OverflowError: can't convert negative int to unsigned`)

2) For sequence pairs, Python tokenizers are fine if the first arg (`text`) is a list of ints and the second arg (`text_pair`) is a list of strings. The Rust tokenizers throw an exception `ValueError: PreTokenizedInputSequence must be Union[List[str], Tuple[str]]`. (Furthermore, the typehints for these arguments indicate that a string, a list of strings, or a list of ints are all fine.)

3) Leaving the `is_split_into_words` kwarg at its default value (`False`), then running `tokenizer.encode_plus(list of ints)` works fine for the Python tokenizers. The Rust tokenizers raise an exception `ValueError: TextInputSequence must be str`.

4) When running on a pair of sequences and setting `return_tensors=None`, the Python tokenizers return an output dict with input_ids (and other elements) as a list of ints i.e.`input_ids = [id1, id2, ...]` whereas the Rust tokenizers return a dict with input_ids as a list of list of ints i.e. `input_ids = [[id1, id2, ...]]`. I also noticed that if you set `return_tensors="pt"`, both the Python and Rust tokenizers return `input_ids = tensor([[id1, id2, ...]])`. 

5) When `return_overflowing_tokens=True`, the Python tokenizers return a list of the overflowing tokens at key `overflowing_tokens` as expected. The Rust tokenizers return them at key `overflow_to_sample_mapping` which is not documented anywhere, as far as I can tell. The values seem to be different for the Python output vs. Rust output.

6) Running the same procedure on the same input twice produces the same result each time for the Python tokenizer. For the Rust tokenizer, the result of the second run is **different**. I am not familiar enough with the Rust tokenizer internals at this point to have a theory as to why this is the case. Anyway, this is the point at which I stopped debugging and decided to file an issue.

## To reproduce

Steps to reproduce the behavior:

1. Download squad 2.0 dataset from ["official" squad website](https://rajpurkar.github.io/SQuAD-explorer/)
2. Make fix in `run_squad_training.py` described above to correctly instantiate a Rust tokenizer
3. Run script: `python examples/question-answering/run_squad_trainer.py --model_name_or_path bert-base-uncased --use_fast --output_dir "./outputs-squad" --do_train --data_dir "./squad-data" --version_2_with_negative`

Also see gist detailing issues described above: https://gist.github.com/k8si/a143346dfa875c28d98e95cba1f82f1b

## Expected behavior

1) I expected `run_squad_trainer.py` to use a Rust tokenizer when the `use_fast` arg was set to True
2) I expected `SquadProcessor.squad_convert_example_to_features` to not raise exceptions when processing squad data when using a Rust tokenizer
3) I expected `tokenizer.encode_plus` to return the same outputs given the same inputs, regardless of whether the tokenizer is a Rust tokenizer or a Python tokenizer
