

TITLE
Add missing tokenizer test files [:building_construction: in progress]

COMMENTS
28

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 1
heart: 0
rocket: 2
eyes: 0

BODY
# 🚀 Add missing tokenizer test files
Several tokenizers currently have no associated tests. I think that adding the test file for one of these tokenizers could be a very good way to make a first contribution to transformers.

## Tokenizers concerned

### not yet claimed
_none_

### claimed

- [ ] LED @nnlnr
- [ ] Flaubert @anmolsjoshi
- [ ] Longformer @tgadeliya
- [ ] Electra @Rajathbharadwaj
- [ ] Splinter @farahdian
- [ ] ConvBert @elusenji
- [ ] RemBert @danhphan 

### with an ongoing PR
_none_

### with an accepted PR

- [ ] MobileBert @leondz #16896
- [ ] RetriBert @mpoemsl #17017

## How to contribute? 

1. Claim a tokenizer

    a. Choose a tokenizer from the list of "not yet claimed" tokenizers
    
    b. Check that no one in the messages for this issue has indicated that they care about this tokenizer
    
    c. Put a message in the issue that you are handling this tokenizer

2. Create a local development setup (if you have not already done it)

    I refer you to section ["start-contributing-pull-requests" of the Contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests) where everything is explained. Don't be afraid with step 5. For this contribution you will only need to test locally the tests you add.

3. Follow the instructions on the readme inside [the `templates/adding_a_missing_tokenization_test` folder](https://github.com/huggingface/transformers/tree/main/templates/adding_a_missing_tokenization_test) to generate the template with cookie cutter for the new test file you will be adding. Don't forget to move the new test file at the end of the template generation to the sub-folder named after the model for which you are adding the test file in the `tests` folder. Some details about questionnaire - assuming that the name of the lowcase model is `brand_new_bert`:
    -   "has_slow_class": Set true there is a `tokenization_brand_new_bert.py` file in the folder `src/transformers/models/brand_new_bert`
    -   "has_fast_class": Set true there is a `tokenization_brand_new_bert_fast.py` file the folder `src/transformers/models/brand_new_bert`.
    -   "slow_tokenizer_use_sentencepiece": Set true if the tokenizer defined in the `tokenization_brand_new_bert.py` file uses sentencepiece. If this tokenizer don't have a ``tokenization_brand_new_bert.py` file set False.

4. Complete the `setUp` method in the generated test file, you can take inspiration for how it is done for the other tokenizers.

5. Try to run all the added tests. It is possible that some tests will not pass, so it will be important to understand why, sometimes the common test is not suited for a tokenizer and sometimes a tokenizer can have a bug. You can also look at what is done in similar tokenizer tests, if there are big problems or you don't know what to do we can discuss this in the PR (step 7.).

6. (Bonus) Try to get a good understanding of the tokenizer to add custom tests to the tokenizer 

7. Open an PR with the new test file added, remember to fill in the RP title and message body (referencing this PR) and request a review from @LysandreJik and @SaulLu.

## Tips

Do not hesitate to read the questions / answers in this issue :newspaper: 