text: 
  
  TITLE
  Possible bug in `prepare_for_model` when using fast tokenizers
  
  COMMENTS
  3
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  ## Environment info
  <!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
       Don't forget to fill out the missing fields in that output! -->
  
  - `transformers` version: 3.5.1
  - `tokenizers` version: 0.9.3
  - Platform: Linux
  - Python version: 3.7.2
  - PyTorch version (GPU?):
  - Tensorflow version (GPU?):
  - Using GPU in script?:
  - Using distributed or parallel set-up in script?:
  
  Also confirmed with `transformers==4.2.2` & `tokenizers==0.9.4`
  
  ### Who can help
  tokenizers: @n1t0, @LysandreJik
  <!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
   If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
   Please tag fewer than 3 people.
  
  Models:
  
  - albert, bert, xlm: @LysandreJik
  - blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
  - longformer, reformer, transfoxl, xlnet: @patrickvonplaten
  - fsmt: @stas00
  - funnel: @sgugger
  - gpt2: @patrickvonplaten, @LysandreJik
  - rag: @patrickvonplaten, @lhoestq
  - tensorflow: @jplu
  
  Library:
  
  - benchmarks: @patrickvonplaten
  - deepspeed: @stas00
  - ray/raytune: @richardliaw, @amogkam
  - text generation: @patrickvonplaten
  - tokenizers: @n1t0, @LysandreJik
  - trainer: @sgugger
  - pipelines: @LysandreJik
  
  Documentation: @sgugger
  
  HF projects:
  
  - nlp datasets: [different repo](https://github.com/huggingface/nlp)
  - rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
  
  Examples:
  
  - maintained examples (not research project or legacy): @sgugger, @patil-suraj
  - research_projects/bert-loses-patience: @JetRunner
  - research_projects/distillation: @VictorSanh
  
   -->
  
  ## Information
  
  I am building my own data preprocessing script which requires me to first know the number of tokens in each sentence before, then match sentences pairs and prepare them as input for a model, in this case BERT. I would like to use the fast tokenizer to speed things up in large datasets, however, I encounter the next assertion problem which is not true since I do provide `return_special_tokens_mask=True` in the function call.
  
  The problem arises when using:
  * [ ] the official example scripts: (give details below)
  * [x] my own modified scripts: (give details below)
  
  The tasks I am working on is:
  * [ ] an official GLUE/SQUaD task: (give the name)
  * [x] my own task or dataset: (give details below)
  
  ## To reproduce
  
  Steps to reproduce the behavior:
  
  The following code snippet reproduces the problem for me:
  ```python
  from transformers import AutoTokenizer
  tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased', use_fast=True)
  s1 = tokenizer.convert_tokens_to_ids(tokenizer.tokenize("Halt! Who goes there?"))
  s2 = tokenizer.convert_tokens_to_ids(tokenizer.tokenize("It is I, Arthur, son of Uther Pendragon, from the castle of Camelot. King of the Britons, defeator of the Saxons, sovereign of all England!"))
  tokenizer.prepare_for_model(s1, s2, return_special_tokens_mask=True)
  ```
  
  I get the following assertion error when running the code:
  ```
  AssertionError                            Traceback (most recent call last)
  <ipython-input-5-889164fb3ae8> in <module>
        2 s1 = tokenizer.convert_tokens_to_ids(tokenizer.tokenize("Halt! Who goes there?"))
        3 s2 = tokenizer.convert_tokens_to_ids(tokenizer.tokenize("It is I, Arthur, son of Uther Pendragon, from the castle of Camelot. King of the Britons, defeator of the Saxons, sovereign of all England!"))
  ----> 4 tokenizer.prepare_for_model(s1, s2, return_special_tokens_mask=True)
  
  ~/p/.venv/lib/python3.7/site-packages/transformers/tokenization_utils_base.py in prepare_for_model(self, ids, pair_ids, add_special_tokens, padding, truncation, max_length, stride, pad_to_multiple_of, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, prepend_batch_axis, **kwargs)
     2724         if return_special_tokens_mask:
     2725             if add_special_tokens:
  -> 2726                 encoded_inputs["special_tokens_mask"] = self.get_special_tokens_mask(ids, pair_ids)
     2727             else:
     2728                 encoded_inputs["special_tokens_mask"] = [0] * len(sequence)
  
  ~/p/.venv/lib/python3.7/site-packages/transformers/tokenization_utils_base.py in get_special_tokens_mask(self, token_ids_0, token_ids_1, already_has_special_tokens)
     3031         """
     3032         assert already_has_special_tokens and token_ids_1 is None, (
  -> 3033             "You cannot use ``already_has_special_tokens=False`` with this tokenizer. "
     3034             "Please use a slow (full python) tokenizer to activate this argument."
     3035             "Or set `return_special_token_mask=True` when calling the encoding method "
  
  AssertionError: You cannot use ``already_has_special_tokens=False`` with this tokenizer. Please use a slow (full python) tokenizer to activate this argument.Or set `return_special_token_mask=True` when calling the encoding method to get the special tokens mask in any tokenizer. 
  ```
  
  <!-- If you have code snippets, error messages, stack traces please provide them here as well.
       Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
       Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->
  
  ## Expected behavior
  For the fast tokenizer to provide the same output as the slow (python) tokenizer:
  ```python
  {'input_ids': [101, 9190, 999, 2040, 3632, 2045, 1029, 102, 2009, 2003, 1045, 1010, 4300, 1010, 2365, 1997, 21183, 5886, 7279, 7265, 7446, 1010, 2013, 1996, 3317, 1997, 19130, 4140, 1012, 2332, 1997, 1996, 28101, 5644, 1010, 4154, 2953, 1997, 1996, 28267, 1010, 11074, 1997, 2035, 2563, 999, 102], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'special_tokens_mask': [1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}
  ```
  
  <!-- A clear and concise description of what you would expect to happen. -->
  
  ## Possible Fix
  Make the following change in:
  https://github.com/huggingface/transformers/blob/d1b14c9b548de34b6606946482946008622967db/src/transformers/tokenization_utils_base.py#L2855
  Since we know beforehand if the special tokens were added and the text pair was already concatenated with or without special tokens, I think the following change will be valid, however, I didn't test it beyond my own use case
  ```python
          if return_special_tokens_mask:
              if add_special_tokens:
                  encoded_inputs["special_tokens_mask"] = self.get_special_tokens_mask(sequence, already_has_special_tokens=True)
              else:
                  encoded_inputs["special_tokens_mask"] = [0] * len(sequence)
  ```