text: 
  
  TITLE
  Difference in decoded strings between a tokenizer and the corresponding fast tokenizer
  
  COMMENTS
  2
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  ## Environment info
  
  - `transformers` version: 4.2.0
  - Platform: Linux-4.15.0-130-generic-x86_64-with-debian-10.5
  - Python version: 3.7.8
  - PyTorch version (GPU?): 1.7.1+cpu (False)
  - Tensorflow version (GPU?): 2.4.0 (False)
  - Using GPU in script?: No
  - Using distributed or parallel set-up in script?: No
  
  ### Who can help
   tokenizers: @mfuntowicz
  
  ## Information
  
  I want to feed a word-based sequence to a tokenizer and get a word-based output decoded from logits.
  To leave spaces before punctuation marks, I specified `tokenizer.decode(ids, clean_up_tokenization_spaces=False)`, but a fast tokenizer removes such spaces while the corresponding non-fast tokenizer preserves them.
  
  ## To reproduce
  
  ```py
  from transformers import BertTokenizer, BertTokenizerFast
  
  seq = ['Cheerfully', ',', 'Hello', 'World', '!']
  
  tokenizer = BertTokenizer.from_pretrained('bert-base-cased')
  ids = tokenizer(seq, is_split_into_words=True).input_ids
  print(ids)  # => [101, 20394, 8284, 5834, 117, 8667, 1291, 106, 102]
  print(tokenizer.decode(ids, clean_up_tokenization_spaces=False))  # => [CLS] Cheerfully , Hello World ! [SEP]
  
  tokenizer = BertTokenizerFast.from_pretrained('bert-base-cased')
  ids = tokenizer(seq, is_split_into_words=True).input_ids
  print(ids)  # => [101, 20394, 8284, 5834, 117, 8667, 1291, 106, 102]
  print(tokenizer.decode(ids, clean_up_tokenization_spaces=False))  # => [CLS] Cheerfully, Hello World! [SEP]
  ```
  
  This happens because the underlying tokenizer ([huggingface/tokenizers](https://github.com/huggingface/tokenizers/)) removes them at the [transformers/tokenization_utils_fast.py#L495](https://github.com/huggingface/transformers/blob/v4.2.0/src/transformers/tokenization_utils_fast.py#L495), whether `clean_up_tokenization_spaces` is `True` or `False`.
  
  To avoid this issue, I tried to use `tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(ids))`, but this also did not work.
  
  ## Expected behavior
  
  A tokenizer and its corresponding fast tokenizer must return the same decoded string.
  