text: 
  
  TITLE
  odd whitespace handling with imported sentencepiece models
  
  COMMENTS
  11
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  ## Environment info
  
  - `transformers` version: 4.7.0
  - Platform: Linux-4.15.0-143-generic-x86_64-with-Ubuntu-18.04-bionic
  - Python version: 3.6.9
  - PyTorch version (GPU?): 1.9.0+cu102 (False)
  - Tensorflow version (GPU?): not installed (NA)
  - Using GPU in script?: no
  - Using distributed or parallel set-up in script?: no
  
  ### Who can help
  @LysandreJik Although my example uses ReformerTokenizer, I think this problem is present in several of the model architectures using sentencepiece tokenizers.
  
  ## Information
  
  Model I am using (Bert, XLNet ...): Reformer
  
  The problem arises when using:
  * [x] the official example scripts: (give details below)
  * [ ] my own modified scripts: (give details below)
  
  The tasks I am working on is:
  * [ ] an official GLUE/SQUaD task: (give the name)
  * [x] my own task or dataset: (give details below)
  
  ## To reproduce
  
  Steps to reproduce the behavior:
  
  ```
  #!/usr/bin/env python3
  
  import sentencepiece as spm
  import transformers as tr
  
  src = (
      'Lorem Ipsum dolor sit amet, consectetur adipiscing elit, sed do',
      'eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut',
      'enim ad minim veniam, quis nostrud exercitation ullamco laboris',
      'nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in',
      'reprehenderit in voluptate velit esse cillum dolore eu fugiat',
      'nulla pariatur.  Excepteur sint occaecat cupidatat non proident,',
      'sunt in culpa qui officia deserunt mollit anim id est laborum.',
  )
  
  spm.SentencePieceTrainer.train(
      sentence_iterator=iter(src),
      model_prefix='test',
      vocab_size=96,
      treat_whitespace_as_suffix=True,
      user_defined_symbols=['<pad>', '<mask>'],
      minloglevel=1,
  )
  
  def show(label, toks):
      print('%14s %2d: %s' % (label, len(toks), toks))
  
  text = 'Lo<mask>m Ipsum'
  
  tok = spm.SentencePieceProcessor(model_file='test.model')
  show('sentencepiece', tok.encode(text, out_type=str))
  
  tok = tr.models.reformer.ReformerTokenizerFast('test.model',
      mask_token='<mask>',
      pad_token='<pad>')
  show('transformers', tok.tokenize(text))
  
  tok.save_pretrained('test')
  
  tr.models.reformer.ReformerConfig().save_pretrained('test')
  tok = tr.AutoTokenizer.from_pretrained('test')
  show('AutoTokenizer', tok.tokenize(text))
  ```
  is giving
  ```
   sentencepiece  9: ['L', 'o', '<mask>', 'm▁', 'I', 'p', 's', 'um', '▁']
    transformers 10: ['▁', 'L', 'o', '<mask>', 'm', '▁', 'I', 'p', 's', 'um']
   AutoTokenizer 11: ['▁', 'L', 'o', '<mask>', '▁', 'm', '▁', 'I', 'p', 's', 'um']
  ```
  
  ## Expected behavior
  
  I believe the tokenization of input text should be more consistent.  I think these variations are cropping up between my attempts to pretrain a language model and then later finetune the saved model, resulting in model accuracy problems.
  
  The use of `treat_whitespace_as_suffix=True` in `sentencepiece` makes this problem worse, but using a sentencepiece model without this flag still shows the `AutoTokenizer.from_pretrained()` created tokenizer inserting whitespace that was not present in the source text.  I haven't been able to track down where this is coming from or how to avoid it.