text: 
  
  TITLE
  potential mismatch between `save_pretrained` and `from_pretrained` for `AutoTokenizer`
  
  COMMENTS
  4
  
  REACTIONS
  +1: 2
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  ## Environment info
  
  - `transformers` version: 4.5.1
  - Platform: Linux-4.15.0-48-generic-x86_64-with-debian-buster-sid
  - Python version: 3.7.3
  - PyTorch version (GPU?): 1.6.0+cu101 (True)
  - Tensorflow version (GPU?): not installed (NA)
  - Using GPU in script?: No
  - Using distributed or parallel set-up in script?: No
  
  
  ### Who can help
  
  Since it is related to tokenizer, I guess @LysandreJik could help to have a look.
  
  
  ## Information
  
  Model I am using (Bert, XLNet ...): `AutoTokenizer`
  
  I tried to save the official pretrained tokenizer `bert-base-multilingual-cased` into a local folder so I can load it from this folder next time.
  
  I got the official `bert-base-multilingual-cased` tokenizer:
  ```python
  from transformers import AutoTokenizer
  tokenizer = AutoTokenizer.from_pretrained(
      pretrained_model_name_or_path='bert-base-multilingual-cased',
      use_fast=True,
  )
  ```
  The tokenizer looks like this:
  ```
  PreTrainedTokenizerFast(name_or_path='bert-base-multilingual-cased', vocab_size=119547, model_max_len=512, is_fast=True, padding_side='right', special_tokens={'unk_token': '[UNK]', 'sep_token': '[SEP]', 'pad_token': '[PAD]', 'cls_token': '[CLS]', 'mask_token': '[MASK]'})
  ```
  
  Tried to save it locally:
  ```python
  tokenizer.save_pretrained('bert-base-multilingual-cased-good')
  ```
  The folder structure after saving:
  ```
  bert-base-multilingual-cased-good/
  ├── special_tokens_map.json
  ├── tokenizer_config.json
  └── vocab.txt
  ```
  
  However, when I tried to load the tokenizer from the local folder, I got this error:
  ```python
  fail_tokenizer = AutoTokenizer.from_pretrained(
      pretrained_model_name_or_path='bert-base-multilingual-cased-good',
      use_fast=True,
  )
  ```
  error message:
  ```
  OSError: Can't load config for 'bert-base-multilingual-cased-good'. Make sure that:
  
  - 'bert-base-multilingual-cased-good' is a correct model identifier listed on 'https://huggingface.co/models'
  - or 'bert-base-multilingual-cased-good' is the correct path to a directory containing a config.json file
  ```
  Seems the code was looking for `config.json` under this folder. :thinking: 
  According to the codes below, I guess the assumption should be correct:
  - https://github.com/huggingface/transformers/blob/04ab2ca639ee6fd1002ce0a498d892245f0c9093/src/transformers/configuration_utils.py#L456
  - https://github.com/huggingface/transformers/blob/04ab2ca639ee6fd1002ce0a498d892245f0c9093/src/transformers/file_utils.py#L218
  
  After googling, I found an answer from stackoverflow that we could just rename the `tokenizer_config.json` as `config.json`. That sounded nice, so I just tried it, now my folder structure became: (I use a new folder `bert-base-multilingual-cased-bad` just for showcase)
  ```
  bert-base-multilingual-cased-bad
  ├── config.json
  ├── special_tokens_map.json
  └── vocab.txt
  ```
  Then I tried to load from it:
  ```python
  bad_tokenizer = AutoTokenizer.from_pretrained(
      pretrained_model_name_or_path='bert-base-multilingual-cased-bad',
      use_fast=True,
  )
  ```
  It works! :100: However, now the tokenizer became very weird:
  ```
  PreTrainedTokenizerFast(name_or_path='bert-base-multilingual-cased-bad', vocab_size=119547, model_max_len=1000000000000000019884624838656, is_fast=True, padding_side='right', special_tokens={'unk_token': '[UNK]', 'sep_token': '[SEP]', 'pad_token': '[PAD]', 'cls_token': '[CLS]', 'mask_token': '[MASK]'})
  ```
  Well, the `model_max_len` became `1000000000000000019884624838656` instead of `512`. After further checking other attributes that were included in the original `tokenizer_config.json`, I found even `do_lower_case` now becomes `True`, scary... :fearful: 
  
  I bet the `bad_tokenizer` didn't take into account the settings from `config.json`, it might just initialize with the default settings instead. So I tried to copy `config.json` and renamed it as `tokenizer_config.json`, now my folder structure looks like this: (I got both `config.json` and `tokenizer_config.json` although they are just the same file with different names)
  ```
  bert-base-multilingual-cased-work
  ├── config.json
  ├── special_tokens_map.json
  ├── tokenizer_config.json
  └── vocab.txt
  ```
  Although it looks weird, it works as expected: (I use a new folder `bert-base-multilingual-cased-work` just for showcase)
  ```python
  work_tokenizer = AutoTokenizer.from_pretrained(
      pretrained_model_name_or_path='bert-base-multilingual-cased-work',
      use_fast=True,
  )
  ```
  Now the tokenizer seems fine: 
  ```
  PreTrainedTokenizerFast(name_or_path='bert-base-multilingual-cased-work', vocab_size=119547, model_max_len=512, is_fast=True, padding_side='right', special_tokens={'unk_token': '[UNK]', 'sep_token': '[SEP]', 'pad_token': '[PAD]', 'cls_token': '[CLS]', 'mask_token': '[MASK]'})
  ```
  
  ### Why I think this could be severe?
  Initially, I just renamed `tokenizer_config.json` as `config.json` and deployed my NLP model (that's why I need to save the tokenizer locally) without checking the tokenizer in detail. However, I found the results from the deployed model didn't match with the local ones. So I did some investigation and found that this issue. 
  If you also encountered similar issues, maybe you can try to check your tokenizer.
  
  I think it doesn't make sense that the code looks for `config.json` in function `from_pretrained` while there is no `config.json` generated when function `save_pretrained` is called (https://github.com/huggingface/transformers/blob/04ab2ca639ee6fd1002ce0a498d892245f0c9093/src/transformers/tokenization_utils_base.py#L110). It's better to unify the config naming system, otherwise, it might cause severe issues that are very hard to spot. 