text: 
  
  TITLE
  Saving Pretrained Tokenizer
  
  COMMENTS
  2
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  I created a custom `tokenizers.Tokenizer` and saved it as follows
  
  ```
  tokenizer.model.save("./tokenizer")
  tokenizer.save("./tokenizer.json") 
  ```
  
  This produces 3 files, merges.txt, vocab.json and tokenizer.json
  
  Then I created a `transformers.RobertaTokenizerFast` and saved it to the same folder
  
  ```
  tokenizer = RobertaTokenizerFast.from_pretrained("./tokenizer")  
  tokenizer.save_pretrained("./tokenizer") 
  ```
  
  This adds `special_tokens_map.json` and `tokenizer_config.json`
  
  I then saved it to another folder to simulate what happens when I train my model
  
  ```
  tokenizer.save_pretrained("./model") 
  tokenizer = RobertaTokenizerFast.from_pretrained("./model")  
  ```
  
  What I noticed was `tokenizer_config.json` contains a key `name_or_path` which still points to `./tokenizer`, so what seems to be happening is `RobertaTokenizerFast.from_pretrained("./model") ` is loading files from two places (./model and ./tokenizer)
  
  Not sure if this is expected, it seems that the tokenizer_config.json should be updated in save_pretrained, and tokenizer.json should be saved with it?
  
  Or perhaps this is just an issue because I'm training my tokenizer in a subdirectory of the model folder?
  