

TITLE
config.json not found!

COMMENTS
2

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
### System Info

```shell
- `transformers` version: 4.18.0
- Platform: Linux-5.4.0-1063-azure-x86_64-with-glibc2.10
- Python version: 3.8.3
- Huggingface_hub version: 0.5.1
- PyTorch version (GPU?): 1.7.1+cu110 (True)
- Tensorflow version (GPU?): 2.4.1 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: yes
```


### Who can help?

@sgugger 
I am training a NER model following tutorial:
```python
from transformers import TrainingArguments

args = TrainingArguments(
    "saved_models_bert-finetuned-ner-100examples-with-aug",
    
    learning_rate=2e-5,
    num_train_epochs=100,
    weight_decay=0.01,
    per_device_train_batch_size = 32,
    per_device_eval_batch_size = 32,
    
    evaluation_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end = True,  
    metric_for_best_model = 'f1'
)

from transformers import Trainer

trainer = Trainer(
    model=model,
    args=args,
    train_dataset=new_training_dataset,
    eval_dataset=tokenized_datasets["validation"].select(range(100)),
    data_collator=data_collator,
    compute_metrics=compute_metrics,
    tokenizer=tokenizer,
)

trainer.train()
```

Then I got this error:
```shell
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
Input In [28], in <cell line: 14>()
      1 from transformers import Trainer
      3 trainer = Trainer(
      4     model=model,
      5     args=args,
   (...)
     11     tokenizer=tokenizer,
     12 )
---> 14 trainer.train()

File /opt/conda/lib/python3.8/site-packages/transformers/trainer.py:1512, in Trainer.train(self, resume_from_checkpoint, trial, ignore_keys_for_eval, **kwargs)
   1509     self.control.should_training_stop = True
   1511 self.control = self.callback_handler.on_epoch_end(args, self.state, self.control)
-> 1512 self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
   1514 if DebugOption.TPU_METRICS_DEBUG in self.args.debug:
   1515     if is_torch_tpu_available():
   1516         # tpu-comment: Logging debug metrics for PyTorch/XLA (compile, execute times, ops, etc.)

File /opt/conda/lib/python3.8/site-packages/transformers/trainer.py:1628, in Trainer._maybe_log_save_evaluate(self, tr_loss, model, trial, epoch, ignore_keys_for_eval)
   1625     self._report_to_hp_search(trial, epoch, metrics)
   1627 if self.control.should_save:
-> 1628     self._save_checkpoint(model, trial, metrics=metrics)
   1629     self.control = self.callback_handler.on_save(self.args, self.state, self.control)

File /opt/conda/lib/python3.8/site-packages/transformers/trainer.py:1700, in Trainer._save_checkpoint(self, model, trial, metrics)
   1697     self.store_flos()
   1699 output_dir = os.path.join(run_dir, checkpoint_folder)
-> 1700 self.save_model(output_dir, _internal_call=True)
   1701 if self.deepspeed:
   1702     # under zero3 model file itself doesn't get saved since it's bogus! Unless deepspeed
   1703     # config `stage3_gather_16bit_weights_on_model_save` is True
   1704     self.deepspeed.save_checkpoint(output_dir)

File /opt/conda/lib/python3.8/site-packages/transformers/trainer.py:2128, in Trainer.save_model(self, output_dir, _internal_call)
   2125             self.deepspeed.save_checkpoint(output_dir)
   2127 elif self.args.should_save:
-> 2128     self._save(output_dir)
   2130 # Push to the Hub when `save_model` is called by the user.
   2131 if self.args.push_to_hub and not _internal_call:

File /opt/conda/lib/python3.8/site-packages/transformers/trainer.py:2180, in Trainer._save(self, output_dir, state_dict)
   2178         torch.save(state_dict, os.path.join(output_dir, WEIGHTS_NAME))
   2179 else:
-> 2180     self.model.save_pretrained(output_dir, state_dict=state_dict)
   2181 if self.tokenizer is not None:
   2182     self.tokenizer.save_pretrained(output_dir)

File /opt/conda/lib/python3.8/site-packages/transformers/modeling_utils.py:1352, in PreTrainedModel.save_pretrained(self, save_directory, save_config, state_dict, save_function, push_to_hub, max_shard_size, **kwargs)
   1350 # Save the config
   1351 if save_config:
-> 1352     model_to_save.config.save_pretrained(save_directory)
   1354 # Save the model
   1355 if state_dict is None:

File /opt/conda/lib/python3.8/site-packages/transformers/configuration_utils.py:440, in PretrainedConfig.save_pretrained(self, save_directory, push_to_hub, **kwargs)
    437 # If we save using the predefined names, we can load using `from_pretrained`
    438 output_config_file = os.path.join(save_directory, CONFIG_NAME)
--> 440 self.to_json_file(output_config_file, use_diff=True)
    441 logger.info(f"Configuration saved in {output_config_file}")
    443 if push_to_hub:

File /opt/conda/lib/python3.8/site-packages/transformers/configuration_utils.py:805, in PretrainedConfig.to_json_file(self, json_file_path, use_diff)
    794 def to_json_file(self, json_file_path: Union[str, os.PathLike], use_diff: bool = True):
    795     """
    796     Save this instance to a JSON file.
    797 
   (...)
    803             is serialized to JSON file.
    804     """
--> 805     with open(json_file_path, "w", encoding="utf-8") as writer:
    806         writer.write(self.to_json_string(use_diff=use_diff))

FileNotFoundError: [Errno 2] No such file or directory: 'saved_models_bert-finetuned-ner-100examples-with-aug/checkpoint-6/config.json'
```

This is so wired! From my understanding, the config.json file should be written, so such error shouldn't occur.


### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I am not sure if this can be reproduced in another machine. btw, I am using A100.

### Expected behavior

```shell
A normal training...
```
