

TITLE
Eval_loss in prediction is very high : transformers/examples/token-classification/run_ner.py 

COMMENTS
5

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
## Environment info
- `transformers` version: 3.3.1
- Platform: Linux-4.9.0-12-amd64-x86_64-with-debian-9.12
- Python version: 3.7.6
- PyTorch version (GPU?): 1.4.0 
- Tensorflow version (GPU?): not installed 
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

     

### Who can help
 examples/token-classification: @stefan-it

## Information

 I am using NER "Emerging and Rare Entities task: WNUT’17 (English NER) dataset"
I am executing the steps as prescribed in https://github.com/huggingface/transformers/tree/08ba4b4902df5a18f5ad41d9490c50fe0a4c970f/examples/token-classification

The problem arises when using prediction:
* [ ] the official example script: wnut_17.json
{
"data_dir": "/home/priya/data_wnut_17",
"labels": "/home/priya/data_wnut_17/labels.txt",
"model_name_or_path": "bert-large-cased",
"output_dir": "wnut-17-model-1",
"max_seq_length": 128,
"num_train_epochs": 3,
"per_device_train_batch_size": 16,
"save_steps": 425,
"seed": 1,
"do_train": true,
"do_eval": true,
"do_predict": true,
"fp16": false
}
* [ ] my own modified scripts: wnut_17_mod.json
{
"data_dir": "/home/priya/data_wnut_17",
"labels": "/home/priya/data_wnut_17/labels.txt",
"model_name_or_path": "bert-large-cased",
"output_dir": "wnut-17-model-1",
"max_seq_length": 128,
"num_train_epochs": 3,
"per_device_train_batch_size": 16,
"save_steps": 425,
"seed": 1,
"do_train": **false,**
"do_eval": **false**,
"do_predict": true,
"fp16": false,
"overwrite_output_dir":false
}


The tasks I am working on is:
* [ ] re-run WNUT’17 dataset. 

My end-goal is to identify abbreviation and explanation from sentences (labels B-abbr, I-abbr, B-expl, I-expl and O). For the example sentence

> Here GAAP stands for Generally accepted accounting principles

we should get token classified as

> Here O
> GAAP B-abbr
> stands O
> for O
> Generally B-expl
> accepted I-expl
> accounting I-expl
> principles I-expl

## To reproduce

Steps to reproduce the behavior:

1.python run_ner.py wnut_17.json
Prediction: 100%|████████████████████████████████████████████████████████| 162/162 [01:08<00:00,  2.38it/s]
10/06/2020 07:21:15 - INFO - __main__ -     eval_loss = 0.2851179020827574
10/06/2020 07:21:15 - INFO - __main__ -     eval_accuracy_score = 0.9511413182867402
10/06/2020 07:21:15 - INFO - __main__ -     eval_precision = 0.5997392438070405
10/06/2020 07:21:15 - INFO - __main__ -     eval_recall = 0.4263206672845227
10/06/2020 07:21:15 - INFO - __main__ -     eval_f1 = 0.49837486457204777

2.python run_ner.py wnut_17_mod.json
Prediction: 100%|████████████████████████████████████████████████████████| 162/162 [01:08<00:00,  2.38it/s]
/opt/conda/lib/python3.7/site-packages/transformers/trainer.py:1175: FutureWarning: This method is deprecated, use `Trainer.is_world_process_zero()` instead.
  warnings.warn("This method is deprecated, use `Trainer.is_world_process_zero()` instead.", FutureWarning)
10/06/2020 08:30:41 - INFO - __main__ -     eval_loss = 2.827890293463151
10/06/2020 08:30:41 - INFO - __main__ -     eval_accuracy_score = 0.016072497221509788
10/06/2020 08:30:41 - INFO - __main__ -     eval_precision = 0.0065180614986565825
10/06/2020 08:30:41 - INFO - __main__ -     eval_recall = 0.12140871177015755
10/06/2020 08:30:41 - INFO - __main__ -     eval_f1 = 0.012371912924399112

## Expected behavior
I am seeing a 10-fold increase in eval-loss from 0.28 to 2.8. 
Other than the changes in  wnut_17_mod.json , I have done no other changes. Please advice how to achieve the published eval_loss and performance.

Thanks,
