

TITLE
LayoutLM Token Classification not learning

COMMENTS
4

REACTIONS
+1: 1
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
## Environment info
- `transformers` version: 3.4.0
- Platform: in docker based on image: nvidia/cuda:10.1-cudnn7-devel-ubuntu18.04
- Python version: 3.7.9
- PyTorch version (GPU?): 1.5.1+cu92 (True)
- Tensorflow version (GPU?): 2.2.0-rc0 (True)
- Using GPU in script?: True
- Using distributed or parallel set-up in script?: <fill in>

## Information

Model I am using (Bert, XLNet ...): LayoutLMForTokenClassification

The problem arises when using: my own scripts

The tasks I am working on is: my own task
NER task. I've reproduced the implementation of Dataset, compute metrics (and other helper functions) as in the original repo [microsoft/layoutlm repo](https://github.com/microsoft/unilm/blob/master/layoutlm/examples/seq_labeling/run_seq_labeling.py)

When initially trying with the original repo and training script the model managed to learn and provided reasonable results after very few epochs. After implementing with Huggingface the model doesn't learn at all even after a much higher number of epochs.


## To reproduce
Model loading and trainer configuration:
```
config = LayoutLMConfig.from_pretrained(
    <path_layoutlm_base_uncased>,
    num_labels=<num_labels>,
    cache_dir=None
)
model = LayoutLMForTokenClassification.from_pretrained(
    <path_layoutlm_base_uncased>
    from_tf=bool(".ckpt" in <path_layoutlm_base_uncased>),
    config=config,
    cache_dir=None,
)

device = torch.device("cuda")
model.train().to(device)

TrainingArguments(
    output_dir=<pytorch_model_dir>,  # output directory
    do_train=True,
    do_eval=True,
    do_predict=False,
    evaluation_strategy=EvaluationStrategy.EPOCH,
    num_train_epochs=<epochs>,  # total # of training epochs
    per_device_train_batch_size=<batch_size>,  # batch size per device during training
    per_device_eval_batch_size=<batch_size>,  # batch size for evaluation
    weight_decay=<weight_decay>,  # strength of weight decay
    learning_rate=<learning_rate>,
    adam_epsilon=<adam_epsilon>,
    logging_dir=<profile_logs>,  # Tensorboard log directory
    logging_steps=0,  # it logs when running evaluation so no need to log on step interval
    save_steps=0,
    seed=seed,
    overwrite_output_dir=True,
    disable_tqdm=False,
    load_best_model_at_end=True,
    save_total_limit=10,
    fp16=True,
)

trainer = MetaMazeTrainer(
    model=model,  # the instantiated 🤗 Transformers model to be trained
    args=training_args,  # training arguments, defined above
    train_dataset=train_dataset,  # training dataset
    eval_dataset=test_dataset,  # evaluation dataset
    compute_metrics=compute_metrics,
)

```

## Expected behavior
Similar results to the original repo as the given the same parameters to the trainer and the Dataset being the same after processing the data.

Is this due to the ongoing integration of this model? Is the setup wrong?