

TITLE
Pytorch Vs Onnx: Pytorch is faster and provides different output

COMMENTS
7

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 1

BODY
## Environment info

- `transformers` version: 3.5.0
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.7.0+cu101 (True)
- Tensorflow version (GPU?): 2.3.0 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help

albert, bert, GPT2, XLM: @LysandreJik 
Onnx: @mfuntowicz


## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

https://colab.research.google.com/drive/1UwgWgUF4k_GPJ5TcziHo4eH_rRFQeNVL?usp=sharing


## Expected behavior

I have followed the  the onnx export tutorial:
https://github.com/huggingface/transformers/blob/master/notebooks/04-onnx-export.ipynb
However, I have found 2 issues:
1. Pytorch is faster than Onnx.
2. Onnx produce different embedding output than Pytorch.

Could anyone help me to figure out the issue ?