

TITLE
Failed in predict function after converting xlnet model to onnx format

COMMENTS
2

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version:
- Platform:ubuntu 18.04
- Python version:3.7.9
- PyTorch version (1.7.0):
- Using GPU in script:No
- Using distributed or parallel set-up in script?:No
- onnx                1.8.0
- onnxruntime         1.5.2
- transformers        3.5.1


### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 
 TransfoXL/XLNet: @TevenLeScao

 -->
TransfoXL/XLNet: 
@TevenLeScao
## Information

Model I am using (XLNet):

The problem arises when using:
* [x] my own modified scripts: (give details below)

## To reproduce

Steps to reproduce the behavior:

1.train a pytorch model by XLNetForSequenceClassification successfully
2.convert to onnx format by convert_graph_to_onnx.py with some warning:
python ../convert_graph_to_onnx.py --pipeline sentiment-analysis --framework pt --opset 12 ....

====== Converting model to ONNX ======
ONNX opset version set to: 12
Loading pipeline (model: /home/yhq/onnx/icd_model/9-4, tokenizer: /home/yhq/onnx/icd_model/9-4)
Using framework PyTorch: 1.7.0
Found input input_ids with shape: {0: 'batch', 1: 'sequence'}
Found input token_type_ids with shape: {0: 'batch', 1: 'sequence'}
Found input attention_mask with shape: {0: 'batch', 1: 'sequence'}
Found output output_0 with shape: {0: 'batch'}
Found output output_1 with shape: {1: 'batch', 0: 'sequence'}
Found output output_2 with shape: {1: 'batch', 0: 'sequence'}
Found output output_3 with shape: {1: 'batch', 0: 'sequence'}
Found output output_4 with shape: {1: 'batch', 0: 'sequence'}
Found output output_5 with shape: {1: 'batch', 0: 'sequence'}
Found output output_6 with shape: {1: 'batch', 0: 'sequence'}
Found output output_7 with shape: {1: 'batch', 0: 'sequence'}
Found output output_8 with shape: {1: 'batch', 0: 'sequence'}
Found output output_9 with shape: {1: 'batch', 0: 'sequence'}
Found output output_10 with shape: {1: 'batch', 0: 'sequence'}
Found output output_11 with shape: {1: 'batch', 0: 'sequence'}
Found output output_12 with shape: {1: 'batch', 0: 'sequence'}
Ensuring inputs are in correct order
mems is not present in the generated input list.
Generated inputs order: ['input_ids', 'attention_mask']
/home/yhq/anaconda3/envs/convert/lib/python3.7/site-packages/torch/onnx/utils.py:1109: UserWarning: Provided key token_type_ids for dynamic axes is not a valid input/output name
  warnings.warn("Provided key {} for dynamic axes is not a valid input/output name".format(key))
/home/yhq/anaconda3/envs/convert/lib/python3.7/site-packages/transformers/modeling_xlnet.py:1160: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  non_tgt_mask = -torch.eye(qlen).to(attn_mask)
/home/yhq/anaconda3/envs/convert/lib/python3.7/site-packages/transformers/modeling_utils.py:1673: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  input_tensor.shape == tensor_shape for input_tensor in input_tensors

3.the model I trained works well while predict by loading the model in pytorch format, however failed while loading the model in onnx format,errors as follow:
onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running Add node. Name:'Add_26' Status Message: /onnxruntime_src/onnxruntime/core/providers/cpu/math/element_wise_ops.h:475 void onnxruntime::BroadcastIterator::Init(int64_t, int64_t) axis == 1 || axis == largest was false. Attempting to broadcast an axis by a dimension other than 1. 6 by 48

my codes:

import onnxruntime,os,torch
from tqdm.auto import tqdm
from torch.utils.data import TensorDataset,SequentialSampler,DataLoader
from transformers import XLNetTokenizer

sess_options = onnxruntime.SessionOptions()
sess_options.graph_optimization_level = onnxruntime.GraphOptimizationLevel.ORT_ENABLE_ALL
sess_options.intra_op_num_threads = 1
session = onnxruntime.InferenceSession(os.path.join(onnx_path,'onnx_model.onnx'), sess_options)
tokenizer = XLNetTokenizer.from_pretrained(onnx_path)
input_ids = []
attention_masks = []
for name in name_list:
    encoded_dict = tokenizer.encode_plus(
        name,
        add_special_tokens=True,
        max_length=max_len,
        pad_to_max_length=True,
        return_attention_mask=True,
        return_tensors='pt',
    )
    input_ids.append(encoded_dict['input_ids'])
    attention_masks.append(encoded_dict['attention_mask'])
input_ids = torch.cat(input_ids, dim=0)
attention_mask = torch.cat(attention_masks, dim=0)

prediction_data = TensorDataset(input_ids,attention_mask)
prediction_sampler = SequentialSampler(prediction_data)
prediction_dataloader = DataLoader(prediction_data, sampler=prediction_sampler, batch_size=1)
for step in tqdm(prediction_dataloader, desc="Predicting"):
    step = tuple(t.detach().cpu().numpy() for t in step)
    ort_inputs = {'input_ids': step[0],
                  'attention_mask': step[1]
                  }
    logits = session.run(None, ort_inputs) #failed


<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
