

TITLE
Blenderbot export issue in ONNX and C++

COMMENTS
10

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
# 📚 Migration



Hi everyone, my issue is regarding blenderbot: facebook/blenderbot-400M-distill. I did the exportation with: 

`python -m transformers.onnx --model=facebook/blenderbot-400M-distill onnx/`

And I get:

```
Some weights of the model checkpoint at facebook/blenderbot-400M-distill were not used when initializing BlenderbotModel: ['final_logits_bias', 'lm_head.weight']
- This IS expected if you are initializing BlenderbotModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BlenderbotModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).   
Using framework PyTorch: 1.10.0+cu102
Overriding 1 configuration item(s)
        - use_cache -> False      
...\venv\lib\site-packages\transformers\models\blenderbot\modeling_blenderbot.py:219: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if attn_weights.size() != (bsz * self.num_heads, tgt_len, src_len):
...\venv\lib\site-packages\transformers\models\blenderbot\modeling_blenderbot.py:225: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if attention_mask.size() != (bsz, 1, tgt_len, src_len):
...\venv\lib\site-packages\transformers\models\blenderbot\modeling_blenderbot.py:256: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if attn_output.size() != (bsz * self.num_heads, tgt_len, self.head_dim):
...\venv\lib\site-packages\transformers\models\blenderbot\modeling_blenderbot.py:845: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if input_shape[-1] > 1:
Validating ONNX model...
        -[✓] ONNX model output names match reference model ({'last_hidden_state'})
        - Validating ONNX Model output "last_hidden_state":
                -[✓] (2, 8, 1280) matches (2, 8, 1280)     
                -[✓] all values close (atol: 1e-05)        
All good, model saved at: onnx/model.onnx
```

**The warning: "TracerWarning: Converting a tensor to a Python boolean...", I think it is the classic problem of control flow**. I have tried to export blenderbot with torch.jit.script(model). However, I get:

```
return any(hasattr(m, "gradient_checkpointing") and m.gradient_checkpointing for m in self.modules())
                   ~ <--- HERE
```

**I understand that TORCHSCRIPT dosen't support the function: any. Therefore, there isn't possibility of export blenderbot as a script and overcomes the problems of control flow**.

Anyway, I wrote the next onnx programm in C++:

```
#include <iostream>
#include <onnxruntime_cxx_api.h>

using namespace std;

void max(std::vector<float>& input, int64_t size_input_ids, int64_t size_embedding, std::vector<int>& output) {
	double bigger = 0.0;
	int index = 0;
	int k = 0;

	output.clear();
	for (size_t i = 0; i < size_input_ids; i++) {//128
		bigger = 0.0;
		index = 0;
		for (size_t j = 0; j < size_embedding; j++) {//1280 or 8008
            if (input[k] > bigger) {
				bigger = input[k];
				index = j;
			}
			k++;
		}
		output.push_back(index);
	}
}

int main()
{

	Ort::Env env;
	Ort::RunOptions runOptions;
	Ort::Session session(nullptr);

	auto modelPath = L"D:\\Nexus\\NexusV_1_0_0\\x64\\Release\\Conversational\\Model\\model.onnx";

	// create session
	session = Ort::Session(env, modelPath, Ort::SessionOptions{ nullptr });
	
	constexpr int64_t size_input = 128;
	constexpr int64_t size_output = 128;
	constexpr int64_t output_01 = 1280;//8008
	constexpr int64_t numOutputElements = size_output * output_01;
	constexpr int64_t output_02 = 1280;
	constexpr int64_t numOutputElements_02 = size_output * output_02;

	// define shape
	const array<int64_t, 2> inputShape = { 1, size_input};
	const array<int64_t, 3> outputShape = { 1, size_output, output_01 };
	const array<int64_t, 3> outputShape_02 = { 1, size_output, output_02 };

	std::vector<int64_t> input_ids(size_input);
	std::vector<int64_t> attention_mask(size_input);
	std::vector<int64_t> decoder_input_ids(size_input);
	std::vector<int64_t> decoder_attention_mask(size_input);

	std::vector<float> last_hidden_state(numOutputElements);
	std::vector<float> last_output_02(numOutputElements_02);

	for (size_t  i = 0; i < input_ids.size(); i++) {
		input_ids[i] = 0;
		attention_mask[i] = 0;
		decoder_input_ids[i] = 0;
		decoder_attention_mask[i] = 0;
	}

	//IDs: I want to order a Pizza
	input_ids[0] = 281;
	input_ids[1] = 538;
	input_ids[2] = 287;
	input_ids[3] = 1831;
	input_ids[4] = 265;
	input_ids[5] = 440;
	input_ids[6] = 4425;
	input_ids[7] = 2;

	attention_mask[0] = 1;
	attention_mask[1] = 1;
	attention_mask[2] = 1;
	attention_mask[3] = 1;
	attention_mask[4] = 1;
	attention_mask[5] = 1;
	attention_mask[6] = 1;
	attention_mask[7] = 1;
		
	for (size_t  i = 0; i < last_hidden_state.size(); i++)
		last_hidden_state[i] = 0;

	for (size_t i = 0; i < last_output_02.size(); i++)
		last_output_02[i] = 0;

	// define Tensor
	auto memory_info = Ort::MemoryInfo::CreateCpu(OrtDeviceAllocator, OrtMemTypeCPU);
				
	std::vector<Ort::Value> inputTensors;
	inputTensors.push_back(Ort::Value::CreateTensor<int64_t>(memory_info, input_ids.data(), input_ids.size(), inputShape.data(), inputShape.size()));
	inputTensors.push_back(Ort::Value::CreateTensor<int64_t>(memory_info, attention_mask.data(), attention_mask.size(), inputShape.data(), inputShape.size()));
	inputTensors.push_back(Ort::Value::CreateTensor<int64_t>(memory_info, decoder_input_ids.data(), decoder_input_ids.size(), inputShape.data(), inputShape.size()));
	inputTensors.push_back(Ort::Value::CreateTensor<int64_t>(memory_info, decoder_attention_mask.data(), decoder_attention_mask.size(), inputShape.data(), inputShape.size()));

	std::vector<Ort::Value> outputTensors;
	outputTensors.push_back(Ort::Value::CreateTensor<float>(memory_info, last_hidden_state.data(), last_hidden_state.size(), outputShape.data(), outputShape.size()));
	outputTensors.push_back(Ort::Value::CreateTensor<float>(memory_info, last_output_02.data(), last_output_02.size(), outputShape_02.data(), outputShape_02.size()));
		
	// define names
	Ort::AllocatorWithDefaultOptions allocator;
	std::vector < const char*> inputNames;
	inputNames.push_back(session.GetInputName(0, allocator));
	inputNames.push_back(session.GetInputName(1, allocator));
	inputNames.push_back(session.GetInputName(2, allocator));
	inputNames.push_back(session.GetInputName(3, allocator));
    
	std::vector < const char*> outputNames;
	outputNames.push_back(session.GetOutputName(0, allocator));
	outputNames.push_back(session.GetOutputName(1, allocator));
		
	// run inference
	try {
		session.Run(Ort::RunOptions{ nullptr }, inputNames.data(), inputTensors.data(), 4, outputNames.data(), outputTensors.data(), 2);
	}
	catch (Ort::Exception& e) {
		cout << e.what() << endl;
		return 1;
	}
	
	std::vector<int> output_ids;
	max(last_hidden_state, size_input, output_01, output_ids);

	for (size_t i = 0; i < output_ids.size(); i++) {
		cout << output_ids[i] << ", ";
	}
	cout << endl;
  
	return 0;
}
```

I get the output:

`652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652, 652,`

**I think the reason why the ids are the same is regarding the control flow problem** (or maybe there is something wrong in my program).

I understand that blenderbot is supported: [(https://huggingface.co/docs/transformers/main/en/serialization#torchscript)]

**Please, could you give me a suggestion to export correctly blenderbot or get the correct output ids?**

Note: I also use this tutorial ([https://www.kaggle.com/code/danieliusv/blenderbot1-0-test-and-export/notebook]), but I get the same results.

**Environment**

Windows 10
PyCharm 2021.3 (Community Edition)
Visual Studio 2019
