juierror

/

flan-t5-text2sql-with-schema

like 30

@cioo

Hi, to be frank, I have never export this model to onnx and use it with C#, so I'm not sure what happen, but from the error, maybe some of the first layer is missing (not sure).

If I have time, I will try to reproduce it with python onnx.

Hello,
thank you for your feedback!
I simply exported btw with: optimum-cli export onnx --model "juierror/text-to-sql-with-table-schema" model_onnx/

Then I tried to load decoder_model.onnx (https://learn.microsoft.com/en-us/azure/machine-learning/how-to-use-automl-onnx-model-dotnet?view=azureml-api-2).
I dont know how exactly the input-data (encoder_attention_mask, input_ids, encoder_hidden_states) should be initialized.
For input_ids I used the output of: prepare_input(question=myquestion, table=mytable) - in python.
encoder_attention_mask = 1 and encoder_hidden_states = 1, but get the error "encoder_hidden_states Got: 2 Expected: 3 Please fix either the inputs or the model.".

I'm curious about your experiences. Best Regards

Hi,

I already try to export and inference the onnx model. The T5 model is a encoder-decoder model, so it different from the tutorial you are provide which is encoder only model https://onnxruntime.ai/docs/tutorials/csharp/bert-nlp-csharp-console-app.html.

When I run optimum-cli export onnx --model "juierror/text-to-sql-with-table-schema" model_onnx/, I get 4 onnx models inside model_onnx/, but I use only encoder_model.onnx and decoder_model.onnx.

From my understanding, we have to run encoder model first and then use the encoder output as one of input to the decoder model.

I can get the result with this method, but the output is incorrect, so I'm not sure if my step is wrong or not.

this is input

this is output

this is the code https://github.com/juierror/experiment_code/blob/master/test_onnx_clean.ipynb

I also see optimum has a way to run the model
https://huggingface.co/docs/transformers/serialization#exporting-a-transformers-model-to-onnx-with-cli
https://huggingface.co/docs/optimum/v1.2.1/en/onnxruntime/modeling_ort#optimum.onnxruntime.ORTModel

Thus, maybe you can check how optimum inference and try it with C#.

Hi,

I ran encoder model first and then used the encoder output as one of input to the decoder model, like this:

I also can get the decoder output. The decoder models output is "logits", a float array.
Do I need to use "google/flan-t5-base"-Tokenizer to get the words from logits? Because in .Net there are only a few Tokenizer, e.g. BertUncasedLargeTokenizer.

I also tried your training code with "google/flan-t5-base" model and BertTokenizer, but result was:
question: Who is the director of the episode that corresponds to the total episodes number 14?
table: ['Total#', 'Series#', 'Title', 'Writer', 'Director', 'Original air date']
model result: [unused19] [unused19] [unused19] [unused19] [unused19] [unused19] [unused19] [unused19] [unused19] [unused1]
real result: SELECT Director FROM table WHERE Total# = 14

Is it important that the model was trained with the same tokenizer as it will be called? If that's the case, I would need a model trained with the BertTokenizer. Do you have any idea why simply swapping the tokenizers in your training code doesn't work?

Best regards!

Hi @cioo

About the first question, after you get logits, you should use the same tokenizer as training phase, because each logits value represent the token which is different for each tokenizer.

For the second question, from my understanding, the T5 model is pre-trained on T5 tokenizer, so it is recognize which logit is which token. If you change the tokenizer when finetune, the model might confuse.

Hello @cioo @juierror ,
I it possible to create single onnx file instead of creating 2 files that is encoder.onnx and decoder.onnx . same thing i am trying to convert a model to tflite getting same issue. please le me know is there any way to do this.

@muhammadbaasit

Hi, sorry for late reply, to be frank, I am not not sure how to merge encoder.onnx and decoder.onnx into 1 file or export it in single file.