

TITLE
BART Summarization : Torchscript Export / Inference Triton Server

COMMENTS
3

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
# 📚 Migration

@sshleifer maybe you can help ?? (thanks for all your work bud!)

## Information

**Objective** : Performance gain, clocking 1-1.5 sec per transaction at the moment, target : under 100 ms. It seems exporting model via TorchScript & running on Triton Server may be plausible solution.

I am exporting BART Large CNN (for generating Summaries) using TorchScript. I have fine-tuned the model with localized data, but I am unclear on how to use **model.generate(input)** which seems to wrap **model(input)**, whereas **model(input)** is being triggered default at the time of inference from the exported model. For simplicity & ability to reproduce, I pasting the issue/code details as if the model is vanilla pre-trained (and not fine-tuned)

Model: **facebook/bart-large-cnn** 

Language: **English**

The problem arises when using: **torch.jit.trace(<model>, <dummy_input>)**

## Details
1. model.pt gets generated without any issues
2. However, the generated trace is producing output for mask-filling like model(input), what I am hoping to generate is model.**generate**(input).
3. Not sure, how to handle this, at the time of export or later during inference. Can you please help.

Code Block Below:

**Step 1**: Generate model.pt file
```
import torch

from transformers import (
    AutoModelForSeq2SeqLM,
    AutoTokenizer,
    AutoConfig
)

dummy_input = torch.tensor([512 * [1]])

BART_CNN_PATH = 'facebook/bart-large-cnn'
BART_CNN_MODEL = AutoModelForSeq2SeqLM.from_pretrained(BART_CNN_PATH)

BART_CNN_MODEL.eval()

traced_model = torch.jit.trace(BART_CNN_MODEL, dummy_input)
traced_model.save("exportedModelsForTritan/bart_large_cnn_fl/1/model.pt")
```

**Step2**: Inference **(+ Error Details)**

```
BART_CNN_TOKENIZER = AutoTokenizer.from_pretrained(BART_CNN_PATH)
input_tokenized = BART_CNN_TOKENIZER.encode(input_text, return_tensors="pt", max_length=512, truncation=True, padding='max_length')

## Test Inference, If I do not use .generate() code works fine ...
## but then it would attempt mask-filling instead of summaries?...
## With model.generate(input), it returns [1xn] where n is the length of summary
## where-as with mode(input), it's generating tuple with length 3 perhaps logits and possibly hidden state weights..
## which I do not know is of significance for summaries ...
model_output = traced_model.generate(input_tokenized)
```

**Error**: ModuleAttributeError: 'RecursiveScriptModule' object has no attribute 'generate'

**Working Code Prior Export as Reference: <Notice the use of model.generate()>**

```
BART_CNN_PATH = 'facebook/bart-large-cnn'
BART_CNN_MODEL = AutoModelForSeq2SeqLM.from_pretrained(BART_CNN_PATH)
BART_CNN_TOKENIZER = AutoTokenizer.from_pretrained(BART_CNN_PATH)

def bart_cnn_summarize_automl(input_text, num_beams=4, num_words=50):
    input_text = str(input_text)
    input_tokenized = BART_CNN_TOKENIZER.encode(input_text, return_tensors="pt", max_length=512)
    summary_ids = BART_CNN_MODEL.generate(input_tokenized, 
                                       max_length=100, 
                                       min_length=40, 
                                       length_penalty=2.0, 
                                       num_beams=num_beams, 
                                       early_stopping=True)
    output = [BART_CNN_TOKENIZER.decode(id, skip_special_tokens=True, clean_up_tokenization_spaces=False) for id in summary_ids]
    return str(output[0])
```

## Environment info
- Python version:Python 3.6.9
- PyTorch version (GPU?): GPU (T4), 1.7.1
- Docker Image: huggingface/transformers-pytorch-gpu:4.2.1



