

TITLE
Generate function does not work with GPU

COMMENTS
11

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
Hello,

I want to use generate function with single GPU. Specifically, I fine tuned a GPT-2 model (on GPU) and subsequently, I want to generate text with it. 

When I run this 

```
input_ids.to(device)
sample_output = model.generate(
    input_ids, 
    do_sample=True, 
    max_length=150,
    top_k=50,
    top_p=0.92
)
```
I get this error

`RuntimeError: Input, output and indices must be on the current device`

When i move the model and the input `.to('cpu')`, it works.