

TITLE
Support fp16 for inference

COMMENTS
6

REACTIONS
+1: 4
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
# 🚀 Feature request - support fp16 inference

Right now most models support mixed precision for model training, but not for inference. Naively calling `model= model.haf()` makes the model generate junk instead of valid results for text generation, even though mixed-precision works fine in training. 

If there's a way to make the model produce stable behavior at 16-bit precision at inference, the throughput can potentially double on most modern GPUS. 

## Motivation

Double the speed is always attractive, especially since transformers are compute-intensive. 

