

TITLE
T5base memory usage for interface

COMMENTS
5

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
### System Info

```shell
Ubuntu 18.04 RTX 2080
```


### Who can help?

@patrickvonplaten

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Just load t5 base from any example.

### Expected behavior


Is it normal that T5 base uses around 1000 MB of GPU memory in half precision (FP-16) for interface.

Many other models use roughly 50 percent of FP-32 model size for FP-16 which should be only 400-500 MB.

I read here that the model https://huggingface.co/google/t5-efficient-small-el16 is 350 mb in size so it uses roughly 50% of the size (175 MB) when used for interface.

Is there anything to do to reduce the memory usage. T5-large for example  also in Fp16 uses around 2GB which is much better.

I expected memory usage to be 50% of model size for fp 16.

