

TITLE
[models] why aren't .bin files compressed for faster download?

COMMENTS
9

REACTIONS
+1: 4
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
Why are not the pretrained model files compressed? 

It took ~25minutes yesterday to download 45GB t5-11b on a slow connection. 

I did a quick test on a random `pytorch_model.bin` with default gzip options and it's 1/3rd less in size. And surely there must be a better compressor that can be used - but will require to be on the client's side so gzip might be good enough. This is not much of a difference for under 1GB files, but for large models this starts to add up.

It's not like you can diff a .bin file, so there is little value in having it stored as is from the RCS point of view. But perhaps I'm missing other aspects. 

Perhaps 2 versions can be stored and the retriever could favor the compressed version for large files?

Cost-wise this change would introduce some 60% increase in storage if both versions are to be stored, but will have a huge saving in downloads.

The compression process can be a simple cronjob, so that it won't need to make users do anything special.

@julien-c, @LysandreJik, @patrickvonplaten, @sgugger 