

TITLE
Loading a pytorch quantized model

COMMENTS
2

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
Hi, I quantized a pre-trained model via **torch.quantization.quantize_dynamic** and saved it using **save_pretrained**.
based on https://colab.research.google.com/github/pytorch/tutorials/blob/gh-pages/_downloads/dynamic_quantization_bert_tutorial.ipynb#scrollTo=foe-dVxHIgOC . 
On reloading it using **from_pretrained** again the model blows up to its original size and the resultant predictions are garbage as well.
Is there a way to properly save the quantized weights and reload them?
Opening an issue because I couldn't find a resolution for the same.
@mfuntowicz @VictorSanh @patrickvonplaten 