memory used
Hello. I'm trying to execute this model and it's killed as it needs more memory than I have in the computer. How much memory it needs? I have to dedice how much to add.
Thanks.
Hi
@cdani
, are you talking about CPU memory or GPU memory?
Just to load the model, you will need ~90GB of CPU memory.
Ah! Ok, thanks, was just that memory. I understand that a cuda GPU its not necessary, isn't it? Will it be very slow without it?
Great!
Yes, technically, running inference on CPU is possible although it will be very slow (an order of magnitude slower than using GPU). There are a few pointers here: https://github.com/bigscience-workshop/t-zero/tree/master/inference
Could it be possible to upload the model weights in multiples bin files (as bloom does) to be able to load with less amount of memory?
Thanks!
Hi
@juanluisdb
,
yes we can! and actually it has been done on the branch "sharded" -> https://huggingface.co/bigscience/T0pp/tree/sharded
You can use the sharded ones by adding a tag in the from_pretrained call: AutoModelForSeq2SeqLM.from_pretrained("bigscience/T0pp", revision="sharded").