Adding More Quant
#2
by
PantonePeri
- opened
Hi! Thank you for the awesome work you've done! I was wondering if it's possible to add 2.5bpw quant?
Also, may I ask how much RAM and VRAM would be required for a 123B exl2 quantization like this?
Like to create the quantization? I was able to do it on just my 3090
I can probably throw up a 2.5 bpw..
Yeah, I also have a 3090 that I want to try quantizing on Windows, but I feel like my 64GB RAM might not be enough, especially for such a large model. Are you quantizing on Linux, and have you set up swap?