5.0 bpw exl2 quant request
I notice in your description you tested the model using 5.0 bpw exl2 quant request. Can you possbly share it? Thanks! Also, I just want to say i am a big fan of your models.
I'll be happy to publish some EXL2 quants if someone else doesn't beat me to it before my upload of the 103B fp16 weights finally finishes in a few days. This is why I rely on other people for quants haha. I looked into upgrading to symmetrical Internet but it isn't available in my area yet, so... yeah. I'm uploading at 1.4MB/s over here.
5.0 quant uploaded to https://huggingface.co/Dracones/Midnight-Miqu-70B-v1.0_exl2_5.0bpw
Will upload a 4.65 quant when it finishes.
@Dracones Thank you! I added the link to the model card.
New EXL2 quants are up:
4.65bpw: https://huggingface.co/Dracones/Midnight-Miqu-70B-v1.0_exl2_4.65bpw
4.0bpw: https://huggingface.co/Dracones/Midnight-Miqu-70B-v1.0_exl2_4.0bpw
3.0bpw: https://huggingface.co/Dracones/Midnight-Miqu-70B-v1.0_exl2_3.0bpw
2.24bpw: https://huggingface.co/Dracones/Midnight-Miqu-70B-v1.0_exl2_2.24bpw
This is an impressive model. I usually use Miqu with a bot creator card for creating complex bots off a simple prompt following the very specific format of the creation card and Midnight handled it perfectly. I attached Midnight's created card, Seraphina, in the above model pages.