Thank's for the quants, happy with your exl2 quants. This model is fit on 3x3060 at 32k contexts, gpu-split 9,10,12 with cache_8bit. This model much more comply than Smaug 34b 5.0bpw or Mixtral 8x7B 4.0bpw. Will try 3.5bpw, since 4.0bpw is OOM.
· Sign up or log in to comment
