ARM quants coming?
Asking for a friend.
Just stopping on by to show my support for this guy's friend getting his ARM quants, because I believe in him. Godspeed.
You can tell your friend that ARM quants are coming, so he can run this on his phone for example. Not today though :)
Actually, now that we're generating the imatrix quants, I remembered we only do arm quants for microscopically tiny models. Drat.
You could enable them anyways just for the memes. Would be such a stupid waste of everyone’s resources. But to be fair there are ARM servers like the "Gigabyte ARM Server R182-P91 Ampere Altra Max" with 32 DIMM slots that support up to 8 TiB of RAM. Enough to run this model in 32-bit.
You are really tempting me there, nico...
arm quants delivered (well, one is still uploading), at only about 1TB per quant. a drop in the ocean.