104b - Command-R-Plus
A new model has been released, probably based on a new architecture. There has been some reports that it can do a more human roleplay than Miqu, though requiring a jailbreak.
Well, let's see how llama.cpp fares. It should have support for that architecture...
Yeah, indeed, it seems to have a different architetcure, despite not saying so in the config file. At least conversion fails:
Can not map tensor 'model.layers.0.self_attn.k_norm.weight'
If you learn about support in llama.cpp, I can try again at a later time.
Looks like some people are on the case, they should start work in a couple of days. Probably less than a week, since the changes for Command-R+ are minor, apparently?
Yes, I would expect the changes to be relatively small. Thanks for providing the PR. If it gets merged and I overlook it, feel free to remind me!
Seems static and imatrix quants have been provided elsewhere by now, so I will likely not bother, unless somebody wants specific quants not provided yet. Or somebody weirdly prefers my imatrix quants.
Maybe I was overly optimistic. In any case, I am not quite the right person for bleeding edge :)
Seems it's no longer really bleeding edge, and it's now in the queue. static quants are being generated right now, and imatrix quants will likely follow.