Lewdiculous

/

Model-Requests

like 37

[Required] Model name:
WestLake-10.7B-v2

[Required] Model link
https://huggingface.co/froggeric/WestLake-10.7B-v2

[Required] Brief description:
Based on Mistral-7B and trained on a new dataset, testing very well on benchmarks for RP/ERP/creative writing and through experience, plays quite well in chat/instruct. This is a self-merge from the original 7B but saw significant improvements in quality.

[Required] An image/direct image link to represent the model (square shaped):
From the 7B version:

[Optional] Additional quants (if you want any):

Quants are pretty limited for this model from its creator, but I am interested in testing PP performance on the Tesla P40, especially with large context sizes. The doesn't seem to respond well to anything larger than 8k, but I am hoping RoPE scaling can improve this, even though it otherwise plays fantastic.

Beyond improving the model's short-term memory, K quants may be slower than traditional quants which may be made worse by the limited compute power of the P40 compared to current generation GPUs (a P40 is comparable to a 1080, just with 24GB VRAM) . I'd like to test this theory, along with benchmarking the difference in using i-quants and imat real effect on PP or TG on these cards (Aforementioned limited compute may have a more pronounced effect).

Finally, this model could potentially fit in very low context sizes (4k or less), but could use every advantage it can get to produce quality output in 6GB (Q3_K_S would just barely) or even 4GB of RAM. While no guarantee this is attainable, it seems like a valuable target if we could succeed.

Default list of quants for reference:

"Q4_K_M", "Q4_K_S", "IQ4_XS", "Q5_K_M", "Q5_K_S",
"Q6_K", "Q8_0", "IQ3_M", "IQ3_S", "IQ3_XXS"

Existing GGUF only has Q4_K_S, Q5_K_S, Q6_K, and Q8_0. Any additional ones beyond those requested for my benchmarks would likely be useful to the community. Comments also mentioned this imat may have been used in preparing the 7B model: https://huggingface.co/Joseph717171/Imatrices/blob/main/WestLake-7B-v2_wikitext-2-raw.imatrix .

Future ambitious revisions to the model could potentially include LASER or DPO improvements, but those are outside of my scope for the moment. I'd like to encourage its development as showing just how good small models can be at specific tasks and maybe help pave the way for more such specialized models that can be swapped by the user as needed.

Thanks for your help, you're doing the Lewd's work. :)