|
--- |
|
license: cc-by-4.0 |
|
language: |
|
- en |
|
base_model: FallenMerick/MN-Violet-Lotus-12B |
|
library_name: transformers |
|
tags: |
|
- storywriting |
|
- text adventure |
|
- creative |
|
- story |
|
- writing |
|
- fiction |
|
- roleplaying |
|
- rp |
|
- mergekit |
|
- merge |
|
- llama-cpp |
|
- gguf-my-repo |
|
--- |
|
|
|
# Triangle104/MN-Violet-Lotus-12B-Q5_K_M-GGUF |
|
This model was converted to GGUF format from [`FallenMerick/MN-Violet-Lotus-12B`](https://huggingface.co/FallenMerick/MN-Violet-Lotus-12B) using llama.cpp via the ggml.ai's [GGUF-my-repo](https://huggingface.co/spaces/ggml-org/gguf-my-repo) space. |
|
Refer to the [original model card](https://huggingface.co/FallenMerick/MN-Violet-Lotus-12B) for more details on the model. |
|
|
|
--- |
|
Model details: |
|
- |
|
This is the model I was trying to create when Chunky-Lotus emerged. Not only does this model score higher on my local EQ benchmarks (80.00 w/ 100% parsed @ 8-bit), but it does an even better job at roleplaying and producing creative outputs while still adhering to wide ranges of character personalities. The high levels of emotional intelligence are really quite noticeable as well. |
|
|
|
Once again, models tend to score higher on my local tests when compared to their posted scores, but this has become the new high score for models I've personally tested. |
|
|
|
I really like the way this model writes, and I hope you'll enjoy using it as well! |
|
|
|
Merge Details |
|
- |
|
This is a merge of pre-trained language models created using mergekit. |
|
|
|
Merge Method |
|
- |
|
This model was merged using the Model Stock merge method. |
|
Models Merged |
|
|
|
The following models were included in the merge: |
|
|
|
Epiculous/Violet_Twilight-v0.2 |
|
NeverSleep/Lumimaid-v0.2-12B |
|
flammenai/Mahou-1.5-mistral-nemo-12B |
|
Sao10K/MN-12B-Lyra-v4 |
|
|
|
Configuration |
|
- |
|
The following YAML configuration was used to produce this model: |
|
|
|
models: |
|
- model: FallenMerick/MN-Twilight-Maid-SLERP-12B #(unreleased) |
|
- model: Sao10K/MN-12B-Lyra-v4 |
|
- model: flammenai/Mahou-1.5-mistral-nemo-12B |
|
merge_method: model_stock |
|
base_model: mistralai/Mistral-Nemo-Instruct-2407 |
|
parameters: |
|
normalize: true |
|
dtype: bfloat16 |
|
|
|
In this recipe, Violet Twilight and Lumimaid were first blended using the SLERP method to create a strong roleplaying foundation. Lyra v4 is then added to the mix for its great creativity and roleplaying performance, along with Mahou to once again curtail the outputs and prevent the resulting model from becoming too wordy. Model Stock was used for the final merge in order to really push the resulting weights in the proper direction while using Nemo Instruct as a strong anchor point. |
|
|
|
--- |
|
## Use with llama.cpp |
|
Install llama.cpp through brew (works on Mac and Linux) |
|
|
|
```bash |
|
brew install llama.cpp |
|
|
|
``` |
|
Invoke the llama.cpp server or the CLI. |
|
|
|
### CLI: |
|
```bash |
|
llama-cli --hf-repo Triangle104/MN-Violet-Lotus-12B-Q5_K_M-GGUF --hf-file mn-violet-lotus-12b-q5_k_m.gguf -p "The meaning to life and the universe is" |
|
``` |
|
|
|
### Server: |
|
```bash |
|
llama-server --hf-repo Triangle104/MN-Violet-Lotus-12B-Q5_K_M-GGUF --hf-file mn-violet-lotus-12b-q5_k_m.gguf -c 2048 |
|
``` |
|
|
|
Note: You can also use this checkpoint directly through the [usage steps](https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#usage) listed in the Llama.cpp repo as well. |
|
|
|
Step 1: Clone llama.cpp from GitHub. |
|
``` |
|
git clone https://github.com/ggerganov/llama.cpp |
|
``` |
|
|
|
Step 2: Move into the llama.cpp folder and build it with `LLAMA_CURL=1` flag along with other hardware-specific flags (for ex: LLAMA_CUDA=1 for Nvidia GPUs on Linux). |
|
``` |
|
cd llama.cpp && LLAMA_CURL=1 make |
|
``` |
|
|
|
Step 3: Run inference through the main binary. |
|
``` |
|
./llama-cli --hf-repo Triangle104/MN-Violet-Lotus-12B-Q5_K_M-GGUF --hf-file mn-violet-lotus-12b-q5_k_m.gguf -p "The meaning to life and the universe is" |
|
``` |
|
or |
|
``` |
|
./llama-server --hf-repo Triangle104/MN-Violet-Lotus-12B-Q5_K_M-GGUF --hf-file mn-violet-lotus-12b-q5_k_m.gguf -c 2048 |
|
``` |
|
|
