|
--- |
|
library_name: transformers |
|
tags: |
|
- not-for-all-audiences |
|
- llama-cpp |
|
- gguf-my-repo |
|
datasets: |
|
- crestf411/LimaRP-DS |
|
- Gryphe/Sonnet3.5-Charcard-Roleplay |
|
- anthracite-org/c2_logs_32k_mistral-v3_v1.2_no_system |
|
- anthracite-org/kalo-opus-instruct-22k-no-refusal-no-system |
|
- anthracite-org/kalo-opus-instruct-3k-filtered-no-system |
|
- anthracite-org/nopm_claude_writing_fixed |
|
base_model: crestf411/MS-sunfall-v0.7.0 |
|
--- |
|
|
|
# Triangle104/MS-sunfall-v0.7.0-Q5_K_S-GGUF |
|
This model was converted to GGUF format from [`crestf411/MS-sunfall-v0.7.0`](https://huggingface.co/crestf411/MS-sunfall-v0.7.0) using llama.cpp via the ggml.ai's [GGUF-my-repo](https://huggingface.co/spaces/ggml-org/gguf-my-repo) space. |
|
Refer to the [original model card](https://huggingface.co/crestf411/MS-sunfall-v0.7.0) for more details on the model. |
|
|
|
--- |
|
Model details: |
|
- |
|
Sunfall (2024-10-28) v0.7.0 on top of Mistral Small Instruct 2409. |
|
|
|
It also contains samples from Antracite.Org datasets. See bottom for details. |
|
|
|
Significant revamping of the dataset metadata generation process, resulting in higher quality dataset overall. The "Diamond Law" experiment has been removed as it didn't seem to affect the model output enough to warrant set up complexity. |
|
|
|
Recommended starting point: |
|
- |
|
Temperature: 1 |
|
MinP: 0.05~0.1 |
|
DRY: 0.8 1.75 2 0 |
|
|
|
At early context, I recommend keeping XTC disabled. Once you hit higher context sizes (10k+), enabling XTC at 0.1 / 0.5 seems to significantly improve the output, but YMMV. If the output drones on and is uninspiring, XTC can be extremely effective. |
|
|
|
General heuristic: |
|
- |
|
Lots of slop? Temperature is too low. Raise it, or enable XTC. For early context, temp bump is probably preferred. |
|
Is the model making mistakes about subtle or obvious details in the scene? Temperature is too high, OR XTC is enabled and/or XTC settings are too high. Lower temp and/or disable XTC. |
|
|
|
Mergers/fine-tuners: there is a LoRA of this model. Consider merging that instead of merging this model. |
|
|
|
This model has been trained on context that mimics that of Silly Tavern's "Mistral V2 & V3" preset, with character names added. |
|
|
|
Silly Tavern output example (Henry is the human, Beth the bot): |
|
|
|
[INST] Henry: I poke Beth.[/INST] Beth: Beth yelps. |
|
|
|
The model has also been trained to do interactive storywriting. You may steer the model towards specific content by "responding" to the model like so: |
|
|
|
Continue writing adhering to the following scenario: (things you want to happen next) |
|
|
|
Additional inclusions (random sampled sub-set, cursorily quality-checked) from: |
|
- |
|
Gryphe/Sonnet3.5-Charcard-Roleplay |
|
anthracite-org/c2_logs_32k_mistral-v3_v1.2_no_system |
|
anthracite-org/kalo-opus-instruct-22k-no-refusal-no-system |
|
anthracite-org/kalo-opus-instruct-3k-filtered-no-system |
|
anthracite-org/nopm_claude_writing_fixed |
|
|
|
As such, the dataset is not 100% slop free, but this addition likely helps the model be a better roleplayer. At some point, I intend to clean up and release the samples, deslopped. |
|
|
|
--- |
|
## Use with llama.cpp |
|
Install llama.cpp through brew (works on Mac and Linux) |
|
|
|
```bash |
|
brew install llama.cpp |
|
|
|
``` |
|
Invoke the llama.cpp server or the CLI. |
|
|
|
### CLI: |
|
```bash |
|
llama-cli --hf-repo Triangle104/MS-sunfall-v0.7.0-Q5_K_S-GGUF --hf-file ms-sunfall-v0.7.0-q5_k_s.gguf -p "The meaning to life and the universe is" |
|
``` |
|
|
|
### Server: |
|
```bash |
|
llama-server --hf-repo Triangle104/MS-sunfall-v0.7.0-Q5_K_S-GGUF --hf-file ms-sunfall-v0.7.0-q5_k_s.gguf -c 2048 |
|
``` |
|
|
|
Note: You can also use this checkpoint directly through the [usage steps](https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#usage) listed in the Llama.cpp repo as well. |
|
|
|
Step 1: Clone llama.cpp from GitHub. |
|
``` |
|
git clone https://github.com/ggerganov/llama.cpp |
|
``` |
|
|
|
Step 2: Move into the llama.cpp folder and build it with `LLAMA_CURL=1` flag along with other hardware-specific flags (for ex: LLAMA_CUDA=1 for Nvidia GPUs on Linux). |
|
``` |
|
cd llama.cpp && LLAMA_CURL=1 make |
|
``` |
|
|
|
Step 3: Run inference through the main binary. |
|
``` |
|
./llama-cli --hf-repo Triangle104/MS-sunfall-v0.7.0-Q5_K_S-GGUF --hf-file ms-sunfall-v0.7.0-q5_k_s.gguf -p "The meaning to life and the universe is" |
|
``` |
|
or |
|
``` |
|
./llama-server --hf-repo Triangle104/MS-sunfall-v0.7.0-Q5_K_S-GGUF --hf-file ms-sunfall-v0.7.0-q5_k_s.gguf -c 2048 |
|
``` |
|
|
