File size: 3,925 Bytes
1766a34 58a941f 1766a34 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 |
---
base_model: allura-org/TQ2.5-14B-Aletheia-v1
library_name: transformers
tags:
- mergekit
- merge
- llama-cpp
- gguf-my-repo
license: apache-2.0
language:
- en
---
# Triangle104/TQ2.5-14B-Aletheia-v1-Q5_K_S-GGUF
This model was converted to GGUF format from [`allura-org/TQ2.5-14B-Aletheia-v1`](https://huggingface.co/allura-org/TQ2.5-14B-Aletheia-v1) using llama.cpp via the ggml.ai's [GGUF-my-repo](https://huggingface.co/spaces/ggml-org/gguf-my-repo) space.
Refer to the [original model card](https://huggingface.co/allura-org/TQ2.5-14B-Aletheia-v1) for more details on the model.
---
Model details:
-
RP/Story hybrid model, merge of Sugarquill and Neon. As with Gemma version, I wanted to preserve Sugarquill's creative spark, while making the model more steerable for RP. It proved to be more difficult this time, but I quite like the result regardless, even if the model is still somewhat temperamental.
Should work for both RP and storywriting, either on raw completion or with back-and-forth cowriting in chat mode. Seems to be quite sensitive to low depth instructions and samplers.
Thanks to Toasty and Fizz for testing and giving feedback
Model was created by Auri.
Notes about merging
-
It took me 20 something attempts to make this model. TIES didn't work at all, producing broken or nearly broken results every time. SLERP worked much better and after just 3 attempts I got something I like. Sugarquill was really prone to overtaking the merge, so I had to reduce it's part a lot, and still model has a lot of influence from it.
Format
-
Model responds to ChatML instruct formatting, exactly like it's base model.
<|im_start|>system
{system message}<|im_end|>
<|im_start|>user
{user message}<|im_end|>
<|im_start|>assistant
{response}<|im_end|>
Recommended Samplers
-
This one is a bit of a special snowflake, with special tastes. Those seem to work pretty well:
Temperature - 0.8
Top-A - 0.3
TFS - 0.75
DRY - Multiplier 0.8 - Base 1.75 - Allowed length 3 - Range 1024
As a starting point, you can try this ST Master Import
Merge Method
-
This model was merged using the SLERP merge method.
Models Merged
-
The following models were included in the merge:
allura-org/TQ2.5-14B-Neon-v1
allura-org/TQ2.5-14B-Sugarquill-v1
Configuration
-
The following YAML configuration was used to produce this model:
base_model: allura-org/TQ2.5-14B-Sugarquill-v1
dtype: bfloat16
merge_method: slerp
parameters:
t:
- value: 0.7
slices:
- sources:
- layer_range: [0, 48]
model: allura-org/TQ2.5-14B-Neon-v1
- layer_range: [0, 48]
model: allura-org/TQ2.5-14B-Sugarquill-v1
---
## Use with llama.cpp
Install llama.cpp through brew (works on Mac and Linux)
```bash
brew install llama.cpp
```
Invoke the llama.cpp server or the CLI.
### CLI:
```bash
llama-cli --hf-repo Triangle104/TQ2.5-14B-Aletheia-v1-Q5_K_S-GGUF --hf-file tq2.5-14b-aletheia-v1-q5_k_s.gguf -p "The meaning to life and the universe is"
```
### Server:
```bash
llama-server --hf-repo Triangle104/TQ2.5-14B-Aletheia-v1-Q5_K_S-GGUF --hf-file tq2.5-14b-aletheia-v1-q5_k_s.gguf -c 2048
```
Note: You can also use this checkpoint directly through the [usage steps](https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#usage) listed in the Llama.cpp repo as well.
Step 1: Clone llama.cpp from GitHub.
```
git clone https://github.com/ggerganov/llama.cpp
```
Step 2: Move into the llama.cpp folder and build it with `LLAMA_CURL=1` flag along with other hardware-specific flags (for ex: LLAMA_CUDA=1 for Nvidia GPUs on Linux).
```
cd llama.cpp && LLAMA_CURL=1 make
```
Step 3: Run inference through the main binary.
```
./llama-cli --hf-repo Triangle104/TQ2.5-14B-Aletheia-v1-Q5_K_S-GGUF --hf-file tq2.5-14b-aletheia-v1-q5_k_s.gguf -p "The meaning to life and the universe is"
```
or
```
./llama-server --hf-repo Triangle104/TQ2.5-14B-Aletheia-v1-Q5_K_S-GGUF --hf-file tq2.5-14b-aletheia-v1-q5_k_s.gguf -c 2048
```
|