Update README.md
Browse files
README.md
CHANGED
@@ -115,8 +115,40 @@ Refer to the Provided Files table below to see what files use which methods, and
|
|
115 |
| [synthia-70b-v1.1.ggmlv3.Q5_0.bin](https://huggingface.co/TheBloke/Synthia-70B-v1.1-GGML/blob/main/synthia-70b-v1.1.ggmlv3.Q5_0.bin) | Q5_0 | 5 | 47.46 GB| 49.96 GB | Original quant method, 5-bit. Higher accuracy, higher resource usage and slower inference. |
|
116 |
| [synthia-70b-v1.1.ggmlv3.Q5_K_S.bin](https://huggingface.co/TheBloke/Synthia-70B-v1.1-GGML/blob/main/synthia-70b-v1.1.ggmlv3.Q5_K_S.bin) | Q5_K_S | 5 | 47.46 GB| 49.96 GB | New k-quant method. Uses GGML_TYPE_Q5_K for all tensors |
|
117 |
| [synthia-70b-v1.1.ggmlv3.Q5_K_M.bin](https://huggingface.co/TheBloke/Synthia-70B-v1.1-GGML/blob/main/synthia-70b-v1.1.ggmlv3.Q5_K_M.bin) | Q5_K_M | 5 | 48.75 GB| 51.25 GB | New k-quant method. Uses GGML_TYPE_Q6_K for half of the attention.wv and feed_forward.w2 tensors, else GGML_TYPE_Q5_K |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
118 |
|
119 |
-
**Note**: the above RAM figures assume no GPU offloading. If layers are offloaded to the GPU, this will reduce RAM usage and use VRAM instead.
|
120 |
|
121 |
## How to run in `llama.cpp`
|
122 |
|
|
|
115 |
| [synthia-70b-v1.1.ggmlv3.Q5_0.bin](https://huggingface.co/TheBloke/Synthia-70B-v1.1-GGML/blob/main/synthia-70b-v1.1.ggmlv3.Q5_0.bin) | Q5_0 | 5 | 47.46 GB| 49.96 GB | Original quant method, 5-bit. Higher accuracy, higher resource usage and slower inference. |
|
116 |
| [synthia-70b-v1.1.ggmlv3.Q5_K_S.bin](https://huggingface.co/TheBloke/Synthia-70B-v1.1-GGML/blob/main/synthia-70b-v1.1.ggmlv3.Q5_K_S.bin) | Q5_K_S | 5 | 47.46 GB| 49.96 GB | New k-quant method. Uses GGML_TYPE_Q5_K for all tensors |
|
117 |
| [synthia-70b-v1.1.ggmlv3.Q5_K_M.bin](https://huggingface.co/TheBloke/Synthia-70B-v1.1-GGML/blob/main/synthia-70b-v1.1.ggmlv3.Q5_K_M.bin) | Q5_K_M | 5 | 48.75 GB| 51.25 GB | New k-quant method. Uses GGML_TYPE_Q6_K for half of the attention.wv and feed_forward.w2 tensors, else GGML_TYPE_Q5_K |
|
118 |
+
| synthia-70b-v1.1.ggmlv3.q5_1.bin | q5_1 | 5 | 51.76 GB | 54.26 GB | Original quant method, 5-bit. Higher accuracy, slower inference than q5_0. |
|
119 |
+
| synthia-70b-v1.1.ggmlv3.q6_K.bin | q6_K | 6 | 56.59 GB | 59.09 GB | New k-quant method. Uses GGML_TYPE_Q8_K - 6-bit quantization - for all tensors |
|
120 |
+
| synthia-70b-v1.1.ggmlv3.q8_0.bin | q8_0 | 8 | 73.23 GB | 75.73 GB | Original llama.cpp quant method, 8-bit. Almost indistinguishable from float16. High resource use and slow. Not recommended for most users. |
|
121 |
+
|
122 |
+
### q5_1, q6_K and q8_0 files require expansion from archive
|
123 |
+
|
124 |
+
**Note:** HF does not support uploading files larger than 50GB. Therefore I have uploaded the q6_K and q8_0 files as multi-part ZIP files. They are not compressed, they are just for storing a .bin file in two parts.
|
125 |
+
|
126 |
+
<details>
|
127 |
+
<summary>Click for instructions regarding q5_1, q6_K and q8_0 files</summary>
|
128 |
+
|
129 |
+
### q5_1
|
130 |
+
Please download:
|
131 |
+
* `synthia-70b-v1.1.ggmlv3.q5_1.zip`
|
132 |
+
* `synthia-70b-v1.1.ggmlv3.q5_1.z01`
|
133 |
+
|
134 |
+
### q6_K
|
135 |
+
Please download:
|
136 |
+
* `synthia-70b-v1.1.ggmlv3.q6_K.zip`
|
137 |
+
* `synthia-70b-v1.1.ggmlv3.q6_K.z01`
|
138 |
+
|
139 |
+
### q8_0
|
140 |
+
Please download:
|
141 |
+
* `synthia-70b-v1.1.ggmlv3.q8_0.zip`
|
142 |
+
* `synthia-70b-v1.1.ggmlv3.q8_0.z01`
|
143 |
+
|
144 |
+
Then extract the .zip archive. This will will expand both parts automatically. On Linux I found I had to use `7zip` - the basic `unzip` tool did not work. Example:
|
145 |
+
```
|
146 |
+
sudo apt update -y && sudo apt install 7zip
|
147 |
+
7zz x synthia-70b-v1.1.ggmlv3.q6_K.zip
|
148 |
+
```
|
149 |
+
|
150 |
+
</details>
|
151 |
|
|
|
152 |
|
153 |
## How to run in `llama.cpp`
|
154 |
|