|
--- |
|
license: llama3.1 |
|
--- |
|
|
|
Experimental .GGUF quants for https://huggingface.co/google/gemma-2-9b-it accordingly to LCPP PR |
|
(based on b_3529 and now b_3565 for the newer ones) : https://github.com/ggerganov/llama.cpp/pull/8836 |
|
|
|
These experimental quant strategies revisiting Ikawrakow's work are displaying a slight decrease of perplexity, |
|
including per bpw (from 10%+ for the lowest quants to 0.x% for the highest ones). |
|
This is significant enough to encourage you folks to test them, and provide feedback if pertinent. |
|
|
|
The iMatrix I use is based on Group Merged V3 and enriched with a bit of French, |
|
a bit of Serbian, and a bit of Croatian languages. |
|
|
|
As usual, the name of the quants are a bit pompous, |
|
because they are numbered on the type of tensor quant mainly used as a base for the FFN. |
|
|
|
|
|
ARC and PPL-512 DATA (Get the last data on the main post of the PR thread) : |
|
|
|
``` |
|
IQ1_XS - Unusable on <30B models |
|
PR |
|
1.94 GB (1.93 BPW) |
|
1.81 GiB (1.93 BPW) |
|
PPL over 564 chunks for n_ctx=512 = 40.0024 +/- 0.27710 |
|
|
|
PR2 |
|
1.98 GB (1.97 BPW) |
|
1.84 GiB (1.97 BPW) |
|
PPL over 564 chunks for n_ctx=512 = 33.5198 +/- 0.24187 |
|
|
|
|
|
IQ1_S - Unusable on <30B models |
|
Master |
|
2.01 GB (2.00 BPW) |
|
1.87 GiB (2.00 BPW) |
|
PPL over 564 chunks for n_ctx=512 = 61.2817 +/- 0.41707 |
|
|
|
PR |
|
2.05 GB (2.04 BPW) |
|
1.91 GiB (2.04 BPW) |
|
PPL over 564 chunks for n_ctx=512 = 25.2524 +/- 0.17651 |
|
|
|
PR2 |
|
2.06 GB (2.05 BPW) |
|
1.91 GiB (2.05 BPW) |
|
PPL over 564 chunks for n_ctx=512 = 24.2661 +/- 0.16923 |
|
|
|
|
|
IQ1_M |
|
Master |
|
2.15 GB (2.15 BPW) |
|
2.01 GiB (2.15 BPW) |
|
PPL over 564 chunks for n_ctx=512 = 26.3761 +/- 0.18200 |
|
|
|
PR |
|
2.14 GB (2.13 BPW) |
|
1.99 GiB (2.13 BPW) |
|
PPL over 564 chunks for n_ctx=512 = 20.0588 +/- 0.14001 |
|
|
|
PR2 |
|
2.15 GB (2.14 BPW) |
|
2.00 GiB (2.14 BPW) |
|
PPL over 564 chunks for n_ctx=512 = 18.8721 +/- 0.13233 |
|
|
|
PR3 |
|
2.16 GB (2.15 BPW) |
|
2.01 GiB (2.15 BPW) |
|
PPL over 564 chunks for n_ctx=512 = 18.7469 +/- 0.13140 |
|
|
|
|
|
IQ1_XL |
|
PR |
|
2.21 GB (2.21 BPW) |
|
2.06 GiB (2.21 BPW) |
|
PPL over 564 chunks for n_ctx=512 = 18.5500 +/- 0.12753 |
|
|
|
PR2 |
|
2.23 GB (2.22 BPW) |
|
2.08 GiB (2.22 BPW) |
|
PPL over 564 chunks for n_ctx=512 = 17.4537 +/- 0.11995 |
|
|
|
PR3 |
|
2.25 GB (2.25 BPW) |
|
2.10 GiB (2.25 BPW) |
|
PPL over 564 chunks for n_ctx=512 = 17.3669 +/- 0.11928 |
|
|
|
PR4 |
|
2.28 GB (2.27 BPW) |
|
2.12 GiB (2.27 BPW) |
|
PPL over 564 chunks for n_ctx=512 = 15.5944 +/- 0.10615 |
|
|
|
PR5 |
|
2.28 GB (2.27 BPW) |
|
2.12 GiB (2.27 BPW) |
|
PPL over 564 chunks for n_ctx=512 = 15.4713 +/- 0.10524 |
|
|
|
|
|
IQ2_XXS |
|
Master |
|
2.39 GB (2.38 BPW) |
|
2.23 GiB (2.38 BPW) |
|
PPL over 564 chunks for n_ctx=512 = 15.2572 +/- 0.10267 |
|
|
|
PR |
|
2.38 GB (2.37 BPW) |
|
2.22 GiB (2.37 BPW) |
|
PPL over 564 chunks for n_ctx=512 = 13.8073 +/- 0.09290 |
|
|
|
PR2 |
|
2.40 GB (2.39 BPW) |
|
2.23 GiB (2.39 BPW) |
|
PPL over 564 chunks for n_ctx=512 = 12.9671 +/- 0.08687 |
|
|
|
PR3 |
|
2.42 GB (2.41 BPW) |
|
2.25 GiB (2.41 BPW) |
|
PPL over 564 chunks for n_ctx=512 = 12.5074 +/- 0.08360 |
|
|
|
PR4 |
|
2.42 GB (2.41 BPW) |
|
2.26 GiB (2.41 BPW) |
|
PPL over 564 chunks for n_ctx=512 = 12.4244 +/- 0.08294 |
|
|
|
|
|
IQ2_XS |
|
Master |
|
2.60 GB (2.59 BPW) |
|
2.42 GiB (2.59 BPW) |
|
PPL over 564 chunks for n_ctx=512 = 11.7483 +/- 0.07776 |
|
|
|
PR |
|
2.52 GB (2.51 BPW) |
|
2.35 GiB (2.51 BPW) |
|
PPL over 564 chunks for n_ctx=512 = 11.6639 +/- 0.07805 |
|
|
|
PR2 |
|
2.53 GB (2.52 BPW) |
|
2.36 GiB (2.52 BPW) |
|
PPL over 564 chunks for n_ctx=512 = 11.5685 +/- 0.07742 |
|
|
|
PR3 |
|
2.58 GB (2.57 BPW) |
|
2.40 GiB (2.57 BPW) |
|
PPL over 564 chunks for n_ctx=512 = 11.3031 +/- 0.07514 |
|
|
|
PR4 |
|
2.59 GB (2.58 BPW) |
|
2.42 GiB (2.58 BPW) |
|
PPL over 564 chunks for n_ctx=512 = 10.9291 +/- 0.07270 |
|
|
|
PR5 |
|
2.60 GB (2.59 BPW) |
|
2.42 GiB (2.59 BPW) |
|
PPL over 564 chunks for n_ctx=512 = 10.8794 +/- 0.07229 |
|
|
|
|
|
IQ2_S |
|
Master |
|
2.75 GB (2.74 BPW) |
|
2.56 GiB (2.74 BPW) |
|
PPL over 564 chunks for n_ctx=512 = 10.5180 +/- 0.06976 |
|
|
|
PR (fail) |
|
2.71 GB (2.70 BPW) |
|
2.52 GiB (2.70 BPW) |
|
PPL over 564 chunks for n_ctx=512 = 10.7010 +/- 0.07027 |
|
|
|
PR2 |
|
2.75 GB (2.74 BPW) |
|
2.56 GiB (2.74 BPW) |
|
PPL over 564 chunks for n_ctx=512 = 10.3728 +/- 0.06806 |
|
|
|
|
|
IQ2_M |
|
Master |
|
2.94 GB (2.93 BPW) |
|
2.74 GiB (2.93 BPW) |
|
PPL over 564 chunks for n_ctx=512 = 9.5935 +/- 0.06228 |
|
|
|
PR |
|
2.93 GB (2.92 BPW) |
|
2.73 GiB (2.92 BPW) |
|
PPL over 564 chunks for n_ctx=512 = 9.4125 +/- 0.06039 |
|
|
|
|
|
IQ2_XL |
|
PR |
|
2.99 GB (2.98 BPW) |
|
2.78 GiB (2.98 BPW) |
|
PPL over 564 chunks for n_ctx=512 = 9.3122 +/- 0.05973 |
|
|
|
PR2 |
|
3.11 GB (3.10 BPW) |
|
2.90 GiB (3.10 BPW) |
|
PPL over 564 chunks for n_ctx=512 = 9.0378 +/- 0.05764 |
|
|
|
PR3 |
|
3.14 GB (3.13 BPW) |
|
2.93 GiB (3.13 BPW) |
|
PPL over 564 chunks for n_ctx=512 = 8.8604 +/- 0.05620 |
|
|
|
|
|
IQ3_XXS |
|
|
|
Master |
|
Size : 3.04 GiB (3.25 BPW) |
|
PPL 512 wikitext : 8.4985 +/- 0.05402 |
|
|
|
PR (good) |
|
Size : 3.11 GiB (3.32 BPW) |
|
PPL 512 wikitext : 8.3274 +/- 0.05334 |
|
|
|
PR2 (so so) |
|
llm_load_print_meta: model size = 3.08 GiB (3.29 BPW) |
|
llm_load_print_meta: model size = 3.30 GB (3.29 BPW) |
|
Final estimate: PPL 512 = 8.3906 +/- 0.05329 |
|
|
|
Let's keep the first PR |
|
|
|
|
|
IQ3_XS |
|
|
|
Master |
|
Size : 3.27 GiB (3.50 BPW) |
|
PPL 512 wikitext : 8.2019 +/- 0.05167 |
|
|
|
PR (ok) |
|
Size : 3.24 GiB (3.47 BPW) |
|
PPL 512 wikitext : 8.1762 +/- 0.05176 |
|
|
|
|
|
IQ3_S |
|
|
|
Master |
|
Size : 3.42 GiB (3.66 BPW) |
|
PPL 512 wikitext : 7.9894 +/- 0.05020 |
|
|
|
PR (good) |
|
Size : 3.41 GiB (3.64 BPW) |
|
PPL 512 wikitext : 7.9067 +/- 0.05022 |
|
|
|
|
|
IQ3_M |
|
|
|
Master |
|
Size : 3.52 GiB (3.76 BPW) |
|
PPL 512 wikitext : 7.9263 +/- 0.04943 |
|
|
|
PR (good) |
|
Size : 3.49 GiB (3.73 BPW) |
|
PPL 512 wikitext : 7.8704 +/- 0.04951 |
|
|
|
|
|
IQ3_XL |
|
|
|
PR (good) |
|
Size : 3.71 GiB (3.97 BPW) |
|
PPL 512 wikitext : 7.7225 +/- 0.04946 |
|
|
|
|
|
IQ3_XXL |
|
|
|
PR (good, the benefit seems meager but the token embeddings pushed form IQ3_S to IQ4_XS explains +0.05BPW of it, |
|
and this tensor doesn't run in VRAM but in RAM) |
|
Size : 3.83 GiB (4.09 BPW) |
|
PPL 512 wikitext : 7.6720 +/- 0.04892 |
|
|
|
|
|
IQ3_XXXL |
|
|
|
PR (good) |
|
Size : 3.97 GiB (4.24 BPW) |
|
PPL 512 wikitext : 7.5920 +/- 0.04839 |
|
|
|
|
|
IQ4_XS |
|
|
|
Master |
|
Size : 4.13 GiB (4.42 BPW) |
|
Arc-C 299 49.16387960 |
|
Arc-E 570 72.10526316 |
|
PPL 512 wikitext : 7.5226 +/- 0.04820 |
|
|
|
|
|
IQ4_XSR |
|
|
|
PR (good) |
|
Size : 4.16 GiB (4.45 BPW) |
|
Arc-C 299 |
|
Arc-E 570 |
|
PPL 512 wikitext : 7.5072 +/- 0.04814 |
|
|
|
|
|
FP16 |
|
|
|
MASTER : Gemma 2 9b It F16. |
|
Size : 14.96 GiB (16.00 BPW) |
|
Arc-C 299 49.49832776 |
|
Arc-E 570 73.85964912 |
|
PPL 512 wikitext : 7.3224 +/- 0.04674 |
|
|
|
``` |
