|
--- |
|
language: |
|
- en |
|
library_name: transformers |
|
license: apache-2.0 |
|
tags: |
|
- gpt |
|
- llm |
|
- large language model |
|
- h2o-llmstudio |
|
thumbnail: >- |
|
https://h2o.ai/etc.clientlibs/h2o/clientlibs/clientlib-site/resources/images/favicon.ico |
|
pipeline_tag: text-generation |
|
quantized_by: h2oai |
|
--- |
|
|
|
# h2o-danube2-1.8b-chat-GGUF |
|
- Model creator: [H2O.ai](https://huggingface.co/h2oai) |
|
- Original model: [h2o-danube2-1.8b-chat](https://huggingface.co/h2oai/h2o-danube2-1.8b-chat) |
|
|
|
## Description |
|
|
|
This repo contains GGUF format model files for [h2o-danube2-1.8b-chat](https://huggingface.co/h2oai/h2o-danube2-1.8b-chat) quantized using [llama.cpp](https://github.com/ggerganov/llama.cpp/) framework. |
|
|
|
Table below summarizes different quantized versions of [h2o-danube2-1.8b-chat](https://huggingface.co/h2oai/h2o-danube2-1.8b-chat). It shows the trade-off between size, speed and quality of the models. |
|
|
|
|
|
| Name | Quant method | Model size | MT-Bench AVG | Perplexity | Tokens per second | |
|
|:----------------------------------|:----------------------------------:|:----------:|:------------:|:------------:|:-------------------:| |
|
| [h2o-danube2-1.8b-chat-F16.gguf](https://huggingface.co/h2oai/h2o-danube2-1.8b-chat-GGUF/blob/main/h2o-danube2-1.8b-chat-F16.gguf) | F16 | 3.66 GB | 5.60 | 8.02 | 797 | |
|
| [h2o-danube2-1.8b-chat-Q8_0.gguf](https://huggingface.co/h2oai/h2o-danube2-1.8b-chat-GGUF/blob/main/h2o-danube2-1.8b-chat-Q8_0.gguf) | Q8_0 | 1.95 GB | 5.51 | 8.02 | 1156 | |
|
| [h2o-danube2-1.8b-chat-Q6_K.gguf](https://huggingface.co/h2oai/h2o-danube2-1.8b-chat-GGUF/blob/main/h2o-danube2-1.8b-chat-Q6_K.gguf) | Q6_K | 1.50 GB | 5.51 | 8.03 | 1131 | |
|
| [h2o-danube2-1.8b-chat-Q5_K_M.gguf](https://huggingface.co/h2oai/h2o-danube2-1.8b-chat-GGUF/blob/main/h2o-danube2-1.8b-chat-Q5_K_M.gguf) | Q5_K_M | 1.30 GB | 5.56 | 8.10 | 1172 | |
|
| [h2o-danube2-1.8b-chat-Q5_K_S.gguf](https://huggingface.co/h2oai/h2o-danube2-1.8b-chat-GGUF/blob/main/h2o-danube2-1.8b-chat-Q5_K_S.gguf) | Q5_K_S | 1.27 GB | 5.49 | 8.12 | 1107 | |
|
| [h2o-danube2-1.8b-chat-Q4_K_M.gguf](https://huggingface.co/h2oai/h2o-danube2-1.8b-chat-GGUF/blob/main/h2o-danube2-1.8b-chat-Q4_K_M.gguf) | Q4_K_M | 1.11 GB | 5.60 | 8.27 | 1162 | |
|
| [h2o-danube2-1.8b-chat-Q4_K_S.gguf](https://huggingface.co/h2oai/h2o-danube2-1.8b-chat-GGUF/blob/main/h2o-danube2-1.8b-chat-Q4_K_S.gguf) | Q4_K_S | 1.06 GB | 5.59 | 8.34 | 1270 | |
|
| [h2o-danube2-1.8b-chat-Q3_K_L.gguf](https://huggingface.co/h2oai/h2o-danube2-1.8b-chat-GGUF/blob/main/h2o-danube2-1.8b-chat-Q3_K_L.gguf) | Q3_K_L | 0.98 GB | 5.23 | 8.72 | 1442 | |
|
| [h2o-danube2-1.8b-chat-Q3_K_M.gguf](https://huggingface.co/h2oai/h2o-danube2-1.8b-chat-GGUF/blob/main/h2o-danube2-1.8b-chat-Q3_K_M.gguf) | Q3_K_M | 0.91 GB | 4.91 | 8.81 | 1107 | |
|
| [h2o-danube2-1.8b-chat-Q3_K_S.gguf](https://huggingface.co/h2oai/h2o-danube2-1.8b-chat-GGUF/blob/main/h2o-danube2-1.8b-chat-Q3_K_S.gguf) | Q3_K_S | 0.82 GB | 4.03 | 10.12 | 1103 | |
|
| [h2o-danube2-1.8b-chat-Q2_K.gguf](https://huggingface.co/h2oai/h2o-danube2-1.8b-chat-GGUF/blob/main/h2o-danube2-1.8b-chat-Q2_K.gguf) | Q2_K | 0.71 GB | 3.03 | 12.56 | 1160 | |
|
|
|
Columns in the table are: |
|
* Name -- model name and link |
|
* Quant method -- quantization method |
|
* Model size -- size of the model in gigabytes |
|
* MT-Bench AVG -- [MT-Bench](https://arxiv.org/abs/2306.05685) benchmark score. The score is from 1 to 10, the higher, the better |
|
* Perplexity -- perplexity metric on WikiText-2 dataset. It's reported in a perplexity test from llama.cpp. The lower, the better |
|
* Tokens per second -- generation speed in tokens per second, as reported in a perplexity test from llama.cpp. The higher, the better. Speed tests are done on a single H100 GPU |
|
|
|
|
|
## Prompt template |
|
``` |
|
<|prompt|>Why is drinking water so healthy?</s><|answer|> |
|
``` |
|
|
