|
--- |
|
base_model: google/gemma-2b-it |
|
inference: false |
|
language: |
|
- en |
|
model_creator: google |
|
model_name: gemma-2b-it |
|
model_type: gemma |
|
pipeline_tag: text-generation |
|
license: other |
|
license_name: gemma-terms-of-use |
|
license_link: https://ai.google.dev/gemma/terms |
|
quantized_by: brittlewis12 |
|
--- |
|
|
|
# Gemma 2B Instruct GGUF |
|
|
|
**Original model**: [gemma-2b-it](https://huggingface.co/google/gemma-2b-it) |
|
|
|
**Model creator**: [google](https://huggingface.co/google) |
|
|
|
This repo contains GGUF format model files for Google’s Gemma-2B-it. |
|
|
|
> Gemma is a family of lightweight, state-of-the-art open models from Google, |
|
> built from the same research and technology used to create the Gemini models. |
|
> They are text-to-text, decoder-only large language models, available in English, |
|
> with open weights, pre-trained variants, and instruction-tuned variants. Gemma |
|
> models are well-suited for a variety of text generation tasks, including |
|
> question answering, summarization, and reasoning. Their relatively small size |
|
> makes it possible to deploy them in environments with limited resources such as |
|
> a laptop, desktop or your own cloud infrastructure, democratizing access to |
|
> state of the art AI models and helping foster innovation for everyone. |
|
|
|
Learn more on Google’s [Model page](https://ai.google.dev/gemma/docs). |
|
|
|
### What is GGUF? |
|
|
|
GGUF is a file format for representing AI models. It is the third version of the format, introduced by the llama.cpp team on August 21st 2023. It is a replacement for GGML, which is no longer supported by llama.cpp. |
|
Converted using llama.cpp build 2226 (revision [eccd7a2](https://github.com/ggerganov/llama.cpp/commit/eccd7a26ddbff19e4b8805648f5f14c501957859)) |
|
|
|
--- |
|
|
|
## Download & run with [cnvrs](https://twitter.com/cnvrsai) on iPhone, iPad, and Mac! |
|
|
|
 |
|
|
|
[cnvrs](https://testflight.apple.com/join/sFWReS7K) is the best app for private, local AI on your device: |
|
- create & save **Characters** with custom system prompts & temperature settings |
|
- download and experiment with any **GGUF model** you can [find on HuggingFace](https://huggingface.co/models?library=gguf)! |
|
- make it your own with custom **Theme colors** |
|
- powered by Metal ⚡️ & [Llama.cpp](https://github.com/ggerganov/llama.cpp), with **haptics** during response streaming! |
|
- **try it out** yourself today, on [Testflight](https://testflight.apple.com/join/sFWReS7K)! |
|
- follow [cnvrs on twitter](https://twitter.com/cnvrsai) to stay up to date |
|
|
|
--- |
|
|
|
## Original Model Evaluation |
|
|
|
| Benchmark | Metric | 2B Params | 7B Params | |
|
| ------------------------------ | ------------- | ----------- | --------- | |
|
| [MMLU](https://arxiv.org/abs/2009.03300) | 5-shot, top-1 | 42.3 | 64.3 | |
|
| [HellaSwag](https://arxiv.org/abs/1905.07830) | 0-shot |71.4 | 81.2 | |
|
| [PIQA](https://arxiv.org/abs/1911.11641) | 0-shot | 77.3 | 81.2 | |
|
| [SocialIQA](https://arxiv.org/abs/1904.09728) | 0-shot | 59.7 | 51.8 | |
|
| [BooIQ](https://arxiv.org/abs/1905.10044) | 0-shot | 69.4 | 83.2 | |
|
| [WinoGrande](https://arxiv.org/abs/1907.10641) | partial score | 65.4 | 72.3 | |
|
| [CommonsenseQA](https://arxiv.org/abs/1811.00937) | 7-shot | 65.3 | 71.3 | |
|
| [OpenBookQA](https://arxiv.org/abs/1809.02789) | | 47.8 | 52.8 | |
|
| [ARC-e](https://arxiv.org/abs/1911.01547) | | 73.2 | 81.5 | |
|
| [ARC-c](https://arxiv.org/abs/1911.01547) | | 42.1 | 53.2 | |
|
| [TriviaQA](https://arxiv.org/abs/1705.03551) | 5-shot | 53.2 | 63.4 | |
|
| [Natural Questions](https://github.com/google-research-datasets/natural-questions) | 5-shot | - | 23 | |
|
| [HumanEval](https://arxiv.org/abs/2107.03374) | pass@1 | 22.0 | 32.3 | |
|
| [MBPP](https://arxiv.org/abs/2108.07732) | 3-shot | 29.2 | 44.4 | |
|
| [GSM8K](https://arxiv.org/abs/2110.14168) | maj@1 | 17.7 | 46.4 | |
|
| [MATH](https://arxiv.org/abs/2108.07732) | 4-shot | 11.8 | 24.3 | |
|
| [AGIEval](https://arxiv.org/abs/2304.06364) | | 24.2 | 41.7 | |
|
| [BIG-Bench](https://arxiv.org/abs/2206.04615) | | 35.2 | 55.1 | |
|
| **Average** | | **54.0** | **56.4** | |
|
|
|
|
|
| Benchmark | Metric | 2B Params | 7B Params | |
|
| ------------------------------ | ------------- | ----------- | --------- | |
|
| [RealToxicity](https://arxiv.org/abs/2009.11462) | average | 6.86 | 7.90 | |
|
| [BOLD](https://arxiv.org/abs/2101.11718) | | 45.57 | 49.08 | |
|
| [CrowS-Pairs](https://aclanthology.org/2020.emnlp-main.154/) | top-1 | 45.82 | 51.33 | |
|
| [BBQ Ambig](https://arxiv.org/abs/2110.08193v2) | 1-shot, top-1 | 62.58 | 92.54 | |
|
| [BBQ Disambig](https://arxiv.org/abs/2110.08193v2) | top-1 | 54.62 | 71.99 | |
|
| [Winogender](https://arxiv.org/abs/1804.09301) | top-1 | 51.25 | 54.17 | |
|
| [TruthfulQA](https://arxiv.org/abs/2109.07958) | | 44.84 | 31.81 | |
|
| [Winobias 1_2](https://arxiv.org/abs/1804.06876) | | 56.12 | 59.09 | |
|
| [Winobias 2_2](https://arxiv.org/abs/1804.06876) | | 91.10 | 92.23 | |
|
| [Toxigen](https://arxiv.org/abs/2203.09509) | | 29.77 | 39.59 | |
|
|
