File size: 3,496 Bytes
92d0cf9 dee436e 7401b4b 92d0cf9 2ff5a95 92d0cf9 4af1377 92d0cf9 d6eead6 4af1377 afc9edc 4af1377 afc9edc 4af1377 afc9edc 4af1377 92d0cf9 ccf0bb1 92d0cf9 ccf0bb1 4af1377 ccf0bb1 92d0cf9 ccf0bb1 92d0cf9 ccf0bb1 92d0cf9 ccf0bb1 92d0cf9 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 |
---
library_name: transformers
license: apache-2.0
pipeline_tag: text-generation
tags:
- bitsandbytes
- quantized
- 4bit
- Mistral
- Mistral-7B
- bnb
---
# Model Card for Mistral-7B-Instruct-v0.2-bnb-4bit
<!-- Provide a quick summary of what the model is/does. -->
This repo contains 4-bit quantized (using bitsandbytes) model Mistral AI_'s Mistral-7B-Instruct-v0.2
## Model Details
- Model creator: [Mistral AI_](https://huggingface.co/mistralai)
- Original model: [Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2)
### About 4 bit quantization using bitsandbytes
- QLoRA: Efficient Finetuning of Quantized LLMs: [arXiv - QLoRA: Efficient Finetuning of Quantized LLMs](https://arxiv.org/abs/2305.14314)
- Hugging Face Blog post on 4-bit quantization using bitsandbytes: [Making LLMs even more accessible with bitsandbytes, 4-bit quantization and QLoRA](https://huggingface.co/blog/4bit-transformers-bitsandbytes)
- bitsandbytes github repo: [bitsandbytes github repo](https://github.com/TimDettmers/bitsandbytes)
# How to Get Started with the Model
Use the code below to get started with the model.
## How to run from Python code
#### First install the package
```shell
pip install -q -U bitsandbytes accelerate torch huggingface_hub
pip install -q -U git+https://github.com/huggingface/transformers.git # Install latest version of transformers
pip install -q -U git+https://github.com/huggingface/peft.git
pip install flash-attn --no-build-isolation
```
# Import
```python
import torch
import os
from torch import bfloat16
from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline, BitsAndBytesConfig, LlamaForCausalLM
```
# Use a pipeline as a high-level helper
```python
model_id_mistral = "alokabhishek/Mistral-7B-Instruct-v0.2-bnb-4bit"
tokenizer_mistral = AutoTokenizer.from_pretrained(model_id_mistral, use_fast=True)
model_mistral = AutoModelForCausalLM.from_pretrained(
model_id_mistral,
device_map="auto"
)
pipe_mistral = pipeline(model=model_mistral, tokenizer=tokenizer_mistral, task='text-generation')
prompt_mistral = "Tell me a funny joke about Large Language Models meeting a Blackhole in an intergalactic Bar."
output_mistral = pipe_llama(prompt_mistral, max_new_tokens=512)
print(output_mistral[0]["generated_text"])
```
## Uses
<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
### Direct Use
<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
[More Information Needed]
### Downstream Use [optional]
<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
[More Information Needed]
### Out-of-Scope Use
<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
[More Information Needed]
## Bias, Risks, and Limitations
<!-- This section is meant to convey both technical and sociotechnical limitations. -->
[More Information Needed]
## Evaluation
<!-- This section describes the evaluation protocols and provides the results. -->
#### Metrics
<!-- These are the evaluation metrics being used, ideally with a description of why. -->
[More Information Needed]
### Results
[More Information Needed]
## Model Card Authors [optional]
[More Information Needed]
## Model Card Contact
[More Information Needed] |