Suparious commited on
Commit
594f30d
1 Parent(s): c15fd1c

Add default model card

Browse files
Files changed (1) hide show
  1. README.md +79 -0
README.md ADDED
@@ -0,0 +1,79 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: transformers
3
+ tags:
4
+ - 4-bit
5
+ - AWQ
6
+ - text-generation
7
+ - autotrain_compatible
8
+ - endpoints_compatible
9
+ pipeline_tag: text-generation
10
+ inference: false
11
+ quantized_by: Suparious
12
+ ---
13
+ # R136a1/BeyondInfinity-v2-4x7B AWQ
14
+
15
+ - Model creator: [R136a1](https://huggingface.co/R136a1)
16
+ - Original model: [BeyondInfinity-v2-4x7B](https://huggingface.co/R136a1/BeyondInfinity-v2-4x7B)
17
+
18
+
19
+
20
+ ## How to use
21
+
22
+ ### Install the necessary packages
23
+
24
+ ```bash
25
+ pip install --upgrade autoawq autoawq-kernels
26
+ ```
27
+
28
+ ### Example Python code
29
+
30
+ ```python
31
+ from awq import AutoAWQForCausalLM
32
+ from transformers import AutoTokenizer, TextStreamer
33
+
34
+ model_path = "solidrust/BeyondInfinity-v2-4x7B-AWQ"
35
+ system_message = "You are BeyondInfinity-v2-4x7B, incarnated as a powerful AI. You were created by R136a1."
36
+
37
+ # Load model
38
+ model = AutoAWQForCausalLM.from_quantized(model_path,
39
+ fuse_layers=True)
40
+ tokenizer = AutoTokenizer.from_pretrained(model_path,
41
+ trust_remote_code=True)
42
+ streamer = TextStreamer(tokenizer,
43
+ skip_prompt=True,
44
+ skip_special_tokens=True)
45
+
46
+ # Convert prompt to tokens
47
+ prompt_template = """\
48
+ <|im_start|>system
49
+ {system_message}<|im_end|>
50
+ <|im_start|>user
51
+ {prompt}<|im_end|>
52
+ <|im_start|>assistant"""
53
+
54
+ prompt = "You're standing on the surface of the Earth. "\
55
+ "You walk one mile south, one mile west and one mile north. "\
56
+ "You end up exactly where you started. Where are you?"
57
+
58
+ tokens = tokenizer(prompt_template.format(system_message=system_message,prompt=prompt),
59
+ return_tensors='pt').input_ids.cuda()
60
+
61
+ # Generate output
62
+ generation_output = model.generate(tokens,
63
+ streamer=streamer,
64
+ max_new_tokens=512)
65
+ ```
66
+
67
+ ### About AWQ
68
+
69
+ AWQ is an efficient, accurate and blazing-fast low-bit weight quantization method, currently supporting 4-bit quantization. Compared to GPTQ, it offers faster Transformers-based inference with equivalent or better quality compared to the most commonly used GPTQ settings.
70
+
71
+ AWQ models are currently supported on Linux and Windows, with NVidia GPUs only. macOS users: please use GGUF models instead.
72
+
73
+ It is supported by:
74
+
75
+ - [Text Generation Webui](https://github.com/oobabooga/text-generation-webui) - using Loader: AutoAWQ
76
+ - [vLLM](https://github.com/vllm-project/vllm) - version 0.2.2 or later for support for all model types.
77
+ - [Hugging Face Text Generation Inference (TGI)](https://github.com/huggingface/text-generation-inference)
78
+ - [Transformers](https://huggingface.co/docs/transformers) version 4.35.0 and later, from any code or client that supports Transformers
79
+ - [AutoAWQ](https://github.com/casper-hansen/AutoAWQ) - for use from Python code