Hugging Face's logo

audreyt
/
Breeze-7B-Instruct-64k-v0.1-GGUF

Text Generation
Transformers
GGUF
Chinese
Model card Files Community
1
audreyt commited on
Commit
62b85f6
•
1 Parent(s): 3ab38e9

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +242 -0
README.md CHANGED
@@ -1,3 +1,245 @@
1
  ---
2
  license: apache-2.0
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: apache-2.0
3
+ language:
4
+ - zh
5
+ library_name: transformers
6
+ pipeline_tag: text-generation
7
+ inference: false
8
+ quantized_by: audreyt
9
  ---
10
+ # Breeze-7B-Instruct-64k-v0.1-GGUF
11
+
12
+ - Model creator: [MediaTek Research](https://huggingface.co/MediaTek-Research)
13
+ - Original model: [Breeze-7B-Instruct-64k-v0.1](https://huggingface.co/MediaTek-Research/Breeze-7B-Instruct-64k-v0.1)
14
+
15
+ ## Description
16
+
17
+ This repo contains GGUF format model files for MediaTek Research's [Breeze-7B-Instruct-64k-v0.1](https://huggingface.co/MediaTek-Research/Breeze-7B-Instruct-64k-v0.1).
18
+
19
+ <!-- README_GGUF.md-about-gguf start -->
20
+ ### About GGUF
21
+
22
+ GGUF is a new format introduced by the llama.cpp team on August 21st 2023. It is a replacement for GGML, which is no longer supported by llama.cpp.
23
+
24
+ Here is an incomplete list of clients and libraries that are known to support GGUF:
25
+
26
+ * [llama.cpp](https://github.com/ggerganov/llama.cpp). The source project for GGUF. Offers a CLI and a server option.
27
+ * [text-generation-webui](https://github.com/oobabooga/text-generation-webui), the most widely used web UI, with many features and powerful extensions. Supports GPU acceleration.
28
+ * [KoboldCpp](https://github.com/LostRuins/koboldcpp), a fully featured web UI, with GPU accel across all platforms and GPU architectures. Especially good for story telling.
29
+ * [GPT4All](https://gpt4all.io/index.html), a free and open source local running GUI, supporting Windows, Linux and macOS with full GPU accel.
30
+ * [LM Studio](https://lmstudio.ai/), an easy-to-use and powerful local GUI for Windows and macOS (Silicon), with GPU acceleration. Linux available, in beta as of 27/11/2023.
31
+ * [LoLLMS Web UI](https://github.com/ParisNeo/lollms-webui), a great web UI with many interesting and unique features, including a full model library for easy model selection.
32
+ * [Faraday.dev](https://faraday.dev/), an attractive and easy to use character-based chat GUI for Windows and macOS (both Silicon and Intel), with GPU acceleration.
33
+ * [llama-cpp-python](https://github.com/abetlen/llama-cpp-python), a Python library with GPU accel, LangChain support, and OpenAI-compatible API server.
34
+ * [candle](https://github.com/huggingface/candle), a Rust ML framework with a focus on performance, including GPU support, and ease of use.
35
+ * [ctransformers](https://github.com/marella/ctransformers), a Python library with GPU accel, LangChain support, and OpenAI-compatible AI server. Note, as of time of writing (November 27th 2023), ctransformers has not been updated in a long time and does not support many recent models.
36
+
37
+ <!-- README_GGUF.md-about-gguf end -->
38
+
39
+ # Original model card
40
+
41
+ Breeze-7B is a language model family that builds on top of [Mistral-7B](https://huggingface.co/mistralai/Mistral-7B-v0.1), specifically intended for Traditional Chinese use.
42
+
43
+ [Breeze-7B-Base](https://huggingface.co/MediaTek-Research/Breeze-7B-Base-v0.1) is the base model for the Breeze-7B series.
44
+ It is suitable for use if you have substantial fine-tuning data to tune it for your specific use case.
45
+
46
+ [Breeze-7B-Instruct](https://huggingface.co/MediaTek-Research/Breeze-7B-Instruct-v0.1) derives from the base model Breeze-7B-Base, making the resulting model amenable to be used as-is for commonly seen tasks.
47
+
48
+ [Breeze-7B-Instruct-64k](https://huggingface.co/MediaTek-Research/Breeze-7B-Instruct-64k-v0.1) is a slightly modified version of
49
+ Breeze-7B-Instruct to enable a 64k-token context length. Roughly speaking, that is equivalent to 88k Traditional Chinese characters.
50
+
51
+ The current release version of Breeze-7B is v0.1.
52
+
53
+ Practicality-wise:
54
+ - Breeze-7B-Base expands the original vocabulary with additional 30,000 Traditional Chinese tokens. With the expanded vocabulary, everything else being equal, Breeze-7B operates at twice the inference speed for Traditional Chinese to Mistral-7B and Llama 7B. [See [Inference Performance](#inference-performance).]
55
+ - Breeze-7B-Instruct can be used as is for common tasks such as Q&A, RAG, multi-round chat, and summarization.
56
+ - In particular, Breeze-7B-Instruct-64k can perform tasks at a document level, not a chapter level.
57
+
58
+ Performance-wise:
59
+ - Breeze-7B-Instruct demonstrates impressive performance in benchmarks for Traditional Chinese, when compared to similar sized open-source contemporaries such as Taiwan-LLM-7B/13B-chat, QWen-7B-Chat, and Yi-6B-Chat. [See [Chat Model Performance](#chat-model-performance).]
60
+ - Breeze-7B-Instruct shows comparable results to Mistral-7B-Instruct-v0.1 on the MMLU and MT-Bench benchmarks. [See [Chat Model Performance](#chat-model-performance).]
61
+
62
+
63
+ *A project by the members (in alphabetical order): Chan-Jan Hsu 許湛然, Chang-Le Liu 劉昶樂, Feng-Ting Liao 廖峰挺, Po-Chun Hsu 許博竣, Yi-Chang Chen 陳宜昌, and the supervisor Da-Shan Shiu 許大山.*
64
+
65
+ ## Features
66
+
67
+ - Breeze-7B-Base-v0.1
68
+ - Expanding the vocabulary dictionary size from 32k to 62k to better support Traditional Chinese
69
+ - 8k-token context length
70
+ - Breeze-7B-Instruct-v0.1
71
+ - Expanding the vocabulary dictionary size from 32k to 62k to better support Traditional Chinese
72
+ - 8k-token context length
73
+ - Multi-turn dialogue (without special handling for harmfulness)
74
+ - Breeze-7B-Instruct-64k-v0.1
75
+ - Expanding the vocabulary dictionary size from 32k to 62k to better support Traditional Chinese
76
+ - 64k-token context length
77
+ - Multi-turn dialogue (without special handling for harmfulness)
78
+
79
+ ## Model Details
80
+
81
+ - Breeze-7B-Base-v0.1
82
+ - Finetuned from: [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1)
83
+ - Model type: Causal decoder-only transformer language model
84
+ - Language: English and Traditional Chinese (zh-tw)
85
+ - Breeze-7B-Instruct-v0.1
86
+ - Finetuned from: [MediaTek-Research/Breeze-7B-Base-v0.1](https://huggingface.co/MediaTek-Research/Breeze-7B-Base-v0.1)
87
+ - Model type: Causal decoder-only transformer language model
88
+ - Language: English and Traditional Chinese (zh-tw)
89
+ - Breeze-7B-Instruct-64k-v0.1
90
+ - Finetuned from: [MediaTek-Research/Breeze-7B-Instruct-v0.1](https://huggingface.co/MediaTek-Research/Breeze-7B-Instruct-v0.1)
91
+ - Model type: Causal decoder-only transformer language model
92
+ - Language: English and Traditional Chinese (zh-tw)
93
+
94
+ ## Base Model Performance
95
+
96
+ **TMMLU+**, **DRCD**, and **Table** source from [MediaTek-Research/TCEval-v2](https://huggingface.co/datasets/MediaTek-Research/TCEval-v2).
97
+ [MediaTek-Research/TCEval-v2](https://huggingface.co/datasets/MediaTek-Research/TCEval-v2) derives from [TCEval-v1](https://github.com/mtkresearch/MR-Models/tree/main/TC-Eval)
98
+ and [ikala/tmmluplus](https://huggingface.co/datasets/ikala/tmmluplus). **MMLU** sources from [hails/mmlu_no_train](https://huggingface.co/datasets/hails/mmlu_no_train).
99
+ We use the code revised from [EleutherAI/lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) to evaluate **TMMLU+**, **DRCD**, **Table**, and **MMLU**.
100
+
101
+
102
+ | Models | |↑ TMMLU+ (ACC) | DRCD (EM) | Table (ACC) | MMLU (ACC) |
103
+ |----------------------------------------------|--------|--------------|-------------|-------------|------------|
104
+ | | |TC, Knowledge |TC, Reasoning|TC, Reasoning|EN, Knowledge|
105
+ | | | 5 shot | 3 shot | 5 shot | 5 shot |
106
+ | [Yi-34B](https://huggingface.co/01-ai/Yi-34B)| 34B | 63.10 | 84.57 | 49.31 | 77.42 |
107
+ | [Qwen-14B](https://huggingface.co/01-ai/Qwen/Qwen-14B)| 14B | 51.30 | 16.95 * | 50.69 | 68.83 |
108
+ | [Yi-6B](https://huggingface.co/01-ai/Yi-6B) | 6B | 49.63 | 76.61 | 34.72 | 65.35 |
109
+ | [Qwen-7B](https://huggingface.co/01-ai/Qwen/Qwen-7B)| 7B | 42.84 | 0.0 * | 39.58 | 61.00 |
110
+ | [**Breeze-7B-Base-v0.1**](https://huggingface.co/MediaTek-Research/Breeze-7B-Base-v0.1) | 7B | 40.35 | 81.13 | 28.47 | 61.63 |
111
+ | [Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1)| 7B | 36.93 | 79.27 | 27.78 | 64.89 |
112
+
113
+
114
+ \* Few-shot learning cannot effectively guide the model to generate the proper answer.
115
+
116
+
117
+ ## Chat Model Performance
118
+
119
+ **TMMLU+**, **DRCD**, **Table**, and **MT-Bench-tw** source from [MediaTek-Research/TCEval-v2](https://huggingface.co/datasets/MediaTek-Research/TCEval-v2).
120
+ [MediaTek-Research/TCEval-v2](https://huggingface.co/datasets/MediaTek-Research/TCEval-v2) derives from [TCEval-v1](https://github.com/mtkresearch/MR-Models/tree/main/TC-Eval)
121
+ and [ikala/tmmluplus](https://huggingface.co/datasets/ikala/tmmluplus). **MMLU** sources from [hails/mmlu_no_train](https://huggingface.co/datasets/hails/mmlu_no_train).
122
+ **MT-Bench** source from [lmsys/mt_bench_human_judgments](https://huggingface.co/datasets/lmsys/mt_bench_human_judgments).
123
+ We use the code revised from [EleutherAI/lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) to evaluate **TMMLU+**, **DRCD**, **Table**, and **MMLU**.
124
+ We use the code revised from [fastchat llm_judge](https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judge) (GPT4 as judge) to evaluate **MT-Bench-tw** and **MT-Bench**.
125
+
126
+
127
+ | Models | |↑ MT-Bench-tw (Score)| TMMLU+ (ACC) | TMMLU+ (ACC) | DRCD (EM) | Table (ACC) | MT-Bench (Score) | MMLU (ACC) | MMLU (ACC) |
128
+ |---------------------------------------------------------------------------------------------------------|--------|--------------------|--------------|--------------|-------------|-------------|------------------|-------------|-------------|
129
+ | | |TC, Chat |TC, Knowledge |TC, Knowledge |TC, Reasoning|TC, Reasoning|EN, Chat |EN, Knowledge|EN, Knowledge|
130
+ | | |0 shot | 0 shot | 5 shot | 3 shot | 0 shot |0 shot | 0 shot | 5 shot |
131
+ | [gpt-3.5-turbo](https://openai.com) | |7.1 | 41.76 | | | |7.9 | 70.00 | |
132
+ | [Yi-34B-Chat](https://huggingface.co/01-ai/Yi-34B-Chat) | 34B |6.9 | 54.87 | | | 36.81 |7.6 | 71.04 | |
133
+ | [Qwen-14B-Chat](https://huggingface.co/Qwen/Qwen-14B-Chat) | 14B |6.4 | 48.41 | | | 41.67 |7.2 | 64.91 | |
134
+ | [**Breeze-7B-Instruct-v0.1**](https://huggingface.co/MediaTek-Research/Breeze-7B-Instruct-v0.1) | 7B |5.7 | 41.61 | | | 45.83 |7.1 | 63.26 | |
135
+ | [**Breeze-7B-Instruct-64k-v0.1**](https://huggingface.co/MediaTek-Research/Breeze-7B-Instruct-64k-v0.1) | 7B |5.5 | 40.99 | | | 36.11 |7.1 | 63.68 | |
136
+ | [Qwen-7B-Chat](https://huggingface.co/Qwen/Qwen-7B-Chat) | 7B |5.4 | 40.02 | | | 33.33 |6.2 | 55.94 | |
137
+ | [Yi-6B-Chat](https://huggingface.co/01-ai/Yi-6B-Chat) | 6B |5.0 | 44.79 | | | 25.69 |6.0 | 59.45 | |
138
+ | [Taiwan-LLM-13B-v2.0-chat](https://huggingface.co/yentinglin/Taiwan-LLM-13B-v2.0-chat) | 13B |5.0 | 29.47 | | | 23.61 |-* | 50.50 | |
139
+ | [Taiwan-LLM-7B-v2.1-chat](https://huggingface.co/yentinglin/Taiwan-LLM-7B-v2.1-chat) | 7B |4.2 | 28.08 | | | 31.25 | -* | 42.72 | |
140
+
141
+ \* Taiwan-LLM models responds to multi-turn questions (English) in Traditional Chinese.
142
+
143
+ **Category Score of MT-Bench-tw (0 shot)**
144
+
145
+ | Models | STEM |Extraction|Reasoning| Math | Coding | Roleplay| Writing |Humanities|↑ AVG |
146
+ |-----------------------------------------------------|---------|---------|---------|---------|---------|---------|---------|---------|---------|
147
+ | gpt-3.5-turbo | 7.8 | 6.1 | 5.1 | 6.4 | 6.2 | 8.7 | 7.4 | 9.3 | 7.1 |
148
+ | Yi-34B-Chat | 9.0 | 4.8 | 5.7 | 4.0 | 4.7 | 8.5 | 8.7 | 9.8 | 6.9 |
149
+ | Qwen-14B-Chat | 7.6 | 5.7 | 4.5 | 4.2 | 5.3 | 7.5 | 7.3 | 9.1 | 6.4 |
150
+ | **Breeze-7B-Instruct-v0.1** | 6.5 | 5.6 | 3.9 | 3.6 | 4.3 | 6.9 | 5.7 | 9.3 | 5.7 |
151
+ | **Breeze-7B-Instruct-64k-v0.1** | 6.1 | 5.3 | 3.7 | 2.9 | 4.2 | 7.0 | 6.7 | 8.3 | 5.5 |
152
+ | Qwen-7B-Chat | 6.6 | 4.5 | 4.8 | 2.9 | 3.6 | 6.2 | 6.8 | 8.2 | 5.4 |
153
+ | Yi-6B-Chat | 7.3 | 2.7 | 3.1 | 3.3 | 2.3 | 7.2 | 5.2 | 8.8 | 5.0 |
154
+ | Taiwan-LLM-13B-v2.0-chat | 6.1 | 3.4 | 4.1 | 2.3 | 3.1 | 7.4 | 6.6 | 6.8 | 5.0 |
155
+ | Taiwan-LLM-7B-v2.1-chat | 5.2 | 2.6 | 2.3 | 1.2 | 3.4 | 6.6 | 5.7 | 6.8 | 4.2 |
156
+
157
+ **Category ACC of TMMLU+ (0 shot)**
158
+
159
+ | Model | STEM | Social Science | Humanities | Other | ↑ AVG |
160
+ |-----------------------------------------------------|--------------|----------------|------------|------------|---------|
161
+ | Yi-34B-Chat | 47.65 | 64.25 | 52.73 | 54.91 | 54.87 |
162
+ | Qwen-14B-Chat | 43.83 | 55.00 | 48.55 | 46.22 | 48.41 |
163
+ | Yi-6B-Chat | 37.80 | 51.74 | 45.36 | 44.25 | 44.79 |
164
+ | gpt-3.5-turbo | 41.56 | 46.72 | 36.73 | 42.03 | 41.76 |
165
+ | **Breeze-7B-Instruct-v0.1** | 37.41 | 46.81 | 42.06 | 40.16 | 41.61 |
166
+ | **Breeze-7B-Instruct-64k-v0.1** | 37.88 | 46.35 | 40.31 | 39.40 | 40.99 |
167
+ | Qwen-7B-Chat | 35.44 | 46.22 | 38.35 | 40.06 | 40.02 |
168
+ | Taiwan-LLM-13B-v2.0-chat | 27.74 | 33.69 | 27.03 | 29.43 | 29.47 |
169
+ | Taiwan-LLM-7B-v2.1-chat | 25.58 | 31.76 | 27.36 | 27.61 | 28.08 |
170
+
171
+
172
+
173
+ ## Inference Performance
174
+ In this test, we use the first 700 characters of the [web article](https://health.udn.com/health/story/5976/7699252?from=udn_ch1005_main_index) as the input and ask the model to write the same article again.
175
+ All inferences run on 2 RTX A6000 GPUs (using `vllm`, with a tensor-parallel size of 2).
176
+
177
+ | Models | ↓ Inference Time (sec)|Estimated Max Input Length (Char)|
178
+ |--------------------------------------------------------------------|-------------------|--------------------------|
179
+ | Yi-6B | 10.62 | 5.2k |
180
+ | **Breeze-7B-Instruct-v0.1** | 10.74 | 11.1k |
181
+ | **Breeze-7B-Instruct-64k-v0.1** | 10.74 | 88.8k |
182
+ | Qwen-7B | 10.86 | 9.8k |
183
+ | Qwen-14B | 18.89 | 9.8k |
184
+ | Mistral-7B-v0.1 | 20.48 | 5.1k |
185
+ | Taiwan-LLM-7B-v2.1-base | 26.26 | 2.2k |
186
+ | Taiwan-LLM-13B-v2.0-base | 36.80 | 2.2k |
187
+ | Yi-34B | 43.71 | 4.5k |
188
+
189
+ ## Long-context Performance
190
+
191
+ TBD
192
+
193
+ ## Examples
194
+
195
+ TBD
196
+
197
+ ## Use in Transformers
198
+
199
+ First install direct dependencies:
200
+ ```
201
+ pip install transformers torch accelerate
202
+ ```
203
+ If you want faster inference using flash-attention2, you need to install these dependencies:
204
+ ```bash
205
+ pip install packaging ninja
206
+ pip install flash-attn
207
+ ```
208
+ Then load the model in transformers:
209
+ ```python
210
+ from transformers import pipeline, AutoModelForCausalLM, AutoTokenizer
211
+
212
+ model = AutoModelForCausalLM.from_pretrained("MediaTek-Research/Breeze-7B-Instruct-v0.1")
213
+ tokenizer = AutoTokenizer.from_pretrained("MediaTek-Research/Breeze-7B-Instruct-v0.1")
214
+
215
+ # you can also using pipeline
216
+ generator = pipeline("text-generation", model=model, tokenizer=tokenizer)
217
+ generator(
218
+ "請問台灣最高的山是",
219
+ max_length=30,
220
+ num_return_sequences=1,
221
+ )
222
+
223
+ ```
224
+
225
+ The structure of the query template follows that of Mistral-7B-Instruct, as shown below.
226
+ ```txt
227
+ <s> SYS_PROMPT [INST] QUERY1 [/INST] RESPONSE1 [INST] QUERY2 [/INST]
228
+ ```
229
+ where `SYS_PROMPT`, `QUERY1`, `RESPONSE1`, and `QUERY2` can be provided by the user.
230
+
231
+ The suggested default `SYS_PROMPT` is
232
+ ```txt
233
+ You are a helpful AI assistant built by MediaTek Research. The user you are helping speaks Traditional Chinese and comes from Taiwan.
234
+ ```
235
+
236
+ ## Citation
237
+
238
+ ```
239
+ @article{breeze7b2024,
240
+ title={},
241
+ author={},
242
+ journal={arXiv},
243
+ year={2024}
244
+ }
245
+ ```