|
|
--- |
|
|
library_name: transformers |
|
|
license: apache-2.0 |
|
|
pipeline_tag: text-generation |
|
|
base_model: |
|
|
- Qwen/Qwen3-4B-Thinking-2507 |
|
|
tags: |
|
|
- neuralmagic |
|
|
- redhat |
|
|
- llmcompressor |
|
|
- quantized |
|
|
- INT4 |
|
|
--- |
|
|
|
|
|
# Qwen3-4B-Thinking-2507.w4a16 |
|
|
|
|
|
## Model Overview |
|
|
- **Model Architecture:** Qwen3ForCausalLM |
|
|
- **Input:** Text |
|
|
- **Output:** Text |
|
|
- **Model Optimizations:** |
|
|
- **Weight quantization:** INT4 |
|
|
- **Intended Use Cases:** |
|
|
- Reasoning. |
|
|
- Function calling. |
|
|
- Subject matter experts via fine-tuning. |
|
|
- Multilingual instruction following. |
|
|
- Translation. |
|
|
- **Out-of-scope:** Use in any manner that violates applicable laws or regulations (including trade compliance laws). |
|
|
- **Release Date:** 05/05/2025 |
|
|
- **Version:** 1.0 |
|
|
- **Model Developers:** RedHat (Neural Magic) |
|
|
|
|
|
### Model Optimizations |
|
|
|
|
|
This model was obtained by quantizing the weights of [Qwen/Qwen3-4B-Thinking-2507](https://huggingface.co/Qwen/Qwen3-4B-Thinking-2507) to INT4 data type. |
|
|
This optimization reduces the number of bits per parameter from 16 to 4, reducing the disk size and GPU memory requirements by approximately 75%. |
|
|
|
|
|
Only the weights of the linear operators within transformers blocks are quantized. |
|
|
Weights are quantized using a symmetric per-group scheme, with group size 128. |
|
|
The [GPTQ](https://arxiv.org/abs/2210.17323) algorithm is applied for quantization, as implemented in the [llm-compressor](https://github.com/vllm-project/llm-compressor) library. |
|
|
|
|
|
|
|
|
## Deployment |
|
|
|
|
|
This model can be deployed efficiently using the [vLLM](https://docs.vllm.ai/en/latest/) backend, as shown in the example below. |
|
|
|
|
|
```python |
|
|
from vllm import LLM, SamplingParams |
|
|
from transformers import AutoTokenizer |
|
|
|
|
|
model_id = "RedHatAI/Qwen3-4B-Thinking-2507.w4a16" |
|
|
number_gpus = 1 |
|
|
sampling_params = SamplingParams(temperature=0.6, top_p=0.95, top_k=20, min_p=0, max_tokens=256) |
|
|
|
|
|
messages = [ |
|
|
{"role": "user", "content": prompt} |
|
|
] |
|
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained(model_id) |
|
|
|
|
|
messages = [{"role": "user", "content": "Give me a short introduction to large language model."}] |
|
|
|
|
|
prompts = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False) |
|
|
|
|
|
llm = LLM(model=model_id, tensor_parallel_size=number_gpus) |
|
|
|
|
|
outputs = llm.generate(prompts, sampling_params) |
|
|
|
|
|
generated_text = outputs[0].outputs[0].text |
|
|
print(generated_text) |
|
|
``` |
|
|
|
|
|
vLLM aslo supports OpenAI-compatible serving. See the [documentation](https://docs.vllm.ai/en/latest/) for more details. |
|
|
|
|
|
## Creation |
|
|
|
|
|
<details> |
|
|
<summary>Creation details</summary> |
|
|
This model was created with [llm-compressor](https://github.com/vllm-project/llm-compressor) by running the code snippet below. |
|
|
|
|
|
|
|
|
```python |
|
|
from llmcompressor.modifiers.quantization import GPTQModifier |
|
|
from llmcompressor.transformers import oneshot |
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
|
|
|
|
# Load model |
|
|
model_stub = "Qwen/Qwen3-4B-Thinking" |
|
|
model_name = model_stub.split("/")[-1] |
|
|
|
|
|
num_samples = 1024 |
|
|
max_seq_len = 8192 |
|
|
|
|
|
model = AutoModelForCausalLM.from_pretrained(model_stub) |
|
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained(model_stub) |
|
|
|
|
|
def preprocess_fn(example): |
|
|
return {"text": tokenizer.apply_chat_template(example["messages"], add_generation_prompt=False, tokenize=False)} |
|
|
|
|
|
ds = load_dataset("neuralmagic/LLM_compression_calibration", split="train") |
|
|
ds = ds.map(preprocess_fn) |
|
|
|
|
|
# Configure the quantization algorithm and scheme |
|
|
recipe = GPTQModifier( |
|
|
ignore: ["lm_head"] |
|
|
config_groups={"group_0": {"targets": ["Linear"], "weights": { "num_bits": 4, "type": int, "strategy": "group", "group_size": 128, "symmetric": true, "dynamic": false, "observer": "mse" } } }, |
|
|
dampening_frac=0.01, |
|
|
) |
|
|
|
|
|
# Apply quantization |
|
|
oneshot( |
|
|
model=model, |
|
|
dataset=ds, |
|
|
recipe=recipe, |
|
|
max_seq_length=max_seq_len, |
|
|
num_calibration_samples=num_samples, |
|
|
) |
|
|
|
|
|
# Save to disk in compressed-tensors format |
|
|
save_path = model_name + "-quantized.w4a16" |
|
|
model.save_pretrained(save_path) |
|
|
tokenizer.save_pretrained(save_path) |
|
|
print(f"Model and tokenizer saved to: {save_path}") |
|
|
``` |
|
|
</details> |
|
|
|
|
|
|
|
|
|
|
|
## Evaluation |
|
|
|
|
|
The model was evaluated on the OpenLLM leaderboard tasks versions 2, using [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness), and on reasoning tasks using [lighteval](https://github.com/neuralmagic/lighteval/tree/reasoning). |
|
|
[vLLM](https://docs.vllm.ai/en/stable/) was used for all evaluations. |
|
|
|
|
|
<details> |
|
|
<summary>Evaluation details</summary> |
|
|
|
|
|
**lm-evaluation-harness** |
|
|
``` |
|
|
lm_eval \ |
|
|
--model vllm \ |
|
|
--model_args pretrained="RedHatAI/Qwen3-4B-Thinking-2507.w4a16",dtype=auto,gpu_memory_utilization=0.5,max_model_len=15000,enable_chunk_prefill=True,tensor_parallel_size=1 \ |
|
|
--tasks openllm \ |
|
|
--apply_chat_template\ |
|
|
--fewshot_as_multiturn \ |
|
|
--batch_size auto |
|
|
``` |
|
|
|
|
|
``` |
|
|
lm_eval \ |
|
|
--model vllm \ |
|
|
--model_args pretrained="RedHatAI/Qwen3-4B-Thinking-2507.w4a16",dtype=auto,gpu_memory_utilization=0.5,max_model_len=15000,enable_chunk_prefill=True,tensor_parallel_size=1 \ |
|
|
--tasks mgsm \ |
|
|
--apply_chat_template\ |
|
|
--batch_size auto |
|
|
``` |
|
|
|
|
|
``` |
|
|
lm_eval \ |
|
|
--model vllm \ |
|
|
--model_args pretrained="RedHatAI/Qwen3-4B-Thinking-2507.w4a16",dtype=auto,gpu_memory_utilization=0.5,max_model_len=15000,enable_chunk_prefill=True,tensor_parallel_size=1 \ |
|
|
--tasks leaderboard \ |
|
|
--apply_chat_template\ |
|
|
--fewshot_as_multiturn \ |
|
|
--batch_size auto |
|
|
``` |
|
|
|
|
|
**lighteval** |
|
|
|
|
|
lighteval_model_arguments.yaml |
|
|
```yaml |
|
|
model_parameters: |
|
|
model_name: RedHatAI/Qwen3-4B-Thinking-2507.w4a16 |
|
|
dtype: auto |
|
|
gpu_memory_utilization: 0.9 |
|
|
max_model_length: 40960 |
|
|
generation_parameters: |
|
|
temperature: 0.6 |
|
|
top_k: 20 |
|
|
min_p: 0.0 |
|
|
top_p: 0.95 |
|
|
max_new_tokens: 32000 |
|
|
``` |
|
|
|
|
|
``` |
|
|
lighteval vllm \ |
|
|
--model_args lighteval_model_arguments.yaml \ |
|
|
--tasks lighteval|aime24|0|0 \ |
|
|
--use_chat_template = true |
|
|
``` |
|
|
|
|
|
``` |
|
|
lighteval vllm \ |
|
|
--model_args lighteval_model_arguments.yaml \ |
|
|
--tasks lighteval|aime25|0|0 \ |
|
|
--use_chat_template = true |
|
|
``` |
|
|
|
|
|
``` |
|
|
lighteval vllm \ |
|
|
--model_args lighteval_model_arguments.yaml \ |
|
|
--tasks lighteval|math_500|0|0 \ |
|
|
--use_chat_template = true |
|
|
``` |
|
|
|
|
|
``` |
|
|
lighteval vllm \ |
|
|
--model_args lighteval_model_arguments.yaml \ |
|
|
--tasks lighteval|gpqa:diamond|0|0 \ |
|
|
--use_chat_template = true |
|
|
``` |
|
|
|
|
|
``` |
|
|
lighteval vllm \ |
|
|
--model_args lighteval_model_arguments.yaml \ |
|
|
--tasks extended|lcb:codegeneration \ |
|
|
--use_chat_template = true |
|
|
``` |
|
|
|
|
|
</details> |
|
|
|
|
|
### Accuracy |
|
|
|
|
|
<table> |
|
|
<tr> |
|
|
<th>Category |
|
|
</th> |
|
|
<th>Benchmark |
|
|
</th> |
|
|
<th>Qwen3-4B-Thinking |
|
|
</th> |
|
|
<th>Qwen3-4B-Thinking-2507.w4a16<br>(this model) |
|
|
</th> |
|
|
<th>Recovery |
|
|
</th> |
|
|
</tr> |
|
|
<tr> |
|
|
<td rowspan="1" ><strong>LLM as a judge</strong> |
|
|
</td> |
|
|
<td>Arena Hard v2 |
|
|
</td> |
|
|
<td>50 (48.3 / 51.8) |
|
|
</td> |
|
|
<td>43.2 (41.4 / 44.7) |
|
|
</td> |
|
|
<td>94.7% |
|
|
</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td rowspan="7" ><strong>OpenLLM v2</strong> |
|
|
</td> |
|
|
<td>MMLU-Pro (5-shot) |
|
|
</td> |
|
|
<td>47.45 |
|
|
</td> |
|
|
<td>45.38 |
|
|
</td> |
|
|
<td>95.6% |
|
|
</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>IFEval (0-shot) |
|
|
</td> |
|
|
<td>86.26 |
|
|
</td> |
|
|
<td>84.86 |
|
|
</td> |
|
|
<td>98.4% |
|
|
</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>BBH (3-shot) |
|
|
</td> |
|
|
<td>XX.XX |
|
|
</td> |
|
|
<td>XX.XX |
|
|
</td> |
|
|
<td>XX.X% |
|
|
</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>Math-lvl-5 (4-shot) |
|
|
</td> |
|
|
<td>XX.XX |
|
|
</td> |
|
|
<td>XX.XX |
|
|
</td> |
|
|
<td>XX.X% |
|
|
</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>GPQA (0-shot) |
|
|
</td> |
|
|
<td>XX.XX |
|
|
</td> |
|
|
<td>XX.XX |
|
|
</td> |
|
|
<td>XX.X% |
|
|
</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>MuSR (0-shot) |
|
|
</td> |
|
|
<td>XX.XX |
|
|
</td> |
|
|
<td>XX.XX |
|
|
</td> |
|
|
<td>XX.X% |
|
|
</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td><strong>Average</strong> |
|
|
</td> |
|
|
<td><strong>XX.XX</strong> |
|
|
</td> |
|
|
<td><strong>xx.xx</strong> |
|
|
</td> |
|
|
<td><strong>XX.X%</strong> |
|
|
</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td rowspan="6" ><strong>Reasoning<br>(generation)</strong> |
|
|
</td> |
|
|
<td>AIME 2025 |
|
|
</td> |
|
|
<td>70.0 |
|
|
</td> |
|
|
<td>63.33 |
|
|
</td> |
|
|
<td>91.5% |
|
|
</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>GPQA diamond |
|
|
</td> |
|
|
<td>69.70 |
|
|
</td> |
|
|
<td>62.12 |
|
|
</td> |
|
|
<td>89.1% |
|
|
</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>Math-lvl-5 |
|
|
</td> |
|
|
<td>96.4 |
|
|
</td> |
|
|
<td>97.20 |
|
|
</td> |
|
|
<td>100.1% |
|
|
</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td><strong>Average</strong> |
|
|
</td> |
|
|
<td><strong>78.70</strong> |
|
|
</td> |
|
|
<td><strong>74.22</strong> |
|
|
</td> |
|
|
<td><strong>94.3%</strong> |
|
|
</td> |
|
|
</tr> |
|
|
</table> |
