---
license: mit
---
# Chuxin-1.6B-Base
<br>

## 介绍 (Introduction)

**Chuxin-1.6B-Base**是16亿参数规模的模型。Chuxin-1.6B完全基于开源数据构建，在经过超大规模数据训练后，Chuxin-1.6B在各类下游任务上具有非常强的竞争力。

**Chuxin-1.6B-1M**是基于Chuxin-1.6B-base模型在1M窗口下训练后的结果，大海捞针实验显示其具有非常强的上下文检索能力。


如果您想了解更多关于Chuxin-1.6B开源模型的细节，我们建议您参阅我们的[技术报告](https://arxiv.org/pdf/2405.04828)

**Chuxin-1.6B-Base** is a model with 1.6 billion parameters. Chuxin-1.6B is built entirely on open-source data. After being trained with large-scale data, Chuxin has very competitive capabilities in various downstream tasks.

**Chuxin-1.6B-1M** is the result of training the Chuxin-1.6B-base model with a 1M windows. Experiments such as searching for a needle in a haystack demonstrate its strong contextual retrieval abilities.

If you would like to learn more about the Chuxin-1.6B open-source model, we suggest you refer to our [technical report](https://arxiv.org/pdf/2405.04828).
<br>

## 快速使用（Quickstart）

您可以通过以下代码轻松调用：

You can easily call the model with the following code:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("chuxin-llm/Chuxin-1.6B-Base", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("chuxin-llm/Chuxin-1.6B-Base", device_map="auto", trust_remote_code=True, bf16=True).eval()
inputs = tokenizer('蒙古国的首都是乌兰巴托（Ulaanbaatar）\n冰岛的首都是雷克雅未克（Reykjavik）\n埃塞俄比亚的首都是', return_tensors='pt')
inputs = inputs.to(model.device)
pred = model.generate(**inputs, max_new_tokens=15, do_sample=False)
print(tokenizer.decode(pred.cpu()[0], skip_special_tokens=True))
# 蒙古国的首都是乌兰巴托（Ulaanbaatar）\n冰岛的首都是雷克雅未克（Reykjavik）\n埃塞俄比亚的首都是亚的斯亚贝巴（Addis Ababa）...
```

## 评测效果（Evaluation）

### (常识推理和阅读理解)  Common Sense Reasoning and Reading Comprehension tasks

| Model         | size | ARC-c |ARC-e |Boolq |Copa |Hellaswag |OpenbookQA |Piqa |Sciq |Winogrande |Avg|
|:--------------|:----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|
| Gemma     |    2B   |     48.98    |  78.45    |  69.51   |  84    |  71.73     |  39.8    |  78.02   |  94.3    |  65.51    |  70.03      |
| H2O-Danube†     |    1.8B  |     35.84   |  62.29   |  65.81   |  -   |  68.20     |  37.6    |  76.93   |  -    |  61.96   |  -     |
| Qwen1.5    |    1.8B   |      37.03     |  67.51    |  66.64    |  78    |  61.60     |  34.40    |  73.99     |  93     |  61.56     |  63.74    | 
| StableLM 2     |    1.6B    |      43.52   |69.44     | 75.5    | 84     | 70.3      | 39.6     | 76.82    | 96.1      | 64.17     | 68.82     |
| OpenLlama†   |    3B    |      34   |69| 68| -| 49| 40| 75| -| 62 |-|
| CT-LLM |  2B |  34.81   |  65.49   | 62.45    | 74    | 54.77      | 33.4     | 71.38   | 90.6     | 57.85     | 60.63     | 
| TinyLLama |  1.1B  |  34.81   | 67.47     | 63.15      | 74     | 60    | 34.6      | 73.12     | 88.8     | 58.88     | 61.64    |
| OLMo |  1B |  34.22   | 67.55     | 61.4     | 82     | 63.96      | 36.4     | 75.1    | 86.7     | 60.3      | 63.07     |
| Chuxin-1.6B-Base |  1.6B |  39.68  | 71.38     | 71.25      | 83    | 66.09     | 35.00      | 77.09     | 95     | 63.54      | 66.89     |

带有†的模型表示我们直接报告了相应论文中的分数，其他的则来自于我们重新测试的结果。

Models with † denote that we directly report the scores from the corresponding paper, and others are from our implementation.

### Open LLM LeaderBoard

| Model         | size | ARC-c  |HellaSwag|MMLU |TruthfulQA |Winogrande |GSM-8k |Avg |Avg wo GSM|
|:--------------|:----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|
| Gemma     |    2B   |     48.98   |  71.73     |  42.47    |  33   |  65.51    |10.08|  45.3    |  52.34      |
| H2O-Danube    |    1.8B    |      39. 68    | 69.75    | 25.97    | 33.63   | 64.17| 2.05    | 39.21     |46.64|
| Qwen1.5†     |    1.8B   |      37.88     |   61.42    |  46.71   |  39.43    |  60.3     |  33.59     |  46.55   | 49.15|
| StableLM 2     |    1.6B    |      43.52 |70.3  | 39.8     | 36.61     | 64.17   | 17.29      | 45.28     | 50.88    |
| OpenLlama†      |     3B    |    39.9  | 71.6    | 27.1    | 34.8     | 67   | 0.9      |40.3|48.08|
| CT-LLM |  2B |  34.81   | 54.77      | 37.81     | 39.81   | 57.85     | 7.35     | 38.73     | 45.01|
| TinyLLama |  1.1B  |  33.87  | 60.31   | 26.04     | 37.32   | 59.51    | 1.44     | 36.42    |43.41|
| OLMo |  1B |  34.22   | 63.96      | 35.44     | 35.53    | 62.67     | 9.86     | 41.81    |48.2|
| Chuxin-1.6B-Base |  1.6B |  39.68  | 66.09     | 41.07      | 37.65    | 63.54    | 12.66     | 43.45    |49.61|

带有†的模型表示我们直接报告 Open LLM排行榜的分数，其他的则来自于我们重新测试的结果。

Models with † denote that we directly report the scores from the Open LLM Leaderboard, and others are from our implementation.

### CMMLU, C-Eval and HumanEval

| Model         | size | C-Eval  |CMMLU|HUMANEVAL |
|:--------------|:----------:|:-----------:|:-----------:|:-----------:|
| Gemma     |    2B   |     31   |  31.06    |  9.51| 
| Qwen1.5    |    1.8B   |      59.38     |   57.08  |  23.17   | 
| StableLM 2     |    1.6B    |      29.27 |30.1 | 7.32     | 
| CT-LLM |  2B |  36.78  | 36.4      | 9.15     | 
| Chuxin-1.6B-Base |  1.6B |  39.31  | 37.11     | 9.76 |

## 引用 (Citation)

如果你觉得我们的工作对你有帮助，欢迎引用！

If you find our work helpful, feel free to give us a cite.

```
@article{chuxin,
  title={CHUXIN: 1.6B TECHNICAL REPORT},
  author={Xiaomin Zhuang, Yufan Jiang, Qiaozhi He, Zhihua Wu},
  journal={arXiv preprint arXiv:2405.04828},
  year={2024}
}
```
<br>