---
license: apache-2.0

inference: false

---

# XVERSE-13B-Chat

## 模型介绍

**XVERSE-13B-Chat**为[**XVERSE-13B**](https://huggingface.co/xverse/XVERSE-13B)模型对齐后的版本。

**XVERSE-13B** 是由深圳元象科技自主研发的支持多语言的大语言模型（Large Language Model），主要特点如下：

- **模型结构**：XVERSE-13B 使用主流 Decoder-only 的标准 Transformer 网络结构，支持 8K 的上下文长度（Context Length），为同尺寸模型中最长，能满足更长的多轮对话、知识问答与摘要等需求，模型应用场景更广泛。
- **训练数据**：构建了 1.4 万亿 token 的高质量、多样化的数据对模型进行充分训练，包含中、英、俄、西等 40 多种语言，通过精细化设置不同类型数据的采样比例，使得中英两种语言表现优异，也能兼顾其他语言效果。
- **分词**：基于 BPE（Byte-Pair Encoding）算法，使用上百 GB 语料训练了一个词表大小为 100,278 的分词器，能够同时支持多语言，而无需额外扩展词表。
- **训练框架**：自主研发多项关键技术，包括高效算子、显存优化、并行调度策略、数据-计算-通信重叠、平台和框架协同等，让训练效率更高，模型稳定性强，在千卡集群上的峰值算力利用率可达到 58.5%，位居业界前列。

## Model Introduction

**XVERSE-13B-Chat** is the aligned version of model [**XVERSE-13B**](https://huggingface.co/xverse/XVERSE-13B)

**XVERSE-13B** is a multilingual large language model, independently developed by Shenzhen Yuanxiang Technology. Its key features are as follows:

- **Model Structure**: XVERSE-13B uses the mainstream Decoder-only Transformer network structure, supports 8k context length, the longest one among models of the same size, which can meet the need of longer multi-round dialogues, knowledge question-answering, and summarization. This makes the model more versatile in application scenarios.
- **Training Data**: The model has been thoroughly trained on a diversified and high-quality dataset consisting of 1.4 trillion of tokens, including more than 40 languages such as Chinese, English, Russian, and Spanish. The sampling ratio of different types of data is finely set, which makes the performance of Chinese and English excellent, and also takes into account the effect of other languages.
- **Tokenization**: Based on the BPE (Byte-Pair Encoding) algorithm, a tokenizer with a vocabulary size of 100,278 has been trained using hundreds of gigabytes of language data. This tokenizer is capable of supporting multilingual without the need for additional vocabulary expansion.
- **Training Framework**: Several key technologies have also been independently developed, including efficient operators, memory optimization, parallel scheduling strategies, overlap of data-computation-communication, and synergy between platforms and frameworks. These advancements enhance training efficiency and model stability. With these technologies, the peak computational power utilization rate on a thousand-card cluster can reach 58.5%, ranking at the forefront of the industry.

## 评测结果

为验证模型的各项能力，我们选取了多个学科综合能力评测集，包括 [MMLU](https://arxiv.org/abs/2009.03300)（英文）、 [C-Eval](https://cevalbenchmark.com/)（中文）、[AGIEval](https://arxiv.org/abs/2304.06364)（中英） 、[GAOKAO-Bench](https://github.com/OpenLMLab/GAOKAO-Bench)（中英）、[GAOKAO-English](https://github.com/ExpressAI/AI-Gaokao)（英文），评测结果如下：

|        模型         |       类型       |       MMLU       |      C-Eval      | AGIEval<sup>1</sup> | GAOKAO-Bench<sup>1</sup> | GAOKAO-English<sup>1</sup> |
| :------------------------: | :--------------: | :--------------: | :--------------: | :-----------------: | :----------------------: | :------------------------: |
|        Baichuan-13B       |      底座       | 51.6<sup>2</sup> | 53.6<sup>3</sup> |        40.5         |           45.9           |            56.9            |
|     Baichuan-13B-Chat     |     对话        | 52.1<sup>2</sup> | 51.5<sup>2</sup> |        34.6         |           46.7           |            63.8            |
|    Chinese-Alpaca-2-13B   |     对话        |       53.2       |       41.3       |        36.6         |           38.4           |            65.1            |
|        Llama-1-13B        |     底座        | 46.9<sup>4</sup> |       28.8       |        27.3         |           26.4           |            38.1            |
|        Llama-2-13B        |     底座        | 54.8<sup>4</sup> |       35.6       |        33.4         |           35.4           |            60.6            |
|  moss-moon-003-base (16B) |     底座        |       24.7       | 33.1<sup>3</sup> |        26.8         |           28.5           |            34.7            |
|  moss-moon-003-sft (16B)  |     对话    |       25.5       |       33.6       |        27.6         |           28.8           |            29.2            |
|       OpenLLaMA-13B       |     底座    |       42.4       |       24.7       |        24.0         |           25.6           |            33.3            |
|          OPT-13B          |     底座    |       25.2       |       25.0       |        24.2         |           24.4           |            31.1            |
|         Pythia-12B        |     底座    |       25.1       |       26.2       |        25.3         |           25.3           |            26.8            |
|      Vicuna-13B-v1.5      |     对话    |       53.5       |       27.9       |        29.7         |           31.6           |            52.9            |
| Ziya-LLaMA-13B-Pretrain-v1|     底座    |       43.9       |       30.2       |        27.2         |           26.4           |            37.6            |
|    Ziya-LLaMA-13B-v1.1    |     对话    |       50.6       |       29.3       |        23.6         |           26.7           |            27.3            |
|       **XVERSE-13B**      |     底座    |     **55.1**     |     **54.7**     |      **41.4**       |         **53.9**         |          **66.5**          |
|    **XVERSE-13B-Chat**    |     对话    |     **60.2**     |     **53.1**     |      **48.3**       |         **50.7**         |          **80.6**          |

> <sup>1：只针对其中的单项选择题进行测试，即排除了填空题、开放性问题和多项选择题</sup>   
> <sup>2：来源于 [Baichuan-13B](https://github.com/baichuan-inc/Baichuan-13B) 的汇报结果</sup>   
> <sup>3：来源于 [C-Eval](https://cevalbenchmark.com/) 的汇报结果</sup>   
> <sup>4：来源于[Llama 2 论文](https://arxiv.org/abs/2307.09288)的汇报结果</sup>
>
> 对于 MMLU ，我们采用作者提供的[评测工具](https://github.com/hendrycks/test)，C-Eval、AGIEval、GAOKAO-Bench、GAOKAO-English 与 MMLU 的评测方式相同，且统一采用 **5-shot** 构造测试样本。

## Model Evaluation

In order to validate the various abilities of the model, we have chosen several comprehensive capability benchmarks across multiple disciplines, including [MMLU](https://arxiv.org/abs/2009.03300) (English), [C-Eval](https://cevalbenchmark.com/) (Chinese), [AGIEval](https://arxiv.org/abs/2304.06364) (Chinese and English), [GAOKAO-Bench](https://github.com/OpenLMLab/GAOKAO-Bench) (Chinese and English), [GAOKAO-English](https://github.com/ExpressAI/AI-Gaokao) (English), the evaluation results are as follows:


|        Models         |       Type       |       MMLU       |      C-Eval      | AGIEval<sup>1</sup> | GAOKAO-Bench<sup>1</sup> | GAOKAO-English<sup>1</sup> |
| :------------------------: | :--------------: | :--------------: | :--------------: | :-----------------: | :----------------------: | :------------------------: |
|        Baichuan-13B       |      pretrained       | 51.6<sup>2</sup> | 53.6<sup>3</sup> |        40.5         |           45.9           |            56.9            |
|     Baichuan-13B-Chat     |     fine-tuned        | 52.1<sup>2</sup> | 51.5<sup>2</sup> |        34.6         |           46.7           |            63.8            |
|    Chinese-Alpaca-2-13B   |     fine-tuned        |       53.2       |       41.3       |        36.6         |           38.4           |            65.1            |
|        Llama-1-13B        |     pretrained        | 46.9<sup>4</sup> |       28.8       |        27.3         |           26.4           |            38.1            |
|        Llama-2-13B        |     pretrained        | 54.8<sup>4</sup> |       35.6       |        33.4         |           35.4           |            60.6            |
|  moss-moon-003-base (16B) |     pretrained        |       24.7       | 33.1<sup>3</sup> |        26.8         |           28.5           |            34.7            |
|  moss-moon-003-sft (16B)  |     fine-tuned    |       25.5       |       33.6       |        27.6         |           28.8           |            29.2            |
|       OpenLLaMA-13B       |     pretrained    |       42.4       |       24.7       |        24.0         |           25.6           |            33.3            |
|          OPT-13B          |     pretrained    |       25.2       |       25.0       |        24.2         |           24.4           |            31.1            |
|         Pythia-12B        |     pretrained    |       25.1       |       26.2       |        25.3         |           25.3           |            26.8            |
|      Vicuna-13B-v1.5      |     fine-tuned    |       53.5       |       27.9       |        29.7         |           31.6           |            52.9            |
| Ziya-LLaMA-13B-Pretrain-v1|     pretrained    |       43.9       |       30.2       |        27.2         |           26.4           |            37.6            |
|    Ziya-LLaMA-13B-v1.1    |     fine-tuned    |       50.6       |       29.3       |        23.6         |           26.7           |            27.3            |
|       **XVERSE-13B**      |     pretrained    |     **55.1**     |     **54.7**     |      **41.4**       |         **53.9**         |          **66.5**          |
|    **XVERSE-13B-Chat**    |     fine-tuned    |     **60.2**     |     **53.1**     |      **48.3**       |         **50.7**         |          **80.6**          |


> <sup>1: Tests are conducted only on single-answer multiple-choice questions, thus excluding fill-in-the-blanks, open-ended questions, and multiple-answer multiple-choice questions.</sup>   
> <sup>2: Reporting results from [Baichuan-13B](https://github.com/baichuan-inc/Baichuan-13B).</sup>   
> <sup>3: Reporting results from [C-Eval](https://cevalbenchmark.com/).</sup>   
> <sup>4: Reporting results from [Llama 2](https://arxiv.org/abs/2307.09288).</sup>
>
> For MMLU, we adopt the [evaluation tools](https://github.com/hendrycks/test) provided by the authors, C-Eval, AGIEval, GAOKAO-Bench, GAOKAO-English are the same as MMLU, and uniformly use **5-shot** to construct the test samples.

### MMLU 各类别指标

MMLU Category Results

|         Models          |         Type          | Average  |   STEM   | Social Science | Humanities |  Others  |
| :------------------------: | :------------------------: | :------: | :------: | :------------: | :--------: | :------: |
|        Baichuan-13B        |   pretrained   |   51.6   |   41.6   |      60.9      |    47.4    |   58.5   |
|     Baichuan-13B-Chat      |   fine-tuned   |   52.1   |   40.9   |      60.9      |    48.8    |   59.0   |
|    Chinese-Alpaca-2-13B    |   fine-tuned   |   53.2   |   41.8   |      61.2      |    51.3    |   59.2   |
|        Llama-1-13B         |   pretrained   |   46.9   |   35.8   |      53.8      |    45.0    |   53.3   |
|        Llama-2-13B         |   pretrained   |   54.8   |   44.1   |      62.6      |    52.8    |   61.1   |
|  moss-moon-003-base (16B)  |   pretrained   |   24.7   |   23.0   |      24.0      |    25.2    |   26.3   |
|  moss-moon-003-sft (16B)   |   fine-tuned   |   25.5   |   25.9   |      23.8      |    27.1    |   24.4   |
|       OpenLLaMA-13B        |   pretrained   |   42.4   |   34.7   |      48.6      |    40.0    |   47.1   |
|          OPT-13B           |   pretrained   |   25.2   |   23.9   |      24.1      |    25.9    |   26.3   |
|         Pythia-12B         |   pretrained   |   25.1   |   24.8   |      23.0      |    26.1    |   26.0   |
|      Vicuna-13B-v1.5       |   fine-tuned   |   53.5   |   42.3   |      61.3      |    50.3    |   60.9   |
| Ziya-LLaMA-13B-Pretrain-v1 |   pretrained   |   43.9   |   36.3   |      48.8      |    41.1    |   50.3   |
|    Ziya-LLaMA-13B-v1.1     |   fine-tuned   |   50.6   |   40.7   |      57.8      |    48.1    |   56.7   |
|       **XVERSE-13B**       |   pretrained   | **55.1** | **44.5** |    **64.4**    |  **50.5**  | **62.9** |
|    **XVERSE-13B-Chat**     |   fine-tuned   | **60.2** | **48.1** |    **67.7**    |  **56.4**  | **68.0** |

### C-Eval 各类别指标

C-Eval Category Results

|         Models          |         Type          | Average  |   STEM   | Social Science | Humanities |  Others  |
| :------------------------: | :------------------------: | :------: | :------: | :------------: | :--------: | :------: |
|        Baichuan-13B        |   pretrained  |   53.6   |   47.0   |      66.8      |    57.3    |   49.8   |
|     Baichuan-13B-Chat      |   fine-tuned  |   51.5   |   43.7   |      64.6      |    56.2    |   49.2   |
|    Chinese-Alpaca-2-13B    |   fine-tuned  |   41.3   |   37.8   |      51.1      |    42.4    |   37.8   |
|        Llama-1-13B         |   pretrained  |   28.8   |   27.5   |      33.9      |    27.7    |   27.7   |
|        Llama-2-13B         |   pretrained  |   35.6   |   34.5   |      39.8      |    36.2    |   33.2   |
|  moss-moon-003-base (16B)  |   pretrained  |   33.1   |   31.6   |      37.0      |    33.4    |   32.1   |
|  moss-moon-003-sft (16B)   |   fine-tuned  |   33.6   |   31.4   |      38.6      |    33.8    |   32.9   |
|       OpenLLaMA-13B        |   pretrained  |   24.7   |   25.5   |      23.5      |    24.2    |   24.7   |
|          OPT-13B           |   pretrained  |   25.0   |   24.4   |      24.6      |    25.9    |   25.4   |
|         Pythia-12B         |   pretrained  |   26.2   |   26.8   |      25.1      |    26.7    |   25.4   |
|      Vicuna-13B-v1.5       |   fine-tuned  |   27.9   |   25.4   |      33.2      |    29.3    |   26.2   |
| Ziya-LLaMA-13B-Pretrain-v1 |   pretrained  |   30.2   |   27.8   |      34.3      |    32.0    |   29.0   |
|    Ziya-LLaMA-13B-v1.1     |   fine-tuned  |   29.3   |   27.5   |      32.8      |    29.7    |   29.0   |
|       **XVERSE-13B**       |   pretrained  | **54.7** | **45.6** |    **66.2**    |  **58.3**  | **56.9** |
|    **XVERSE-13B-Chat**     |   fine-tuned  | **53.1** | **44.5** |    **65.3**    |  **56.5**  | **54.3** |

### Loading with Transformers

可通过以下代码加载 XVERSE-13B-Chat 模型进行对话：

The XVERSE-13B-Chat model can be loaded for chat using the following code:

```python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from transformers.generation.utils import GenerationConfig
model_path = "xverse/XVERSE-13B-Chat"
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForCausalLM.from_pretrained(model_path, trust_remote_code=True, torch_dtype=torch.bfloat16, device_map='auto')
model.generation_config = GenerationConfig.from_pretrained(model_path)
model = model.eval()
history = [{"role": "user", "content": "1955年谁是美国总统？他是什么党派？"}]
response = model.chat(tokenizer, history)
print(response)
history.append({"role": "assistant", "content": response})
history.append({"role": "user", "content": "他任职了多少年"})
response = model.chat(tokenizer, history)
print(response)
```

更多细节，包括对话demo、模型微调及量化等，请参考我们的[Github](https://github.com/xverse-ai/XVERSE-13B)。

For more details, including chat demo, model fine-tuning and quantization, please refer to our [Github](https://github.com/xverse-ai/XVERSE-13B).

## 局限性与免责申明

XVERSE-13B-Chat 与其他所有 LLM 一样，在某些情况下可能会产生不准确、有偏见或其他令人反感的内容。因此，请谨慎使用模型生成的内容，请勿将生成的有害内容进行传播，在部署任何 XVERSE-13B-Chat 的应用之前，开发人员应根据其具体应用对模型进行安全测试和调优。

我们强烈警告不要将 XVERSE-13B-Chat 模型用于制造或传播有害信息，或进行任何可能损害公众、国家、社会安全或违反法规的活动。如果使用 XVERSE-13B-Chat 模型产生任何问题，无论是数据安全问题、公共舆论风险，还是模型被误解、滥用、传播或不合规使用所引发的任何风险和问题，我们将不承担任何责任。

## Limitations and Disclaimer

Like all other Large Language Models (LLMs), XVERSE-13B-Chat may produce inaccurate, biased, or otherwise offensive content under certain circumstances. Therefore, please use the content generated by the model with caution and refrain from disseminating harmful content. Before deploying any application of XVERSE-13B-Chat, developers should conduct safety tests and optimization of the model according to its specific application.

We strongly warn against the use of the XVERSE-13B-Chat model for producing or spreading harmful information, or conducting any activities that might harm the public, national, or social security, or violate regulations. We assume no responsibility for any problems arising from the use of the XVERSE-13B-Chat model, whether it be data security issues, public opinion risks, or any risks and issues caused by misunderstanding, misuse, dissemination, or non-compliance with the model.

## 模型开源协议

使用本仓库的源码需要遵循 [Apache-2.0](https://github.com/xverse-ai/XVERSE-13B/blob/main/LICENSE) 开源协议，使用 XVERSE-13B-Chat 的模型权重则需要遵循[模型许可协议](MODEL_LICENSE.pdf)。

XVERSE-13B-Chat 模型权重对学术研究**完全开放**，并且支持**免费商用**，商用需申请商业使用授权，可以发送邮件到 <opensource@xverse.cn> 进行申请。

## Open Source License

The use of the source code in this repository must follow the [Apache-2.0](https://github.com/xverse-ai/XVERSE-13B/blob/main/LICENSE) open-source license, while the use of the model weights of XVERSE-13B-Chat needs to adhere to the [Model License Agreement](MODEL_LICENSE.pdf).

The XVERSE-13B-Chat model weights are **fully open** to academic research and support **free commercial use**. Commercial use requires an application for a commercial use license by sending an email to <opensource@xverse.cn>.