Update README.md
Browse files
README.md
CHANGED
@@ -2,27 +2,22 @@
|
|
2 |
license: apache-2.0
|
3 |
---
|
4 |
|
5 |
-
|
6 |
-
|
7 |
-
APUS-xDAN-4.0-MOE
|
8 |
-
|
9 |
Introduction
|
10 |
-
APUS-xDAN-4.0-MOE is a transformer-based MoE decoder-only language model 对齐在 on a large amount of data.
|
11 |
|
12 |
-
|
|
|
|
|
13 |
|
14 |
Model Details
|
15 |
-
APUS-xDAN-4.0-MOE employs Mixture of Experts (MoE) architecture, where the models are upcycled from dense language models. For instance, APUS-xDAN-4.0-MOE is upcycled from xDAN-L2 Series which are high performance alignModels. It has 136B parameters in total and 30B activated parameters during runtime.
|
16 |
-
进过先进量化技术优化,我们的开源版本仅仅只有42GB大小,是可以很好的在消费级显卡例如4090,3090上运行。
|
17 |
|
|
|
18 |
|
19 |
Requirements
|
20 |
-
The code of APUS-xDAN-4.0-MOE has been in the latest Hugging face transformers and we advise you to build from source with command pip install git+https://github.com/huggingface/transformers, or you might encounter the following error:
|
21 |
-
|
22 |
-
|
23 |
-
Usage
|
24 |
-
llama.cpp
|
25 |
|
|
|
26 |
|
|
|
|
|
27 |
License
|
28 |
-
|
|
|
|
2 |
license: apache-2.0
|
3 |
---
|
4 |
|
|
|
|
|
|
|
|
|
5 |
Introduction
|
|
|
6 |
|
7 |
+
APUS-xDAN-4.0-MOE is a transformer-based decoder-only language model, developed on a vast corpus of data to ensure robust performance.
|
8 |
+
|
9 |
+
For more comprehensive information, please visit our blog post and GitHub repository.
|
10 |
|
11 |
Model Details
|
|
|
|
|
12 |
|
13 |
+
APUS-xDAN-4.0-MOE leverages the innovative Mixture of Experts (MoE) architecture, incorporating components from dense language models. Specifically, it inherits its capabilities from the highly performant xDAN-L2 Series. With a total of 136 billion parameters, of which 30 billion are activated during runtime, APUS-xDAN-4.0-MOE demonstrates unparalleled efficiency. Through advanced quantization techniques, our open-source version occupies a mere 42GB, making it seamlessly compatible with consumer-grade GPUs like the 4090 and 3090.
|
14 |
|
15 |
Requirements
|
|
|
|
|
|
|
|
|
|
|
16 |
|
17 |
+
The codebase for APUS-xDAN-4.0-MOE is integrated into the latest Hugging Face transformers library. We recommend building from source using the command pip install git+https://github.com/huggingface/transformers to ensure compatibility. Failure to do so may result in encountering the following error:
|
18 |
|
19 |
+
Copy code
|
20 |
+
Usage llama.cpp
|
21 |
License
|
22 |
+
|
23 |
+
APUS-xDAN-4.0-MOE is distributed under the LLAMA 2 Community License, Copyright (c) Meta Platforms, Inc. All Rights Reserved.
|