Update README.md
Browse files
README.md
CHANGED
|
@@ -25,7 +25,7 @@ This Model is a test to combine [Jamba](https://huggingface.co/ai21labs/Jamba-v0
|
|
| 25 |
|
| 26 |
The goal is to developpe and test if this kind of architectures have not too much quality loss for a fast inference.
|
| 27 |
|
| 28 |
-
Only 17.8M parameter over
|
| 29 |
|
| 30 |
|
| 31 |
- **Model type:** Mixture of attention head mixture of depth and mixture of expert 1.58bit linear layers **excepted for attention layer**
|
|
|
|
| 25 |
|
| 26 |
The goal is to developpe and test if this kind of architectures have not too much quality loss for a fast inference.
|
| 27 |
|
| 28 |
+
Only 17.8M parameter over 1025 is in bf16 precision wich is ~ 1.7% of the total number of parameters
|
| 29 |
|
| 30 |
|
| 31 |
- **Model type:** Mixture of attention head mixture of depth and mixture of expert 1.58bit linear layers **excepted for attention layer**
|