metadata
base_model:
- appvoid/arco
- h2oai/h2o-danube3-500m-base
library_name: transformers
tags:
- mergekit
- merge
arco+
This is an untrained passthrough model based on arco and danube as a first effort to train a small enough reasoning language model that generalizes across all kind of reasoning tasks.
Benchmarks
| Parameters | Model | MMLU | ARC | HellaSwag | PIQA | Winogrande | Average |
|---|---|---|---|---|---|---|---|
| 488m | arco-lite | 23.22 | 33.45 | 56.55 | 69.70 | 59.19 | 48.46 |
| 773m | arco-plus | 23.06 | 36.43 | 60.09 | 72.36 | 60.46 | 50.48 |
Configuration
The following YAML configuration was used to produce this model:
slices:
- sources:
- model: appvoid/arco
layer_range: [0, 14]
- sources:
- model: h2oai/h2o-danube3-500m-base
layer_range: [4, 16]
merge_method: passthrough
dtype: float16