File size: 3,815 Bytes
909228d 9b21fe1 909228d 9b21fe1 909228d 9b21fe1 99d9151 9b21fe1 aa3fc92 e34cc8f 492629e aa3fc92 492629e 7bf4c71 492629e 7bf4c71 aa3fc92 7bf4c71 9b21fe1 0475aa6 e34cc8f 0475aa6 e34cc8f 0475aa6 e34cc8f 0475aa6 e34cc8f 0475aa6 aa3fc92 9b21fe1 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 |
---
language:
- en
- ko
license: cc-by-nc-4.0
datasets:
- kyujinpy/KOR-gugugu-platypus-set
base_model:
- LDCC/LDCC-SOLAR-10.7B
pipeline_tag: text-generation
---
# LDCC-SOLAR-gugutypus-10.7B
<img src="logo.png" height=350, width=350>
---
## Model Details
**Model Developers**
- DongGeon Lee ([oneonlee](https://huggingface.co/oneonlee))
**Model Architecture**
- **LDCC-SOLAR-gugutypus-10.7B** is a instruction fine-tuned auto-regressive language model, based on the [SOLAR](https://huggingface.co/upstage/SOLAR-10.7B-v1.0) transformer architecture.
**Base Model**
- [LDCC/LDCC-SOLAR-10.7B](https://huggingface.co/LDCC/LDCC-SOLAR-10.7B)
**Training Dataset**
- [kyujinpy/KOR-gugugu-platypus-set](https://huggingface.co/datasets/kyujinpy/KOR-gugugu-platypus-set)
---
## Model comparisons
- **Ko-LLM leaderboard (2024/03/01)** [[link]](https://huggingface.co/spaces/upstage/open-ko-llm-leaderboard)
| Model | Average | Ko-ARC | Ko-HellaSwag | Ko-MMLU | Ko-TruthfulQA | Ko-CommonGen V2 |
| ----------------------------------------- | ----------- | ------ | ------------ | ------- | ------------- | --------------- |
| **[oneonlee/KoSOLAR-v0.2-gugutypus-10.7B](https://huggingface.co/oneonlee/KoSOLAR-v0.2-gugutypus-10.7B)** | **51.17** | 47.78 | 58.29 | 47.27 | 48.31 | 54.19 |
| oneonlee/LDCC-SOLAR-gugutypus-10.7B | 49.45 | 45.9 | 55.46 | 47.96 | 48.93 | 49 |
<br>
- **(KOR) AI-Harness evaluation** [[link]](https://github.com/Beomi/ko-lm-evaluation-harness)
| Tasks |Version|Filter|n-shot|Metric|Value | |Stderr|
|-------------------------|-------|------|-----:|------|-----:|---|-----:|
|KMMLU |N/A |none | 0|acc |0.3329|± |0.0794|
|KMMLU |N/A |none | 5|acc |0.3969|± |0.0816|
|KoBEST-HellaSwag | 0|none | 0|acc |0.4260|± |0.0221|
|KoBEST-HellaSwag | 0|none | 5|acc |0.4260|± |0.0221|
|KoBEST-BoolQ | 0|none | 0|acc |0.7792|± |0.0111|
|KoBEST-BoolQ | 0|none | 5|acc |0.8925|± |0.0083|
|KoBEST-COPA | 0|none | 0|acc |0.6670|± |0.0149|
|KoBEST-COPA | 0|none | 5|acc |0.7070|± |0.0144|
|KoBEST-SentiNeg | 0|none | 0|acc |0.7582|± |0.0215|
|KoBEST-SentiNeg | 0|none | 5|acc |0.9219|± |0.0135|
<br>
- **(ENG) AI-Harness evaluation** [[link]](https://github.com/EleutherAI/lm-evaluation-harness)
| Tasks |Version|Filter|n-shot|Metric|Value | |Stderr|
|------------------|-------|------|-----:|------|-----:|---|-----:|
|MMLU |N/A |none | 0|acc |0.5826|± |0.1432|
|MMLU |N/A |none | 5|acc |0.6124|± |0.1275|
|HellaSwag | 1|none | 0|acc |0.6075|± |0.0049|
|HellaSwag | 1|none | 5|acc |0.6534|± |0.0047|
|BoolQ | 2|none | 0|acc |0.8737|± |0.0058|
|BoolQ | 2|none | 5|acc |0.8878|± |0.0055|
|COPA | 1|none | 0|acc |0.8300|± |0.0378|
|COPA | 1|none | 5|acc |0.9300|± |0.0256|
|truthfulqa |N/A |none | 0|acc |0.4249|± |0.0023|
|truthfulqa |N/A |none | 5|acc | - |± | - |
---
## Implementation Code
```python
### LDCC-SOLAR-gugutypus
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
repo = "oneonlee/LDCC-SOLAR-gugutypus-10.7B"
model = AutoModelForCausalLM.from_pretrained(
repo,
return_dict=True,
torch_dtype=torch.float16,
device_map='auto'
)
tokenizer = AutoTokenizer.from_pretrained(repo)
``` |