|
--- |
|
library_name: transformers |
|
license: apache-2.0 |
|
pipeline_tag: text-generation |
|
datasets: |
|
- maywell/ko_Ultrafeedback_binarized |
|
base model: |
|
- yanolja/EEVE-Korean-Instruct-10.8B-v1.0 |
|
--- |
|
|
|
 |
|
|
|
|
|
|
|
|
|
# T3Q-LLM-MG-v1.0 |
|
|
|
## Model Developers Chihoon Lee(chihoonlee10), T3Q |
|
|
|
### Python code |
|
```python |
|
import torch |
|
from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer |
|
|
|
MODEL_DIR = "chihoonlee10/T3Q-LLM-MG-v1.0" |
|
model = AutoModelForCausalLM.from_pretrained(MODEL_DIR, torch_dtype=torch.float16).to("cuda") |
|
tokenizer = AutoTokenizer.from_pretrained(MODEL_DIR) |
|
streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True) |
|
|
|
s = "한국의 수도는 어디?" |
|
conversation = [{'role': 'user', 'content': s}] |
|
inputs = tokenizer.apply_chat_template( |
|
conversation, |
|
tokenize=True, |
|
add_generation_prompt=True, |
|
return_tensors='pt').to("cuda") |
|
_ = model.generate(inputs, streamer=streamer, max_new_tokens=1024) |
|
|
|
``` |
|
hf (pretrained=chihoonlee10/T3Q-LLM-MG-v1.0), limit: None, provide_description: False, num_fewshot: 0, batch_size: None |
|
|
|
| Task |Version| Metric |Value | |Stderr| |
|
|----------------|------:|--------|-----:|---|-----:| |
|
|kobest_boolq | 0|acc |0.9523|± |0.0057| |
|
| | |macro_f1|0.9523|± |0.0057| |
|
|kobest_copa | 0|acc |0.7740|± |0.0132| |
|
| | |macro_f1|0.7737|± |0.0133| |
|
|kobest_hellaswag| 0|acc |0.4980|± |0.0224| |
|
| | |acc_norm|0.5920|± |0.0220| |
|
| | |macro_f1|0.4950|± |0.0223| |
|
|kobest_sentineg | 0|acc |0.7254|± |0.0224| |
|
| | |macro_f1|0.7106|± |0.0234| |
|
|
|
|
|
|
|
### T3Q-LLM/T3Q-LLM-sft1.0-dpo1.0 |
|
|
|
| Task |Version| Metric |Value | |Stderr| |
|
|----------------|------:|--------|-----:|---|-----:| |
|
|kobest_boolq | 0|acc |0.9387|± |0.0064| |
|
| | |macro_f1|0.9387|± |0.0064| |
|
|kobest_copa | 0|acc |0.7590|± |0.0135| |
|
| | |macro_f1|0.7585|± |0.0135| |
|
|kobest_hellaswag| 0|acc |0.5080|± |0.0224| |
|
| | |acc_norm|0.5580|± |0.0222| |
|
| | |macro_f1|0.5049|± |0.0224| |
|
|kobest_sentineg | 0|acc |0.8489|± |0.0180| |
|
| | |macro_f1|0.8483|± |0.0180| |
|
|
