T3Q-LLM-MG-v1.0 / README.md
chihoonlee10's picture
Update README.md
b1366b4 verified
metadata
library_name: transformers
license: apache-2.0
pipeline_tag: text-generation
datasets:
  - maywell/ko_Ultrafeedback_binarized
base model:
  - yanolja/EEVE-Korean-Instruct-10.8B-v1.0

image/png

T3Q-LLM-MG-v1.0

Model Developers Chihoon Lee(chihoonlee10), T3Q

Python code

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer

MODEL_DIR = "chihoonlee10/T3Q-LLM-MG-v1.0"
model = AutoModelForCausalLM.from_pretrained(MODEL_DIR, torch_dtype=torch.float16).to("cuda")
tokenizer = AutoTokenizer.from_pretrained(MODEL_DIR)
streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)

s = "한국의 수도는 어디?"
conversation = [{'role': 'user', 'content': s}]
inputs = tokenizer.apply_chat_template(
    conversation,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors='pt').to("cuda")
_ = model.generate(inputs, streamer=streamer, max_new_tokens=1024)

hf (pretrained=chihoonlee10/T3Q-LLM-MG-v1.0), limit: None, provide_description: False, num_fewshot: 0, batch_size: None

Task Version Metric Value Stderr
kobest_boolq 0 acc 0.9523 ± 0.0057
macro_f1 0.9523 ± 0.0057
kobest_copa 0 acc 0.7740 ± 0.0132
macro_f1 0.7737 ± 0.0133
kobest_hellaswag 0 acc 0.4980 ± 0.0224
acc_norm 0.5920 ± 0.0220
macro_f1 0.4950 ± 0.0223
kobest_sentineg 0 acc 0.7254 ± 0.0224
macro_f1 0.7106 ± 0.0234

T3Q-LLM/T3Q-LLM-sft1.0-dpo1.0

Task Version Metric Value Stderr
kobest_boolq 0 acc 0.9387 ± 0.0064
macro_f1 0.9387 ± 0.0064
kobest_copa 0 acc 0.7590 ± 0.0135
macro_f1 0.7585 ± 0.0135
kobest_hellaswag 0 acc 0.5080 ± 0.0224
acc_norm 0.5580 ± 0.0222
macro_f1 0.5049 ± 0.0224
kobest_sentineg 0 acc 0.8489 ± 0.0180
macro_f1 0.8483 ± 0.0180