metadata

base_model: llm-jp/llm-jp-3-13b
tags:
  - text-generation-inference
  - transformers
  - unsloth
  - llama
  - trl
license: apache-2.0
language:
  - en
datasets:
  - kinokokoro/ichikara-instruction-003

Uploaded model

Developed by: Rumi
License: apache-2.0
Finetuned from model : llm-jp/llm-jp-3-13b

This llama model was trained 2x faster with Unsloth and Huggingface's TRL library.

How to conduct inference

from unsloth import FastLanguageModel
from peft import PeftModel
import torch
import json
from tqdm import tqdm
import re

# Base model id and LoRA adapter ID
base_model_id = "llm-jp/llm-jp-3-13b"
adapter_id = "Rumi/llm-jp_SFT_rn_2024-12-14_06"

# Log in with your Hugging Face token
HF_TOKEN = "hogehoge"
from huggingface_hub import login
login(HF_TOKEN)

# Download the original model
dtype = None
load_in_4bit = True

base_model, tokenizer = FastLanguageModel.from_pretrained(
    model_name=base_model_id,
    dtype=dtype,
    load_in_4bit=load_in_4bit,
    trust_remote_code=True,
)

# Merge adapter to the base model
model = PeftModel.from_pretrained(base_model, adapter_id, token = HF_TOKEN)

# Read evaluation dataset
datasets = []
with open("./elyza-tasks-100-TV_0.jsonl", "r") as f:
    item = ""
    for line in f:
      line = line.strip()
      item += line
      if item.endswith("}"):
        datasets.append(json.loads(item))
        item = ""

# Change the format and conduct the evaluation
FastLanguageModel.for_inference(model)

results = []
for dt in tqdm(datasets):
  input = dt["input"]

  prompt = f"""### 指示\n{input}\n### 回答\n"""

  inputs = tokenizer([prompt], return_tensors = "pt").to(model.device)

  outputs = model.generate(**inputs, max_new_tokens = 512, use_cache = True, do_sample=False, repetition_penalty=1.2)
  prediction = tokenizer.decode(outputs[0], skip_special_tokens=True).split('\n### 回答')[-1]

  results.append({"task_id": dt["task_id"], "input": input, "output": prediction})

# Save result in the jsonl format
json_file_id = re.sub(".*/", "", adapter_id)
with open(f"/content/{json_file_id}_output.jsonl", 'w', encoding='utf-8') as f:
    for result in results:
        json.dump(result, f, ensure_ascii=False)
        f.write('\n')