README.md

---
library_name: transformers
tags:
- llama
- trl
datasets:
- elyza/ELYZA-tasks-100
language:
- ja
base_model:
- llm-jp/llm-jp-3-13b
---

# Model Card for Model ID




## Model Details

### Model Description

東大松尾研LLM講座2024の最終課題向けのelyza-tasks-100-TV_0.jsonlの出力用にFinetuningしたモデルです。
モデルの利用については、提供いただいたOmmniCampusの環境およびサンプルコードに沿ったものとなっております。

- **Developed by:** maktag
- **Language(s) (NLP):** Japanese
- **Finetuned from model [optional]:** llm-jp/llm-jp-3-13b


## How to Get Started with the Model

```
from transformers import AutoTokenizer, AutoModelForCausalLM

# Load the fine-tuned model and tokenizer
base_model_id = "llm-jp/llm-jp-3-13b" 
adapter_id = "maktag/llm-jp-3-13b-finetune8"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

# QLoRA config
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)

# Load model
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=bnb_config,
    device_map="auto",
    token = HF_TOKEN
)

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True, token = HF_TOKEN)

# 元のモデルにLoRAのアダプタを統合。
model = PeftModel.from_pretrained(model, adapter_id, token = HF_TOKEN)
```

[More Information Needed]

## Training Details

- Fine-Tuning Framework: LoRA-based PEFT (Parameter-Efficient Fine-Tuning).
- Dataset: Proprietary Japanese instruction-following dataset.
- Sequence Length: 512 tokens.
- Hyperparameters:
  - Batch size: 32
  - Learning rate: 1e-5
  - Epochs: 3

### Training Data

<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

- [elyza/ELYZA-tasks-100](https://huggingface.co/datasets/elyza/ELYZA-tasks-100)
- [Ichikara Instruction](https://liat-aip.sakura.ne.jp/wp/llmのための日本語インストラクションデータ作成/llmのための日本語インストラクションデータ-公開/)