shakebenn
/

llm-jp-3-13b-SFT-LoRA

@@ -2,199 +2,109 @@
 library_name: transformers
 tags:
 - unsloth
 ---
-# Model Card for Model ID
-<!-- Provide a quick summary of what the model is/does. -->
-## Model Details
-### Model Description
-<!-- Provide a longer summary of what this model is. -->
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-### Model Sources [optional]
-<!-- Provide the basic links for the model. -->
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-## Uses
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-### Direct Use
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-[More Information Needed]
-### Downstream Use [optional]
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-[More Information Needed]
-### Out-of-Scope Use
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-[More Information Needed]
-## Bias, Risks, and Limitations
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-[More Information Needed]
-### Recommendations
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-## How to Get Started with the Model
-Use the code below to get started with the model.
-[More Information Needed]
-## Training Details
-### Training Data
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-[More Information Needed]
-### Training Procedure
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-#### Preprocessing [optional]
-[More Information Needed]
-#### Training Hyperparameters
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-#### Speeds, Sizes, Times [optional]
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-[More Information Needed]
-## Evaluation
-<!-- This section describes the evaluation protocols and provides the results. -->
-### Testing Data, Factors & Metrics
-#### Testing Data
-<!-- This should link to a Dataset Card if possible. -->
-[More Information Needed]
-#### Factors
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-[More Information Needed]
-#### Metrics
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-[More Information Needed]
-### Results
-[More Information Needed]
-#### Summary
-## Model Examination [optional]
-<!-- Relevant interpretability work for the model goes here -->
-[More Information Needed]
-## Environmental Impact
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-## Technical Specifications [optional]
-### Model Architecture and Objective
-[More Information Needed]
-### Compute Infrastructure
-[More Information Needed]
-#### Hardware
-[More Information Needed]
-#### Software
-[More Information Needed]
-## Citation [optional]
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-**BibTeX:**
-[More Information Needed]
-**APA:**
-[More Information Needed]
-## Glossary [optional]
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-[More Information Needed]
-## More Information [optional]
-[More Information Needed]
-## Model Card Authors [optional]
-[More Information Needed]
-## Model Card Contact
-[More Information Needed]

 library_name: transformers
 tags:
 - unsloth
+- japanese
+- llm-jp
+- lora
+datasets:
+- GENIAC-Team-Ozaki/Hachi-Alpaca_newans
+- llm-jp/magpie-sft-v1.0
+language:
+- ja
+base_model:
+- llm-jp/llm-jp-3-13b
 ---
+# llm-jp-3-13b-SFT-LoRA モデルカード
+llm-jp-3-13bをベースに、QLoRAとUnslothを用いて効率的なファインチューニングを行った日本語言語モデルです。
+## モデルの詳細
+### モデルの説明
+- **開発者:** GENIAC Team
+- **共有者:** GENIAC Team
+- **モデルタイプ:** 言語モデル（デコーダーのみ）
+- **言語:** 日本語
+- **ライセンス:** ベースモデルに準拠
+- **ベースモデル:** llm-jp/llm-jp-3-13b
+### モデルソース
+- **リポジトリ:** https://huggingface.co/llm-jp/llm-jp-3-13b
+## 使用方法
+### 直接利用
+このモデルは以下のような用途に適しています：
+- 質問応答
+- テキスト生成
+- 文章要約
+- その他の自然言語処理タスク
+### 対象外の使用
+以下の用途での使用は推奨されません：
+- 商用利用
+- 重要な意思決定
+- 医療・法律アドバイス
+- 有害なコンテンツの生成
+## バイアス、リスク、制限事項
+- 学習データに起因するバイアスが存在する可能性があります
+- 事実と異なる情報を生成する可能性があります
+- 有害なコンテンツを生成する可能性があります
+### 推奨事項
+- 出力内容の検証を必ず行ってください
+- センシティブな用途での使用は避けてください
+- 生成された内容の責任は使用者が負うものとします
+## モデルの使用開始方法
+## 学習の詳細
+### 学習データ
+以下のデータセットを使用:
+- GENIAC-Team-Ozaki/Hachi-Alpaca_newans
+- llm-jp/magpie-sft-v1.0
+### 学習手順
+#### 前処理
+- 指示文と回答のペアにフォーマット
+- コンテキスト長を512トークンに制限
+#### 学習ハイパーパラメータ
+- **学習手法:** QLoRA with Unsloth
+- **量子化:** 4-bit
+- **LoRA設定:**
+  - rank (r): 32
+  - alpha: 32
+  - dropout: 0.05
+  - target_modules: ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]
+- **トレーニング設定:**
+  - バッチサイズ: 2
+  - 勾配累積: 4
+  - エポック数: 1
+  - 学習率: 2e-4
+  - シーケンス長: 512
+## 技術仕様
+### 計算インフラ
+#### ハードウェア要件
+- CUDA対応GPU
+- 最小8GB VRAM推奨
+#### ソフトウェア要件
+- Python 3.10以上
+- PyTorch 2.0以上
+- Transformers最新版
+- Unsloth（推奨）