Update README.md

README.md

@@ -1,67 +1,144 @@
 ---
-base_model: Aratako/Llama-Gemma-2-27b-Simpo-trial3-iter1
+base_model: Aratako/Llama-Gemma-2-27b-CPO_SimPO-iter2
 library_name: transformers
-model_name: fft-orpo-iterative-iter3
 tags:
 - generated_from_trainer
 - axolotl
 - trl
 - orpo
 licence: license
+license: 
+  - llama3.1
+  - gemma
 ---
 
-# Model Card for fft-orpo-iterative-iter3
+# Llama-Gemma-2-27b-ORPO-iter3
 
-This model is a fine-tuned version of [Aratako/Llama-Gemma-2-27b-Simpo-trial3-iter1](https://huggingface.co/Aratako/Llama-Gemma-2-27b-Simpo-trial3-iter1).
-It has been trained using [TRL](https://github.com/huggingface/trl).
+## 概要
 
-## Quick start
+[google/gemma-2-27b](https://huggingface.co/google/gemma-2-27b)を教師あり学習と2回の[CPO_SimPO](https://github.com/fe1ixxu/CPO_SIMPO)によりInstruction Tuningしたモデルである[Aratako/Llama-Gemma-2-27b-CPO_SimPO-iter2](https://huggingface.co/Aratako/Llama-Gemma-2-27b-CPO_SimPO-iter2)に対して、
+[ORPO](https://arxiv.org/abs/2403.07691)を適用したモデルです。
 
-```python
-from transformers import pipeline
+[松尾研大規模言語モデル講座2024](https://weblab.t.u-tokyo.ac.jp/lecture/course-list/large-language-model/)のコンペ用の提出モデル作成の一環として作成・公開しています。
 
-question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
-generator = pipeline("text-generation", model="Aratako/fft-orpo-iterative-iter3", device="cuda")
-output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
-print(output["generated_text"])
-```
+This model is built with Llama and Qwen.
+
+## 使用データセット
+
+- [Aratako/iterative-dpo-data-for-ORPO-iter3](https://huggingface.co/datasets/Aratako/iterative-dpo-data-for-ORPO-iter3)
+
+## ライセンス
+
+本モデルは学習に利用したデータの関係で以下のライセンスの影響を受けます。
+
+- [META LLAMA 3.1 COMMUNITY LICENSE](https://www.llama.com/llama3_1/license/)を継承します。
+- [Gemma Terms of Use](https://ai.google.dev/gemma/terms)を継承します。
+- [Qwen LICENSE AGREEMENT](https://huggingface.co/Qwen/Qwen2.5-72B-Instruct/blob/main/LICENSE)の影響を受けます。ライセンスは継承しませんが、「Built with Qwen」のような文言を記載する必要があります。
+
+## 学習に関する詳細
+
+本モデルの学習には[axolotl](https://github.com/axolotl-ai-cloud/axolotl)を使いました。パラメータ等の学習の設定は下記の設定ファイルをご確認ください。
+
+[<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/axolotl-ai-cloud/axolotl)
+<details><summary>See axolotl config</summary>
+
+axolotl version: `0.5.2`
+```yaml
+base_model: Aratako/Llama-Gemma-2-27b-CPO_SimPO-iter2
+model_type: AutoModelForCausalLM
+tokenizer_type: AutoTokenizer
+
+hub_model_id: Aratako/fft-orpo-iterative-iter3
+hub_strategy: "end"
+push_dataset_to_hub:
+hf_use_auth_token: true
+
+plugins:
+  - axolotl.integrations.liger.LigerPlugin
+liger_cross_entropy: false
+liger_rope: true
+liger_rms_norm: true
+liger_swiglu: true
+liger_fused_linear_cross_entropy: true
+
+load_in_8bit: false
+load_in_4bit: false
+strict: false
+
+chat_template: tokenizer_default
+rl: orpo
+orpo_alpha: 0.1
+max_prompt_len: 512
+max_length: 2560
+
+
+datasets:
+  - path: Aratako/iterative-dpo-data-for-ORPO-iter3
+    type: chat_template.argilla
+    train_on_split: train
+
+
+shuffle_merged_datasets: true
+dataset_prepared_path: /workspace/data/fft-orpo-iterative-iter3-data
+output_dir: /workspace/data/27b-fft-orpo-iterative-iter3
 
-## Training procedure
+sequence_len: 2560
+sample_packing: false
+eval_sample_packing: false
+pad_to_sequence_len: true
 
-[<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="150" height="24"/>](https://wandb.ai/aratako-lm/27b-fft/runs/5o3squgn)
+adapter:
+lora_model_dir:
+lora_r:
+lora_alpha:
+lora_dropout:
+lora_target_linear:
+lora_fan_in_fan_out:
 
-This model was trained with ORPO, a method introduced in [ORPO: Monolithic Preference Optimization without Reference Model](https://huggingface.co/papers/2403.07691).
+wandb_project: 27b-fft
+wandb_entity: aratako-lm
+wandb_watch:
+wandb_name: orpo-iter3
+wandb_log_model:
 
-### Framework versions
+gradient_accumulation_steps: 16
+micro_batch_size: 1
+num_epochs: 1
+optimizer: paged_adamw_8bit
+lr_scheduler: cosine
+cosine_min_lr_ratio: 0.1
+learning_rate: 8e-7
 
-- TRL: 0.12.0
-- Transformers: 4.46.3
-- Pytorch: 2.3.1+cu121
-- Datasets: 3.1.0
-- Tokenizers: 0.20.3
+train_on_inputs: false
+group_by_length: false
+bf16: auto
+fp16:
+tf32: false
 
-## Citations
+gradient_checkpointing: true
+early_stopping_patience:
+auto_resume_from_checkpoints: true
+local_rank:
+logging_steps: 1
+xformers_attention:
+flash_attention: true
 
-Cite ORPO as:
+save_strategy: steps
+save_steps: 100
+save_total_limit: 1
 
-```bibtex
-@article{hong2024orpo,
-    title        = {{ORPO: Monolithic Preference Optimization without Reference Model}},
-    author       = {Jiwoo Hong and Noah Lee and James Thorne},
-    year         = 2024,
-    eprint       = {arXiv:2403.07691}
-}
+warmup_steps: 20
+eval_steps:
+eval_batch_size:
+eval_table_size:
+eval_max_new_tokens:
+debug:
+deepspeed: /workspace/axolotl/deepspeed_configs/zero3_bf16.json
+weight_decay: 0.01
+fsdp:
+fsdp_config:
+special_tokens:
+  pad_token: <pad>
 ```
 
-Cite TRL as:
-    
-```bibtex
-@misc{vonwerra2022trl,
-	title        = {{TRL: Transformer Reinforcement Learning}},
-	author       = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallouédec},
-	year         = 2020,
-	journal      = {GitHub repository},
-	publisher    = {GitHub},
-	howpublished = {\url{https://github.com/huggingface/trl}}
-}
-```
+</details><br>