--- license: llama3.1 base_model: meta-llama/Meta-Llama-3.1-8B tags: - generated_from_trainer model-index: - name: workspace/axolotl/dolphin-2.9.4-llama3.1-8b results: [] --- [Built with Axolotl](https://github.com/axolotl-ai-cloud/axolotl)
See axolotl config axolotl version: `0.4.1` ```yaml base_model: meta-llama/Meta-Llama-3.1-8B model_type: LlamaForCausalLM tokenizer_type: AutoTokenizer load_in_8bit: false # load_in_4bit: true strict: false datasets: - path: /workspace/datasets/dolphin-2.9.4/dolphin201-sharegpt2.jsonl type: sharegpt conversation: chatml chat_template: chatml # adapter: qlora # lora_r: 128 # lora_alpha: 16 # lora_modules_to_save: [embed_tokens, lm_head] # lora_dropout: 0.05 # lora_target_linear: true unfrozen_parameters: - input_layernorm - model.norm - post_attention_layernorm - self_attn.rotary_emb - ^lm_head.weight$ - ^model.embed_tokens.weight$ # mlp.down_proj layers - model.layers.1.mlp.down_proj - model.layers.0.mlp.down_proj - model.layers.30.mlp.down_proj - model.layers.2.mlp.down_proj - model.layers.21.mlp.down_proj - model.layers.22.mlp.down_proj - model.layers.29.mlp.down_proj - model.layers.5.mlp.down_proj - model.layers.4.mlp.down_proj - model.layers.20.mlp.down_proj - model.layers.23.mlp.down_proj - model.layers.19.mlp.down_proj - model.layers.3.mlp.down_proj - model.layers.17.mlp.down_proj - model.layers.6.mlp.down_proj - model.layers.31.mlp.down_proj # mlp.up_proj layers - model.layers.4.mlp.up_proj - model.layers.3.mlp.up_proj - model.layers.0.mlp.up_proj - model.layers.5.mlp.up_proj - model.layers.7.mlp.up_proj - model.layers.6.mlp.up_proj - model.layers.2.mlp.up_proj - model.layers.1.mlp.up_proj - model.layers.8.mlp.up_proj - model.layers.12.mlp.up_proj - model.layers.14.mlp.up_proj - model.layers.9.mlp.up_proj - model.layers.15.mlp.up_proj - model.layers.17.mlp.up_proj - model.layers.13.mlp.up_proj - model.layers.19.mlp.up_proj # self_attn.k_proj layers - model.layers.29.self_attn.k_proj - model.layers.25.self_attn.k_proj - model.layers.23.self_attn.k_proj - model.layers.28.self_attn.k_proj - model.layers.21.self_attn.k_proj - model.layers.19.self_attn.k_proj - model.layers.22.self_attn.k_proj - model.layers.20.self_attn.k_proj - model.layers.24.self_attn.k_proj - model.layers.31.self_attn.k_proj - model.layers.27.self_attn.k_proj - model.layers.26.self_attn.k_proj - model.layers.17.self_attn.k_proj - model.layers.11.self_attn.k_proj - model.layers.18.self_attn.k_proj - model.layers.14.self_attn.k_proj # self_attn.o_proj layers - model.layers.14.self_attn.o_proj - model.layers.7.self_attn.o_proj - model.layers.5.self_attn.o_proj - model.layers.11.self_attn.o_proj - model.layers.6.self_attn.o_proj - model.layers.24.self_attn.o_proj - model.layers.9.self_attn.o_proj - model.layers.13.self_attn.o_proj - model.layers.10.self_attn.o_proj - model.layers.12.self_attn.o_proj - model.layers.8.self_attn.o_proj - model.layers.25.self_attn.o_proj - model.layers.21.self_attn.o_proj - model.layers.23.self_attn.o_proj - model.layers.15.self_attn.o_proj - model.layers.16.self_attn.o_proj # self_attn.q_proj layers - model.layers.8.self_attn.q_proj - model.layers.13.self_attn.q_proj - model.layers.9.self_attn.q_proj - model.layers.14.self_attn.q_proj - model.layers.10.self_attn.q_proj - model.layers.11.self_attn.q_proj - model.layers.0.self_attn.q_proj - model.layers.15.self_attn.q_proj - model.layers.1.self_attn.q_proj - model.layers.6.self_attn.q_proj - model.layers.5.self_attn.q_proj - model.layers.7.self_attn.q_proj - model.layers.12.self_attn.q_proj - model.layers.16.self_attn.q_proj - model.layers.17.self_attn.q_proj - model.layers.26.self_attn.q_proj # self_attn.v_proj layers - model.layers.26.self_attn.v_proj - model.layers.17.self_attn.v_proj - model.layers.3.self_attn.v_proj - model.layers.28.self_attn.v_proj - model.layers.29.self_attn.v_proj - model.layers.21.self_attn.v_proj - model.layers.15.self_attn.v_proj - model.layers.16.self_attn.v_proj - model.layers.20.self_attn.v_proj - model.layers.25.self_attn.v_proj - model.layers.6.self_attn.v_proj - model.layers.23.self_attn.v_proj - model.layers.4.self_attn.v_proj - model.layers.1.self_attn.v_proj - model.layers.22.self_attn.v_proj - model.layers.14.self_attn.v_proj # mlp.gate_proj layers - model.layers.1.mlp.gate_proj - model.layers.2.mlp.gate_proj - model.layers.3.mlp.gate_proj - model.layers.4.mlp.gate_proj - model.layers.0.mlp.gate_proj - model.layers.25.mlp.gate_proj - model.layers.26.mlp.gate_proj - model.layers.5.mlp.gate_proj - model.layers.24.mlp.gate_proj - model.layers.28.mlp.gate_proj - model.layers.23.mlp.gate_proj - model.layers.27.mlp.gate_proj - model.layers.21.mlp.gate_proj - model.layers.22.mlp.gate_proj - model.layers.29.mlp.gate_proj - model.layers.20.mlp.gate_proj dataset_prepared_path: /workspace/axolotl/dolph-2.9.4-nemo-prepared val_set_size: 0.01 output_dir: /workspace/axolotl/dolphin-2.9.4-llama3.1-8b sequence_len: 8192 sample_packing: true pad_to_sequence_len: true wandb_project: dolphin-2.9.4-llama3.1-8b wandb_watch: wandb_run_id: wandb_log_model: gradient_accumulation_steps: 16 micro_batch_size: 2 num_epochs: 3 optimizer: adamw_torch lr_scheduler: cosine learning_rate: 5e-6 train_on_inputs: false group_by_length: false bf16: auto fp16: tf32: gradient_checkpointing: true gradient_checkpointing_kwargs: use_reentrant: false early_stopping_patience: resume_from_checkpoint: logging_steps: 1 xformers_attention: flash_attention: true warmup_steps: 100 # evals_per_epoch: 4 eval_table_size: saves_per_epoch: 1 save_total_limit: 2 save_steps: debug: deepspeed: deepspeed_configs/zero3_bf16.json weight_decay: 0.1 special_tokens: eos_token: "<|im_end|>" bos_token: "<|begin_of_text|>" pad_token: "<|finetune_right_pad_id|>" tokens: - "<|im_start|>" # fsdp: # - full_shard # - auto_wrap # fsdp_config: # fsdp_limit_all_gathers: true # fsdp_sync_module_states: true # fsdp_offload_params: true # fsdp_use_orig_params: false # fsdp_cpu_ram_efficient_loading: true # fsdp_transformer_layer_cls_to_wrap: MixtralSparseMoeBlock # fsdp_state_dict_type: FULL_STATE_DICT # fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP # fsdp_sharding_strategy: FULL_SHARD # fsdp_forward_prefetch: false # fsdp_backward_prefetch: BACKWARD_PRE ```

# workspace/axolotl/dolphin-2.9.4-llama3.1-8b This model is a fine-tuned version of [meta-llama/Meta-Llama-3.1-8B](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B) on the None dataset. It achieves the following results on the evaluation set: - Loss: 0.5655 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 5e-06 - train_batch_size: 2 - eval_batch_size: 2 - seed: 42 - distributed_type: multi-GPU - num_devices: 8 - gradient_accumulation_steps: 16 - total_train_batch_size: 256 - total_eval_batch_size: 16 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: cosine - lr_scheduler_warmup_steps: 100 - num_epochs: 3 ### Training results | Training Loss | Epoch | Step | Validation Loss | |:-------------:|:------:|:----:|:---------------:| | 0.5837 | 1.0180 | 1161 | 0.5814 | | 0.5525 | 2.0179 | 2322 | 0.5671 | | 0.5514 | 2.9624 | 3420 | 0.5655 | ### Framework versions - Transformers 4.44.0.dev0 - Pytorch 2.4.0+cu121 - Datasets 2.19.1 - Tokenizers 0.19.1