--- license: cc-by-nc-nd-4.0 library_name: peft tags: - generated_from_trainer base_model: rizla/rizla-17 model-index: - name: lorazapam-out results: [] --- [Built with Axolotl](https://github.com/OpenAccess-AI-Collective/axolotl)
See axolotl config axolotl version: `0.4.0` ```yaml base_model: rizla/rizla-17 model_type: AutoModelForCausalLM tokenizer_type: LlamaTokenizer trust_remote_code: true load_in_8bit: false load_in_4bit: true strict: false datasets: - path: meta-math/MetaMathQA-40K type: system_prompt: "You are an expert problem solver who is great at teaching how to solve problems via first principles reasoning" field_system: system field_instruction: query field_output: response format: "[INST] {instruction} [/INST]" no_input_format: "[INST] {instruction} [/INST]" dataset_prepared_path: last_run_prepared val_set_size: 0.0 output_dir: ./lorazapam-out ## You can optionally freeze the entire model and unfreeze a subset of parameters # - lm_head.* # - model.embed_tokens.* # - model.layers.2[0-9]+.block_sparse_moe.gate.* # - model.layers.2[0-9]+.block_sparse_moe.experts.* # - model.layers.3[0-9]+.block_sparse_moe.gate.* # - model.layers.3[0-9]+.block_sparse_moe.experts.* model_config: output_router_logits: true adapter: qlora lora_model_dir: sequence_len: 512 sample_packing: true pad_to_sequence_len: true lora_r: 32 lora_alpha: 16 lora_dropout: 0.05 lora_target_linear: true lora_fan_in_fan_out: wandb_project: wandb_entity: wandb_watch: wandb_name: wandb_log_model: gradient_accumulation_steps: 2 micro_batch_size: 16 num_epochs: 1 optimizer: adamw_bnb_8bit lr_scheduler: cosine learning_rate: 0.0002 train_on_inputs: false group_by_length: false bf16: true fp16: tf32: false gradient_checkpointing: true early_stopping_patience: resume_from_checkpoint: local_rank: logging_steps: 1 xformers_attention: false flash_attention: true loss_watchdog_threshold: 5.0 loss_watchdog_patience: 3 warmup_steps: 10 evals_per_epoch: 1 eval_table_size: eval_table_max_new_tokens: 128 saves_per_epoch: 1 debug: # deepspeed: deepspeed_configs/zero_1.json weight_decay: 0.0 fsdp: fsdp_config: special_tokens: bos_token: "" eos_token: "" unk_token: "" ```

# lorazapam-out This model is a fine-tuned version of [rizla/rizla-17](https://huggingface.co/rizla/rizla-17) on the None dataset. ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 0.0002 - train_batch_size: 16 - eval_batch_size: 16 - seed: 42 - distributed_type: multi-GPU - gradient_accumulation_steps: 2 - total_train_batch_size: 32 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: cosine - lr_scheduler_warmup_steps: 10 - num_epochs: 1 ### Training results ### Framework versions - PEFT 0.8.2 - Transformers 4.38.0.dev0 - Pytorch 2.1.2+cu121 - Datasets 2.16.1 - Tokenizers 0.15.0