nl-to-logql/gemma-2-logql

See axolotl config

axolotl version: 0.4.1

base_model: google/gemma-2-9b

sequence_len: 1024

# base model weight quantization
load_in_8bit: true
# load_in_4bit: true

# attention implementation
flash_attention: true

# finetuned adapter config
adapter: lora
lora_model_dir:
lora_r: 16
lora_alpha: 32
lora_dropout: 0.05
lora_target_linear: true
lora_fan_in_fan_out:
lora_modules_to_save: # required when adding new tokens to LLaMA/Mistral
  - embed_tokens
  - lm_head

# if training fails, uncomment above
# for details, see https://github.com/huggingface/peft/issues/334#issuecomment-1561727994

###
# Dataset Configuration: sqlqa
###
# datasets:
#   - path: data.jsonl
#     type: alpaca

datasets:
  - path: public_train_data.jsonl
    ds_type: json
    type:
      field_instruction: instruction
      field_input: input
      field_output: output
      format: |-
        [INST] {instruction}
        {input} [/INST]

chat_template: gemma
tokens:
  - "[INST]"
  - " [/INST]"
  - "[QL]"
  - " [/QL]"
  - "[EXPLANATION]"
  - " [/EXPLANATION]"
# dataset formatting config

special_tokens:
  pad_token: <|end_of_text|>

val_set_size: 0.05

###
# Training Configuration
###

# masks the input messages so that the model learns and understands the language w/o being reliant on the input
train_on_inputs: false
# random seed for better reproducibility
seed: 117

# optimizer config
optimizer: adamw_bnb_8bit
learning_rate: 0.0001
lr_scheduler: cosine
num_epochs: 4
micro_batch_size: 4
gradient_accumulation_steps: 1
warmup_steps: 10

# axolotl saving config
dataset_prepared_path: last_run_prepared
output_dir: ./lora-out

# logging and eval config
logging_steps: 1
eval_steps: 0.05

# training performance optimization config
bf16: auto
tf32: false
gradient_checkpointing: true

###
# Miscellaneous Configuration
###

# when true, prevents over-writing the config from the CLI
strict: false

# "Don't mess with this, it's here for accelerate and torchrun" -- axolotl docs
local_rank:

# WANDB
wandb_mode:
wandb_project:
wandb_watch:
wandb_name:
wandb_run_id:

# Multi-GPU
# deepspeed: /root/axolotl/deepspeed_configs/zero3_bf16.json
# deepspeed: zero3_bf16.json
# deepspeed: /workspace/axolotl/deepspeed_configs/zero2.json
deepspeed:
fsdp:
fsdp_config:

lora-out

This model is a fine-tuned version of google/gemma-2-9b on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.0077

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 4
• eval_batch_size: 4
• seed: 117
• distributed_type: multi-GPU
• num_devices: 4
• total_train_batch_size: 16
• total_eval_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 10
• num_epochs: 4

Training results

Training Loss	Epoch	Step	Validation Loss
1.7925	0.0385	1	2.0412
1.6872	0.2308	6	1.6089
0.6967	0.4615	12	0.6328
0.3327	0.6923	18	0.2711
0.1784	0.9231	24	0.1733
0.1136	1.1538	30	0.1190
0.0891	1.3846	36	0.0850
0.0746	1.6154	42	0.0626
0.0522	1.8462	48	0.0465
0.033	2.0769	54	0.0282
0.0333	2.3077	60	0.0225
0.0171	2.5385	66	0.0203
0.0172	2.7692	72	0.0144
0.0095	3.0	78	0.0119
0.0088	3.2308	84	0.0099
0.0054	3.4615	90	0.0089
0.0073	3.6923	96	0.0085
0.0059	3.9231	102	0.0077

Framework versions

• PEFT 0.13.0
• Transformers 4.45.1
• Pytorch 2.3.1+cu121
• Datasets 2.21.0
• Tokenizers 0.20.0