BrunoGR/JUST_HEAR_ME-PEFT_Adapter

Sólo Escúchame: PEFT Adapter

This model is a fine-tuned version of meta-llama/Llama-2-7b-chat-hf on BrunoGR/HEAR-Hispanic_Emotional_Accompaniment_Responses. It achieves the following results on the evaluation set:

• Loss: 0.8265

Model Description

Sólo Escúchame is an emotional accompaniment chatbot designed to provide psychological support in Spanish. This model, built upon the LLaMA-2-7b-Chat architecture, has been fine-tuned to deliver empathetic responses tailored to users' emotional states.

Intended Uses & Limitations

This model is intended for use in providing emotional support and companionship. It is not a substitute for professional mental health services. Users should be aware of its limitations in handling severe psychological conditions and emergencies.

Training and Evaluation Data

The model was trained using the "BrunoGR/HEAR-Hispanic_Emotional_Accompaniment_Responses" dataset, which includes diverse emotional responses to simulate a wide range of psychological support scenarios.

Training Procedure

Training Hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 5
• eval_batch_size: 8
• seed: 1
• gradient_accumulation_steps: 3
• total_train_batch_size: 15
• optimizer: Adam with betas=(0.9,0.95) and epsilon=2e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 300
• num_epochs: 3

LoRA Parameters:

• r: 64
• Alpha: 128
• LoRA_DROPOUT: 0.1
• bias: "none"
• task_type: "CAUSAL_LM"
• LoRA_TARGET_MODULES = ["q_proj", "v_proj"]

Training Results

Training Loss	Epoch	Step	Validation Loss
0.7717	0.16	450	0.9065
0.7716	0.32	900	0.8863
0.7856	0.48	1350	0.8679
0.7463	0.64	1800	0.8602
0.7298	0.81	2250	0.8531
0.7426	0.97	2700	0.8415
0.6991	1.13	3150	0.8425
0.6923	1.29	3600	0.8382
0.6917	1.45	4050	0.8361
0.6681	1.61	4500	0.8328
0.6764	1.77	4950	0.8297
0.6684	1.93	5400	0.8275
0.6395	2.09	5850	0.8268
0.6735	2.25	6300	0.8272
0.6416	2.42	6750	0.8275
0.651	2.58	7200	0.8265
0.6514	2.74	7650	0.8265
0.6535	2.9	8100	0.8265

Framework Versions

• Transformers 4.35.2
• Pytorch 2.1.1+cu121
• Datasets 2.16.0
• Tokenizers 0.15.0
• torch: 2.1.1
• trl: 0.7.5
• tokenizers: 0.15.0
• peft: 0.7.0
• evaluate: 0.4.1
• accelerate: 0.25.0
• seaborn: 0.13.0

License

Sólo Escúchame is released under the MIT License.

Citation

If you use this Adapter in your research, please cite the following paper:

@misc{ramírez2024soloescuchamespanishemotional,
      title={S\'olo Esc\'uchame: Spanish Emotional Accompaniment Chatbot}, 
      author={Bruno Gil Ramírez and Jessica López Espejel and María del Carmen Santiago Díaz and Gustavo Trinidad Rubín Linares},
      year={2024},
      eprint={2408.01852},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2408.01852}, 
}

Contact

For any questions or inquiries, please contact:

• Bruno Gil Ramírez: bruno.gilram@gmail.com