mohamedbouhali/finetuned_gemma_3_1b_lora32

Model Card for Model ID

This repository provides a LoRA rank-32 adapter for google/gemma-3-1b-it , fine-tuned on the CounselChat dataset of therapist Q&A. The model is designed to generate short, empathetic, counseling-style responses while being lightweight enough for local/on-premise deployment in privacy-sensitive environments (e.g., clinics).

⚠️ Disclaimer: This model is for research purposes only. It is not a replacement for therapy, nor has it been clinically validated. Do not use it for medical decision-making.

Model Details

Lightweight LoRA rank-32 adapter trained on therapist Q&A from CounselChat to make google/gemma-3-1b-it more responsive for short, task-oriented counseling prompts. This repo contains only the adapter; load it on top of the base model. :contentReference[oaicite:0]{index=0}

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

base_id    = "google/gemma-3-1b-it"
adapter_id = "mohamedbouhali/finetuned_gemma_3_1b_lora32"

tok = AutoTokenizer.from_pretrained(base_id)
base = AutoModelForCausalLM.from_pretrained(base_id, torch_dtype="auto")
model = PeftModel.from_pretrained(base, adapter_id)

prompt = "How can I avoid thinking much?,I start thinking deeply about everything I may do or say and about anything that may happen. I really want to avoid it since it really bothers me."
chat = tok.apply_chat_template([{"role": "user", "content": prompt}], tokenize=False, add_generation_prompt=True)
inputs = tok(chat, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=200, temperature=0.7)
print(tok.decode(out[0], skip_special_tokens=True))

Model Details

• Developed by: Mohamed Bouhali, Nikhil Sharma, Mohammad Zainnuddin
• Funded by [optional]: Chair of Health and Informatics of Technical University of Munich
• Language(s) (NLP): English
• Finetuned from model [optional]: Gemma-3-1b-it

Training

• Dataset: 863 therapist Q&A pairs from CounselChat, cleaned and stratified

• Split: ~80/20 by topic (rare topics only in train set)

• Training setup:

• Epoch :2 (43 steps), AdamW optimizer

• Learning rate: 1e-4 (cosine schedule)

• Batch size: 1 (grad accumulation = 16)

• LoRA rank explored: {4, 8, 16, 32}

• Mixed precision (fp16), single NVIDIA GTX Titan X

• This release: LoRA-32 with chain-of-thought prompting — best on overlap metrics (BLEU-1/4, METEOR)

Limitations:

• Dataset is small (863 pairs) → may not generalize well across all therapy domains.

• Empathy metric is automatic, not clinically validated.

• Performance varies by topic; sensitive domains (e.g., LGBTQ, legal) remain weaker.

Citation

@online{bouhali2025empathy, title = {Analyzing Empathy in Counseling Conversations with Finetuned Small Language Models}, author = {Bouhali, Mohamed and Zainuddin, Mohammad and Sharma, Nikhil}, year = {2025}, url = {https://huggingface.co/mohamedbouhali/finetuned_gemma_3_1b_lora32}, note = {Code: https://github.com/mohamedbouhali/PSLM} }