README.md

---
base_model: llava-hf/llava-v1.6-mistral-7b-hf
library_name: peft
license: apache-2.0
datasets:
- mirzaei2114/stackoverflowVQA-filtered-small
language:
- en
tags:
- llava
- llava-next
- fine-tuned
- stack-overflow
- qlora
- images
- vqa
- 4bit
---

# Model Card for Model ID

Finetuned LLaVA-Next model for Visual QA on Stack Overflow questions with images.

## Model Details

### Model Description

This model is a finetuned version of **LLaVA-Next (llava-hf/llava-v1.6-mistral-7b-hf)** specifically for visual question answering (VQA)
on Stack Overflow questions containing images. The model was finetuned using **QLoRA** with 4-bit quantization, optimized to handle both
text and image inputs.

The training dataset was filtered from the **mirzaei2114/stackoverflowVQA-filtered-small** dataset.
Only samples with a maximum input length of 1024 (for both question and answer combined) were used. Images were kept
to size to capture detail needed for methods such as optical character recognition.



- **Developed by:** Adam Cassidy
- **Model type:** Visual QA
- **Language(s) (NLP):** EN
- **License:** Apache License, Version 2.0
- **Finetuned from model:** llava-hf/llava-v1.6-mistral-7b-hf

### Model Sources

- **Repository:** [llava-hf/llava-v1.6-mistral-7b-hf](https://huggingface.co/llava-hf/llava-v1.6-mistral-7b-hf)

## Uses

Drag a snipping rectangle for a screenshot around the exact focus/context for a question related to software development(usually front end)
and accompany it with the question for inference.

### Direct Use

Visual Question Answering (VQA) on technical Stack Overflow (software-adjacent) questions with accompanying images.

### Out-of-Scope Use

General-purpose VQA tasks, though performance on non-technical domains may vary.

## Bias, Risks, and Limitations

Model Capacity: The model was trained using 4-bit QLoRA.
Dataset Size: The training dataset is relatively small, and this may impact generalization to other VQA datasets or domains outside of Stack Overflow.

## How to Get Started with the Model

To use this model, ensure you have the following dependencies installed:
torch==2.4.1+cu121
transformers==4.45.1

Do inference according to this multi-image inference llava-next example: https://huggingface.co/docs/transformers/en/model_doc/llava_next#:~:text=skip_special_tokens%3DTrue))-,Multi%20image%20inference,-LLaVa%2DNext%20can

## Training Details

### Training Data

[mirzaei2114/stackoverflowVQA-filtered-small](https://huggingface.co/datasets/mirzaei2114/stackoverflowVQA-filtered-small/viewer/default/train)

### Training Procedure

#### Training Hyperparameters

  TrainingArguments(
      per_device_train_batch_size=4,
      per_device_eval_batch_size=4,
      max_grad_norm=0.1,
      evaluation_strategy="steps",
      eval_steps=15,
      group_by_length=True,
      logging_steps=15,
      gradient_checkpointing=True,
      gradient_accumulation_steps=2,
      num_train_epochs=3,
      weight_decay=0.1,
      warmup_steps=10,
      lr_scheduler_type="cosine",
      learning_rate=1e-5,
      save_steps=15,
      save_total_limit=5,
      bf16=True,
      remove_unused_columns=False
  )

#### Speeds, Sizes, Times

checkpoint-240

## Evaluation

Evaluation Loss (Pre-finetuning): 2.93
Validation Loss (Post-finetuning): 1.78

### Testing Data, Factors & Metrics

#### Testing Data

[mirzaei2114/stackoverflowVQA-filtered-small](https://huggingface.co/datasets/mirzaei2114/stackoverflowVQA-filtered-small/viewer/default/test)

### Compute Infrastructure

#### Hardware

L4 GPU

#### Software

Google Colab

### Framework versions

- PEFT 0.13.1.dev0
- PyTorch 2.4.1+cu121
- Transformers 4.45.1