README.md

---
{}
---
# Model Card for Fine-Tuned Paligemma-3B-PT-224 Model

This model is a fine-tuned version of `google/paligemma-3b-pt-224` using the `peft` library. The fine-tuning process involved the `Multimodal-Fatima/VQAv2_sample_train` dataset, focusing on vision-language tasks.

## Model Details

### Model Description

This model is designed for vision-language tasks, fine-tuned to answer questions based on images and textual prompts. It leverages advanced quantization techniques and specific configurations to optimize performance and efficiency.

- **Developed by:** [AmmarAbdelhady](https://ammar-abdelhady-ai.github.io/Ammar-Abdelhady-Portfolio/)
- **Model type:** Vision-Language Model
- **Language(s) (NLP):** English
- **Finetuned from model:** `google/paligemma-3b-pt-224`

### Model Sources

- **Repository:** [Vision-Language-Model-Fine-Tuning Notebook](https://github.com/Ammar-Abdelhady-ai/Vision-Language-Model-Fine-Tuning/blob/main/fine-tuning-of-paligemma-vision-language-model.ipynb)
- **Demo:** [Vision-Language-Model-Fine-Tuning](https://github.com/Ammar-Abdelhady-ai/Vision-Language-Model-Fine-Tuning)

## Uses

### Direct Use

This model can be used directly for vision-language tasks, including image captioning and visual question answering.

### Downstream Use

The model can be fine-tuned further for specific tasks or integrated into larger systems requiring vision-language capabilities.

### Out-of-Scope Use

The model is not suitable for tasks unrelated to vision-language processing, such as purely text-based or purely image-based tasks without multimodal interaction.

## Bias, Risks, and Limitations

The model may inherit biases from the training dataset, particularly in terms of visual and textual content. It is crucial to evaluate and mitigate these biases in downstream applications.

### Recommendations

Users should be aware of the model's limitations and potential biases. It is recommended to perform thorough evaluations on diverse datasets to understand the model's performance across different scenarios.

## How to Get Started with the Model

```python
from transformers import PaliGemmaForConditionalGeneration, PaliGemmaProcessor
import torch
from PIL import Image
import requests

model = PaliGemmaForConditionalGeneration.from_pretrained('your_model_path')
processor = PaliGemmaProcessor.from_pretrained('your_model_path')

prompt = "What is on the flower?"
image_url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg?download=true"
raw_image = Image.open(requests.get(image_url, stream=True).raw)
inputs = processor(prompt, raw_image, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=20)

print(processor.decode(output[0], skip_special_tokens=True))