Hugging Face's logo

kdqemre
/
SmolVLM-Base-vqav2

PEFT
TensorBoard
Safetensors
Generated from Trainer
Model card Files Metrics Community
SmolVLM-Base-vqav2
File size: 3,468 Bytes 
712dcdc
 
 
 
 
 
 
 
 
 
55bf572
 
 
 
 
 
 
 
 
 
eab1e49
55bf572
712dcdc
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
55bf572
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
 
---
library_name: peft
license: apache-2.0
base_model: HuggingFaceTB/SmolVLM-Base
tags:
- generated_from_trainer
model-index:
- name: SmolVLM-Base-vqav2
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# SmolVLM-Base-vqav2

This model is a fine-tuned version of [HuggingFaceTB/SmolVLM-Base](https://huggingface.co/HuggingFaceTB/SmolVLM-Base) on an unknown dataset.

## Model description

Here is the sample code for how to use.

```python
from transformers import AutoProcessor, AutoModelForImageTextToText
from peft import PeftModel
import torch
from PIL import Image


DEVICE = "cuda:0" if torch.cuda.is_available() else "cpu" ### DEVICE = "cuda:0" instead of  DEVICE = "cuda"  it fixes flash attention warning!!


model_id = "HuggingFaceTB/SmolVLM-Instruct"  # Base Model

base_model = AutoModelForImageTextToText.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    _attn_implementation="flash_attention_2" if DEVICE == "cuda" else "eager"
).to(DEVICE)

print(f"Model is on device: {base_model.device}")


# QLoRA adapter
adapter_path = r"C:\Users\.....\SmolVLM-Base-vqav2\checkpoint-670"
model = PeftModel.from_pretrained(base_model, adapter_path)

model = model.to(DEVICE)  # Check the model device #####################################

# Load the processor
processor = AutoProcessor.from_pretrained(model_id)

# Functıon for load images from local
def load_image_from_file(file_path):
    try:
        image = Image.open(file_path)
        return image
    except Exception as e:
        print(f"Error loading image: {e}")
        return None


image1_path = "C:/Users/.../IMG_4.jpg"
image2_path = "C:/Users/.../IMG_35.jpg"

# Load images
image1 = load_image_from_file(image1_path)
image2 = load_image_from_file(image2_path)

# Check the images
if image1 and image2:

    # Create message type
    messages = [
        {
            "role": "user",
            "content": [
                {"type": "image"},
                {"type": "image"},
                {"type": "text", "text": "Can you describe and compare the two images?"}
            ]
        },
    ]
    
    # Prepare the Prompt

    prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
    inputs = processor(text=prompt, images=[image1, image2], return_tensors="pt")
    inputs = inputs.to(DEVICE)
    
    # Run the model
    generated_ids = model.generate(**inputs, max_new_tokens=500)
    generated_texts = processor.batch_decode(generated_ids, skip_special_tokens=True)

    # Print the result
    print(generated_texts[0])  # Çıktı

else:
    print("Images can not be loaded")


```

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0001
- train_batch_size: 4
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 16
- optimizer: Use adamw_hf with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 50
- num_epochs: 1

### Training results



### Framework versions

- PEFT 0.14.0
- Transformers 4.46.3
- Pytorch 2.5.1+cu121
- Datasets 3.1.0
- Tokenizers 0.20.3