File size: 1,604 Bytes
24580da c39c842 a093431 c39c842 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 |
---
license: apache-2.0
language:
- en
base_model:
- meta-llama/Llama-3.2-11B-Vision-Instruct
---
## Introduction
This model originates from [Xkev/Llama-3.2V-11B-cot](https://huggingface.co/Xkev/Llama-3.2V-11B-cot). This repository simply quantizes the model into the NF4 format using the bitsandbytes library.
All credit goes to the original repository.
## Usage
```
from transformers import MllamaForConditionalGeneration, AutoProcessor, BitsAndBytesConfig
from PIL import Image
import time
# Load model
model_id = "zhangsongbo365/Llama-3.2V-11B-cot-nf4"
model = MllamaForConditionalGeneration.from_pretrained(
model_id,
use_safetensors=True,
device_map="cuda:0",
trust_remote_code=True
)
# Load tokenizer
processor = AutoProcessor.from_pretrained(model_id)
# Caption a local image
IMAGE = Image.open("1.png").convert("RGB") # 改为你的实际图片路径
PROMPT = """<|begin_of_text|><|start_header_id|>user<|end_header_id|>
Caption this image:
<|image|><|eot_id|><|start_header_id|>assistant<|end_header_id|>
"""
inputs = processor(IMAGE, PROMPT, return_tensors="pt").to(model.device)
prompt_tokens = len(inputs['input_ids'][0])
print(f"Prompt tokens: {prompt_tokens}")
t0 = time.time()
generate_ids = model.generate(**inputs, max_new_tokens=256)
t1 = time.time()
total_time = t1 - t0
generated_tokens = len(generate_ids[0]) - prompt_tokens
time_per_token = generated_tokens/total_time
print(f"Generated {generated_tokens} tokens in {total_time:.3f} s ({time_per_token:.3f} tok/s)")
output = processor.decode(generate_ids[0][prompt_tokens:]).replace('<|eot_id|>', '')
```
|