File size: 2,089 Bytes
336b5d2 9d3c7cd 336b5d2 e872e5e 9d3c7cd e872e5e |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 |
---
license: mit
pipeline_tag: image-text-to-text
---
## CoLLaVO model
This repository contains the weights of the model presented in [CoLLaVO: Crayon Large Language and Vision mOdel](https://huggingface.co/papers/2402.11248).
### Simple running code is based on [MoAI-Github](https://github.com/ByungKwanLee/MoAI).
You need only the following seven steps.
### Simple running code is based on [CoLLaVO-Github](https://github.com/ByungKwanLee/CoLLaVO).
You need only the following seven steps.
### [0] Download Github Code of CoLLaVO, install the required libraries, set the necessary environment variable (README.md explains in detail! Don't Worry!).
```bash
git clone https://github.com/ByungKwanLee/CoLLaVO
bash install
```
### [1] Loading Image
```python
from PIL import Image
from torchvision.transforms import Resize
from torchvision.transforms.functional import pil_to_tensor
image_path = "figures/crayon_image.jpg"
image = Resize(size=(490, 490), antialias=False)(pil_to_tensor(Image.open(image_path)))
```
### [2] Instruction Prompt
```python
prompt = "Describe this image in detail."
```
### [3] Loading CoLlaVO
```python
from collavo.load_collavo import prepare_collavo
collavo_model, collavo_processor, seg_model, seg_processor = prepare_collavo(collavo_path='BK-Lee/CoLLaVO-7B', bits=4, dtype='fp16')
```
### [4] Pre-processing for CoLLaVO
```python
collavo_inputs = collavo_model.demo_process(image=image,
prompt=prompt,
processor=collavo_processor,
seg_model=seg_model,
seg_processor=seg_processor,
device='cuda:0')
```
### [5] Generate
```python
import torch
with torch.inference_mode():
generate_ids = collavo_model.generate(**collavo_inputs, do_sample=True, temperature=0.9, top_p=0.95, max_new_tokens=256, use_cache=True)
```
### [6] Decoding
```python
answer = collavo_processor.batch_decode(generate_ids, skip_special_tokens=True)[0].split('[U')[0]
print(answer)
``` |