|
--- |
|
license: apache-2.0 |
|
language: |
|
- en |
|
pipeline_tag: Multimodal Small Language Model, Phi-2, VQA |
|
--- |
|
# :smiling_imp: IMP |
|
|
|
The :smiling_imp: IMP project aims to provide a family of a strong multimodal `small` language models (MSLMs). Our `IMP-v0-3B` model is a strong MSLM with only **3B** parameters, which is build upon a small yet powerful SLM [Phi-2 ](https://huggingface.co/microsoft/phi-2)(2.7B) and a powerful visual encoder [SigLIP ](https://huggingface.co/google/siglip-so400m-patch14-384)(0.4B), and trained on the [LLaVA-v1.5](https://github.com/haotian-liu/LLaVA) training set. |
|
|
|
As shown in the Table below, `IMP-v0-3B` significantly outperforms the counterparts of similar model sizes, and even achieves slightly better performance than the strong LLaVA-7B model on various multimodal benchmarks. |
|
|
|
We release our model weights and provide an example below to run our model . Detailed technical report and corresponding training/evaluation code will be released soon on our [GitHub repo](https://github.com/MILVLG/imp). We will persistently improve our model and release the next versions to further improve model performance :) |
|
|
|
## How to use |
|
|
|
You can use the following code for model inference. We minimize the required dependency libraries that only the `transformers` and `torch` packages are used. The format of text instructions is similar to [LLaVA](https://github.com/haotian-liu/LLaVA). |
|
|
|
```Python |
|
import torch |
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
from PIL import Image |
|
|
|
torch.set_default_device("cuda") |
|
|
|
#Create model |
|
model = AutoModelForCausalLM.from_pretrained( |
|
"milvlg/imp-v0", |
|
torch_dtype=torch.float16, |
|
device_map="auto", |
|
trust_remote_code=True) |
|
tokenizer = AutoTokenizer.from_pretrained("milvlg/imp-v0", trust_remote_code=True) |
|
|
|
#Set inputs |
|
text = "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: <image>\nWhat are the colors of the bus in the image? ASSISTANT:" |
|
image = Image.open("images/bus.jpg") |
|
|
|
input_ids = tokenizer(text, return_tensors='pt').input_ids |
|
image_tensor = model.process_images([image]) |
|
|
|
#Generate the answer |
|
output_ids = model.generate( |
|
input_ids, |
|
max_new_tokens=100, |
|
images=image_tensor, |
|
use_cache=True)[0] |
|
print(tokenizer.decode(output_ids[input_ids.shape[1]:], skip_special_tokens=True).strip()) |
|
``` |
|
|
|
## Model evaluation |
|
We perform evaluation on 8 commonly-used benchmarks to validate the effectiveness of our model, including 5 academic VQA benchmarks and 3 recent MLLM benchmarks. |
|
|
|
| Models | Size | VQAv2 | GQA |VisWiz | SQA (IMG) | TextVQA | POPE | MME | MMB |MM-Vet| |
|
|:--------:|:-----:|:----:|:----:|:-------------:|:--------:|:-----:|:----:|:-------:|:-------:|:-------:| |
|
| [LLaVA-v1.5-lora](https://huggingface.co/liuhaotian/llava-v1.5-7b) | 7B |79.10 | **63.00** |47.80 | 68.40 |58.20| 86.40 | **1476.9** | 66.10 |30.2| |
|
| [TinyGPT-V](https://huggingface.co/Tyrannosaurus/TinyGPT-V) | 3B | - | 33.60 | 24.80 | - | - | -| - | - |-| |
|
| [LLaVA-Phi](https://arxiv.org/pdf/2401.02330.pdf) | 3B | 71.40 | - | 35.90 | 68.40 | 48.60 | 85.00 | 1335.1 | 59.80 |28.9| |
|
| [MobileVLM](https://huggingface.co/mtgv/MobileVLM-3B) | 3B | - | 59.00 | - | 61.00 | 47.50 | 84.90 | 1288.9 | 59.60 |-| |
|
| [MC-LLaVA-3b](https://huggingface.co/visheratin/MC-LLaVA-3b) | 3B | 64.24 | 49.6 | 24.88 | - | 38.59 | 80.59 | - | - |-| |
|
| **IMP-v0 (ours)** | 3B | **79.45** | 58.55 | **50.09** |**69.96**| **59.38** | **88.02**| 1434 | **66.49** |**33.1**| |
|
|
|
|
|
## License |
|
This project is licensed under the Apache License 2.0 - see the [LICENSE](https://www.apache.org/licenses/LICENSE-2.0) file for details. |
|
|
|
## About us |
|
Project :smiling_imp: IMP is maintained by the [MILVLG](https://github.com/MILVLG) group led by Prof. Zhou Yu and Jun Yu, and mainly developed by Zhenwei Shao and Xuecheng Ouyang. We hope our model may server as a strong baseline to inspire future research on MSLMs and derivative applications on mobile devices and robotics. |
