| # VisCPM | |
| [简体中文](README.md) | English | |
| <p align="center"> | |
| <p align="left"> | |
| <a href="./LICENSE"><img src="https://img.shields.io/badge/license-Apache%202-dfd.svg"></a> | |
| <a href=""><img src="https://img.shields.io/badge/python-3.8+-aff.svg"></a> | |
| </p> | |
| `VisCPM` is a family of open-source large multimodal models, which support multimodal conversational capabilities (`VisCPM-Chat` model) and text-to-image generation capabilities (`VisCPM-Paint` model) in both Chinese and English, achieving state-of-the-art peformance among Chinese open-source multimodal models. `VisCPM` is trained based on the large language model [CPM-Bee](https://github.com/OpenBMB/CPM-Bee) with 10B parameters, fusing visual encoder (`Q-Former`) and visual decoder (`Diffusion-UNet`) to support visual inputs and outputs. Thanks to the good bilingual capability of `CPM-Bee`, `VisCPM` can be pre-trained with English multimodal data only and well generalize to achieve promising Chinese multimodal capabilities. | |
| ## VisCPM-Chat | |
| `VisCPM-Chat` supports bilingual multimodal conversations involving images in both Chinese and English. The model utilizes `Q-Former` as the visual encoder and CPM-Bee (10B) as the base LLM. It combines visual and language models through language modeling training objectives. The model training consists of two stages: pretraining and instruction fine-tuning. | |
| * Pretrain: `VisCPM-Chat` was pretrained using approximately 100 million high-quality English multimodal data pairs. The data sources include CC3M, CC12M, COCO, Visual Genome, Laion, and others. In this stage, the language model parameters remain fixed, and only the parameters of the `Q-Former` are updated to enable efficient alignment of large-scale visual-language representations. | |
| * Instruction fine-tuning: We utilized the [LLaVA-150K](https://llava-vl.github.io/) dataset, which consists of English multimodal instruction-following dataset. We mixed this data with corresponding translated Chinese data to fine-tune the model and align its multimodal capabilities with user intents. In this phase, we updated all model parameters to improve the utilization efficiency of the instruction fine-tuning data. Interestingly, we observed that even when using only English instruction data for fine-tuning, the model can comprehend Chinese questions but can only respond in English. This indicates that the model has achieved good generalization in terms of its multilingual and multimodal capabilities. By incorporating a small amount of translated Chinese data during the instruction fine-tuning phase, we can align the model's response language with the user's question language. | |
| We evaluated the model on the LLaVA English test set and the translated Chinese test set. The evaluation benchmark examined the model's performance in open-domain conversations, image detail descriptions, and complex reasoning tasks, using GPT-4 for scoring. It is evident that `VisCPM-Chat` achieved the best average performance in Chinese multimodal capabilities, excelling in general-domain conversations and complex reasoning. It also demonstrated commendable English multimodal abilities. | |
| <table> | |
| <tr> | |
| <td align="center" rowspan="2" colspan="2">Model</td> | |
| <td align="center" colspan="4">English</td> | |
| <td align="center" colspan="4">Chinese</td> | |
| </tr> | |
| <tr> | |
| <td align="center">Conversation</td> | |
| <td align="center">Detailed Description</td> | |
| <td align="center">Complex Reasoning</td> | |
| <td align="center">All</td> | |
| <td align="center">Conversation</td> | |
| <td align="center">Detailed Description</td> | |
| <td align="center">Complex Reasoning</td> | |
| <td align="center">All</td> | |
| </tr> | |
| <tr> | |
| <td align="center" rowspan="3">English Model</td> | |
| <td align="center">MiniGPT4</td> | |
| <td align="center">65</td> | |
| <td align="center">67.3</td> | |
| <td align="center">76.6</td> | |
| <td align="center">69.7</td> | |
| <td align="center">-</td> | |
| <td align="center">-</td> | |
| <td align="center">-</td> | |
| <td align="center">-</td> | |
| </tr> | |
| <tr> | |
| <td align="center">InstructBLIP</td> | |
| <td align="center">81.9</td> | |
| <td align="center">68</td> | |
| <td align="center">91.2</td> | |
| <td align="center">80.5</td> | |
| <td align="center">-</td> | |
| <td align="center">-</td> | |
| <td align="center">-</td> | |
| <td align="center">-</td> | |
| </tr> | |
| <tr> | |
| <td align="center">LLaVA</td> | |
| <td align="center">89.5</td> | |
| <td align="center">70.4</td> | |
| <td align="center">96.2</td> | |
| <td align="center">85.6</td> | |
| <td align="center">-</td> | |
| <td align="center">-</td> | |
| <td align="center">-</td> | |
| <td align="center">-</td> | |
| </tr> | |
| <tr> | |
| <td align="center" rowspan="4">En-Zh Bilingual Model</td> | |
| <td align="center">mPLUG-Owl </td> | |
| <td align="center">64.6</td> | |
| <td align="center">47.7</td> | |
| <td align="center">80.1</td> | |
| <td align="center">64.2</td> | |
| <td align="center">76.3</td> | |
| <td align="center">61.2</td> | |
| <td align="center">77.8</td> | |
| <td align="center">72</td> | |
| </tr> | |
| <tr> | |
| <td align="center">VisualGLM</td> | |
| <td align="center">62.4</td> | |
| <td align="center">63</td> | |
| <td align="center">80.6</td> | |
| <td align="center">68.7</td> | |
| <td align="center">76.6</td> | |
| <td align="center">87.8</td> | |
| <td align="center">83.6</td> | |
| <td align="center">82.7</td> | |
| </tr> | |
| <tr> | |
| <td align="center">Ziya (LLaMA 13B)</td> | |
| <td align="center">82.7</td> | |
| <td align="center">69.9</td> | |
| <td align="center">92.1</td> | |
| <td align="center">81.7</td> | |
| <td align="center">85</td> | |
| <td align="center">74.7</td> | |
| <td align="center">82.4</td> | |
| <td align="center">80.8</td> | |
| </tr> | |
| <tr> | |
| <td align="center">VisCPM-Chat</td> | |
| <td align="center">83.3</td> | |
| <td align="center">68.9</td> | |
| <td align="center">90.5</td> | |
| <td align="center">81.1</td> | |
| <td align="center">92.7</td> | |
| <td align="center">76.1</td> | |
| <td align="center">89.2</td> | |
| <td align="center">86.3</td> | |
| </tr> | |
| </table> | |
| # Install | |
| 1. Clone this repository and navigate to source folder | |
| ```bash | |
| git clone <github repo URL> | |
| cd viscpm | |
| ``` | |
| 2. Install Package | |
| ```Shell | |
| conda create -n viscpm python=3.10 -y | |
| conda activate viscpm | |
| pip install setuptools | |
| pip install diffusers jieba matplotlib numpy opencv_python | |
| pip install pandas Pillow psutil pydantic scipy | |
| pip install torch==1.13.1 torchscale==0.2.0 torchvision==0.14.1 timm | |
| pip install transformers==4.28.0 | |
| pip install tqdm typing_extensions | |
| pip install git+https://github.com/thunlp/OpenDelta.git | |
| pip install git+https://github.com/OpenBMB/CPM-Bee.git#egg=cpm-live&subdirectory=src | |
| ``` | |
| `VisCPM` require GPUs with more than 40GB memory. We will soon update more memory-friendly inference methods. | |
| ## How to use | |
| ```python | |
| >>> from transformers import AutoModel, AutoTokenizer, AutoImageProcessor | |
| >>> from PIL import Image | |
| >>> tokenizer = AutoTokenizer.from_pretrained('viscpm', trust_remote_code=True) | |
| >>> processor = AutoImageProcessor.from_pretrained('viscpm', trust_remote_code=True) | |
| >>> model = AutoModel.from_pretrained('viscpm', trust_remote_code=True).to('cuda') | |
| >>> data = [{ | |
| >>> 'context': '', | |
| >>> 'question': 'describe this image in detail.', | |
| >>> 'image': tokenizer.unk_token * model.query_num, | |
| >>> '<ans>': '' | |
| >>> }] | |
| >>> image = Image.open('case.jpg') | |
| >>> result = model.generate(data, tokenizer, processor, image) | |
| >>> print(result[0]['<ans>']) | |
| 这幅图片显示了一群热气球在天空中飞行。这些热气球漂浮在不同的地方，包括山脉、城市和乡村地区。 | |
| ``` | |