README_en.md

VisCPM

简体中文 | English

VisCPM is a family of open-source large multimodal models, which support multimodal conversational capabilities (VisCPM-Chat model) and text-to-image generation capabilities (VisCPM-Paint model) in both Chinese and English, achieving state-of-the-art peformance among Chinese open-source multimodal models. VisCPM is trained based on the large language model CPM-Bee with 10B parameters, fusing visual encoder (Q-Former) and visual decoder (Diffusion-UNet) to support visual inputs and outputs. Thanks to the good bilingual capability of CPM-Bee, VisCPM can be pre-trained with English multimodal data only and well generalize to achieve promising Chinese multimodal capabilities.

VisCPM-Chat

VisCPM-Chat supports bilingual multimodal conversations involving images in both Chinese and English. The model utilizes Q-Former as the visual encoder and CPM-Bee (10B) as the base LLM. It combines visual and language models through language modeling training objectives. The model training consists of two stages: pretraining and instruction fine-tuning.

• Pretrain: VisCPM-Chat was pretrained using approximately 100 million high-quality English multimodal data pairs. The data sources include CC3M, CC12M, COCO, Visual Genome, Laion, and others. In this stage, the language model parameters remain fixed, and only the parameters of the Q-Former are updated to enable efficient alignment of large-scale visual-language representations.

• Instruction fine-tuning: We utilized the LLaVA-150K dataset, which consists of English multimodal instruction-following dataset. We mixed this data with corresponding translated Chinese data to fine-tune the model and align its multimodal capabilities with user intents. In this phase, we updated all model parameters to improve the utilization efficiency of the instruction fine-tuning data. Interestingly, we observed that even when using only English instruction data for fine-tuning, the model can comprehend Chinese questions but can only respond in English. This indicates that the model has achieved good generalization in terms of its multilingual and multimodal capabilities. By incorporating a small amount of translated Chinese data during the instruction fine-tuning phase, we can align the model's response language with the user's question language.

We evaluated the model on the LLaVA English test set and the translated Chinese test set. The evaluation benchmark examined the model's performance in open-domain conversations, image detail descriptions, and complex reasoning tasks, using GPT-4 for scoring. It is evident that VisCPM-Chat achieved the best average performance in Chinese multimodal capabilities, excelling in general-domain conversations and complex reasoning. It also demonstrated commendable English multimodal abilities.

Model		English				Chinese
		Conversation	Detailed Description	Complex Reasoning	All	Conversation	Detailed Description	Complex Reasoning	All
English Model	MiniGPT4	65	67.3	76.6	69.7	-	-	-	-
	InstructBLIP	81.9	68	91.2	80.5	-	-	-	-
	LLaVA	89.5	70.4	96.2	85.6	-	-	-	-
En-Zh Bilingual Model	mPLUG-Owl	64.6	47.7	80.1	64.2	76.3	61.2	77.8	72
	VisualGLM	62.4	63	80.6	68.7	76.6	87.8	83.6	82.7
	Ziya (LLaMA 13B)	82.7	69.9	92.1	81.7	85	74.7	82.4	80.8
	VisCPM-Chat	83.3	68.9	90.5	81.1	92.7	76.1	89.2	86.3

Install

1. Clone this repository and navigate to source folder

git clone <github repo URL>
cd viscpm

2. Install Package

conda create -n viscpm python=3.10 -y
conda activate viscpm
pip install setuptools
pip install diffusers jieba matplotlib numpy opencv_python
pip install pandas Pillow psutil pydantic scipy
pip install torch==1.13.1 torchscale==0.2.0 torchvision==0.14.1 timm
pip install transformers==4.28.0
pip install tqdm typing_extensions
pip install git+https://github.com/thunlp/OpenDelta.git
pip install git+https://github.com/OpenBMB/CPM-Bee.git#egg=cpm-live&subdirectory=src

VisCPM require GPUs with more than 40GB memory. We will soon update more memory-friendly inference methods.

How to use

>>> from transformers import AutoModel, AutoTokenizer, AutoImageProcessor
>>> from PIL import Image

>>> tokenizer = AutoTokenizer.from_pretrained('viscpm', trust_remote_code=True)
>>> processor = AutoImageProcessor.from_pretrained('viscpm', trust_remote_code=True)
>>> model = AutoModel.from_pretrained('viscpm', trust_remote_code=True).to('cuda')

>>> data = [{
>>>     'context': '',
>>>     'question': 'describe this image in detail.',
>>>     'image': tokenizer.unk_token * model.query_num,
>>>     '<ans>': ''
>>>     }]
>>> image = Image.open('case.jpg')
>>> result = model.generate(data, tokenizer, processor, image)
>>> print(result[0]['<ans>'])
这幅图片显示了一群热气球在天空中飞行。这些热气球漂浮在不同的地方，包括山脉、城市和乡村地区。