|
--- |
|
inference: false |
|
license: apache-2.0 |
|
pipeline_tag: video-text-to-text |
|
--- |
|
|
|
<br> |
|
|
|
## LLaVA-NeXT-Video is upgraded 🚀 |
|
|
|
In our [LLaVA-Video blog](https://llava-vl.github.io/blog/2024-04-30-llava-next-video/) released this April, we shared two key observations: |
|
- 🎬 AnyRes provides a shared and flexible representation between images and videos, and thus accommodates capability transfer between the two most common vision signals. Therefore, stronger image LMMs can naturally lead to stronger zero-shot video LMMs. |
|
- 🗂️ There is a lack of high-quality language-video data, including video instruction-following data, and thus naive tuning on existing public data at that time results in performance degradation. Therefore, there is an urgent need to build high-quality video captions and QA datasets to train LMMs for improved video performance. |
|
|
|
Based on the insights, the new LLaVA-NeXT-Video in this release improves from two aspects: |
|
|
|
- 🎬 A stronger image LMMs ([LLaVA-NeXT-32B-Qwen](https://huggingface.co/lmms-lab/llava-next-qwen-32b)), which is built by initializing from Qwen-1.5 32B LLM. We further initialize our video training from this image checkpoint. |
|
- 🗂️ A new high-quality video dataset with 830k samples. It is combined with LLaVA-1.6 image training data, and applying the same image-video mixed training procedure leads to the new video model. |
|
The new model achieves the best open-source performance in several video benchmarks including [Video-MME](https://video-mme.github.io/home_page.html#leaderboard). |
|
|
|
### Resources |
|
- **Inference Script**: |
|
```bash |
|
bash scripts/video/demo/video_demo.sh lmms-lab/LLaVA-NeXT-Video-32B-Qwen 32 2 average after grid True playground/demo/xU25MMA2N4aVtYay.mp4 |
|
``` |
|
|
|
### Evaluation Results |
|
| Model | NextQA-MC | video-mme(overall) | | Egochema | Perception Test (val) | |
|
|-----------------------------|-----------|--------------------|--------|----------|------------------------| |
|
| | | w/o subs | w subs | | | |
|
| **Proprietary** | | | | | | |
|
| GPT-4o | - | 71.9 | 77.2 | 72.2 | - | |
|
| Gemini 1.5 Pro | - | 75.0 | 81.3 | 72.2 | - | |
|
| **Open-Source** | | | | | | |
|
| VideoLLaMA 2 (8x7B) | 76.3* | 47.9 | 50.3 | 53.3 | 51.2* | |
|
| VILA-1.5-34B | 67.89* | 60.1 | 61.1 | 58.04* | 54 | |
|
| LLaVA-NeXT-Video (Qwen-32B) | 77.31 | 60.2 | 63.0 | 60.85 | 59.38 | |
|
|
|
_*Results are reproduced by [lmms-eval](https://github.com/EvolvingLMMs-Lab/lmms-eval). Please refer to the lmms-eval to reproduce the results._ |
|
|
|
### Model details |
|
|
|
**Model type:** |
|
<br> |
|
LLaVA-Next-Video is an open-source chatbot trained by fine-tuning LLM on multimodal instruction-following data. |
|
<br> |
|
Base LLM: [Qwen/Qwen1.5-32B](https://huggingface.co/Qwen/Qwen1.5-32B) |
|
|
|
**Model date:** |
|
<br> |
|
LLaVA-NeXT-Video-32B-Qwen was trained in June 2024. |
|
|
|
**Paper or resources for more information:** |
|
<br> |
|
https://github.com/LLaVA-VL/LLaVA-NeXT |
|
|
|
### License |
|
[Qwen/Qwen1.5-32B](https://huggingface.co/Qwen/Qwen1.5-32B) license. |
|
|
|
|
|
### Where to send questions or comments about the model |
|
https://github.com/LLaVA-VL/LLaVA-NeXT/issues |
|
|
|
### Intended use |
|
**Primary intended uses:** |
|
<br> |
|
The primary use of LLaVA is research on large multimodal models and chatbots. |
|
|
|
**Primary intended users:** |
|
<br> |
|
The primary intended users of the model are researchers and hobbyists in computer vision, natural language processing, machine learning, and artificial intelligence. |
|
|
|
### Training dataset |
|
|
|
### Image |
|
- 558K filtered image-text pairs from LAION/CC/SBU, captioned by BLIP. |
|
- 158K GPT-generated multimodal instruction-following data. |
|
- 500K academic-task-oriented VQA data mixture. |
|
- 50K GPT-4V data mixture. |
|
- 40K ShareGPT data. |
|
### Video |
|
- 830k data |
|
|
|
### Citations |
|
```bibtex |
|
|
|
@misc{zhang2024videoinstructiontuningsynthetic, |
|
title={Video Instruction Tuning With Synthetic Data}, |
|
author={Yuanhan Zhang and Jinming Wu and Wei Li and Bo Li and Zejun Ma and Ziwei Liu and Chunyuan Li}, |
|
year={2024}, |
|
eprint={2410.02713}, |
|
archivePrefix={arXiv}, |
|
primaryClass={cs.CV}, |
|
url={https://arxiv.org/abs/2410.02713}, |
|
} |
|
|
|
@misc{zhang2024llavanextvideo, |
|
title={LLaVA-NeXT: A Strong Zero-shot Video Understanding Model}, |
|
url={https://llava-vl.github.io/blog/2024-04-30-llava-next-video/}, |
|
author={Zhang, Yuanhan and Li, Bo and Liu, haotian and Lee, Yong jae and Gui, Liangke and Fu, Di and Feng, Jiashi and Liu, Ziwei and Li, Chunyuan}, |
|
month={April}, |
|
year={2024} |
|
} |
|
|
|
@misc{li2024llavanext-interleave, |
|
title={LLaVA-NeXT: Tackling Multi-image, Video, and 3D in Large Multimodal Models}, |
|
url={https://llava-vl.github.io/blog/2024-06-16-llava-next-interleave/}, |
|
author={Li, Feng and Zhang, Renrui and Zhang, Hao and Zhang, Yuanhan and Li, Bo and Li, Wei and Ma, Zejun and Li, Chunyuan}, |
|
month={June}, |
|
year={2024} |
|
} |
