OpenGVLab
/

InternVL2-40B

@@ -9,6 +9,8 @@ pipeline_tag: image-text-to-text
 [\[🗨️ Chat Demo\]](https://internvl.opengvlab.com/)  [\[🤗 HF Demo\]](https://huggingface.co/spaces/OpenGVLab/InternVL)  [\[🚀 Quick Start\]](#quick-start)  [\[📖 中文解读\]](https://zhuanlan.zhihu.com/p/706547971)  \[🌟 [魔搭社区](https://modelscope.cn/organization/OpenGVLab) | [教程](https://mp.weixin.qq.com/s/OUaVLkxlk1zhFb1cvMCFjg) \]
 ## Introduction
 We are excited to announce the release of InternVL 2.0, the latest addition to the InternVL series of multimodal large language models. InternVL 2.0 features a variety of **instruction-tuned models**, ranging from 1 billion to 108 billion parameters. This repository contains the instruction-tuned InternVL2-40B model.
@@ -17,6 +19,16 @@ Compared to the state-of-the-art open-source multimodal large language models, I
 InternVL 2.0 is trained with an 8k context window and utilizes training data consisting of long texts, multiple images, and videos, significantly improving its ability to handle these types of inputs compared to InternVL 1.5. For more details, please refer to our blog and GitHub.
 ## Model Details
 InternVL 2.0 is a multimodal large language model series, featuring models of various sizes. For each size, we release instruction-tuned models optimized for multimodal tasks. InternVL2-40B consists of [InternViT-6B-448px-V1-5](https://huggingface.co/OpenGVLab/InternViT-6B-448px-V1-5), an MLP projector, and [Nous-Hermes-2-Yi-34B](https://huggingface.co/NousResearch/Nous-Hermes-2-Yi-34B).
@@ -25,27 +37,28 @@ InternVL 2.0 is a multimodal large language model series, featuring models of va
 ### Image Benchmarks
-|          Benchmark           | GPT-4T-20240409 | Gemini-1.5-Pro | InternVL2-26B | InternVL2-40B |
-| :--------------------------: | :-------------: | :------------: | :-----------: | :-----------: |
-|          Model Size          |        -        |       -        |     25.5B     |      40B      |
-|                              |                 |                |               |               |
-|    DocVQA<sub>test</sub>     |      87.2       |      86.5      |     92.9      |     93.9      |
-|    ChartQA<sub>test</sub>    |      78.1       |      81.3      |     84.9      |     86.2      |
-|    InfoVQA<sub>test</sub>    |        -        |      72.7      |     75.9      |     78.7      |
-|    TextVQA<sub>val</sub>     |        -        |      73.5      |     82.3      |     83.0      |
-|           OCRBench           |       678       |      754       |      825      |      837      |
-|      MME<sub>sum</sub>       |     2070.2      |     2110.6     |    2260.7     |    2315.0     |
-|         RealWorldQA          |      68.0       |      67.5      |     68.3      |     71.8      |
-|     AI2D<sub>test</sub>      |      89.4       |      80.3      |     84.5      |     87.1      |
-|      MMMU<sub>val</sub>      |   63.1 / 61.7   |  58.5 / 60.6   |  48.3 / 50.7  |  53.9 / 55.2  |
-|  MMBench-EN<sub>test</sub>   |      81.0       |      73.9      |     83.4      |     86.8      |
-|  MMBench-CN<sub>test</sub>   |      80.2       |      73.8      |     82.0      |     86.5      |
-|    CCBench<sub>dev</sub>     |      57.3       |      28.4      |     73.5      |     80.6      |
-|  MMVet<sub>GPT-4-0613</sub>  |        -        |       -        |     64.2      |     68.5      |
-| MMVet<sub>GPT-4-Turbo</sub>  |      67.5       |      64.0      |     62.1      |     65.5      |
-|          SEED-Image          |        -        |       -        |     76.8      |     78.2      |
-|   HallBench<sub>avg</sub>    |      43.9       |      45.6      |     50.7      |     56.9      |
-| MathVista<sub>testmini</sub> |      58.1       |      57.7      |     59.4      |     63.7      |
 - We simultaneously use InternVL and VLMEvalKit repositories for model evaluation. Specifically, the results reported for DocVQA, ChartQA, InfoVQA, TextVQA, MME, AI2D, MMBench, CCBench, MMVet, and SEED-Image were tested using the InternVL repository. OCRBench, RealWorldQA, HallBench, and MathVista were evaluated using the VLMEvalKit.
@@ -579,6 +592,16 @@ If you find this project useful in your research, please consider citing:
 InternVL 2.0 使用 8k 上下文窗口进行训练，训练数据包含长文本、多图和视频数据，与 InternVL 1.5 相比，其处理这些类型输入的能力显著提高。更多详细信息，请参阅我们的博客和 GitHub。
 ## 模型细节
 InternVL 2.0 是一个多模态大语言模型系列，包含各种规模的模型。对于每个规模的模型，我们都会发布针对多模态任务优化的指令微调模型。InternVL2-40B 包含 [InternViT-6B-448px-V1-5](https://huggingface.co/OpenGVLab/InternViT-6B-448px-V1-5)、一个 MLP 投影器和 [Nous-Hermes-2-Yi-34B](https://huggingface.co/NousResearch/Nous-Hermes-2-Yi-34B)。
@@ -587,27 +610,28 @@ InternVL 2.0 是一个多模态大语言模型系列，包含各种规模的模
 ### 图像相关评测
-|          评测数据集          | GPT-4T-20240409 | Gemini-1.5-Pro | InternVL2-26B | InternVL2-40B |
-| :--------------------------: | :-------------: | :------------: | :-----------: | :-----------: |
-|           模型大小           |        -        |       -        |     25.5B     |      40B      |
-|                              |                 |                |               |               |
-|    DocVQA<sub>test</sub>     |      87.2       |      86.5      |     92.9      |     93.9      |
-|    ChartQA<sub>test</sub>    |      78.1       |      81.3      |     84.9      |     86.2      |
-|    InfoVQA<sub>test</sub>    |        -        |      72.7      |     75.9      |     78.7      |
-|    TextVQA<sub>val</sub>     |        -        |      73.5      |     82.3      |     83.0      |
-|           OCRBench           |       678       |      754       |      825      |      837      |
-|      MME<sub>sum</sub>       |     2070.2      |     2110.6     |    2260.7     |    2315.0     |
-|         RealWorldQA          |      68.0       |      67.5      |     68.3      |     71.8      |
-|     AI2D<sub>test</sub>      |      89.4       |      80.3      |     84.5      |     87.1      |
-|      MMMU<sub>val</sub>      |   63.1 / 61.7   |  58.5 / 60.6   |  48.3 / 50.7  |  53.9 / 55.2  |
-|  MMBench-EN<sub>test</sub>   |      81.0       |      73.9      |     83.4      |     86.8      |
-|  MMBench-CN<sub>test</sub>   |      80.2       |      73.8      |     82.0      |     86.5      |
-|    CCBench<sub>dev</sub>     |      57.3       |      28.4      |     73.5      |     80.6      |
-|  MMVet<sub>GPT-4-0613</sub>  |        -        |       -        |     64.2      |     68.5      |
-| MMVet<sub>GPT-4-Turbo</sub>  |      67.5       |      64.0      |     62.1      |     65.5      |
-|          SEED-Image          |        -        |       -        |     76.8      |     78.2      |
-|   HallBench<sub>avg</sub>    |      43.9       |      45.6      |     50.7      |     56.9      |
-| MathVista<sub>testmini</sub> |      58.1       |      57.7      |     59.4      |     63.7      |
 - 我们同时使用 InternVL 和 VLMEvalKit 仓库进行模型评估。具体来说，DocVQA、ChartQA、InfoVQA、TextVQA、MME、AI2D、MMBench、CCBench、MMVet 和 SEED-Image 的结果是使用 InternVL 仓库测试的。MMMU、OCRBench、RealWorldQA、HallBench 和 MathVista 是使用 VLMEvalKit 进行评估的。

 [\[🗨️ Chat Demo\]](https://internvl.opengvlab.com/)  [\[🤗 HF Demo\]](https://huggingface.co/spaces/OpenGVLab/InternVL)  [\[🚀 Quick Start\]](#quick-start)  [\[📖 中文解读\]](https://zhuanlan.zhihu.com/p/706547971)  \[🌟 [魔搭社区](https://modelscope.cn/organization/OpenGVLab) | [教程](https://mp.weixin.qq.com/s/OUaVLkxlk1zhFb1cvMCFjg) \]
+[切换至中文版](#简介)
 ## Introduction
 We are excited to announce the release of InternVL 2.0, the latest addition to the InternVL series of multimodal large language models. InternVL 2.0 features a variety of **instruction-tuned models**, ranging from 1 billion to 108 billion parameters. This repository contains the instruction-tuned InternVL2-40B model.
 InternVL 2.0 is trained with an 8k context window and utilizes training data consisting of long texts, multiple images, and videos, significantly improving its ability to handle these types of inputs compared to InternVL 1.5. For more details, please refer to our blog and GitHub.
+|      Model Name      |                                     Vision Part                                     |                                        Language Part                                         |                             HF Link                              |                                MS Link                                 |
+| :------------------: | :---------------------------------------------------------------------------------: | :------------------------------------------------------------------------------------------: | :--------------------------------------------------------------: | :--------------------------------------------------------------------: |
+|     InternVL2-1B     |    [InternViT-300M-448px](https://huggingface.co/OpenGVLab/InternViT-300M-448px)    |            [Qwen2-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2-0.5B-Instruct)            |     [🤗 link](https://huggingface.co/OpenGVLab/InternVL2-1B)     |     [🤖 link](https://modelscope.cn/models/OpenGVLab/InternVL2-1B)     |
+|     InternVL2-2B     |    [InternViT-300M-448px](https://huggingface.co/OpenGVLab/InternViT-300M-448px)    |          [internlm2-chat-1_8b](https://huggingface.co/internlm/internlm2-chat-1_8b)          |     [🤗 link](https://huggingface.co/OpenGVLab/InternVL2-2B)     |     [🤖 link](https://modelscope.cn/models/OpenGVLab/InternVL2-2B)     |
+|     InternVL2-4B     |    [InternViT-300M-448px](https://huggingface.co/OpenGVLab/InternViT-300M-448px)    |    [Phi-3-mini-128k-instruct](https://huggingface.co/microsoft/Phi-3-mini-128k-instruct)     |     [🤗 link](https://huggingface.co/OpenGVLab/InternVL2-4B)     |     [🤖 link](https://modelscope.cn/models/OpenGVLab/InternVL2-4B)     |
+|     InternVL2-8B     |    [InternViT-300M-448px](https://huggingface.co/OpenGVLab/InternViT-300M-448px)    |          [internlm2_5-7b-chat](https://huggingface.co/internlm/internlm2_5-7b-chat)          |     [🤗 link](https://huggingface.co/OpenGVLab/InternVL2-8B)     |     [🤖 link](https://modelscope.cn/models/OpenGVLab/InternVL2-8B)     |
+|    InternVL2-26B     | [InternViT-6B-448px-V1-5](https://huggingface.co/OpenGVLab/InternViT-6B-448px-V1-5) |           [internlm2-chat-20b](https://huggingface.co/internlm/internlm2-chat-20b)           |    [🤗 link](https://huggingface.co/OpenGVLab/InternVL2-26B)     |    [🤖 link](https://modelscope.cn/models/OpenGVLab/InternVL2-26B)     |
+|    InternVL2-40B     | [InternViT-6B-448px-V1-5](https://huggingface.co/OpenGVLab/InternViT-6B-448px-V1-5) |       [Nous-Hermes-2-Yi-34B](https://huggingface.co/NousResearch/Nous-Hermes-2-Yi-34B)       |    [🤗 link](https://huggingface.co/OpenGVLab/InternVL2-40B)     |    [🤖 link](https://modelscope.cn/models/OpenGVLab/InternVL2-40B)     |
+| InternVL2-Llama3-76B | [InternViT-6B-448px-V1-5](https://huggingface.co/OpenGVLab/InternViT-6B-448px-V1-5) | [Hermes-2-Theta-Llama-3-70B](https://huggingface.co/NousResearch/Hermes-2-Theta-Llama-3-70B) | [🤗 link](https://huggingface.co/OpenGVLab/InternVL2-Llama3-76B) | [🤖 link](https://modelscope.cn/models/OpenGVLab/InternVL2-Llama3-76B) |
 ## Model Details
 InternVL 2.0 is a multimodal large language model series, featuring models of various sizes. For each size, we release instruction-tuned models optimized for multimodal tasks. InternVL2-40B consists of [InternViT-6B-448px-V1-5](https://huggingface.co/OpenGVLab/InternViT-6B-448px-V1-5), an MLP projector, and [Nous-Hermes-2-Yi-34B](https://huggingface.co/NousResearch/Nous-Hermes-2-Yi-34B).
 ### Image Benchmarks
+|            Benchmark            | GPT-4T-20240409 | Gemini-1.5-Pro | InternVL2-26B | InternVL2-40B |
+| :-----------------------------: | :-------------: | :------------: | :-----------: | :-----------: |
+|           Model Size            |        -        |       -        |     25.5B     |      40B      |
+|                                 |                 |                |               |               |
+|      DocVQA<sub>test</sub>      |      87.2       |      86.5      |     92.9      |     93.9      |
+|     ChartQA<sub>test</sub>      |      78.1       |      81.3      |     84.9      |     86.2      |
+|     InfoVQA<sub>test</sub>      |        -        |      72.7      |     75.9      |     78.7      |
+|      TextVQA<sub>val</sub>      |        -        |      73.5      |     82.3      |     83.0      |
+|            OCRBench             |       678       |      754       |      825      |      837      |
+|        MME<sub>sum</sub>        |     2070.2      |     2110.6     |    2260.7     |    2315.0     |
+|           RealWorldQA           |      68.0       |      67.5      |     68.3      |     71.8      |
+|       AI2D<sub>test</sub>       |      89.4       |      80.3      |     84.5      |     87.1      |
+|       MMMU<sub>val</sub>        |   63.1 / 61.7   |  58.5 / 60.6   |  48.3 / 50.7  |  53.9 / 55.2  |
+|    MMBench-EN<sub>test</sub>    |      81.0       |      73.9      |     83.4      |     86.8      |
+|    MMBench-CN<sub>test</sub>    |      80.2       |      73.8      |     82.0      |     86.5      |
+|      CCBench<sub>dev</sub>      |      57.3       |      28.4      |     73.5      |     80.6      |
+|   MMVet<sub>GPT-4-0613</sub>    |        -        |       -        |     64.2      |     68.5      |
+|   MMVet<sub>GPT-4-Turbo</sub>   |      67.5       |      64.0      |     62.1      |     65.5      |
+|           SEED-Image            |        -        |       -        |     76.8      |     78.2      |
+|     HallBench<sub>avg</sub>     |      43.9       |      45.6      |     50.7      |     56.9      |
+|  MathVista<sub>testmini</sub>   |      58.1       |      57.7      |     59.4      |     63.7      |
+| OpenCompass<sub>avg-score</sub> |      63.5       |      64.4      |     66.4      |     69.7      |
 - We simultaneously use InternVL and VLMEvalKit repositories for model evaluation. Specifically, the results reported for DocVQA, ChartQA, InfoVQA, TextVQA, MME, AI2D, MMBench, CCBench, MMVet, and SEED-Image were tested using the InternVL repository. OCRBench, RealWorldQA, HallBench, and MathVista were evaluated using the VLMEvalKit.
 InternVL 2.0 使用 8k 上下文窗口进行训练，训练数据包含长文本、多图和视频数据，与 InternVL 1.5 相比，其处理这些类型输入的能力显著提高。更多详细信息，请参阅我们的博客和 GitHub。
+|       模型名称       |                                      视觉部分                                       |                                           语言部分                                           |                             HF 链接                              |                                MS 链接                                 |
+| :------------------: | :---------------------------------------------------------------------------------: | :------------------------------------------------------------------------------------------: | :--------------------------------------------------------------: | :--------------------------------------------------------------------: |
+|     InternVL2-1B     |    [InternViT-300M-448px](https://huggingface.co/OpenGVLab/InternViT-300M-448px)    |            [Qwen2-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2-0.5B-Instruct)            |     [🤗 link](https://huggingface.co/OpenGVLab/InternVL2-1B)     |     [🤖 link](https://modelscope.cn/models/OpenGVLab/InternVL2-1B)     |
+|     InternVL2-2B     |    [InternViT-300M-448px](https://huggingface.co/OpenGVLab/InternViT-300M-448px)    |          [internlm2-chat-1_8b](https://huggingface.co/internlm/internlm2-chat-1_8b)          |     [🤗 link](https://huggingface.co/OpenGVLab/InternVL2-2B)     |     [🤖 link](https://modelscope.cn/models/OpenGVLab/InternVL2-2B)     |
+|     InternVL2-4B     |    [InternViT-300M-448px](https://huggingface.co/OpenGVLab/InternViT-300M-448px)    |    [Phi-3-mini-128k-instruct](https://huggingface.co/microsoft/Phi-3-mini-128k-instruct)     |     [🤗 link](https://huggingface.co/OpenGVLab/InternVL2-4B)     |     [🤖 link](https://modelscope.cn/models/OpenGVLab/InternVL2-4B)     |
+|     InternVL2-8B     |    [InternViT-300M-448px](https://huggingface.co/OpenGVLab/InternViT-300M-448px)    |          [internlm2_5-7b-chat](https://huggingface.co/internlm/internlm2_5-7b-chat)          |     [🤗 link](https://huggingface.co/OpenGVLab/InternVL2-8B)     |     [🤖 link](https://modelscope.cn/models/OpenGVLab/InternVL2-8B)     |
+|    InternVL2-26B     | [InternViT-6B-448px-V1-5](https://huggingface.co/OpenGVLab/InternViT-6B-448px-V1-5) |           [internlm2-chat-20b](https://huggingface.co/internlm/internlm2-chat-20b)           |    [🤗 link](https://huggingface.co/OpenGVLab/InternVL2-26B)     |    [🤖 link](https://modelscope.cn/models/OpenGVLab/InternVL2-26B)     |
+|    InternVL2-40B     | [InternViT-6B-448px-V1-5](https://huggingface.co/OpenGVLab/InternViT-6B-448px-V1-5) |       [Nous-Hermes-2-Yi-34B](https://huggingface.co/NousResearch/Nous-Hermes-2-Yi-34B)       |    [🤗 link](https://huggingface.co/OpenGVLab/InternVL2-40B)     |    [🤖 link](https://modelscope.cn/models/OpenGVLab/InternVL2-40B)     |
+| InternVL2-Llama3-76B | [InternViT-6B-448px-V1-5](https://huggingface.co/OpenGVLab/InternViT-6B-448px-V1-5) | [Hermes-2-Theta-Llama-3-70B](https://huggingface.co/NousResearch/Hermes-2-Theta-Llama-3-70B) | [🤗 link](https://huggingface.co/OpenGVLab/InternVL2-Llama3-76B) | [🤖 link](https://modelscope.cn/models/OpenGVLab/InternVL2-Llama3-76B) |
 ## 模型细节
 InternVL 2.0 是一个多模态大语言模型系列，包含各种规模的模型。对于每个规模的模型，我们都会发布针对多模态任务优化的指令微调模型。InternVL2-40B 包含 [InternViT-6B-448px-V1-5](https://huggingface.co/OpenGVLab/InternViT-6B-448px-V1-5)、一个 MLP 投影器和 [Nous-Hermes-2-Yi-34B](https://huggingface.co/NousResearch/Nous-Hermes-2-Yi-34B)。
 ### 图像相关评测
+|           评测数据集            | GPT-4T-20240409 | Gemini-1.5-Pro | InternVL2-26B | InternVL2-40B |
+| :-----------------------------: | :-------------: | :------------: | :-----------: | :-----------: |
+|            模型大小             |        -        |       -        |     25.5B     |      40B      |
+|                                 |                 |                |               |               |
+|      DocVQA<sub>test</sub>      |      87.2       |      86.5      |     92.9      |     93.9      |
+|     ChartQA<sub>test</sub>      |      78.1       |      81.3      |     84.9      |     86.2      |
+|     InfoVQA<sub>test</sub>      |        -        |      72.7      |     75.9      |     78.7      |
+|      TextVQA<sub>val</sub>      |        -        |      73.5      |     82.3      |     83.0      |
+|            OCRBench             |       678       |      754       |      825      |      837      |
+|        MME<sub>sum</sub>        |     2070.2      |     2110.6     |    2260.7     |    2315.0     |
+|           RealWorldQA           |      68.0       |      67.5      |     68.3      |     71.8      |
+|       AI2D<sub>test</sub>       |      89.4       |      80.3      |     84.5      |     87.1      |
+|       MMMU<sub>val</sub>        |   63.1 / 61.7   |  58.5 / 60.6   |  48.3 / 50.7  |  53.9 / 55.2  |
+|    MMBench-EN<sub>test</sub>    |      81.0       |      73.9      |     83.4      |     86.8      |
+|    MMBench-CN<sub>test</sub>    |      80.2       |      73.8      |     82.0      |     86.5      |
+|      CCBench<sub>dev</sub>      |      57.3       |      28.4      |     73.5      |     80.6      |
+|   MMVet<sub>GPT-4-0613</sub>    |        -        |       -        |     64.2      |     68.5      |
+|   MMVet<sub>GPT-4-Turbo</sub>   |      67.5       |      64.0      |     62.1      |     65.5      |
+|           SEED-Image            |        -        |       -        |     76.8      |     78.2      |
+|     HallBench<sub>avg</sub>     |      43.9       |      45.6      |     50.7      |     56.9      |
+|  MathVista<sub>testmini</sub>   |      58.1       |      57.7      |     59.4      |     63.7      |
+| OpenCompass<sub>avg-score</sub> |      63.5       |      64.4      |     66.4      |     69.7      |
 - 我们同时使用 InternVL 和 VLMEvalKit 仓库进行模型评估。具体来说，DocVQA、ChartQA、InfoVQA、TextVQA、MME、AI2D、MMBench、CCBench、MMVet 和 SEED-Image 的结果是使用 InternVL 仓库测试的。MMMU、OCRBench、RealWorldQA、HallBench 和 MathVista 是使用 VLMEvalKit 进行评估的。