|
--- |
|
license: bsd-3-clause |
|
language: |
|
- en |
|
pipeline_tag: visual-question-answering |
|
library_name: transformers |
|
--- |
|
|
|
<br> |
|
<br> |
|
|
|
# LoViM Model Card |
|
|
|
## Model details |
|
|
|
**Model type:** |
|
LoViM is an open-source Vision-Languagde model trained by initializing from InstructBLIP and alignment with Vicuna on multimodal instruction-finetuning data. |
|
It composes of an EVA-CLIP vision encoder, a Q-Former, a projection layer and an auto-regressive language model, based on the decoder only transformer architecture. |
|
|
|
**Model date:** |
|
LoViM_FlanT5 was trained in July 2023. |
|
|
|
**Paper or resources for more information:** |
|
https://project page |
|
|
|
**License:** |
|
BSD 3-Clause License |
|
|
|
**Where to send questions or comments about the model:** |
|
https://github.com/ |
|
|
|
## Intended use |
|
**Primary intended uses:** |
|
The primary use of LoViM FlanT5 is for commercial use on large multimodal models. |
|
|
|
**Primary intended users:** |
|
The primary intended users of this model is for commercial companies in computer vision, natural language processing, machine learning, and artificial intelligence. |
|
|
|
## Training dataset |
|
Pre-train data: 558K filtered image-text pairs from LAION,CC-3M, and SBU. Selected by LLaVA. |
|
|
|
Instruction-finetuning data: COCO-Caption, TextCaps, VQAv2, OKVQA, A-OKVQA, LLaVA-150K, OCR-VQA. |
|
|
|
## Evaluation dataset |
|
For zero-shot evaluation on general image task, we selected Nocaps, Flickr30K, VizWiz, Visual Spaial Reasoning (VSR), IconQA, Visual Dialog, ScienceQA, MSRVTT QA, TextVQA and Hateful Memes. |
|
|
|
For zero-shot evaluation on text-rich image OCR task, we selected ST-VQA, OCR-VQA, Text-VQA, and Doc-VQA. |
|
|
|
More detials are in our github, https://github.com/ |
