LMMs-Lab

community

https://www.lmms-lab.com/

lmmslab

EvolvingLMMs-Lab

Activity Feed

AI & ML interests

Feeling and building the multimodal intelligence.

Recent Activity

Yin-Xie updated a model about 7 hours ago

lmms-lab/LLaVA-OneVision-1.5-8B-Instruct

xiangan new activity 1 day ago

lmms-lab/LLaVA-OneVision-1.5-Instruct-Data:⚠️ Missing images in some SFT dataset subsets — please report affected samples

Yin-Xie updated a dataset 3 days ago

lmms-lab/LLaVA-OneVision-1.5-Instruct-Data

View all activity

lmms-lab 's collections 15

LLaVA-OneVision-1.5

https://github.com/EvolvingLMMs-Lab/LLaVA-OneVision-1.5

lmms-lab/LLaVA-OneVision-1.5-Instruct-Data

Updated 3 days ago • 106k • 50
lmms-lab/LLaVA-One-Vision-1.5-Mid-Training-85M

Preview • Updated 4 days ago • 116k • 40
lmms-lab/LLaVA-OneVision-1.5-8B-Instruct

Image-Text-to-Text • 9B • Updated about 7 hours ago • 3.55k • 43
lmms-lab/LLaVA-OneVision-1.5-4B-Instruct

Image-Text-to-Text • 5B • Updated 9 days ago • 2.25k • 8

MMSearch-R1

MMSearch-R1 is a solution designed to train LMMs to perform on-demand multimodal search in real-world environment.

lmms-lab/MMSearch-R1-7B-0807

8B • Updated Aug 7 • 20
lmms-lab/MMSearch-R1-7B

8B • Updated Jul 30 • 34 • 7
lmms-lab/FVQA

Viewer • Updated Aug 9 • 6.66k • 353 • 6
MMSearch-R1: Incentivizing LMMs to Search

Paper • 2506.20670 • Published Jun 25 • 64

EgoLife

CVPR 2025 - EgoLife: Towards Egocentric Life Assistant. Homepage: https://egolife-ai.github.io/

EgoLife: Towards Egocentric Life Assistant

Paper • 2503.03803 • Published Mar 5 • 44
Runtime error

14

14

EgoGPT

👁

Analyze video to describe actions and transcribe audio
lmms-lab/EgoIT-99K

Viewer • Updated Mar 7 • 199k • 4.99k • 7
lmms-lab/EgoLife

Viewer • Updated Mar 13 • 32k • 10.9k • 15

Multimodal-SAE

The collection of the sae that hooked on llava

Running on Zero

9

9

Multimodal SAE

💬

Demo for Multimodal-SAE
Large Multi-modal Models Can Interpret Features in Large Multi-modal Models

Paper • 2411.14982 • Published Nov 22, 2024 • 19
lmms-lab/llava-sae-explanations-5k

Viewer • Updated Nov 22, 2024 • 9.8k • 90 • 5
lmms-lab/llama3-llava-next-8b-hf-sae-131k

Updated Nov 26, 2024 • 16 • 7

LLaVA-Video

Models focus on video understanding (previously known as LLaVA-NeXT-Video).

Video Instruction Tuning With Synthetic Data

Paper • 2410.02713 • Published Oct 3, 2024 • 39
lmms-lab/LLaVA-Video-178K

Viewer • Updated Oct 11, 2024 • 1.63M • 33.3k • 171
lmms-lab/LLaVA-Video-7B-Qwen2

Video-Text-to-Text • 8B • Updated Oct 25, 2024 • 39.1k • 112
lmms-lab/LLaVA-Video-72B-Qwen2

Text Generation • 73B • Updated Oct 25, 2024 • 357 • 20

LMMs-Eval

Dataset Collection of LMMs-Eval

LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models

Paper • 2407.12772 • Published Jul 17, 2024 • 35
lmms-lab/VQAv2

Viewer • Updated Jan 26, 2024 • 770k • 5.51k • 25
lmms-lab/MME

Viewer • Updated Dec 23, 2023 • 2.37k • 14.1k • 23
lmms-lab/DocVQA

Viewer • Updated Apr 18, 2024 • 16.6k • 13.1k • 55

LLaVA-Next-Interleave

LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models

Paper • 2407.07895 • Published Jul 10, 2024 • 42
lmms-lab/llava-next-interleave-qwen-7b

Text Generation • 8B • Updated Jul 24, 2024 • 216 • 27
lmms-lab/llava-next-interleave-qwen-7b-dpo

Text Generation • 8B • Updated Jul 12, 2024 • 23 • 12
lmms-lab/M4-Instruct-Data

Updated Jul 21, 2024 • 2.6k • 75

LMMs-Eval-Lite

Making Lite version of the dataset to accelerate holistic evaluation during model development!

lmms-lab/CMMMU

Viewer • Updated Mar 8, 2024 • 12k • 478 • 4

LLaVA-Critic-R1

lmms-lab/LLaVA-Critic-R1-7B

8B • Updated Jul 19 • 967
lmms-lab/LLaVA-Critic-R1-7B-Plus-Qwen

8B • Updated Jul 26 • 378 • 4
lmms-lab/LLaVA-Critic-R1-7B-Plus-Mimo

8B • Updated Aug 28 • 103
lmms-lab/LLaVA-Critic-R1-7B-LLaMA32v

11B • Updated Aug 28 • 10

Aero-1-Audio

Runtime error

43

43

Aero 1 Audio Demo

💬

Demo for Aero-1-Audio
lmms-lab/Aero-1-Audio

Text Generation • 2B • Updated Jun 7 • 694 • 88

VideoMMMU

Video-MMMU: Evaluating Knowledge Acquisition from Multi-Discipline Professional Videos

Paper • 2501.13826 • Published Jan 23 • 25
lmms-lab/VideoMMMU

Viewer • Updated May 5 • 900 • 2.62k • 10

LLaVA-Critic

as a general evaluator for assessing model performance

LLaVA-Critic: Learning to Evaluate Multimodal Models

Paper • 2410.02712 • Published Oct 3, 2024 • 37
lmms-lab/llava-critic-7b

8B • Updated Oct 4, 2024 • 654 • 14
lmms-lab/llava-critic-72b

73B • Updated Oct 4, 2024 • 57 • 15
lmms-lab/llava-critic-113k

Viewer • Updated Oct 5, 2024 • 113k • 403 • 28

LLaVA-OneVision

a model good at arbitrary types of visual input

LLaVA-OneVision: Easy Visual Task Transfer

Paper • 2408.03326 • Published Aug 6, 2024 • 60
lmms-lab/LLaVA-OneVision-Mid-Data

Viewer • Updated Aug 26, 2024 • 563k • 314 • 21
lmms-lab/LLaVA-OneVision-Data

Viewer • Updated May 24 • 3.94M • 16.9k • 220
lmms-lab/LLaVA-NeXT-Data

Viewer • Updated Aug 30, 2024 • 779k • 2.26k • 41

LongVA

Long Context Transfer From Text To Vision: https://lmms-lab.github.io/posts/longva/

Long Context Transfer from Language to Vision

Paper • 2406.16852 • Published Jun 24, 2024 • 33
lmms-lab/LongVA-7B

Text Generation • 8B • Updated Jun 26, 2024 • 525 • 15
lmms-lab/LongVA-7B-DPO

Text Generation • 8B • Updated Jun 26, 2024 • 560 • 9
lmms-lab/v_niah_needles

Viewer • Updated Jun 15, 2024 • 5 • 9 • 4

LLaVA-NeXT

Some powerful image models.

lmms-lab/llava-next-110b

Text Generation • 112B • Updated May 14, 2024 • 13 • 21
lmms-lab/llava-next-72b

Text Generation • 73B • Updated Aug 22, 2024 • 48 • 14
lmms-lab/llava-next-qwen-32b

Text Generation • 33B • Updated Jul 16, 2024 • 88 • 7
lmms-lab/llama3-llava-next-8b

Text Generation • 8B • Updated Aug 17, 2024 • 3.04k • 100

LLaVA-OneVision-1.5

https://github.com/EvolvingLMMs-Lab/LLaVA-OneVision-1.5

lmms-lab/LLaVA-OneVision-1.5-Instruct-Data

Updated 3 days ago • 106k • 50
lmms-lab/LLaVA-One-Vision-1.5-Mid-Training-85M

Preview • Updated 4 days ago • 116k • 40
lmms-lab/LLaVA-OneVision-1.5-8B-Instruct

Image-Text-to-Text • 9B • Updated about 7 hours ago • 3.55k • 43
lmms-lab/LLaVA-OneVision-1.5-4B-Instruct

Image-Text-to-Text • 5B • Updated 9 days ago • 2.25k • 8

LLaVA-Critic-R1

lmms-lab/LLaVA-Critic-R1-7B

8B • Updated Jul 19 • 967
lmms-lab/LLaVA-Critic-R1-7B-Plus-Qwen

8B • Updated Jul 26 • 378 • 4
lmms-lab/LLaVA-Critic-R1-7B-Plus-Mimo

8B • Updated Aug 28 • 103
lmms-lab/LLaVA-Critic-R1-7B-LLaMA32v

11B • Updated Aug 28 • 10

MMSearch-R1

MMSearch-R1 is a solution designed to train LMMs to perform on-demand multimodal search in real-world environment.

lmms-lab/MMSearch-R1-7B-0807

8B • Updated Aug 7 • 20
lmms-lab/MMSearch-R1-7B

8B • Updated Jul 30 • 34 • 7
lmms-lab/FVQA

Viewer • Updated Aug 9 • 6.66k • 353 • 6
MMSearch-R1: Incentivizing LMMs to Search

Paper • 2506.20670 • Published Jun 25 • 64

Aero-1-Audio

Runtime error

43

43

Aero 1 Audio Demo

💬

Demo for Aero-1-Audio
lmms-lab/Aero-1-Audio

Text Generation • 2B • Updated Jun 7 • 694 • 88

EgoLife

CVPR 2025 - EgoLife: Towards Egocentric Life Assistant. Homepage: https://egolife-ai.github.io/

EgoLife: Towards Egocentric Life Assistant

Paper • 2503.03803 • Published Mar 5 • 44
Runtime error

14

14

EgoGPT

👁

Analyze video to describe actions and transcribe audio
lmms-lab/EgoIT-99K

Viewer • Updated Mar 7 • 199k • 4.99k • 7
lmms-lab/EgoLife

Viewer • Updated Mar 13 • 32k • 10.9k • 15

VideoMMMU

Video-MMMU: Evaluating Knowledge Acquisition from Multi-Discipline Professional Videos

Paper • 2501.13826 • Published Jan 23 • 25
lmms-lab/VideoMMMU

Viewer • Updated May 5 • 900 • 2.62k • 10

Multimodal-SAE

The collection of the sae that hooked on llava

Running on Zero

9

9

Multimodal SAE

💬

Demo for Multimodal-SAE
Large Multi-modal Models Can Interpret Features in Large Multi-modal Models

Paper • 2411.14982 • Published Nov 22, 2024 • 19
lmms-lab/llava-sae-explanations-5k

Viewer • Updated Nov 22, 2024 • 9.8k • 90 • 5
lmms-lab/llama3-llava-next-8b-hf-sae-131k

Updated Nov 26, 2024 • 16 • 7

LLaVA-Critic

as a general evaluator for assessing model performance

LLaVA-Critic: Learning to Evaluate Multimodal Models

Paper • 2410.02712 • Published Oct 3, 2024 • 37
lmms-lab/llava-critic-7b

8B • Updated Oct 4, 2024 • 654 • 14
lmms-lab/llava-critic-72b

73B • Updated Oct 4, 2024 • 57 • 15
lmms-lab/llava-critic-113k

Viewer • Updated Oct 5, 2024 • 113k • 403 • 28

LLaVA-Video

Models focus on video understanding (previously known as LLaVA-NeXT-Video).

Video Instruction Tuning With Synthetic Data

Paper • 2410.02713 • Published Oct 3, 2024 • 39
lmms-lab/LLaVA-Video-178K

Viewer • Updated Oct 11, 2024 • 1.63M • 33.3k • 171
lmms-lab/LLaVA-Video-7B-Qwen2

Video-Text-to-Text • 8B • Updated Oct 25, 2024 • 39.1k • 112
lmms-lab/LLaVA-Video-72B-Qwen2

Text Generation • 73B • Updated Oct 25, 2024 • 357 • 20

LLaVA-OneVision

a model good at arbitrary types of visual input

LLaVA-OneVision: Easy Visual Task Transfer

Paper • 2408.03326 • Published Aug 6, 2024 • 60
lmms-lab/LLaVA-OneVision-Mid-Data

Viewer • Updated Aug 26, 2024 • 563k • 314 • 21
lmms-lab/LLaVA-OneVision-Data

Viewer • Updated May 24 • 3.94M • 16.9k • 220
lmms-lab/LLaVA-NeXT-Data

Viewer • Updated Aug 30, 2024 • 779k • 2.26k • 41

LMMs-Eval

Dataset Collection of LMMs-Eval

LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models

Paper • 2407.12772 • Published Jul 17, 2024 • 35
lmms-lab/VQAv2

Viewer • Updated Jan 26, 2024 • 770k • 5.51k • 25
lmms-lab/MME

Viewer • Updated Dec 23, 2023 • 2.37k • 14.1k • 23
lmms-lab/DocVQA

Viewer • Updated Apr 18, 2024 • 16.6k • 13.1k • 55

LongVA

Long Context Transfer From Text To Vision: https://lmms-lab.github.io/posts/longva/

Long Context Transfer from Language to Vision

Paper • 2406.16852 • Published Jun 24, 2024 • 33
lmms-lab/LongVA-7B

Text Generation • 8B • Updated Jun 26, 2024 • 525 • 15
lmms-lab/LongVA-7B-DPO

Text Generation • 8B • Updated Jun 26, 2024 • 560 • 9
lmms-lab/v_niah_needles

Viewer • Updated Jun 15, 2024 • 5 • 9 • 4

LLaVA-Next-Interleave

LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models

Paper • 2407.07895 • Published Jul 10, 2024 • 42
lmms-lab/llava-next-interleave-qwen-7b

Text Generation • 8B • Updated Jul 24, 2024 • 216 • 27
lmms-lab/llava-next-interleave-qwen-7b-dpo

Text Generation • 8B • Updated Jul 12, 2024 • 23 • 12
lmms-lab/M4-Instruct-Data

Updated Jul 21, 2024 • 2.6k • 75

LLaVA-NeXT

Some powerful image models.

lmms-lab/llava-next-110b

Text Generation • 112B • Updated May 14, 2024 • 13 • 21
lmms-lab/llava-next-72b

Text Generation • 73B • Updated Aug 22, 2024 • 48 • 14
lmms-lab/llava-next-qwen-32b

Text Generation • 33B • Updated Jul 16, 2024 • 88 • 7
lmms-lab/llama3-llava-next-8b

Text Generation • 8B • Updated Aug 17, 2024 • 3.04k • 100

LMMs-Eval-Lite

Making Lite version of the dataset to accelerate holistic evaluation during model development!

lmms-lab/CMMMU

Viewer • Updated Mar 8, 2024 • 12k • 478 • 4

AI & ML interests

Recent Activity

Team members 34

lmms-lab 's collections 15

EgoGPT

Multimodal SAE

Aero 1 Audio Demo

Aero 1 Audio Demo

EgoGPT

Multimodal SAE