SmolVLM - small yet mighty Vision Language Model
β’
107
Merve Noyan
merve
AI & ML interests
VLMs, vision & co
Recent Activity
posted
an
update
1 day ago
small but mighty π₯
you can fine-tune SmolVLM on an L4 with batch size of 4 and it will only take 16.4 GB VRAM π«°π» also with gradient accumulation simulated batch size is 16 β¨
I made a notebook that includes all the goodies: QLoRA, gradient accumulation, gradient checkpointing with explanations on how they work π https://github.com/huggingface/smollm/blob/main/finetuning/Smol_VLM_FT.ipynb
posted
an
update
1 day ago
Last week we were blessed with open-source models! A recap π
https://huggingface.co/collections/merve/nov-29-releases-674ccc255a57baf97b1e2d31
πΌοΈ Multimodal
> At Hugging Face we released SmolVLM, a performant and efficient smol vision language model π
> Show Lab released ShowUI-2B: new vision-language-action model to build GUI/web automation agents π€
> Rhymes AI has released the base model of Aria: Aria-Base-64K and Aria-Base-8K with their respective context length
> ViDoRe team released ColSmolVLM: A new ColPali-like retrieval model based on SmolVLM
> Dataset: Llava-CoT-o1-Instruct: new dataset labelled using Llava-CoT multimodal reasoning modelπ
> Dataset: LLaVA-CoT-100k dataset used to train Llava-CoT released by creators of Llava-CoT π
π¬ LLMs
> Qwen team released QwQ-32B-Preview, state-of-the-art open-source reasoning model, broke the internet π₯
> AliBaba has released Marco-o1, a new open-source reasoning model π₯
> NVIDIA released Hymba 1.5B Base and Instruct, the new state-of-the-art SLMs with hybrid architecture (Mamba + transformer)
β―οΈ Image/Video Generation
> Qwen2VL-Flux: new image generation model based on Qwen2VL image encoder, T5 and Flux for generation
> Lightricks released LTX-Video, a new DiT-based video generation model that can generate 24 FPS videos at 768x512 res β―οΈ
> Dataset: Image Preferences is a new image generation preference dataset made with DIBT community effort of Argilla π·οΈ
Audio
> OuteAI released OuteTTS-0.2-500M new multilingual text-to-speech model based on Qwen-2.5-0.5B trained on 5B audio prompt tokens
updated
a collection
1 day ago
Nov 29 Releases π²π²
Articles
Organizations
Posts
79
Post
1313
small but mighty π₯
you can fine-tune SmolVLM on an L4 with batch size of 4 and it will only take 16.4 GB VRAM π«°π» also with gradient accumulation simulated batch size is 16 β¨
I made a notebook that includes all the goodies: QLoRA, gradient accumulation, gradient checkpointing with explanations on how they work π https://github.com/huggingface/smollm/blob/main/finetuning/Smol_VLM_FT.ipynb
you can fine-tune SmolVLM on an L4 with batch size of 4 and it will only take 16.4 GB VRAM π«°π» also with gradient accumulation simulated batch size is 16 β¨
I made a notebook that includes all the goodies: QLoRA, gradient accumulation, gradient checkpointing with explanations on how they work π https://github.com/huggingface/smollm/blob/main/finetuning/Smol_VLM_FT.ipynb
Post
1779
Last week we were blessed with open-source models! A recap π
merve/nov-29-releases-674ccc255a57baf97b1e2d31
πΌοΈ Multimodal
> At Hugging Face we released SmolVLM, a performant and efficient smol vision language model π
> Show Lab released ShowUI-2B: new vision-language-action model to build GUI/web automation agents π€
> Rhymes AI has released the base model of Aria: Aria-Base-64K and Aria-Base-8K with their respective context length
> ViDoRe team released ColSmolVLM: A new ColPali-like retrieval model based on SmolVLM
> Dataset: Llava-CoT-o1-Instruct: new dataset labelled using Llava-CoT multimodal reasoning modelπ
> Dataset: LLaVA-CoT-100k dataset used to train Llava-CoT released by creators of Llava-CoT π
π¬ LLMs
> Qwen team released QwQ-32B-Preview, state-of-the-art open-source reasoning model, broke the internet π₯
> AliBaba has released Marco-o1, a new open-source reasoning model π₯
> NVIDIA released Hymba 1.5B Base and Instruct, the new state-of-the-art SLMs with hybrid architecture (Mamba + transformer)
β―οΈ Image/Video Generation
> Qwen2VL-Flux: new image generation model based on Qwen2VL image encoder, T5 and Flux for generation
> Lightricks released LTX-Video, a new DiT-based video generation model that can generate 24 FPS videos at 768x512 res β―οΈ
> Dataset: Image Preferences is a new image generation preference dataset made with DIBT community effort of Argilla π·οΈ
Audio
> OuteAI released OuteTTS-0.2-500M new multilingual text-to-speech model based on Qwen-2.5-0.5B trained on 5B audio prompt tokens
merve/nov-29-releases-674ccc255a57baf97b1e2d31
πΌοΈ Multimodal
> At Hugging Face we released SmolVLM, a performant and efficient smol vision language model π
> Show Lab released ShowUI-2B: new vision-language-action model to build GUI/web automation agents π€
> Rhymes AI has released the base model of Aria: Aria-Base-64K and Aria-Base-8K with their respective context length
> ViDoRe team released ColSmolVLM: A new ColPali-like retrieval model based on SmolVLM
> Dataset: Llava-CoT-o1-Instruct: new dataset labelled using Llava-CoT multimodal reasoning modelπ
> Dataset: LLaVA-CoT-100k dataset used to train Llava-CoT released by creators of Llava-CoT π
π¬ LLMs
> Qwen team released QwQ-32B-Preview, state-of-the-art open-source reasoning model, broke the internet π₯
> AliBaba has released Marco-o1, a new open-source reasoning model π₯
> NVIDIA released Hymba 1.5B Base and Instruct, the new state-of-the-art SLMs with hybrid architecture (Mamba + transformer)
β―οΈ Image/Video Generation
> Qwen2VL-Flux: new image generation model based on Qwen2VL image encoder, T5 and Flux for generation
> Lightricks released LTX-Video, a new DiT-based video generation model that can generate 24 FPS videos at 768x512 res β―οΈ
> Dataset: Image Preferences is a new image generation preference dataset made with DIBT community effort of Argilla π·οΈ
Audio
> OuteAI released OuteTTS-0.2-500M new multilingual text-to-speech model based on Qwen-2.5-0.5B trained on 5B audio prompt tokens
spaces
104
models
87
merve/paligemma_vqav2
Image-Text-to-Text
β’
Updated
β’
266
β’
11
merve/google-ckpts
Updated
merve/google-tokenizers
Updated
merve/idefics3-llama-vqav2
Updated
merve/idefics3llama-vqav2
Updated
β’
8
merve/flux-dreambooth-lora
Updated
β’
1
merve/trained-flux-lora-lego
Text-to-Image
β’
Updated
β’
10
β’
β’
1
merve/flux-lego-lora-dreambooth
Text-to-Image
β’
Updated
β’
2.62k
β’
β’
13
merve/sam2-hiera-large
Mask Generation
β’
Updated
β’
6.27k
β’
2
merve/sam2-hiera-base-plus
Mask Generation
β’
Updated
β’
41
datasets
26
merve/model-test-inputs
Updated
β’
42
merve/vqav2-small
Viewer
β’
Updated
β’
21.4k
β’
1.01k
β’
8
merve/SGinW
Preview
β’
Updated
β’
460
merve/pascal-voc
Viewer
β’
Updated
β’
336k
β’
528
merve/YouCook2
Viewer
β’
Updated
β’
2k
β’
61
merve/faiss_embeddings
Updated
β’
19
merve/pokemon-ds-embeddings
Viewer
β’
Updated
β’
833
β’
68
β’
4
merve/tr-h4-norobots
Updated
β’
73
β’
10
merve/lego_sets_latest
Viewer
β’
Updated
β’
61
β’
151
β’
4
merve/ai-tube-dummy
Updated
β’
56