SmolVLM - small yet mighty Vision Language Model
β’
108
Merve Noyan
merve
AI & ML interests
VLMs, vision & co
Recent Activity
posted
an
update
1 day ago
small but mighty π₯
you can fine-tune SmolVLM on an L4 with batch size of 4 and it will only take 16.4 GB VRAM π«°π» also with gradient accumulation simulated batch size is 16 β¨
I made a notebook that includes all the goodies: QLoRA, gradient accumulation, gradient checkpointing with explanations on how they work π https://github.com/huggingface/smollm/blob/main/finetuning/Smol_VLM_FT.ipynb
posted
an
update
2 days ago
Last week we were blessed with open-source models! A recap π
https://huggingface.co/collections/merve/nov-29-releases-674ccc255a57baf97b1e2d31
πΌοΈ Multimodal
> At Hugging Face we released SmolVLM, a performant and efficient smol vision language model π
> Show Lab released ShowUI-2B: new vision-language-action model to build GUI/web automation agents π€
> Rhymes AI has released the base model of Aria: Aria-Base-64K and Aria-Base-8K with their respective context length
> ViDoRe team released ColSmolVLM: A new ColPali-like retrieval model based on SmolVLM
> Dataset: Llava-CoT-o1-Instruct: new dataset labelled using Llava-CoT multimodal reasoning modelπ
> Dataset: LLaVA-CoT-100k dataset used to train Llava-CoT released by creators of Llava-CoT π
π¬ LLMs
> Qwen team released QwQ-32B-Preview, state-of-the-art open-source reasoning model, broke the internet π₯
> AliBaba has released Marco-o1, a new open-source reasoning model π₯
> NVIDIA released Hymba 1.5B Base and Instruct, the new state-of-the-art SLMs with hybrid architecture (Mamba + transformer)
β―οΈ Image/Video Generation
> Qwen2VL-Flux: new image generation model based on Qwen2VL image encoder, T5 and Flux for generation
> Lightricks released LTX-Video, a new DiT-based video generation model that can generate 24 FPS videos at 768x512 res β―οΈ
> Dataset: Image Preferences is a new image generation preference dataset made with DIBT community effort of Argilla π·οΈ
Audio
> OuteAI released OuteTTS-0.2-500M new multilingual text-to-speech model based on Qwen-2.5-0.5B trained on 5B audio prompt tokens
updated
a collection
2 days ago
Nov 29 Releases π²π²
Articles
Organizations
merve's activity
posted
an
update
1 day ago
Post
1802
small but mighty π₯
you can fine-tune SmolVLM on an L4 with batch size of 4 and it will only take 16.4 GB VRAM π«°π» also with gradient accumulation simulated batch size is 16 β¨
I made a notebook that includes all the goodies: QLoRA, gradient accumulation, gradient checkpointing with explanations on how they work π https://github.com/huggingface/smollm/blob/main/finetuning/Smol_VLM_FT.ipynb
you can fine-tune SmolVLM on an L4 with batch size of 4 and it will only take 16.4 GB VRAM π«°π» also with gradient accumulation simulated batch size is 16 β¨
I made a notebook that includes all the goodies: QLoRA, gradient accumulation, gradient checkpointing with explanations on how they work π https://github.com/huggingface/smollm/blob/main/finetuning/Smol_VLM_FT.ipynb
posted
an
update
2 days ago
Post
2242
Last week we were blessed with open-source models! A recap π
merve/nov-29-releases-674ccc255a57baf97b1e2d31
πΌοΈ Multimodal
> At Hugging Face we released SmolVLM, a performant and efficient smol vision language model π
> Show Lab released ShowUI-2B: new vision-language-action model to build GUI/web automation agents π€
> Rhymes AI has released the base model of Aria: Aria-Base-64K and Aria-Base-8K with their respective context length
> ViDoRe team released ColSmolVLM: A new ColPali-like retrieval model based on SmolVLM
> Dataset: Llava-CoT-o1-Instruct: new dataset labelled using Llava-CoT multimodal reasoning modelπ
> Dataset: LLaVA-CoT-100k dataset used to train Llava-CoT released by creators of Llava-CoT π
π¬ LLMs
> Qwen team released QwQ-32B-Preview, state-of-the-art open-source reasoning model, broke the internet π₯
> AliBaba has released Marco-o1, a new open-source reasoning model π₯
> NVIDIA released Hymba 1.5B Base and Instruct, the new state-of-the-art SLMs with hybrid architecture (Mamba + transformer)
β―οΈ Image/Video Generation
> Qwen2VL-Flux: new image generation model based on Qwen2VL image encoder, T5 and Flux for generation
> Lightricks released LTX-Video, a new DiT-based video generation model that can generate 24 FPS videos at 768x512 res β―οΈ
> Dataset: Image Preferences is a new image generation preference dataset made with DIBT community effort of Argilla π·οΈ
Audio
> OuteAI released OuteTTS-0.2-500M new multilingual text-to-speech model based on Qwen-2.5-0.5B trained on 5B audio prompt tokens
merve/nov-29-releases-674ccc255a57baf97b1e2d31
πΌοΈ Multimodal
> At Hugging Face we released SmolVLM, a performant and efficient smol vision language model π
> Show Lab released ShowUI-2B: new vision-language-action model to build GUI/web automation agents π€
> Rhymes AI has released the base model of Aria: Aria-Base-64K and Aria-Base-8K with their respective context length
> ViDoRe team released ColSmolVLM: A new ColPali-like retrieval model based on SmolVLM
> Dataset: Llava-CoT-o1-Instruct: new dataset labelled using Llava-CoT multimodal reasoning modelπ
> Dataset: LLaVA-CoT-100k dataset used to train Llava-CoT released by creators of Llava-CoT π
π¬ LLMs
> Qwen team released QwQ-32B-Preview, state-of-the-art open-source reasoning model, broke the internet π₯
> AliBaba has released Marco-o1, a new open-source reasoning model π₯
> NVIDIA released Hymba 1.5B Base and Instruct, the new state-of-the-art SLMs with hybrid architecture (Mamba + transformer)
β―οΈ Image/Video Generation
> Qwen2VL-Flux: new image generation model based on Qwen2VL image encoder, T5 and Flux for generation
> Lightricks released LTX-Video, a new DiT-based video generation model that can generate 24 FPS videos at 768x512 res β―οΈ
> Dataset: Image Preferences is a new image generation preference dataset made with DIBT community effort of Argilla π·οΈ
Audio
> OuteAI released OuteTTS-0.2-500M new multilingual text-to-speech model based on Qwen-2.5-0.5B trained on 5B audio prompt tokens
posted
an
update
7 days ago
Post
2078
The authors of ColPali trained a retrieval model based on SmolVLM π€
vidore/colsmolvlm-alpha
TLDR;
- ColSmolVLM performs better than ColPali and DSE-Qwen2 on all English tasks
- ColSmolVLM is more memory efficient than ColQwen2 π
TLDR;
- ColSmolVLM performs better than ColPali and DSE-Qwen2 on all English tasks
- ColSmolVLM is more memory efficient than ColQwen2 π
posted
an
update
8 days ago
Post
3704
Small yet mighty! π«
We are releasing SmolVLM: a new 2B small vision language made for on-device use, fine-tunable on consumer GPU, immensely memory efficient π€
We release three checkpoints under Apache 2.0: SmolVLM-Instruct, SmolVLM-Synthetic and SmolVLM-Base HuggingFaceTB/smolvlm-6740bd584b2dcbf51ecb1f39
Learn more from our blog here: huggingface.co/blog/smolvlm
This release comes with a demo, fine-tuning code, MLX integration and TRL integration for DPO π
Try the demo: HuggingFaceTB/SmolVLM
Fine-tuning Recipe: https://github.com/huggingface/smollm/blob/main/finetuning/Smol_VLM_FT.ipynb
Also TRL integration for DPO π
We are releasing SmolVLM: a new 2B small vision language made for on-device use, fine-tunable on consumer GPU, immensely memory efficient π€
We release three checkpoints under Apache 2.0: SmolVLM-Instruct, SmolVLM-Synthetic and SmolVLM-Base HuggingFaceTB/smolvlm-6740bd584b2dcbf51ecb1f39
Learn more from our blog here: huggingface.co/blog/smolvlm
This release comes with a demo, fine-tuning code, MLX integration and TRL integration for DPO π
Try the demo: HuggingFaceTB/SmolVLM
Fine-tuning Recipe: https://github.com/huggingface/smollm/blob/main/finetuning/Smol_VLM_FT.ipynb
Also TRL integration for DPO π
posted
an
update
12 days ago
Post
2525
What a week! A recap for everything you missed βοΈ
merve/nov-22-releases-673fbbcfc1c97c4f411def07
Multimodal β¨
> Mistral AI
released Pixtral 124B, a gigantic open vision language model
> Llava-CoT (formerly known as Llava-o1) was released, a multimodal reproduction of o1 model by PKU
> OpenGVLab released MMPR: a new multimodal reasoning dataset
> Jina has released Jina-CLIP-v2 0.98B multilingual multimodal embeddings
> Apple released new SotA vision encoders AIMv2
LLMs π¦
> AllenAI dropped a huge release of models, datasets and scripts for TΓΌlu, a family of models based on Llama 3.1 aligned with SFT, DPO and a new technique they have developed called RLVR
> Jina has released embeddings-v3: new multilingual embeddings with longer context
> Hugging Face released SmolTalk: synthetic dataset used to align SmolLM2 using supervised fine-tuning
> Microsoft released orca-agentinstruct-1M-v1: a gigantic instruction dataset of 1M synthetic instruction pairs
Image Generation πΌοΈ
> Black Forest Labs released Flux 1. tools: four new models for different image modifications and two LoRAs to do image conditioning and better steer generations
Lastly Hugging Face released a new library Observers: a lightweight SDK for monitoring interactions with AI APIs and easily store and browse them π
$ pip install observers
merve/nov-22-releases-673fbbcfc1c97c4f411def07
Multimodal β¨
> Mistral AI
released Pixtral 124B, a gigantic open vision language model
> Llava-CoT (formerly known as Llava-o1) was released, a multimodal reproduction of o1 model by PKU
> OpenGVLab released MMPR: a new multimodal reasoning dataset
> Jina has released Jina-CLIP-v2 0.98B multilingual multimodal embeddings
> Apple released new SotA vision encoders AIMv2
LLMs π¦
> AllenAI dropped a huge release of models, datasets and scripts for TΓΌlu, a family of models based on Llama 3.1 aligned with SFT, DPO and a new technique they have developed called RLVR
> Jina has released embeddings-v3: new multilingual embeddings with longer context
> Hugging Face released SmolTalk: synthetic dataset used to align SmolLM2 using supervised fine-tuning
> Microsoft released orca-agentinstruct-1M-v1: a gigantic instruction dataset of 1M synthetic instruction pairs
Image Generation πΌοΈ
> Black Forest Labs released Flux 1. tools: four new models for different image modifications and two LoRAs to do image conditioning and better steer generations
Lastly Hugging Face released a new library Observers: a lightweight SDK for monitoring interactions with AI APIs and easily store and browse them π
$ pip install observers
- 3 replies
posted
an
update
12 days ago
Post
1459
Apple released AIMv2 π a family of state-of-the-art open-set vision encoders
apple/aimv2-6720fe1558d94c7805f7688c
> like CLIP, but add a decoder and train on autoregression π€―
> 19 open models come in 300M, 600M, 1.2B, 2.7B with resolutions of 224, 336, 448
> Load and use with π€ transformers
apple/aimv2-6720fe1558d94c7805f7688c
> like CLIP, but add a decoder and train on autoregression π€―
> 19 open models come in 300M, 600M, 1.2B, 2.7B with resolutions of 224, 336, 448
> Load and use with π€ transformers
posted
an
update
12 days ago
Post
2948
your hugging face profile now has your recent activities π€
posted
an
update
16 days ago
Post
1781
π€ transformers pipelines now support vision language models for easy local inference π«°π»
h/t @yonigozlan for shipping this π©π
you can also use inference API to infer hosted vision LMs (via Python, JS and cURL) https://huggingface.co/docs/api-inference/en/tasks/image-text-to-text
h/t @yonigozlan for shipping this π©π
you can also use inference API to infer hosted vision LMs (via Python, JS and cURL) https://huggingface.co/docs/api-inference/en/tasks/image-text-to-text
reacted to
sayakpaul's
post with β€οΈ
16 days ago
Post
2476
It's been a while we shipped native quantization support in
We currently support
This post is just a reminder of what's possible:
1. Loading a model with a quantization config
2. Saving a model with quantization config
3. Loading a pre-quantized model
4.
5. Training and loading LoRAs into quantized checkpoints
Docs:
https://huggingface.co/docs/diffusers/main/en/quantization/bitsandbytes
diffusers π§¨We currently support
bistandbytes as the official backend but using others like torchao is already very simple. This post is just a reminder of what's possible:
1. Loading a model with a quantization config
2. Saving a model with quantization config
3. Loading a pre-quantized model
4.
enable_model_cpu_offload()5. Training and loading LoRAs into quantized checkpoints
Docs:
https://huggingface.co/docs/diffusers/main/en/quantization/bitsandbytes
- 1 reply
reacted to
BlinkDL's
post with π₯
16 days ago
Post
3106
RWKV-6-world-v3 (+3.1T tokens) is our best multilingual 7B model as of now:
BlinkDL/rwkv-6-world
It's 100% RNN and attention-free. MMLU 54.2% (previous world-v2.1 = 47.9%. note: without eval-boosting tricks such as annealing).
RWKV-7-world-v4 soon :)
It's 100% RNN and attention-free. MMLU 54.2% (previous world-v2.1 = 47.9%. note: without eval-boosting tricks such as annealing).
RWKV-7-world-v4 soon :)
reacted to
davidberenstein1957's
post with π₯π
16 days ago
Post
1909
For anyone who struggles with NER or information extraction with LLM.
We showed an efficient workflow for token classification including zero-shot suggestions and model fine-tuning with Argilla, GliNER, the NuMind NuExtract LLM and SpanMarker. @argilla
Video: https://youtu.be/JvLpaYgNd84?feature=shared
Notebooks and slides included to try it yourself π
We showed an efficient workflow for token classification including zero-shot suggestions and model fine-tuning with Argilla, GliNER, the NuMind NuExtract LLM and SpanMarker. @argilla
Video: https://youtu.be/JvLpaYgNd84?feature=shared
Notebooks and slides included to try it yourself π
reacted to
erikkaum's
post with ππ₯
16 days ago
Post
1680
A while ago I started experimenting with compiling the Python interpreter to WASM.
To build a secure, fast, and lightweight sandbox for code execution β ideal for running LLM-generated Python code.
- Send code simply as a POST request
- 1-2ms startup times
Hack away:
https://github.com/ErikKaum/runner
To build a secure, fast, and lightweight sandbox for code execution β ideal for running LLM-generated Python code.
- Send code simply as a POST request
- 1-2ms startup times
Hack away:
https://github.com/ErikKaum/runner
reacted to
AdinaY's
post with π
16 days ago
Post
1460
LLaMA Mesh π₯ Unifying 3D Mesh Generation with Language Models
Model: Zhengyi/LLaMA-Mesh
Demo: Zhengyi/LLaMA-Mesh
Paper: LLaMA-Mesh: Unifying 3D Mesh Generation with Language Models (2411.09595)
β¨ Unified 3D generation & text understanding.
β¨ 3D meshes as plain text for seamless LLM integration.
β¨ High-quality 3D outputs rivaling specialized models.
Model: Zhengyi/LLaMA-Mesh
Demo: Zhengyi/LLaMA-Mesh
Paper: LLaMA-Mesh: Unifying 3D Mesh Generation with Language Models (2411.09595)
β¨ Unified 3D generation & text understanding.
β¨ 3D meshes as plain text for seamless LLM integration.
β¨ High-quality 3D outputs rivaling specialized models.
reacted to
prithivMLmods's
post with π€
16 days ago
Post
3913
Minimalistic Adapters π
πDemo Here:
prithivMLmods/FLUX-LoRA-DLC
πModel:
{ Quote Tuner } : prithivMLmods/Flux.1-Dev-Quote-LoRA
{ Stamp Art } : prithivMLmods/Flux.1-Dev-Stamp-Art-LoRA
{ Hand Sticky } : prithivMLmods/Flux.1-Dev-Hand-Sticky-LoRA
{ Poster HQ } : prithivMLmods/Flux.1-Dev-Poster-HQ-LoRA
{ Ctoon Min } : prithivMLmods/Flux.1-Dev-Ctoon-LoRA
πCollection:
{ Flux LoRA Collection} : prithivMLmods/flux-lora-collections-66dd5908be2206cfaa8519be
{ LoRA Space Collection } : prithivMLmods/lora-space-collections-6714b72e0d49e1c97fbd6a32
πFor More Visit
https://huggingface.co/strangerzonehf
.
.
.
π€@prithivMLmods
πDemo Here:
prithivMLmods/FLUX-LoRA-DLC
πModel:
{ Quote Tuner } : prithivMLmods/Flux.1-Dev-Quote-LoRA
{ Stamp Art } : prithivMLmods/Flux.1-Dev-Stamp-Art-LoRA
{ Hand Sticky } : prithivMLmods/Flux.1-Dev-Hand-Sticky-LoRA
{ Poster HQ } : prithivMLmods/Flux.1-Dev-Poster-HQ-LoRA
{ Ctoon Min } : prithivMLmods/Flux.1-Dev-Ctoon-LoRA
πCollection:
{ Flux LoRA Collection} : prithivMLmods/flux-lora-collections-66dd5908be2206cfaa8519be
{ LoRA Space Collection } : prithivMLmods/lora-space-collections-6714b72e0d49e1c97fbd6a32
πFor More Visit
https://huggingface.co/strangerzonehf
.
.
.
π€@prithivMLmods
- 3 replies
reacted to
hexgrad's
post with π₯
16 days ago
Post
3200
Kokoro: a small, fast 80M param TTS model hosted on ZeroGPU at
hexgrad/Kokoro-TTS
- 3 replies
posted
an
update
17 days ago
Post
4841
OmniVision-968M: a new local VLM for edge devices, fast & small but performant
π¨ a new vision language model with 9x less image tokens, super efficient
π aligned with DPO for reducing hallucinations
β‘οΈ Apache 2.0 license π₯
Demo hf.co/spaces/NexaAIDev/omnivlm-dpo-demo
Model NexaAIDev/omnivision-968M
π¨ a new vision language model with 9x less image tokens, super efficient
π aligned with DPO for reducing hallucinations
β‘οΈ Apache 2.0 license π₯
Demo hf.co/spaces/NexaAIDev/omnivlm-dpo-demo
Model NexaAIDev/omnivision-968M
- 4 replies
reacted to
AdinaY's
post with ππ₯
19 days ago
Post
2524
Letβs dive into the exciting releases from the Chinese community last week π₯π
More details π https://huggingface.co/zh-ai-community
Code model:
β¨Qwen 2.5 coder by Alibaba Qwen
Qwen/qwen25-coder-66eaa22e6f99801bf65b0c2f
β¨OpenCoder by InflyAI - Fully open code modelπ
infly/opencoder-672cec44bbb86c39910fb55e
Image model:
β¨Hunyuan3D-1.0 by Tencent
tencent/Hunyuan3D-1
MLLM:
β¨JanusFlow by DeepSeek
deepseek-ai/JanusFlow-1.3B
deepseek-ai/JanusFlow-1.3B
β¨Mono-InternVL-2B by OpenGVlab
OpenGVLab/Mono-InternVL-2B
Video model:
β¨CogVideoX 1.5 by ChatGLM
THUDM/CogVideoX1.5-5B-SAT
Audio model:
β¨Fish Agent by FishAudio
fishaudio/fish-agent-v0.1-3b
Dataset:
β¨OPI dataset by BAAIBeijing
BAAI/OPI
More details π https://huggingface.co/zh-ai-community
Code model:
β¨Qwen 2.5 coder by Alibaba Qwen
Qwen/qwen25-coder-66eaa22e6f99801bf65b0c2f
β¨OpenCoder by InflyAI - Fully open code modelπ
infly/opencoder-672cec44bbb86c39910fb55e
Image model:
β¨Hunyuan3D-1.0 by Tencent
tencent/Hunyuan3D-1
MLLM:
β¨JanusFlow by DeepSeek
deepseek-ai/JanusFlow-1.3B
deepseek-ai/JanusFlow-1.3B
β¨Mono-InternVL-2B by OpenGVlab
OpenGVLab/Mono-InternVL-2B
Video model:
β¨CogVideoX 1.5 by ChatGLM
THUDM/CogVideoX1.5-5B-SAT
Audio model:
β¨Fish Agent by FishAudio
fishaudio/fish-agent-v0.1-3b
Dataset:
β¨OPI dataset by BAAIBeijing
BAAI/OPI