DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding
Paper
โข
2412.10302
โข
Published
โข
17
Jฤdrzej Grabala
jgitsolutions
AI & ML interests
Local Drive Human Overseered System of Agents, LLMs, Langchains & other useful stuff on mid-to-low-end of commercial hardware.
Recent Activity
upvoted
a
collection
8 days ago
DeepSeek-VL2
liked
a Space
14 days ago
depth-anything/Video-Depth-Anything
Organizations
jgitsolutions's activity
upvoted
a
paper
8 days ago
upvoted
a
collection
8 days ago
liked
a
Space
14 days ago
reacted to
chansung's
post with ๐
14 days ago
Post
2846
Simple Paper Review #5
I briefly reviewed the paper "SFT Memorizes, RL Generalizes," which compares SFT and RL in post-training of LLM/VLM from HKU, UC Berkeley, Google DeepMind, and New York University
The conclusion suggests SFT excels in memorization, while RL is better for generalization. However, since LLM/VLM should benefit humans beyond just generalization, a mix of SFT and RL is advisable. Typically, some SFT is followed by RL to understand prompt formats and enhance generalization through trial and error.
The study focused on one model, Llama-3.2-Vision-11B, using environments like General Points for arithmetic reasoning and V-IRL for spatial reasoning. Training data was used for both SFT and RL, with evaluations on in-distribution and out-of-distribution data to assess memorization and generalization.
I want to apply RL extensively, but it requires building a similar simulation environment. For domain-specific models, significant investment in creating a "playground" for the model is crucial, as the effort will directly influence the outcomes.
https://arxiv.org/abs/2501.17161
I briefly reviewed the paper "SFT Memorizes, RL Generalizes," which compares SFT and RL in post-training of LLM/VLM from HKU, UC Berkeley, Google DeepMind, and New York University
The conclusion suggests SFT excels in memorization, while RL is better for generalization. However, since LLM/VLM should benefit humans beyond just generalization, a mix of SFT and RL is advisable. Typically, some SFT is followed by RL to understand prompt formats and enhance generalization through trial and error.
The study focused on one model, Llama-3.2-Vision-11B, using environments like General Points for arithmetic reasoning and V-IRL for spatial reasoning. Training data was used for both SFT and RL, with evaluations on in-distribution and out-of-distribution data to assess memorization and generalization.
I want to apply RL extensively, but it requires building a similar simulation environment. For domain-specific models, significant investment in creating a "playground" for the model is crucial, as the effort will directly influence the outcomes.
https://arxiv.org/abs/2501.17161
liked
a
Space
15 days ago
liked
a
Space
17 days ago
upvoted
an
article
17 days ago
Article
Welcome to Inference Providers on the Hub ๐ฅ
โข
377
liked
a
model
22 days ago
liked
a
model
2 months ago
liked
a
Space
2 months ago
liked
5
Spaces
3 months ago
54
tankwar
๐
"One-minute creation by AI Coding Autonomous Agent MOUSE-I"
112
MOUSE-Visual AI Chatbot
๐ฅ
Text2Visual Web Converter with AI Image Generation
1.97k
Anychat
๐ข
Browse and use coding demos from various providers
7.46k
Kolors Virtual Try-On
๐
Try on garments on virtual models
351
Logo In Context
๐ค
Add a logo to anything
reacted to
anakin87's
post with ๐
3 months ago
Post
1107
Ok, you're finally convinced that synthetic data works... โ๏ธ
๐๐จ๐ฐ ๐ฒ๐จ๐ฎ ๐ฐ๐๐ง๐ญ ๐ญ๐จ ๐ ๐๐ง๐๐ซ๐๐ญ๐ ๐๐ง ๐ข๐ง๐ฌ๐ญ๐ซ๐ฎ๐๐ญ๐ข๐จ๐ง ๐๐๐ญ๐๐ฌ๐๐ญ ๐๐จ๐ซ ๐๐ข๐ง๐-๐ญ๐ฎ๐ง๐ข๐ง๐ ๐ข๐ง ๐ ๐ฅ๐๐ง๐ ๐ฎ๐๐ ๐ ๐จ๐ญ๐ก๐๐ซ ๐ญ๐ก๐๐ง ๐๐ง๐ ๐ฅ๐ข๐ฌ๐ก.
But how do you get started?
I explore how to do this with Magpie in my new article
https://huggingface.co/blog/anakin87/multilingual-magpie
---
๐ฆโโฌ ๐๐ก๐๐ญ ๐ข๐ฌ ๐๐๐ ๐ฉ๐ข๐?
It's a recent technique for creating synthetic instruction datasets.
Magpie is based on a simple but ingenious idea ๐
if you prompt an instruction-tuned model with a pre-query template, you can make it generate a plausible user query/instruction
Here's an example:
model: Llama-3-8B-Instruct
pre-query template: "<|begin_of_text|><|start_header_id|>user<|end_header_id|>"
generated user instruction: "What are some of the responsibilities of a commercial pilot?"
You can then feed this instruction back into the same model to get the assistant response.
By repeating this process, it's possible to generate large synthetic datasets with relatively little effort.
๐ช The authors demonstrate that using these datasets for Supervised Fine Tuning (SFT) can yield strong performance, even competitive with the original instruct model.
๐ง๐๐๐ง๐๐ซ๐๐ญ๐ข๐ง๐ ๐ง๐จ๐ง-๐๐ง๐ ๐ฅ๐ข๐ฌ๐ก ๐๐๐ญ๐
Most Language Models are primarily trained on English texts, so they tend to produce data in English.
How can we overcome this?
Earlier approaches were complex or costly.
Then @mrm8488 found a simple solution: add the target language to the pre-query template.
For Spanish, the template becomes "<|begin_of_text|><|start_header_id|>user<|end_header_id|>spanish:".
This method works for Spanish and German!
โ Unfortunately, it does not work well for other languages (๐ฎ๐น, ๐ณ๐ฑ, ...)
๐
๐๐จ๐ฐ ๐ฒ๐จ๐ฎ ๐ฐ๐๐ง๐ญ ๐ญ๐จ ๐ ๐๐ง๐๐ซ๐๐ญ๐ ๐๐ง ๐ข๐ง๐ฌ๐ญ๐ซ๐ฎ๐๐ญ๐ข๐จ๐ง ๐๐๐ญ๐๐ฌ๐๐ญ ๐๐จ๐ซ ๐๐ข๐ง๐-๐ญ๐ฎ๐ง๐ข๐ง๐ ๐ข๐ง ๐ ๐ฅ๐๐ง๐ ๐ฎ๐๐ ๐ ๐จ๐ญ๐ก๐๐ซ ๐ญ๐ก๐๐ง ๐๐ง๐ ๐ฅ๐ข๐ฌ๐ก.
But how do you get started?
I explore how to do this with Magpie in my new article
https://huggingface.co/blog/anakin87/multilingual-magpie
---
๐ฆโโฌ ๐๐ก๐๐ญ ๐ข๐ฌ ๐๐๐ ๐ฉ๐ข๐?
It's a recent technique for creating synthetic instruction datasets.
Magpie is based on a simple but ingenious idea ๐
if you prompt an instruction-tuned model with a pre-query template, you can make it generate a plausible user query/instruction
Here's an example:
model: Llama-3-8B-Instruct
pre-query template: "<|begin_of_text|><|start_header_id|>user<|end_header_id|>"
generated user instruction: "What are some of the responsibilities of a commercial pilot?"
You can then feed this instruction back into the same model to get the assistant response.
By repeating this process, it's possible to generate large synthetic datasets with relatively little effort.
๐ช The authors demonstrate that using these datasets for Supervised Fine Tuning (SFT) can yield strong performance, even competitive with the original instruct model.
๐ง๐๐๐ง๐๐ซ๐๐ญ๐ข๐ง๐ ๐ง๐จ๐ง-๐๐ง๐ ๐ฅ๐ข๐ฌ๐ก ๐๐๐ญ๐
Most Language Models are primarily trained on English texts, so they tend to produce data in English.
How can we overcome this?
Earlier approaches were complex or costly.
Then @mrm8488 found a simple solution: add the target language to the pre-query template.
For Spanish, the template becomes "<|begin_of_text|><|start_header_id|>user<|end_header_id|>spanish:".
This method works for Spanish and German!
โ Unfortunately, it does not work well for other languages (๐ฎ๐น, ๐ณ๐ฑ, ...)
๐
- 1 reply
reacted to
yongchanghao's
post with ๐๐ฅ
3 months ago
Post
3765
We just released a paper (NeuZip) that compresses VRAM in a lossless manner to run larger models. This should be particularly useful when VRAM is insufficient during training/inference. Specifically, we look inside each floating number and find that the exponents are highly compressible (as shown in the figure below).
Read more about the work at NeuZip: Memory-Efficient Training and Inference with Dynamic Compression of Neural Networks (2410.20650)
Read more about the work at NeuZip: Memory-Efficient Training and Inference with Dynamic Compression of Neural Networks (2410.20650)
reacted to
PLB's
post with ๐
3 months ago
Post
1882
โ ๏ธ People selling AI chatbots for websites hate us.
Add an open source chat assistant on your website in 5 minutes: https://github.com/phospho-app/ai-chat-bubble
How does it work ?
- You give an URL
- The AI assistant crawls the website content and embed it
- Add it to your frontend in one line of code
- People on your website can ask the assistant questions
Powered by BAAI/bge-small-en-v1.5 and Mistral AI
Add an open source chat assistant on your website in 5 minutes: https://github.com/phospho-app/ai-chat-bubble
How does it work ?
- You give an URL
- The AI assistant crawls the website content and embed it
- Add it to your frontend in one line of code
- People on your website can ask the assistant questions
Powered by BAAI/bge-small-en-v1.5 and Mistral AI
- 5 replies
reacted to
LukeNeumann's
post with ๐
3 months ago
Post
1864
Hello Hugging Face community!
I wanted to introduce myself and my company @Overlaiapp . We are a collective of filmmakers, photographers, and AI engineers working on high resolution (8K+) training data.
We plan to share a lot of our datasets with the community and are kicking things off with two curated datasets:
- Overlaiai/OregonCoastin4K
- Overlaiai/SubArcticPolarBear
Overlai.ai Dataset Features
๐ฅ Oversampled: Every clip is captured in stunning 8K resolution, delivering rich detail ideal for fine tuning scenic landscapes and ocean dynamics.
๐ธ Variance: Includes close-up details, slow-motion footage of crashing waves, sweeping landscapes, and wildlife shots.
๐ Detailed Metadata: Every clip is paired with structured metadata, including creative descriptions, precise camera movements, lens information, field of view calculations, and shot settings, ensuring AI models can fully understand and replicate real-world cinematography with accuracy.
โ๏ธ Consistency: Re-thinking training data at the point of capture by "overshooting" a subject, enabling models to learn more nuanced relationships and views across scenes.
๐ Light: Shot during early morning and sunset light for optimal color contrast and dynamic range, maximizing visual quality for color and lighting-sensitive tasks.
๐ Curation: Curated specifically for machine learning, providing clean, high-quality data for next generation model training.
I wanted to introduce myself and my company @Overlaiapp . We are a collective of filmmakers, photographers, and AI engineers working on high resolution (8K+) training data.
We plan to share a lot of our datasets with the community and are kicking things off with two curated datasets:
- Overlaiai/OregonCoastin4K
- Overlaiai/SubArcticPolarBear
Overlai.ai Dataset Features
๐ฅ Oversampled: Every clip is captured in stunning 8K resolution, delivering rich detail ideal for fine tuning scenic landscapes and ocean dynamics.
๐ธ Variance: Includes close-up details, slow-motion footage of crashing waves, sweeping landscapes, and wildlife shots.
๐ Detailed Metadata: Every clip is paired with structured metadata, including creative descriptions, precise camera movements, lens information, field of view calculations, and shot settings, ensuring AI models can fully understand and replicate real-world cinematography with accuracy.
โ๏ธ Consistency: Re-thinking training data at the point of capture by "overshooting" a subject, enabling models to learn more nuanced relationships and views across scenes.
๐ Light: Shot during early morning and sunset light for optimal color contrast and dynamic range, maximizing visual quality for color and lighting-sensitive tasks.
๐ Curation: Curated specifically for machine learning, providing clean, high-quality data for next generation model training.