What Matters in Transformers? Not All Attention is Needed
Paper
•
2406.15786
•
Published
•
31
sepilqi
sepilqi
AI & ML interests
None yet
Recent Activity
upvoted
a
paper
4 months ago
What Matters in Transformers? Not All Attention is Needed
liked
a Space
4 months ago
huggingface/paper-central
upvoted
an
article
4 months ago
Exploring the Daily Papers Page on Hugging Face
Organizations
sepilqi's activity
upvoted
a
paper
4 months ago
liked
a
Space
4 months ago
upvoted
2
articles
4 months ago
Article
Exploring the Daily Papers Page on Hugging Face
•
49
Article
A Short Summary of Chinese AI Global Expansion
•
22
upvoted
an
article
6 months ago
Article
Flash Attention 学习笔记
By
•
•
1
published
an
article
6 months ago
Article
Flash Attention 学习笔记
By
•
•
1
upvoted
an
article
7 months ago
Article
Getting Started with Sentiment Analysis using Python
•
38
[Support] Community Articles
80
#5 opened 11 months ago
by
victor
upvoted
2
articles
7 months ago
Article
How NuminaMath Won the 1st AIMO Progress Prize
•
116
Article
Mixture of Experts Explained
•
351
reacted to
akhaliq's
post with 👍
9 months ago
Post
21008
Chameleon
Mixed-Modal Early-Fusion Foundation Models
Chameleon: Mixed-Modal Early-Fusion Foundation Models (2405.09818)
We present Chameleon, a family of early-fusion token-based mixed-modal models capable of understanding and generating images and text in any arbitrary sequence. We outline a stable training approach from inception, an alignment recipe, and an architectural parameterization tailored for the early-fusion, token-based, mixed-modal setting. The models are evaluated on a comprehensive range of tasks, including visual question answering, image captioning, text generation, image generation, and long-form mixed modal generation. Chameleon demonstrates broad and general capabilities, including state-of-the-art performance in image captioning tasks, outperforms Llama-2 in text-only tasks while being competitive with models such as Mixtral 8x7B and Gemini-Pro, and performs non-trivial image generation, all in a single model. It also matches or exceeds the performance of much larger models, including Gemini Pro and GPT-4V, according to human judgments on a new long-form mixed-modal generation evaluation, where either the prompt or outputs contain mixed sequences of both images and text. Chameleon marks a significant step forward in a unified modeling of full multimodal documents.
Mixed-Modal Early-Fusion Foundation Models
Chameleon: Mixed-Modal Early-Fusion Foundation Models (2405.09818)
We present Chameleon, a family of early-fusion token-based mixed-modal models capable of understanding and generating images and text in any arbitrary sequence. We outline a stable training approach from inception, an alignment recipe, and an architectural parameterization tailored for the early-fusion, token-based, mixed-modal setting. The models are evaluated on a comprehensive range of tasks, including visual question answering, image captioning, text generation, image generation, and long-form mixed modal generation. Chameleon demonstrates broad and general capabilities, including state-of-the-art performance in image captioning tasks, outperforms Llama-2 in text-only tasks while being competitive with models such as Mixtral 8x7B and Gemini-Pro, and performs non-trivial image generation, all in a single model. It also matches or exceeds the performance of much larger models, including Gemini Pro and GPT-4V, according to human judgments on a new long-form mixed-modal generation evaluation, where either the prompt or outputs contain mixed sequences of both images and text. Chameleon marks a significant step forward in a unified modeling of full multimodal documents.
updated
a
collection
9 months ago
commented
a
paper
11 months ago