No application file
š
opt-ssm
community
AI & ML interests
None defined yet.
Recent Activity
View all activity
opt-ssm's activity
Andron00eĀ
updated
2
Spaces
5 months ago
Post
2026
Sparse Concept Bottleneck Models: Gumbel Tricks in Contrastive Learning
Paper: Sparse Concept Bottleneck Models: Gumbel Tricks in Contrastive Learning (2404.03323)
Authors propose a novel architecture and method of explainable classification with Concept Bottleneck Models (CBMs): they introduce a new type of layers known as Concept Bottleneck Layers (CBL), and present three methods for training them: with $\ell_1$-loss, contrastive loss and loss function based on Gumbel-Softmax distribution (Sparse-CBM), while final FC layer is still trained with Cross-Entropy. They show a significant increase in accuracy using sparse hidden layers in CLIP-based bottleneck models. Which means that sparse representation of concepts activation vector is meaningful in Concept Bottleneck Models.
Key concepts:
ā Contrastive Gumbel-Softmax loss: the first contrastive variant of Gumbel-Softmax objective which achieves an inner sparse representation of the Concept Bottleneck Layer activations.
ā Sparse $\ell_1$ regularization.
ā Contrastive loss for inner layers of the model.
Methodology:
The approach consists of three main steps:
ā Create a set of concepts based on the labels of the dataset.
ā Supply a multi-modal encoder with CBL.
ā Train this CBL with the picked objective function and train the classifiers head with Cross-Entropy.
Results and Analysis:
The methodology can be applied to the task of interpreted image classification. And the experimental results show the superiority of using sparse hidden representations of concepts.
Paper: Sparse Concept Bottleneck Models: Gumbel Tricks in Contrastive Learning (2404.03323)
Authors propose a novel architecture and method of explainable classification with Concept Bottleneck Models (CBMs): they introduce a new type of layers known as Concept Bottleneck Layers (CBL), and present three methods for training them: with $\ell_1$-loss, contrastive loss and loss function based on Gumbel-Softmax distribution (Sparse-CBM), while final FC layer is still trained with Cross-Entropy. They show a significant increase in accuracy using sparse hidden layers in CLIP-based bottleneck models. Which means that sparse representation of concepts activation vector is meaningful in Concept Bottleneck Models.
Key concepts:
ā Contrastive Gumbel-Softmax loss: the first contrastive variant of Gumbel-Softmax objective which achieves an inner sparse representation of the Concept Bottleneck Layer activations.
ā Sparse $\ell_1$ regularization.
ā Contrastive loss for inner layers of the model.
Methodology:
The approach consists of three main steps:
ā Create a set of concepts based on the labels of the dataset.
ā Supply a multi-modal encoder with CBL.
ā Train this CBL with the picked objective function and train the classifiers head with Cross-Entropy.
Results and Analysis:
The methodology can be applied to the task of interpreted image classification. And the experimental results show the superiority of using sparse hidden representations of concepts.
Andron00eĀ
authored
a
paper
9 months ago