LoRA-Ensemble: Uncertainty Modelling for Self-attention Networks

Michelle Halbheer, Dominik J. Mühlematter, Alexander Becker, Dominik Narnhofer, Helge Aasen, Konrad Schindler and Mehmet Ozgur Turkoglu - 2024

[Paper on ArXiv]

Pre-trained models

This repository contains the pre-trained models corresponding to the code we released on GitHub. The usage of the models with our pipeline is described in the GitHub repository. This repository only contains the models for our final experiments on the CIFAR-10, CIFAR-100 and HAM10000 datasets, not, however, for all intermediate results. The results of the ESC-50 dataset cannot be published at this time, as it would require storing five models per epoch during training in order to store all models of the five-fold cross-validation. This is infeasible on our infrastructure so we only release models for CIFAR-10, CIFAR-100 and HAM10000.

Base models

Alongside our pre-trained models we release the base models we used for our models. This is to ensure the reproducibility of our results even if the weights by torchvision and timm should get changed.

Citation

If you find our work useful or interesting or use our code, please cite our paper as follows

@misc{
  title = {LoRA-Ensemble: Uncertainty Modelling for Self-attention Networks},
  author = {Halbheer, Michelle and M\"uhlematter, Dominik Jan and Becker, Alexander and Narnhofer, Dominik and Aasen, Helge and Schindler, Konrad and Turkoglu, Mehmet Ozgur}
  year = {2024}
  note = {arXiv: 2405.14438}
}

CIFAR-100

The table below shows the evaluation results obtained using different methods. Each method was trained five times with varying random seeds.

Method (ViT)	Accuracy	ECE	Settings name*	Model weights*
Single Network	$76.6\pm 0.276.6\backslash pm0.2$	$0.144\pm 0.0010.144\backslash pm0.001$	CIFAR100_settings_explicit	Deep_Ensemble_ViT_base_32_1_members_CIFAR100_settings_explicit<seed>.pt
Single Network with LoRA	$79.6\pm 0.279.6\backslash pm0.2$	$\mathbf{\text{0.014}}\pm 0.003\backslash textbf\{0.014\}\backslash pm0.003$	CIFAR100_settings_LoRA	LoRA_Former_ViT_base_32_1_members_CIFAR100_settings_LoRA<seed>.pt
MC Dropout	$77.1\pm 0.577.1\backslash pm0.5$	$0.055\pm 0.0020.055\backslash pm0.002$	CIFAR100_settings_MCDropout	MCDropout_ViT_base_32_16_members_CIFAR100_settings_MCDropout<seed>.pt
Explicit Ensemble	$\underset{‾}{79.8}\pm 0.2\backslash underline\{79.8\}\backslash pm0.2$	$0.098\pm 0.0010.098\backslash pm0.001$	CIFAR100_settings_explicit	Deep_Ensemble_ViT_base_32_16_members_CIFAR100_settings_explicit<seed>.pt
LoRA-Ensemble	$\mathbf{\text{82.5}}\pm 0.1\backslash textbf\{82.5\}\backslash pm0.1$	$\underset{‾}{0.035}\pm 0.001\backslash underline\{0.035\}\backslash pm0.001$	CIFAR100_settings_LoRA	LoRA_Former_ViT_base_32_16_members_CIFAR100_settings_LoRA<seed>.pt

* Settings and model names are followed by a number in the range 1-5 indicating the used random seed.

HAM10000

The table below shows the evaluation results obtained using different methods. Each method was trained five times with varying random seeds.

Method (ViT)	Accuracy	ECE	Settings name*	Model weights*
Single Network	$84.3\pm 0.584.3\backslash pm0.5$	$0.136\pm 0.0060.136\backslash pm0.006$	HAM10000_settings_explicit	Deep_Ensemble_ViT_base_32_1_members_HAM10000_settings_explicit<seed>.pt
Single Network with LoRA	$83.2\pm 0.783.2\backslash pm0.7$	$0.085\pm 0.0040.085\backslash pm0.004$	HAM10000_settings_LoRA	LoRA_Former_ViT_base_32_1_members_HAM10000_settings_LoRA<seed>.pt
MC Dropout	$83.7\pm 0.483.7\backslash pm0.4$	$\underset{‾}{0.099}\pm 0.007\backslash underline\{0.099\}\backslash pm0.007$	HAM10000_settings_MCDropout	MCDropout_ViT_base_32_16_members_HAM10000_settings_MCDropout<seed>.pt
Explicit Ensemble	$\underset{‾}{85.7}\pm 0.3\backslash underline\{85.7\}\backslash pm0.3$	$0.106\pm 0.0020.106\backslash pm0.002$	HAM10000_settings_explicit	Deep_Ensemble_ViT_base_32_16_members_HAM10000_settings_explicit<seed>.pt
LoRA-Ensemble	$\mathbf{\text{88.0}}\pm 0.2\backslash textbf\{88.0\}\backslash pm0.2$	$\mathbf{\text{0.037}}\pm 0.002\backslash textbf\{0.037\}\backslash pm0.002$	HAM10000_settings_LoRA	LoRA_Former_ViT_base_32_16_members_HAM10000_settings_LoRA<seed>.pt

* Settings and model names are followed by a number in the range 1-5 indicating the used random seed.