FoamoftheSea/multiformer-m1

Multiformer

Multiformer is a multi-task vision transformer architecture designed to provide strong perception capabilities with a nimble and lightweight architecture.

• Publication
• Training Report
• GitHub

This model uses a custom branch of the transformers library, which can be installed easily using the instructions below.

For training and evaluation, a custom MultitaskTrainer class is used that can handle complex nested losses and successfully log them to wandb.

A training/eval and inference script are both available in the project repository.

Setup Instructions

1. Open a terminal and navigate to your root folder, then run

git clone https://github.com/FoamoftheSea/shift-experiments.git

2. Follow the setup instructions for your operating system found in the README

Quick Load Multiformer-M1

You should now be able to run the following code to load a Multiformer-M1 with pretrained weights:

from transformers import AutoModel

multiformer = AutoModel.from_pretrained("FoamoftheSea/multiformer-m1)