Hugging Face's logo

SMSD75
/
SIGMA

Video Classification
English
Model card Files Community
Mohammadreza Salehidehnavi commited on
Commit
c98a7cc
·
1 Parent(s): 9cba51c

Feat: Instructions has been added

Browse files
Files changed (6) hide show
  1. DATASET.md +39 -0
  2. FINETUNE.md +147 -0
  3. MODEL_ZOO.md +34 -0
  4. NOTICE.md +453 -0
  5. PRETRAIN.md +120 -0
  6. README.md +159 -5
DATASET.md ADDED
@@ -0,0 +1,39 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Data Preparation
2
+
3
+ We have successfully pre-trained and fine-tuned our SIGMA on [Kinetics400](https://deepmind.com/research/open-source/kinetics), [Something-Something-V2](https://developer.qualcomm.com/software/ai-datasets/something-something), [UCF101](https://www.crcv.ucf.edu/data/UCF101.php) and [HMDB51](https://serre-lab.clps.brown.edu/resource/hmdb-a-large-human-motion-database/) with this codebase.
4
+
5
+ - The pre-processing of **Something-Something-V2** can be summarized into 3 steps:
6
+
7
+ 1. Download the dataset from [official website](https://developer.qualcomm.com/software/ai-datasets/something-something).
8
+
9
+ 2. Preprocess the dataset by changing the video extension from `webm` to `.mp4` with the **original** height of **240px**.
10
+
11
+ 3. Generate annotations needed for dataloader ("<path_to_video> <video_class>" in annotations). The annotation usually includes `train.csv`, `val.csv` and `test.csv` ( here `test.csv` is the same as `val.csv`). We **share** our annotation files (train.csv, val.csv, test.csv) via **[Google Drive](https://drive.google.com/drive/folders/1cfA-SrPhDB9B8ZckPvnh8D5ysCjD-S_I?usp=share_link)**. The format of `*.csv` file is like:
12
+
13
+ ```
14
+ dataset_root/video_1.mp4 label_1
15
+ dataset_root/video_2.mp4 label_2
16
+ dataset_root/video_3.mp4 label_3
17
+ ...
18
+ dataset_root/video_N.mp4 label_N
19
+ ```
20
+
21
+ - The pre-processing of **Kinetics400** can be summarized into 3 steps:
22
+
23
+ 1. Download the dataset from [official website](https://deepmind.com/research/open-source/kinetics).
24
+
25
+ 2. Preprocess the dataset by resizing the short edge of video to **320px**. You can refer to [MMAction2 Data Benchmark](https://github.com/open-mmlab/mmaction2) for [TSN](https://github.com/open-mmlab/mmaction2/tree/master/configs/recognition/tsn#kinetics-400-data-benchmark-8-gpus-resnet50-imagenet-pretrain-3-segments) and [SlowOnly](https://github.com/open-mmlab/mmaction2/tree/master/configs/recognition/slowonly#kinetics-400-data-benchmark).
26
+
27
+ 3. Generate annotations needed for dataloader ("<path_to_video> <video_class>" in annotations). The annotation usually includes `train.csv`, `val.csv` and `test.csv` ( here `test.csv` is the same as `val.csv`). The format of `*.csv` file is like:
28
+
29
+ ```
30
+ dataset_root/video_1.mp4 label_1
31
+ dataset_root/video_2.mp4 label_2
32
+ dataset_root/video_3.mp4 label_3
33
+ ...
34
+ dataset_root/video_N.mp4 label_N
35
+ ```
36
+
37
+ ### Note:
38
+
39
+ We use [decord](https://github.com/dmlc/decord) to decode the videos **on the fly** during both pre-training and fine-tuning phases.
FINETUNE.md ADDED
@@ -0,0 +1,147 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Fine-tuning SIGMA
2
+
3
+ ## Original Implementation
4
+
5
+ The implementation of our SIGMA supports **multi-node distributed training**. We provide the **off-the-shelf** scripts in the [scripts folder](scripts).
6
+
7
+ - For example, to fine-tune SIGMA ViT-Base on **Something-Something V2** with 64 GPUs (8 nodes x 8 GPUs), you can run
8
+
9
+ ```bash
10
+ OUTPUT_DIR='YOUR_PATH/ssv2_SIGMA_pretrain_base_patch16_224_frame_16x2_tube_mask_ratio_0.9_e800/eval_lr_5e-4_epoch_50'
11
+ DATA_PATH='YOUR_PATH/list_ssv2'
12
+ MODEL_PATH='YOUR_PATH/ssv2_SIGMA_pretrain_base_patch16_224_frame_16x2_tube_mask_ratio_0.9_e800/checkpoint-799.pth'
13
+
14
+ OMP_NUM_THREADS=1 python -m torch.distributed.launch --nproc_per_node=8 \
15
+ --master_port 12320 --nnodes=8 \
16
+ --node_rank=0 --master_addr=$ip_node_0 \
17
+ run_class_finetuning.py \
18
+ --model vit_base_patch16_224 \
19
+ --data_set SSV2 \
20
+ --nb_classes 174 \
21
+ --data_path ${DATA_PATH} \
22
+ --finetune ${MODEL_PATH} \
23
+ --log_dir ${OUTPUT_DIR} \
24
+ --output_dir ${OUTPUT_DIR} \
25
+ --batch_size 8 \
26
+ --num_sample 1 \
27
+ --input_size 224 \
28
+ --short_side_size 224 \
29
+ --save_ckpt_freq 10 \
30
+ --num_frames 16 \
31
+ --opt adamw \
32
+ --lr 5e-4 \
33
+ --opt_betas 0.9 0.999 \
34
+ --weight_decay 0.05 \
35
+ --epochs 50 \
36
+ --dist_eval \
37
+ --test_num_segment 2 \
38
+ --test_num_crop 3 \
39
+ --enable_deepspeed
40
+ ```
41
+
42
+ on the first node. On other nodes, run the same command with `--node_rank 1`, ..., `--node_rank 7` respectively. `--master_addr` is set as the ip of the node 0.
43
+
44
+ - For example, to fine-tune SIGMA ViT-Base on **Kinetics400** with 64 GPUs (8 nodes x 8 GPUs), you can run
45
+
46
+ ```bash
47
+ OUTPUT_DIR='YOUR_PATH/k400_SIGMA_pretrain_base_patch16_224_frame_16x4_tube_mask_ratio_0.9_e800/eval_lr_1e-3_epoch_100'
48
+ DATA_PATH='YOUR_PATH/list_kinetics-400'
49
+ MODEL_PATH='YOUR_PATH/k400_SIGMA_pretrain_base_patch16_224_frame_16x4_tube_mask_ratio_0.9_e800/checkpoint-799.pth'
50
+
51
+ OMP_NUM_THREADS=1 python -m torch.distributed.launch --nproc_per_node=8 \
52
+ --master_port 12320 --nnodes=8 \
53
+ --node_rank=0 --master_addr=$ip_node_0 \
54
+ run_class_finetuning.py \
55
+ --model vit_base_patch16_224 \
56
+ --data_set Kinetics-400 \
57
+ --nb_classes 400 \
58
+ --data_path ${DATA_PATH} \
59
+ --finetune ${MODEL_PATH} \
60
+ --log_dir ${OUTPUT_DIR} \
61
+ --output_dir ${OUTPUT_DIR} \
62
+ --batch_size 8 \
63
+ --num_sample 1 \
64
+ --input_size 224 \
65
+ --short_side_size 224 \
66
+ --save_ckpt_freq 10 \
67
+ --num_frames 16 \
68
+ --sampling_rate 4 \
69
+ --opt adamw \
70
+ --lr 1e-3 \
71
+ --opt_betas 0.9 0.999 \
72
+ --weight_decay 0.05 \
73
+ --epochs 100 \
74
+ --dist_eval \
75
+ --test_num_segment 5 \
76
+ --test_num_crop 3 \
77
+ --enable_deepspeed
78
+ ```
79
+
80
+ on the first node. On other nodes, run the same command with `--node_rank 1`, ..., `--node_rank 7` respectively. `--master_addr` is set as the ip of the node 0.
81
+
82
+ ### Note:
83
+
84
+ - We perform the **I3D dense sampling** on **Kinetics400** and **uniform sampling** on **Something-Something V2**, respectively.
85
+ - We didn't use `cls token` in our implementation, and directly average the feature of last layer for video classification.
86
+ - Here total batch size = (`batch_size` per gpu) x `nodes` x (gpus per node).
87
+ - `lr` here is the base learning rate. The ` actual lr` is computed by the [linear scaling rule](https://arxiv.org/abs/1706.02677): `` actual lr`` = `lr` * total batch size / 256.
88
+
89
+ ## Slurm
90
+
91
+ To help the community to reproduce our results on slurm cluster, we also provide the the **off-the-shelf** script.
92
+
93
+ For example, to fine-tune SIGMA ViT-Base on **Kinetics400** with 64 GPUs (8 nodes x 8 GPUs), you can run:
94
+
95
+ ```bash
96
+ export MASTER_PORT=$((12000 + $RANDOM % 20000))
97
+ export OMP_NUM_THREADS=1
98
+
99
+ OUTPUT_DIR='YOUR_PATH/k400_SIGMA_pretrain_base_patch16_224_frame_16x4_tube_mask_ratio_0.9_e800/eval_lr_1e-3_epoch_100'
100
+ DATA_PATH='YOUR_PATH/list_kinetics-400'
101
+ MODEL_PATH='YOUR_PATH/k400_SIGMA_pretrain_base_patch16_224_frame_16x4_tube_mask_ratio_0.9_e800/checkpoint-799.pth'
102
+
103
+ JOB_NAME=$1
104
+ PARTITION=${PARTITION:-"video"}
105
+ # 8 for 1 node, 16 for 2 node, etc.
106
+ GPUS=${GPUS:-64}
107
+ GPUS_PER_NODE=${GPUS_PER_NODE:-8}
108
+ CPUS_PER_TASK=${CPUS_PER_TASK:-8}
109
+ SRUN_ARGS=${SRUN_ARGS:-""}
110
+ PY_ARGS=${@:2}
111
+
112
+ # batch_size can be adjusted according to the graphics card
113
+ srun -p $PARTITION \
114
+ --job-name=${JOB_NAME} \
115
+ --gres=gpu:${GPUS_PER_NODE} \
116
+ --ntasks=${GPUS} \
117
+ --ntasks-per-node=${GPUS_PER_NODE} \
118
+ --cpus-per-task=${CPUS_PER_TASK} \
119
+ --kill-on-bad-exit=1 \
120
+ ${SRUN_ARGS} \
121
+ python -u run_class_finetuning.py \
122
+ --model vit_base_patch16_224 \
123
+ --data_set Kinetics-400 \
124
+ --nb_classes 400 \
125
+ --data_path ${DATA_PATH} \
126
+ --finetune ${MODEL_PATH} \
127
+ --log_dir ${OUTPUT_DIR} \
128
+ --output_dir ${OUTPUT_DIR} \
129
+ --batch_size 8 \
130
+ --num_sample 1 \
131
+ --input_size 224 \
132
+ --short_side_size 224 \
133
+ --save_ckpt_freq 10 \
134
+ --num_frames 16 \
135
+ --sampling_rate 4 \
136
+ --opt adamw \
137
+ --lr 1e-3 \
138
+ --opt_betas 0.9 0.999 \
139
+ --weight_decay 0.05 \
140
+ --epochs 100 \
141
+ --dist_eval \
142
+ --test_num_segment 5 \
143
+ --test_num_crop 3 \
144
+ --enable_deepspeed \
145
+ ${PY_ARGS}
146
+ ```
147
+
MODEL_ZOO.md ADDED
@@ -0,0 +1,34 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # VideoMAE Model Zoo
2
+
3
+ ### Kinetics-400
4
+
5
+ | Method | Extra Data | Backbone | Epoch | \#Frame | Pre-train | Fine-tune | Top-1 | Top-5 |
6
+ | :------: | :--------: | :------: | :---: | :-----: | :----------------------------------------------------------: | :----------------------------------------------------------: | :---: | :---: |
7
+ | VideoMAE | ***no*** | ViT-S | 1600 | 16x5x3 | [script](scripts/kinetics/videomae_vit_small_patch16_224_tubemasking_ratio_0.9_epoch_1600/pretrain.sh)/[log](https://drive.google.com/file/d/1fbmQtp3UUw9fro3MVkKCW62Ib_HlZvNz/view?usp=sharing)/[checkpoint](https://drive.google.com/file/d/1nU-H1u3eJ-VuyCveU7v-WIOcAVxs5Hww/view?usp=sharing) | [script](scripts/kinetics/videomae_vit_small_patch16_224_tubemasking_ratio_0.9_epoch_1600/finetune.sh)/[log](https://drive.google.com/file/d/1RuEvCT2OMKPax2gGB1gBsH6ItiXIPH-R/view?usp=sharing)/[checkpoint](https://drive.google.com/file/d/1ygjLRm1kvs9mwGsP3lLxUExhRo6TWnrx/view?usp=sharing) | 79.0 | 93.8 |
8
+ | VideoMAE | ***no*** | ViT-B | 800 | 16x5x3 | [script](scripts/kinetics/videomae_vit_base_patch16_224_tubemasking_ratio_0.9_epoch_800/pretrain.sh)/[log](https://drive.google.com/file/d/1kP3_-465jCL7PRNFq1JcAghPo2BONRWY/view?usp=sharing)/[checkpoint](https://drive.google.com/file/d/1JfrhN144Hdg7we213H1WxwR3lGYOlmIn/view?usp=sharing) | [script](scripts/kinetics/videomae_vit_base_patch16_224_tubemasking_ratio_0.9_epoch_800/finetune.sh)/[log](https://drive.google.com/file/d/1JOJzhlCujgpsjjth0J49k5EwBNxy76xt/view?usp=sharing)/[checkpoint](https://drive.google.com/file/d/18EEgdXY9347yK3Yb28O-GxFMbk41F6Ne/view?usp=sharing)<br />(w/o repeated aug) | 80.0 | 94.4 |
9
+ | VideoMAE | ***no*** | ViT-B | 800 | 16x5x3 | same as above | TODO | 81.0 | 94.8 |
10
+ | VideoMAE | ***no*** | ViT-B | 1600 | 16x5x3 | [script](scripts/kinetics/videomae_vit_base_patch16_224_tubemasking_ratio_0.9_epoch_1600/pretrain.sh)/[log](https://drive.google.com/file/d/1ftVHzzCupEGV4bCHC5JWIUsEwOEeAQcg/view?usp=sharing)/[checkpoint](https://drive.google.com/file/d/1tEhLyskjb755TJ65ptsrafUG2llSwQE1/view?usp=sharing) | [script](scripts/kinetics/videomae_vit_large_patch16_224_tubemasking_ratio_0.9_epoch_1600/finetune.sh)/[log](https://drive.google.com/file/d/1fYXtL2y2ZTMxDtTRqoUOe6leVmdVI5HH/view?usp=sharing)/[checkpoint](https://drive.google.com/file/d/1MzwteHH-1yuMnFb8vRBQDvngV1Zl-d3z/view?usp=sharing) | 81.5 | 95.1 |
11
+ | VideoMAE | ***no*** | ViT-L | 1600 | 16x5x3 | [script](scripts/kinetics/videomae_vit_large_patch16_224_tubemasking_ratio_0.9_epoch_1600/pretrain.sh)/[log](https://drive.google.com/file/d/1X7WBzn_yG4lDWuvBMBBgrtgqDLZVHrc2/view?usp=sharing)/[checkpoint](https://drive.google.com/file/d/1qLOXWb_MGEvaI7tvuAe94CV7S2HXRwT3/view?usp=sharing) | [script](scripts/kinetics/videomae_vit_large_patch16_224_tubemasking_ratio_0.9_epoch_1600/finetune.sh)/[log](https://drive.google.com/file/d/1Doqx6zDQEMnMyPvDdz2knG385o0sZn3f/view?usp=sharing)/[checkpoint](https://drive.google.com/file/d/1jX1CiqxSkCfc94y8FRW1YGHy-GNvHCuD/view?usp=sharing) | 85.2 | 96.8 |
12
+ | VideoMAE | ***no*** | ViT-H | 1600 | 16x5x3 | [script](scripts/kinetics/videomae_vit_huge_patch16_224_tubemasking_ratio_0.9_epoch_1600/pretrain.sh)/[log](https://drive.google.com/file/d/1ZGOGk5_L7cqJ2UkrNQ7c_jcw1OUBqptl/view?usp=sharing)/[checkpoint](https://drive.google.com/file/d/1AJQR1Rsi2N1pDn9tLyJ8DQrUREiBA1bO/view?usp=sharing) | [script](scripts/kinetics/videomae_vit_huge_patch16_224_tubemasking_ratio_0.9_epoch_1600/finetune.sh)/[log](https://drive.google.com/file/d/1NOUjO5wPrHZo4EUfklKvfGM3ScJVmGAK/view?usp=sharing)/[checkpoint](https://drive.google.com/file/d/104ouJZxSVPSAm0LwJXd6IzjdA_RGLqZi/view?usp=sharing) | 86.6 | 97.1 |
13
+
14
+ ### Something-Something V2
15
+
16
+ | Method | Extra Data | Backbone | Epoch | \#Frame | Pre-train | Fine-tune | Top-1 | Top-5 |
17
+ | :------: | :--------: | :------: | :---: | :-----: | :----------------------------------------------------------: | :----------------------------------------------------------: | :---: | :---: |
18
+ | VideoMAE | ***no*** | ViT-S | 2400 | 16x2x3 | [script](scripts/ssv2/videomae_vit_small_patch16_224_tubemasking_ratio_0.9_epoch_2400/pretrain.sh)/[log](https://drive.google.com/file/d/129wqpAtwTCD-T1SQIX7q5nB9CEGchhw0/view?usp=sharing)/[checkpoint](https://drive.google.com/file/d/1p_I1aaONOeUvRmRQw1UT3-L2H8XJClHu/view?usp=sharing) | [script](scripts/ssv2/videomae_vit_small_patch16_224_tubemasking_ratio_0.9_epoch_2400/finetune.sh)/[log](https://drive.google.com/file/d/17X9PcDSBB1Zb1blNqQP3vvnqOuMzJrGp/view?usp=sharing)/[checkpoint](https://drive.google.com/file/d/1ajlMrT06jiiM-5YjNI2X_UFyzsuYbbtZ/view?usp=sharing) | 66.8 | 90.3 |
19
+ | VideoMAE | ***no*** | ViT-B | 800 | 16x2x3 | [script](scripts/ssv2/videomae_vit_base_patch16_224_tubemasking_ratio_0.9_epoch_800/pretrain.sh)/[log](https://drive.google.com/file/d/1eGS18rKvbgEJ3nbsXxokkMSwNGxxoX48/view?usp=sharing)/[checkpoint](https://drive.google.com/file/d/181hLvyrrPW2IOGA46fkxdJk0tNLIgdB2/view?usp=sharing) | [script](scripts/ssv2/videomae_vit_base_patch16_224_tubemasking_ratio_0.9_epoch_800/finetune.sh)/[log](https://drive.google.com/file/d/1jYAHPcs7zt_QMPM2D_geEWoWrf3yHox8/view?usp=sharing)/[checkpoint](https://drive.google.com/file/d/1xZCiaPF4w7lYmLt5o1D5tIZyDdLtJAvH/view?usp=sharing)<br />(w/o repeated aug) | 69.6 | 92.0 |
20
+ | VideoMAE | ***no*** | ViT-B | 2400 | 16x2x3 | [script](scripts/ssv2/videomae_vit_base_patch16_224_tubemasking_ratio_0.9_epoch_2400/pretrain.sh)/[log](https://drive.google.com/file/d/148nURgfcIFBQd3IQH5YhJ9dTwNCc2jkU/view?usp=sharing)/[checkpoint](https://drive.google.com/file/d/1I18dY_7rSalGL8fPWV82c0-foRUDzJJk/view?usp=sharing) | [script](scripts/ssv2/videomae_vit_base_patch16_224_tubemasking_ratio_0.9_epoch_2400/finetune.sh)/[log](https://drive.google.com/file/d/15TPBiUl_K2Q_9l6J41G_vf-2lovVLEHM/view?usp=sharing)/[checkpoint](https://drive.google.com/file/d/1dt_59tBIyzdZd5Ecr22lTtzs_64MOZkT/view?usp=sharing) | 70.8 | 92.4 |
21
+
22
+ ### UCF101
23
+
24
+ | Method | Extra Data | Backbone | Epoch | \#Frame | Pre-train | Fine-tune | Top-1 | Top-5 |
25
+ | :------: | :--------: | :------: | :---: | :-----: | :----------------------------------------------------------: | :----------------------------------------------------------: | :---: | :---: |
26
+ | VideoMAE | ***no*** | ViT-B | 3200 | 16x5x3 | [script](scripts/ucf101/videomae_vit_base_patch16_224_tubemasking_ratio_0.75_epoch_3200/pretrain.sh)/[log](https://drive.google.com/file/d/1kZODk_dQgB-aW6oIwPYZxqZAG6YKNtXC/view?usp=sharing)/[checkpoint](https://drive.google.com/file/d/1BHev4meNgKM0o_8DMRbuzAsKSP3IpQ3o/view?usp=sharing) | [script](scripts/ucf101/videomae_vit_base_patch16_224_tubemasking_ratio_0.75_epoch_3200/finetune.sh)/[log](https://drive.google.com/file/d/17Mq7rlM1TRgV4KKX7UIlmKw653RmwSqe/view?usp=sharing)/[checkpoint](https://drive.google.com/file/d/1MSyon6fPpKz7oqD6WDGPFK4k_Rbyb6fw/view?usp=sharing) | 91.3 | 98.5 |
27
+
28
+ ### Note:
29
+
30
+ - We report the results of VideoMAE finetuned with `I3D dense sampling` on **Kinetics400** and `TSN uniform sampling` on **Something-Something V2**, respectively.
31
+ - \#Frame = #input_frame x #clip x #crop.
32
+ - \#input_frame means how many frames are input for model during the test phase.
33
+ - \#crop means spatial crops (e.g., 3 for left/right/center crop).
34
+ - \#clip means temporal clips (e.g., 5 means repeted temporal sampling five clips with different start indices).
NOTICE.md ADDED
@@ -0,0 +1,453 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ NOTICES AND INFORMATION
2
+
3
+ Do Not Translate or Localize
4
+
5
+ This software incorporates material from third parties.
6
+
7
+ ===============================================================================
8
+
9
+ Component.
10
+
11
+ rwightman/pytorch-image-models
12
+
13
+ Open Source License/Copyright Notice.
14
+
15
+ ```
16
+ Apache License
17
+ Version 2.0, January 2004
18
+ http://www.apache.org/licenses/
19
+
20
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
21
+
22
+ 1. Definitions.
23
+
24
+ "License" shall mean the terms and conditions for use, reproduction,
25
+ and distribution as defined by Sections 1 through 9 of this document.
26
+
27
+ "Licensor" shall mean the copyright owner or entity authorized by
28
+ the copyright owner that is granting the License.
29
+
30
+ "Legal Entity" shall mean the union of the acting entity and all
31
+ other entities that control, are controlled by, or are under common
32
+ control with that entity. For the purposes of this definition,
33
+ "control" means (i) the power, direct or indirect, to cause the
34
+ direction or management of such entity, whether by contract or
35
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
36
+ outstanding shares, or (iii) beneficial ownership of such entity.
37
+
38
+ "You" (or "Your") shall mean an individual or Legal Entity
39
+ exercising permissions granted by this License.
40
+
41
+ "Source" form shall mean the preferred form for making modifications,
42
+ including but not limited to software source code, documentation
43
+ source, and configuration files.
44
+
45
+ "Object" form shall mean any form resulting from mechanical
46
+ transformation or translation of a Source form, including but
47
+ not limited to compiled object code, generated documentation,
48
+ and conversions to other media types.
49
+
50
+ "Work" shall mean the work of authorship, whether in Source or
51
+ Object form, made available under the License, as indicated by a
52
+ copyright notice that is included in or attached to the work
53
+ (an example is provided in the Appendix below).
54
+
55
+ "Derivative Works" shall mean any work, whether in Source or Object
56
+ form, that is based on (or derived from) the Work and for which the
57
+ editorial revisions, annotations, elaborations, or other modifications
58
+ represent, as a whole, an original work of authorship. For the purposes
59
+ of this License, Derivative Works shall not include works that remain
60
+ separable from, or merely link (or bind by name) to the interfaces of,
61
+ the Work and Derivative Works thereof.
62
+
63
+ "Contribution" shall mean any work of authorship, including
64
+ the original version of the Work and any modifications or additions
65
+ to that Work or Derivative Works thereof, that is intentionally
66
+ submitted to Licensor for inclusion in the Work by the copyright owner
67
+ or by an individual or Legal Entity authorized to submit on behalf of
68
+ the copyright owner. For the purposes of this definition, "submitted"
69
+ means any form of electronic, verbal, or written communication sent
70
+ to the Licensor or its representatives, including but not limited to
71
+ communication on electronic mailing lists, source code control systems,
72
+ and issue tracking systems that are managed by, or on behalf of, the
73
+ Licensor for the purpose of discussing and improving the Work, but
74
+ excluding communication that is conspicuously marked or otherwise
75
+ designated in writing by the copyright owner as "Not a Contribution."
76
+
77
+ "Contributor" shall mean Licensor and any individual or Legal Entity
78
+ on behalf of whom a Contribution has been received by Licensor and
79
+ subsequently incorporated within the Work.
80
+
81
+ 2. Grant of Copyright License. Subject to the terms and conditions of
82
+ this License, each Contributor hereby grants to You a perpetual,
83
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
84
+ copyright license to reproduce, prepare Derivative Works of,
85
+ publicly display, publicly perform, sublicense, and distribute the
86
+ Work and such Derivative Works in Source or Object form.
87
+
88
+ 3. Grant of Patent License. Subject to the terms and conditions of
89
+ this License, each Contributor hereby grants to You a perpetual,
90
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
91
+ (except as stated in this section) patent license to make, have made,
92
+ use, offer to sell, sell, import, and otherwise transfer the Work,
93
+ where such license applies only to those patent claims licensable
94
+ by such Contributor that are necessarily infringed by their
95
+ Contribution(s) alone or by combination of their Contribution(s)
96
+ with the Work to which such Contribution(s) was submitted. If You
97
+ institute patent litigation against any entity (including a
98
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
99
+ or a Contribution incorporated within the Work constitutes direct
100
+ or contributory patent infringement, then any patent licenses
101
+ granted to You under this License for that Work shall terminate
102
+ as of the date such litigation is filed.
103
+
104
+ 4. Redistribution. You may reproduce and distribute copies of the
105
+ Work or Derivative Works thereof in any medium, with or without
106
+ modifications, and in Source or Object form, provided that You
107
+ meet the following conditions:
108
+
109
+ (a) You must give any other recipients of the Work or
110
+ Derivative Works a copy of this License; and
111
+
112
+ (b) You must cause any modified files to carry prominent notices
113
+ stating that You changed the files; and
114
+
115
+ (c) You must retain, in the Source form of any Derivative Works
116
+ that You distribute, all copyright, patent, trademark, and
117
+ attribution notices from the Source form of the Work,
118
+ excluding those notices that do not pertain to any part of
119
+ the Derivative Works; and
120
+
121
+ (d) If the Work includes a "NOTICE" text file as part of its
122
+ distribution, then any Derivative Works that You distribute must
123
+ include a readable copy of the attribution notices contained
124
+ within such NOTICE file, excluding those notices that do not
125
+ pertain to any part of the Derivative Works, in at least one
126
+ of the following places: within a NOTICE text file distributed
127
+ as part of the Derivative Works; within the Source form or
128
+ documentation, if provided along with the Derivative Works; or,
129
+ within a display generated by the Derivative Works, if and
130
+ wherever such third-party notices normally appear. The contents
131
+ of the NOTICE file are for informational purposes only and
132
+ do not modify the License. You may add Your own attribution
133
+ notices within Derivative Works that You distribute, alongside
134
+ or as an addendum to the NOTICE text from the Work, provided
135
+ that such additional attribution notices cannot be construed
136
+ as modifying the License.
137
+
138
+ You may add Your own copyright statement to Your modifications and
139
+ may provide additional or different license terms and conditions
140
+ for use, reproduction, or distribution of Your modifications, or
141
+ for any such Derivative Works as a whole, provided Your use,
142
+ reproduction, and distribution of the Work otherwise complies with
143
+ the conditions stated in this License.
144
+
145
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
146
+ any Contribution intentionally submitted for inclusion in the Work
147
+ by You to the Licensor shall be under the terms and conditions of
148
+ this License, without any additional terms or conditions.
149
+ Notwithstanding the above, nothing herein shall supersede or modify
150
+ the terms of any separate license agreement you may have executed
151
+ with Licensor regarding such Contributions.
152
+
153
+ 6. Trademarks. This License does not grant permission to use the trade
154
+ names, trademarks, service marks, or product names of the Licensor,
155
+ except as required for reasonable and customary use in describing the
156
+ origin of the Work and reproducing the content of the NOTICE file.
157
+
158
+ 7. Disclaimer of Warranty. Unless required by applicable law or
159
+ agreed to in writing, Licensor provides the Work (and each
160
+ Contributor provides its Contributions) on an "AS IS" BASIS,
161
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
162
+ implied, including, without limitation, any warranties or conditions
163
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
164
+ PARTICULAR PURPOSE. You are solely responsible for determining the
165
+ appropriateness of using or redistributing the Work and assume any
166
+ risks associated with Your exercise of permissions under this License.
167
+
168
+ 8. Limitation of Liability. In no event and under no legal theory,
169
+ whether in tort (including negligence), contract, or otherwise,
170
+ unless required by applicable law (such as deliberate and grossly
171
+ negligent acts) or agreed to in writing, shall any Contributor be
172
+ liable to You for damages, including any direct, indirect, special,
173
+ incidental, or consequential damages of any character arising as a
174
+ result of this License or out of the use or inability to use the
175
+ Work (including but not limited to damages for loss of goodwill,
176
+ work stoppage, computer failure or malfunction, or any and all
177
+ other commercial damages or losses), even if such Contributor
178
+ has been advised of the possibility of such damages.
179
+
180
+ 9. Accepting Warranty or Additional Liability. While redistributing
181
+ the Work or Derivative Works thereof, You may choose to offer,
182
+ and charge a fee for, acceptance of support, warranty, indemnity,
183
+ or other liability obligations and/or rights consistent with this
184
+ License. However, in accepting such obligations, You may act only
185
+ on Your own behalf and on Your sole responsibility, not on behalf
186
+ of any other Contributor, and only if You agree to indemnify,
187
+ defend, and hold each Contributor harmless for any liability
188
+ incurred by, or claims asserted against, such Contributor by reason
189
+ of your accepting any such warranty or additional liability.
190
+
191
+ END OF TERMS AND CONDITIONS
192
+
193
+ APPENDIX: How to apply the Apache License to your work.
194
+
195
+ To apply the Apache License to your work, attach the following
196
+ boilerplate notice, with the fields enclosed by brackets "{}"
197
+ replaced with your own identifying information. (Don't include
198
+ the brackets!) The text should be enclosed in the appropriate
199
+ comment syntax for the file format. We also recommend that a
200
+ file or class name and description of purpose be included on the
201
+ same "printed page" as the copyright notice for easier
202
+ identification within third-party archives.
203
+
204
+ Copyright 2019 Ross Wightman
205
+
206
+ Licensed under the Apache License, Version 2.0 (the "License");
207
+ you may not use this file except in compliance with the License.
208
+ You may obtain a copy of the License at
209
+
210
+ http://www.apache.org/licenses/LICENSE-2.0
211
+
212
+ Unless required by applicable law or agreed to in writing, software
213
+ distributed under the License is distributed on an "AS IS" BASIS,
214
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
215
+ See the License for the specific language governing permissions and
216
+ limitations under the License.
217
+ ```
218
+
219
+ microsoft/unilm/tree/master/beit
220
+
221
+ Open Source License/Copyright Notice.
222
+
223
+ ```
224
+ The MIT License (MIT)
225
+
226
+ Copyright (c) Microsoft Corporation
227
+
228
+ Permission is hereby granted, free of charge, to any person obtaining a copy
229
+ of this software and associated documentation files (the "Software"), to deal
230
+ in the Software without restriction, including without limitation the rights
231
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
232
+ copies of the Software, and to permit persons to whom the Software is
233
+ furnished to do so, subject to the following conditions:
234
+
235
+ The above copyright notice and this permission notice shall be included in all
236
+ copies or substantial portions of the Software.
237
+
238
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
239
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
240
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
241
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
242
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
243
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
244
+ SOFTWARE.
245
+ ```
246
+
247
+ facebookresearch/SlowFast
248
+
249
+ Open Source License/Copyright Notice.
250
+
251
+ ```
252
+ Apache License
253
+ Version 2.0, January 2004
254
+ http://www.apache.org/licenses/
255
+
256
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
257
+
258
+ 1. Definitions.
259
+
260
+ "License" shall mean the terms and conditions for use, reproduction,
261
+ and distribution as defined by Sections 1 through 9 of this document.
262
+
263
+ "Licensor" shall mean the copyright owner or entity authorized by
264
+ the copyright owner that is granting the License.
265
+
266
+ "Legal Entity" shall mean the union of the acting entity and all
267
+ other entities that control, are controlled by, or are under common
268
+ control with that entity. For the purposes of this definition,
269
+ "control" means (i) the power, direct or indirect, to cause the
270
+ direction or management of such entity, whether by contract or
271
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
272
+ outstanding shares, or (iii) beneficial ownership of such entity.
273
+
274
+ "You" (or "Your") shall mean an individual or Legal Entity
275
+ exercising permissions granted by this License.
276
+
277
+ "Source" form shall mean the preferred form for making modifications,
278
+ including but not limited to software source code, documentation
279
+ source, and configuration files.
280
+
281
+ "Object" form shall mean any form resulting from mechanical
282
+ transformation or translation of a Source form, including but
283
+ not limited to compiled object code, generated documentation,
284
+ and conversions to other media types.
285
+
286
+ "Work" shall mean the work of authorship, whether in Source or
287
+ Object form, made available under the License, as indicated by a
288
+ copyright notice that is included in or attached to the work
289
+ (an example is provided in the Appendix below).
290
+
291
+ "Derivative Works" shall mean any work, whether in Source or Object
292
+ form, that is based on (or derived from) the Work and for which the
293
+ editorial revisions, annotations, elaborations, or other modifications
294
+ represent, as a whole, an original work of authorship. For the purposes
295
+ of this License, Derivative Works shall not include works that remain
296
+ separable from, or merely link (or bind by name) to the interfaces of,
297
+ the Work and Derivative Works thereof.
298
+
299
+ "Contribution" shall mean any work of authorship, including
300
+ the original version of the Work and any modifications or additions
301
+ to that Work or Derivative Works thereof, that is intentionally
302
+ submitted to Licensor for inclusion in the Work by the copyright owner
303
+ or by an individual or Legal Entity authorized to submit on behalf of
304
+ the copyright owner. For the purposes of this definition, "submitted"
305
+ means any form of electronic, verbal, or written communication sent
306
+ to the Licensor or its representatives, including but not limited to
307
+ communication on electronic mailing lists, source code control systems,
308
+ and issue tracking systems that are managed by, or on behalf of, the
309
+ Licensor for the purpose of discussing and improving the Work, but
310
+ excluding communication that is conspicuously marked or otherwise
311
+ designated in writing by the copyright owner as "Not a Contribution."
312
+
313
+ "Contributor" shall mean Licensor and any individual or Legal Entity
314
+ on behalf of whom a Contribution has been received by Licensor and
315
+ subsequently incorporated within the Work.
316
+
317
+ 2. Grant of Copyright License. Subject to the terms and conditions of
318
+ this License, each Contributor hereby grants to You a perpetual,
319
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
320
+ copyright license to reproduce, prepare Derivative Works of,
321
+ publicly display, publicly perform, sublicense, and distribute the
322
+ Work and such Derivative Works in Source or Object form.
323
+
324
+ 3. Grant of Patent License. Subject to the terms and conditions of
325
+ this License, each Contributor hereby grants to You a perpetual,
326
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
327
+ (except as stated in this section) patent license to make, have made,
328
+ use, offer to sell, sell, import, and otherwise transfer the Work,
329
+ where such license applies only to those patent claims licensable
330
+ by such Contributor that are necessarily infringed by their
331
+ Contribution(s) alone or by combination of their Contribution(s)
332
+ with the Work to which such Contribution(s) was submitted. If You
333
+ institute patent litigation against any entity (including a
334
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
335
+ or a Contribution incorporated within the Work constitutes direct
336
+ or contributory patent infringement, then any patent licenses
337
+ granted to You under this License for that Work shall terminate
338
+ as of the date such litigation is filed.
339
+
340
+ 4. Redistribution. You may reproduce and distribute copies of the
341
+ Work or Derivative Works thereof in any medium, with or without
342
+ modifications, and in Source or Object form, provided that You
343
+ meet the following conditions:
344
+
345
+ (a) You must give any other recipients of the Work or
346
+ Derivative Works a copy of this License; and
347
+
348
+ (b) You must cause any modified files to carry prominent notices
349
+ stating that You changed the files; and
350
+
351
+ (c) You must retain, in the Source form of any Derivative Works
352
+ that You distribute, all copyright, patent, trademark, and
353
+ attribution notices from the Source form of the Work,
354
+ excluding those notices that do not pertain to any part of
355
+ the Derivative Works; and
356
+
357
+ (d) If the Work includes a "NOTICE" text file as part of its
358
+ distribution, then any Derivative Works that You distribute must
359
+ include a readable copy of the attribution notices contained
360
+ within such NOTICE file, excluding those notices that do not
361
+ pertain to any part of the Derivative Works, in at least one
362
+ of the following places: within a NOTICE text file distributed
363
+ as part of the Derivative Works; within the Source form or
364
+ documentation, if provided along with the Derivative Works; or,
365
+ within a display generated by the Derivative Works, if and
366
+ wherever such third-party notices normally appear. The contents
367
+ of the NOTICE file are for informational purposes only and
368
+ do not modify the License. You may add Your own attribution
369
+ notices within Derivative Works that You distribute, alongside
370
+ or as an addendum to the NOTICE text from the Work, provided
371
+ that such additional attribution notices cannot be construed
372
+ as modifying the License.
373
+
374
+ You may add Your own copyright statement to Your modifications and
375
+ may provide additional or different license terms and conditions
376
+ for use, reproduction, or distribution of Your modifications, or
377
+ for any such Derivative Works as a whole, provided Your use,
378
+ reproduction, and distribution of the Work otherwise complies with
379
+ the conditions stated in this License.
380
+
381
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
382
+ any Contribution intentionally submitted for inclusion in the Work
383
+ by You to the Licensor shall be under the terms and conditions of
384
+ this License, without any additional terms or conditions.
385
+ Notwithstanding the above, nothing herein shall supersede or modify
386
+ the terms of any separate license agreement you may have executed
387
+ with Licensor regarding such Contributions.
388
+
389
+ 6. Trademarks. This License does not grant permission to use the trade
390
+ names, trademarks, service marks, or product names of the Licensor,
391
+ except as required for reasonable and customary use in describing the
392
+ origin of the Work and reproducing the content of the NOTICE file.
393
+
394
+ 7. Disclaimer of Warranty. Unless required by applicable law or
395
+ agreed to in writing, Licensor provides the Work (and each
396
+ Contributor provides its Contributions) on an "AS IS" BASIS,
397
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
398
+ implied, including, without limitation, any warranties or conditions
399
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
400
+ PARTICULAR PURPOSE. You are solely responsible for determining the
401
+ appropriateness of using or redistributing the Work and assume any
402
+ risks associated with Your exercise of permissions under this License.
403
+
404
+ 8. Limitation of Liability. In no event and under no legal theory,
405
+ whether in tort (including negligence), contract, or otherwise,
406
+ unless required by applicable law (such as deliberate and grossly
407
+ negligent acts) or agreed to in writing, shall any Contributor be
408
+ liable to You for damages, including any direct, indirect, special,
409
+ incidental, or consequential damages of any character arising as a
410
+ result of this License or out of the use or inability to use the
411
+ Work (including but not limited to damages for loss of goodwill,
412
+ work stoppage, computer failure or malfunction, or any and all
413
+ other commercial damages or losses), even if such Contributor
414
+ has been advised of the possibility of such damages.
415
+
416
+ 9. Accepting Warranty or Additional Liability. While redistributing
417
+ the Work or Derivative Works thereof, You may choose to offer,
418
+ and charge a fee for, acceptance of support, warranty, indemnity,
419
+ or other liability obligations and/or rights consistent with this
420
+ License. However, in accepting such obligations, You may act only
421
+ on Your own behalf and on Your sole responsibility, not on behalf
422
+ of any other Contributor, and only if You agree to indemnify,
423
+ defend, and hold each Contributor harmless for any liability
424
+ incurred by, or claims asserted against, such Contributor by reason
425
+ of your accepting any such warranty or additional liability.
426
+
427
+ END OF TERMS AND CONDITIONS
428
+
429
+ APPENDIX: How to apply the Apache License to your work.
430
+
431
+ To apply the Apache License to your work, attach the following
432
+ boilerplate notice, with the fields enclosed by brackets "[]"
433
+ replaced with your own identifying information. (Don't include
434
+ the brackets!) The text should be enclosed in the appropriate
435
+ comment syntax for the file format. We also recommend that a
436
+ file or class name and description of purpose be included on the
437
+ same "printed page" as the copyright notice for easier
438
+ identification within third-party archives.
439
+
440
+ Copyright 2019, Facebook, Inc
441
+
442
+ Licensed under the Apache License, Version 2.0 (the "License");
443
+ you may not use this file except in compliance with the License.
444
+ You may obtain a copy of the License at
445
+
446
+ http://www.apache.org/licenses/LICENSE-2.0
447
+
448
+ Unless required by applicable law or agreed to in writing, software
449
+ distributed under the License is distributed on an "AS IS" BASIS,
450
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
451
+ See the License for the specific language governing permissions and
452
+ limitations under the License.
453
+ ```
PRETRAIN.md ADDED
@@ -0,0 +1,120 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Pre-training SIGMA
2
+
3
+ ## Original Implementation
4
+
5
+ The implementation of our SIGMA supports **multi-node distributed training**. We provide the **off-the-shelf** scripts in the [scripts folder](scripts).
6
+
7
+ - For example, to pre-train SIGMA ViT-Base on **Something-Something V2** with 64 GPUs (8 nodes x 8 GPUs), you can run
8
+
9
+ ```bash
10
+ OUTPUT_DIR='YOUR_PATH/ssv2_SIGMA_pretrain_base_patch16_224_frame_16x2_tube_mask_ratio_0.9_e800'
11
+ DATA_PATH='YOUR_PATH/list_ssv2/train.csv'
12
+
13
+ OMP_NUM_THREADS=1 python -m torch.distributed.launch --nproc_per_node=8 \
14
+ --master_port 12320 --nnodes=8 \
15
+ --node_rank=0 --master_addr=$ip_node_0 \
16
+ run_mae_pretraining.py \
17
+ --data_path ${DATA_PATH} \
18
+ --mask_type tube \
19
+ --mask_ratio 0.9 \
20
+ --model pretrain_videomae_base_patch16_224 \
21
+ --decoder_depth 4 \
22
+ --batch_size 32 \
23
+ --num_frames 16 \
24
+ --sampling_rate 2 \
25
+ --opt adamw \
26
+ --opt_betas 0.9 0.95 \
27
+ --warmup_epochs 40 \
28
+ --save_ckpt_freq 20 \
29
+ --epochs 801 \
30
+ --log_dir ${OUTPUT_DIR} \
31
+ --output_dir ${OUTPUT_DIR}
32
+ ```
33
+
34
+ on the first node. On other nodes, run the same command with `--node_rank 1`, ..., `--node_rank 7` respectively. `--master_addr` is set as the ip of the node 0.
35
+
36
+ - For example, to pre-train SIGMA ViT-Base on **Kinetics400** with 64 GPUs (8 nodes x 8 GPUs), you can run
37
+
38
+ ```bash
39
+ OUTPUT_DIR='YOUR_PATH/k400_SIGMA_pretrain_base_patch16_224_frame_16x4_tube_mask_ratio_0.9_e800'
40
+ DATA_PATH='YOUR_PATH/list_kinetics-400/train.csv'
41
+
42
+ OMP_NUM_THREADS=1 python3 -m torch.distributed.launch --nproc_per_node=8 \
43
+ --master_port 12320 --nnodes=8 \
44
+ --node_rank=0 --master_addr=$your_node_0_ip \
45
+ run_mae_pretraining.py \
46
+ --data_path ${DATA_PATH} \
47
+ --mask_type tube \
48
+ --mask_ratio 0.9 \
49
+ --model pretrain_videomae_base_patch16_224 \
50
+ --decoder_depth 4 \
51
+ --batch_size 32 \
52
+ --num_frames 16 \
53
+ --sampling_rate 4 \
54
+ --opt adamw \
55
+ --opt_betas 0.9 0.95 \
56
+ --warmup_epochs 40 \
57
+ --save_ckpt_freq 20 \
58
+ --epochs 801 \
59
+ --log_dir ${OUTPUT_DIR} \
60
+ --output_dir ${OUTPUT_DIR}
61
+ ```
62
+
63
+ on the first node. On other nodes, run the same command with `--node_rank 1`, ..., `--node_rank 7` respectively. `--master_addr` is set as the ip of the node 0.
64
+
65
+ ### Note:
66
+
67
+ - Here the batch size is 32 (`batch_size` per gpu) * 8 (`nodes`) * 8 (gpus per node) = 2048.
68
+ - `lr` here is the base learning rate and is set to `1.5e-4` as default. The ` actual lr` is computed by the [linear scaling rule](https://arxiv.org/abs/1706.02677): `` actual lr`` = `lr` * total batch size / 256.
69
+ - [Fixed]~~We have observed accidental interrupt in the last epoch when conduct the experiment on V100 GPUs (torch 1.6.0). This interrupt is caused by the scheduler of learning rate. We naively set `--epochs 801` to walk away from issue :)~~
70
+
71
+ ## Slurm
72
+
73
+ To help the community to reproduce our results on slurm cluster, we also provide the the **off-the-shelf** script.
74
+
75
+ For example, to pre-train SIGMA ViT-Base on **Kinetics400** with 64 GPUs (8 nodes x 8 GPUs), you can run
76
+
77
+ ```bash
78
+ export MASTER_PORT=$((12000 + $RANDOM % 20000))
79
+ export OMP_NUM_THREADS=1
80
+
81
+ OUTPUT_DIR='YOUR_PATH/k400_SIGMA_pretrain_base_patch16_224_frame_16x4_tube_mask_ratio_0.9_e800'
82
+ DATA_PATH='YOUR_PATH/list_kinetics-400/train.csv'
83
+
84
+ JOB_NAME=$1
85
+ PARTITION=${PARTITION:-"video"}
86
+ # 8 for 1 node, 16 for 2 node, etc.
87
+ GPUS=${GPUS:-64}
88
+ GPUS_PER_NODE=${GPUS_PER_NODE:-8}
89
+ CPUS_PER_TASK=${CPUS_PER_TASK:-8}
90
+ SRUN_ARGS=${SRUN_ARGS:-""}
91
+ PY_ARGS=${@:2}
92
+
93
+ # batch_size can be adjusted according to the graphics card
94
+ srun -p $PARTITION \
95
+ --job-name=${JOB_NAME} \
96
+ --gres=gpu:${GPUS_PER_NODE} \
97
+ --ntasks=${GPUS} \
98
+ --ntasks-per-node=${GPUS_PER_NODE} \
99
+ --cpus-per-task=${CPUS_PER_TASK} \
100
+ --kill-on-bad-exit=1 \
101
+ ${SRUN_ARGS} \
102
+ python -u run_mae_pretraining.py \
103
+ --data_path ${DATA_PATH} \
104
+ --mask_type tube \
105
+ --mask_ratio 0.9 \
106
+ --model pretrain_SIGMA_base_patch16_224 \
107
+ --decoder_depth 4 \
108
+ --batch_size 32 \
109
+ --num_frames 16 \
110
+ --sampling_rate 4 \
111
+ --opt adamw \
112
+ --opt_betas 0.9 0.95 \
113
+ --warmup_epochs 40 \
114
+ --save_ckpt_freq 20 \
115
+ --epochs 801 \
116
+ --log_dir ${OUTPUT_DIR} \
117
+ --output_dir ${OUTPUT_DIR} \
118
+ ${PY_ARGS}
119
+ ```
120
+
README.md CHANGED
@@ -1,5 +1,159 @@
1
- ---
2
- license: other
3
- license_name: bsd-3-clause-clear-license
4
- license_link: https://github.com/QUVA-Lab/SIGMA/blob/gh-pages/LICENSE
5
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Official PyTorch Implementation of SIGMA(ECCV 2024).
2
+
3
+ ![VideoMAE Framework](figs/method.jpg)
4
+
5
+ [![License: CC BY-NC 4.0](https://img.shields.io/badge/License-CC_BY--NC_4.0-lightgrey.svg)](https://creativecommons.org/licenses/by-nc/4.0/)<br>
6
+
7
+
8
+ <!-- [![Hugging Face Models](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Models-blue)](https://huggingface.co/models?other=videomae)[![Hugging Face Spaces](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue)](https://huggingface.co/spaces/sayakpaul/video-classification-ucf101-subset)[![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/video_classification.ipynb)<br>
9
+ [![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/videomae-masked-autoencoders-are-data-1/action-recognition-in-videos-on-something)](https://paperswithcode.com/sota/action-recognition-in-videos-on-something?p=videomae-masked-autoencoders-are-data-1)<br>
10
+ [![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/videomae-masked-autoencoders-are-data-1/action-classification-on-kinetics-400)](https://paperswithcode.com/sota/action-classification-on-kinetics-400?p=videomae-masked-autoencoders-are-data-1)<br>[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/videomae-masked-autoencoders-are-data-1/action-recognition-on-ava-v2-2)](https://paperswithcode.com/sota/action-recognition-on-ava-v2-2?p=videomae-masked-autoencoders-are-data-1)<br>
11
+ [![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/videomae-masked-autoencoders-are-data-1/self-supervised-action-recognition-on-ucf101)](https://paperswithcode.com/sota/self-supervised-action-recognition-on-ucf101?p=videomae-masked-autoencoders-are-data-1)<br>
12
+ [![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/videomae-masked-autoencoders-are-data-1/self-supervised-action-recognition-on-hmdb51)](https://paperswithcode.com/sota/self-supervised-action-recognition-on-hmdb51?p=videomae-masked-autoencoders-are-data-1) -->
13
+
14
+ <!-- > [**VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking**](https://arxiv.org/abs/2303.16727)<br>
15
+ > [Limin Wang](http://wanglimin.github.io/), [Bingkun Huang](https://github.com/congee524), [Zhiyu Zhao](https://github.com/JerryFlymi), [Zhan Tong](https://github.com/yztongzhan), Yinan He, Yi Wang, Yali Wang, Yu Qiao <br>Nanjing University, Shanghai AI Lab, CAS
16
+
17
+ > [**VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training**](https://arxiv.org/abs/2203.12602)<br>
18
+ > [Zhan Tong](https://github.com/yztongzhan), [Yibing Song](https://ybsong00.github.io/), [Jue Wang](https://juewang725.github.io/), [Limin Wang](http://wanglimin.github.io/)<br>Nanjing University, Tencent AI Lab -->
19
+
20
+ <!-- ## 📰 News
21
+ **[2023.4.3]** VideoMAE V2 is accepted by **CVPR 2023**! 🎉 Code comming soon. <br>
22
+ **[2023.1.16]** Code and pre-trained models for Action Detection are [available](https://github.com/MCG-NJU/VideoMAE-Action-Detection)! <br>
23
+ **[2022.11.20]** 👀 VideoMAE is integrated into [![Hugging Face Spaces](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue)](https://huggingface.co/spaces/sayakpaul/video-classification-ucf101-subset) and [![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/video_classification.ipynb), supported by [@Sayak Paul](https://github.com/sayakpaul).<br>
24
+ **[2022.10.25]** 👀 VideoMAE is integrated into [MMAction2](https://github.com/open-mmlab/mmaction2/tree/dev-1.x/configs/recognition/videomae), the results on Kinetics-400 can be reproduced successfully. <br>
25
+ **[2022.10.20]** The pre-trained models and scripts of **ViT-S** and **ViT-H** are available! <br>
26
+ **[2022.10.19]** The pre-trained models and scripts on **UCF101** are [available](MODEL_ZOO.md#UCF101)! <br>
27
+ **[2022.9.15]** VideoMAE is accepted by **NeurIPS 2022** as a **spotlight** presentation! 🎉 <br>
28
+ **[2022.8.8]** 👀 VideoMAE is integrated into **official** [🤗HuggingFace Transformers](https://huggingface.co/docs/transformers/main/en/model_doc/videomae) now! [![Hugging Face Models](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Models-blue)](https://huggingface.co/models?other=videomae)<br>
29
+ **[2022.7.7]** We have updated new results on downstream AVA 2.2 benchmark. Please refer to our [paper](https://arxiv.org/abs/2203.12602) for details. <br>
30
+ **[2022.4.24]** Code and pre-trained models are available now! <br>
31
+ **[2022.3.24]** ~~Code and pre-trained models will be released here.~~ Welcome to **watch** this repository for the latest updates. -->
32
+
33
+ <!-- ## ✨ Highlights -->
34
+
35
+ ### 🔥 Sinkhorn-Guided Masked Video Modeling
36
+
37
+ Video-based pretraining offers immense potential for learning strong visual representations on an unprecedented scale. Recently, masked video modeling methods have shown promising scalability, yet fall short in capturing higher-level semantics due to reconstructing predefined low-level targets such as pixels. To tackle this, we present Sinkhorn-guided Masked Video Modelling (SIGMA), a novel video pretraining method that jointly learns the video model in addition to a target feature space using a projection network. However, this simple modification means that the regular L2 reconstruction loss will lead to trivial solutions as both networks are jointly optimized. As a solution, we distribute features of space-time tubes evenly across a limited number of learnable clusters. By posing this as an optimal transport problem, we enforce high entropy in the generated features across the batch, infusing semantic and temporal meaning into the feature space. The resulting cluster assignments are used as targets for a symmetric prediction task where the video model predicts cluster assignment of the projection network and vice versa. Experimental results on ten datasets across three benchmarks validate the effectiveness of SIGMA in learning more performant, temporally-aware, and robust video representations improving upon state-of-the-art methods.
38
+
39
+ <!-- ### ⚡️ A Simple, Efficient and Strong Baseline in SSVP
40
+
41
+ VideoMAE uses the simple masked autoencoder and **plain ViT** backbone to perform video self-supervised learning. Due to the extremely high masking ratio, the pre-training time of VideoMAE is **much shorter** than contrastive learning methods (**3.2x** speedup). VideoMAE can serve as **a simple but strong baseline** for future research in self-supervised video pre-training.
42
+
43
+ ### 😮 High performance, but NO extra data required
44
+
45
+ VideoMAE works well for video datasets of different scales and can achieve **87.4%** on Kinects-400, **75.4%** on Something-Something V2, **91.3%** on UCF101, and **62.6%** on HMDB51. To our best knowledge, VideoMAE is the **first** to achieve the state-of-the-art performance on these four popular benchmarks with the **vanilla ViT** backbones while **doesn't need** any extra data or pre-trained models.
46
+
47
+ <!-- ## 🚀 Main Results -->
48
+
49
+ ### ✨ Something-Something V2
50
+
51
+ | Method | Extra Data | Backbone | Resolution | #Frames x Clips x Crops | Epoch | Top-1 |
52
+ | :------: | :--------: | :------: | :--------: | :---------------------: | :---: | :---: |
53
+ | VideoMAE | ***no*** | ViT-S | 224x224 | 16x2x3 | 2400 | 66.8 |
54
+ | VideoMAE | ***no*** | ViT-B | 224x224 | 16x2x3 | 800 | 69.6 |
55
+ | SIGMA |***Img-1k***| ViT-S | 224x224 | 16x2x3 | 2400 | 68.6 |
56
+ | SIGMA |***Img-1k***| ViT-B | 224x224 | 16x2x3 | 800 | 70.9 |
57
+
58
+ ### ✨ Kinetics-400
59
+
60
+
61
+ | Method | Extra Data | Backbone | Resolution | #Frames x Clips x Crops | Epoch | Top-1 |
62
+ | :------: | :--------: | :------: | :--------: | :---------------------: | :---: | :---: |
63
+ | VideoMAE | ***no*** | ViT-S | 224x224 | 16x5x3 | 1600 | 79.0 |
64
+ | VideoMAE | ***no*** | ViT-B | 224x224 | 16x5x3 | 800 | 80.0 |
65
+ | SIGMA |***Img-1k***| ViT-S | 224x224 | 16x5x3 | 800 | 79.4 |
66
+ | SIGMA |***Img-1k***| ViT-B | 224x224 | 16x5x3 | 800 | 81.6 |
67
+
68
+
69
+ <!-- | Method | Extra Data | Backbone | Resolution | #Frames x Clips x Crops | Top-1 | Top-5 |
70
+ | :------: | :--------: | :------: | :--------: | :---------------------: | :---: | :---: |
71
+ | VideoMAE | ***no*** | ViT-S | 224x224 | 16x5x3 | 79.0 | 93.8 |
72
+ | VideoMAE | ***no*** | ViT-B | 224x224 | 16x5x3 | 81.5 | 95.1 |
73
+ | VideoMAE | ***no*** | ViT-L | 224x224 | 16x5x3 | 85.2 | 96.8 |
74
+ | VideoMAE | ***no*** | ViT-H | 224x224 | 16x5x3 | 86.6 | 97.1 |
75
+ | VideoMAE | ***no*** | ViT-L | 320x320 | 32x4x3 | 86.1 | 97.3 |
76
+ | VideoMAE | ***no*** | ViT-H | 320x320 | 32x4x3 | 87.4 | 97.6 | -->
77
+
78
+ <!-- ### ✨ AVA 2.2
79
+
80
+ Please check the code and checkpoints in [VideoMAE-Action-Detection](https://github.com/MCG-NJU/VideoMAE-Action-Detection).
81
+ | Method | Extra Data | Extra Label | Backbone | #Frame x Sample Rate | mAP |
82
+ | :------: | :----------: | :---------: | :------: | :------------------: | :--: |
83
+ | VideoMAE | Kinetics-400 | &cross; | ViT-S | 16x4 | 22.5 |
84
+ | VideoMAE | Kinetics-400 | &check; | ViT-S | 16x4 | 28.4 |
85
+ | VideoMAE | Kinetics-400 | &cross; | ViT-B | 16x4 | 26.7 |
86
+ | VideoMAE | Kinetics-400 | &check; | ViT-B | 16x4 | 31.8 |
87
+ | VideoMAE | Kinetics-400 | &cross; | ViT-L | 16x4 | 34.3 |
88
+ | VideoMAE | Kinetics-400 | &check; | ViT-L | 16x4 | 37.0 |
89
+ | VideoMAE | Kinetics-400 | &cross; | ViT-H | 16x4 | 36.5 |
90
+ | VideoMAE | Kinetics-400 | &check; | ViT-H | 16x4 | 39.5 |
91
+ | VideoMAE | Kinetics-700 | &cross; | ViT-L | 16x4 | 36.1 |
92
+ | VideoMAE | Kinetics-700 | &check; | ViT-L | 16x4 | 39.3 | -->
93
+
94
+ <!-- ### ✨ UCF101 & HMDB51
95
+
96
+ | Method | Extra Data | Backbone | UCF101 | HMDB51 |
97
+ | :------: | :----------: | :------: | :----: | :----: |
98
+ | VideoMAE | ***no*** | ViT-B | 91.3 | 62.6 |
99
+ | VideoMAE | Kinetics-400 | ViT-B | 96.1 | 73.3 | -->
100
+
101
+ ## 🔨 Installation
102
+
103
+ Please follow the instructions in [INSTALL.md](INSTALL.md).
104
+
105
+ ## ➡️ Data Preparation
106
+
107
+ Please follow the instructions in [DATASET.md](DATASET.md) for data preparation.
108
+
109
+ ## 🔄 Pre-training
110
+
111
+ The pre-training instruction is in [PRETRAIN.md](PRETRAIN.md).
112
+
113
+ ## ⤴️ Fine-tuning with pre-trained models
114
+
115
+ The fine-tuning instruction is in [FINETUNE.md](FINETUNE.md).
116
+
117
+ ## 📍Model Zoo
118
+
119
+
120
+ ## ⚠️ Our code is based on [VideoMAE](https://github.com/MCG-NJU/VideoMAE) code base.
121
+
122
+ <!-- We provide pre-trained and fine-tuned models in [MODEL_ZOO.md](MODEL_ZOO.md). -->
123
+
124
+ <!-- ## 👀 Visualization -->
125
+
126
+ <!-- We provide the script for visualization in [`vis.sh`](vis.sh). Colab notebook for better visualization is coming soon. -->
127
+
128
+ <!-- ## ☎️ Contact
129
+
130
+ Zhan Tong: tongzhan@smail.nju.edu.cn
131
+
132
+ ## 👍 Acknowledgements
133
+
134
+ Thanks to [Ziteng Gao](https://sebgao.github.io/), Lei Chen, [Chongjian Ge](https://chongjiange.github.io/), and [Zhiyu Zhao](https://github.com/JerryFlymi) for their kind support.<br>
135
+ This project is built upon [MAE-pytorch](https://github.com/pengzhiliang/MAE-pytorch) and [BEiT](https://github.com/microsoft/unilm/tree/master/beit). Thanks to the contributors of these great codebases.
136
+
137
+ ## 🔒 License
138
+
139
+ The majority of this project is released under the CC-BY-NC 4.0 license as found in the [LICENSE](https://github.com/MCG-NJU/VideoMAE/blob/main/LICENSE) file. Portions of the project are available under separate license terms: [SlowFast](https://github.com/facebookresearch/SlowFast) and [pytorch-image-models](https://github.com/rwightman/pytorch-image-models) are licensed under the Apache 2.0 license. [BEiT](https://github.com/microsoft/unilm/tree/master/beit) is licensed under the MIT license.
140
+
141
+ ## ✏️ Citation
142
+
143
+ If you think this project is helpful, please feel free to leave a star⭐️ and cite our paper:
144
+
145
+ ```
146
+ @inproceedings{tong2022videomae,
147
+ title={Video{MAE}: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training},
148
+ author={Zhan Tong and Yibing Song and Jue Wang and Limin Wang},
149
+ booktitle={Advances in Neural Information Processing Systems},
150
+ year={2022}
151
+ }
152
+
153
+ @article{videomae,
154
+ title={VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training},
155
+ author={Tong, Zhan and Song, Yibing and Wang, Jue and Wang, Limin},
156
+ journal={arXiv preprint arXiv:2203.12602},
157
+ year={2022}
158
+ }
159
+ ``` -->