Hugging Face's logo

Spaces:
nateraw
/
lavila
Runtime error

App Files Community
2
lavila
File size: 5,519 Bytes 
39d5658
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
 
# Preparing datasets for LAVILA

Please download the (selected) datasets from the official websites and place or sim-link them under `$LAVILA_ROOT/datasets/`.

```bash
$LAVILA_ROOT/datasets/
    CharadesEgo/
    EGTEA/
    EK100/
    Ego4D/
```

## Ego4D
1. Download [Ego4D videos](https://ego4d-data.org/docs/start-here/#download-data) (license is required).

2. Preprocess（TBA)

3. Download annotations

    a. Download [egomcq.json](https://drive.google.com/file/d/1-5iRYf4BCHmj4MYQYFRMY4bhsWJUN3rW/view) to `$LAVILA_ROOT/datasets/Ego4D` (if you want to evaluate EgoMCQ).

    b. Download [metadata for train split](https://dl.fbaipublicfiles.com/lavila/metadata/ego4d/ego4d_train.pkl) and [val split](https://dl.fbaipublicfiles.com/lavila/metadata/ego4d/ego4d_val.pkl) to `$LAVILA_ROOT/datasets/Ego4D` ((if you want to train LAVILA from scratch).

The fold should look like this:
```bash
$LAVILA_ROOT/datasets/
    Ego4D/
        ego4d_train.pkl
        ego4d_val.pkl
        egomcq.json
        video_288px/
            000786a7-3f9d-4fe6-bfb3-045b368f7d44.mp4/
                0.mp4
                300.mp4
            000a3525-6c98-4650-aaab-be7d2c7b9402.mp4/
                0.mp4
            ...
```


## EPIC-Kitchens-100 (EK-100)

1. Download annotations

```bash
# Assume that you are under `datasets/EK100/`
git clone https://github.com/epic-kitchens/epic-kitchens-100-annotations
```

2. Download videos.

    a. For raw videos, please download them from [https://epic-kitchens.github.io/](https://epic-kitchens.github.io/).

    b. (Recommended) The raw videos are huge (~1 TB). As an alternative, please check out a [resized version]().

3. (For EK-100 MIR)

    a. Generate the relevancy matrix of train/val splits using [the official code](https://github.com/mwray/Joint-Part-of-Speech-Embeddings).

    b. (Recommended) The generated result has some randomness. Therefore, we also provide the [replica of train split](https://dl.fbaipublicfiles.com/lavila/metadata/EK100/caption_relevancy_EPIC_100_retrieval_train.pkl) and [val split](https://dl.fbaipublicfiles.com/lavila/metadata/EK100/caption_relevancy_EPIC_100_retrieval_test.pkl). Please put them to the folder `$LAVILA_ROOT/datasets/EK100/epic-kitchens-100-annotations/retrieval_annotations/relevancy/`.


The folder should look like this:
```bash
$LAVILA_ROOT/datasets/
    EK100/
        epic-kitchens-100-annotations/
            EPIC_100_train.csv
            EPIC_100_validation.csv
            ...
            retrieval_annotations/relevancy/  # this appears if you do 3.
                caption_relevancy_EPIC_100_retrieval_train.pkl
                caption_relevancy_EPIC_100_retrieval_test.pkl
        video_ht256px/
            P01/
                P01_01.MP4
                P01_02.MP4
                ...
                P01_19.MP4
            P02/
                P02_01.MP4
                P02_02.MP4
                ...
                P02_15.MP4
            ...
```

## CharadesEgo

1. Download annotations at [https://prior.allenai.org/projects/charades-ego](https://prior.allenai.org/projects/charades-ego).
```bash
### Annotations
# Assume that you are under `datasets/CharadesEgo/`
wget https://ai2-public-datasets.s3-us-west-2.amazonaws.com/charades/CharadesEgo.zip
unzip CharadesEgo.zip && rm CharadesEgo.zip
```

2. Download data (~11GB) at [https://prior.allenai.org/projects/charades-ego](https://prior.allenai.org/projects/charades-ego).
```bash
### Data
wget https://ai2-public-datasets.s3-us-west-2.amazonaws.com/charades/CharadesEgo_v1_480.tar
tar -xvf CharadesEgo_v1_480.tar  # Or specify an external path using `-C` and sim-link it to here
rm CharadesEgo_v1_480.tar
```

3. (For fine-tuning CharadesEgo) Download two additional metadata files: [clip-level metadata (train)](https://dl.fbaipublicfiles.com/lavila/metadata/CharadesEgo/metadata_filtered_train.pkl) and [clip-level metadata (val)](https://dl.fbaipublicfiles.com/lavila/metadata/CharadesEgo/metadata_filtered_val.pkl). Put them to the folder `$LAVILA_ROOT/datasets/CharadesEgo/CharadesEgo/`.

The folder should look like this:
```bash
$LAVILA_ROOT/datasets/
    CharadesEgo/
        CharadesEgo/
            CharadesEgo_v1_train_only1st.csv
            CharadesEgo_v1_test_only1st.csv
            ...
            metadata_filtered_train.pkl  # this appears if you do 3.
            metadata_filtered_val.pkl    # this appears if you do 3.
        CharadesEgo_v1_480/
            005BU.mp4
            005BUEGO.mp4
            ...
```


## EGTEA

1. Visit [https://cbs.ic.gatech.edu/fpv/](https://cbs.ic.gatech.edu/fpv/).

2. Download `TRIMMED_ACTION_CLIPS` (~20GB) and `ACTION_ANNOTATIONS` and untar to the current folder `$LAVILA_ROOT/datasets/EGTEA`.

```bash
unzip action_annotation.zip -d EGTEA/ && rm action_annotation.zip
```

The folder should look like this:
```bash
$LAVILA_ROOT/datasets/
    EGTEA/
        train_split1.txt
        test_split1.txt
        cropped_clips/
            OP01-R01-PastaSalad/
                OP01-R01-PastaSalad-1002316-1004005-F024051-F024101.mp4
                OP01-R01-PastaSalad-1004110-1021110-F024057-F024548.mp4
                OP01-R01-PastaSalad-1022590-1024050-F024539-F024581.mp4
                ...
            OP01-R02-TurkeySandwich/
                OP01-R02-TurkeySandwich-102320-105110-F002449-F002529.mp4
                OP01-R02-TurkeySandwich-105440-106460-F002528-F002558.mp4
                OP01-R02-TurkeySandwich-107332-133184-F002513-F003259.mp4
                ...
            ...
```