license: cc-by-nc-4.0
pipeline_tag: robotics
Model Card: VC-1 (Visual Cortex ViT-Large) Last updated: 2023-04-07
Version: 1.0
Code: https://github.com/facebookresearch/eai-vc
Other Links: VC-1 Website, VC-1 Blogpost, VC-1 Paper, VC-1 Demo The VC-1 model is a vision transformer (ViT) pre-trained on over 4,000 hours of egocentric videos from 7 different sources, together with ImageNet. The model is trained using Masked Auto-Encoding (MAE) and is available in two sizes: ViT-B and ViT-L. The model is intended for use for EmbodiedAI tasks, such as object manipulation and indoor navigation. The VC-1 model is a vision transformer (ViT) pre-trained on over 4,000 hours of egocentric videos from 7 different sources, together with ImageNet. The model is trained using Masked Auto-Encoding (MAE) and is available in two sizes: ViT-B and ViT-L. The model is intended for use for EmbodiedAI tasks, such as object manipulation and indoor navigation.
Model Details
- Model Name: VC-1 (Vision Transformer-based model)
- Architecture:
- Patch size: 16x16
- Embedding dimension: 1024
- Number of layers: 24
- Number of heads: 16
- MLP ratio: 4
- QKV bias: True
- Layer normalization: eps=1e-6
- Inputs: Images presented in 224x224x3.
- Outputs: 1024x1 embedding.
- Image Size: 224
- Use of Classification Token: True
- Dropout Rate: 0.0
- Algorithm: MAE
- Epochs trained: 182
- Model authors: Arjun Majumdar, Karmesh Yadav, Sergio Arnaud, Yecheng Jason Ma, Claire Chen, Sneha Silwal, Aryan Jain, Vincent-Pierre Berges, Pieter Abbeel, Jitendra Malik, Dhruv Batra, Yixin Lin, Oleksandr Maksymets, Aravind Rajeswaran, and Franziska Meier.
- Person of Contact: Oleksandr Maksymets (FAIR)
Citation
If you use this model, please cite:
@inproceedings{vc2023,
title={Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?},
author={Arjun Majumdar and Karmesh Yadav and Sergio Arnaud and Yecheng Jason Ma and Claire Chen and Sneha Silwal and Aryan Jain and Vincent-Pierre Berges and Pieter Abbeel and Jitendra Malik and Dhruv Batra and Yixin Lin and Oleksandr Maksymets and Aravind Rajeswaran and Franziska Meier},
year={2023},
eprint={2303.18240},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
Model Data
Training data: The VC-1 model was trained on a large-scale dataset of egocentric videos, consisting of over 5.6 million frames. The dataset includes three modalities: manipulation, navigation, and object recognition. The manipulation modality includes videos of people performing various manipulations, such as cooking, cleaning, and tool use. The navigation modality includes videos of people moving around in indoor environments, such as homes and offices. The object recognition modality includes images from the ImageNet dataset, which contains over 1.2 million images of objects in various categories.
This table provides an overview of the assembled datasets used for scaling hypothesis experiments, including the total number of frames and the frames used for each dataset:
| Dataset | Contains | Total Frames | Frames used |
|---|---|---|---|
| Ego4D | Ego4D | 418,578,043 | 2,790,520 |
| EgoM (Manipulation) | Ego4D | 418,578,043 | 2,790,520 |
| 100DOH | 99,899 | 99,899 | |
| SS-v2 | 25,209,271 | 315,115 | |
| Epic Kitchens | 19,965,439 | 332,757 | |
| Total | 3,538,291 | ||
| EgoO (OpenHouse24) | Ego4D | 418,578,043 | 2,790,520 |
| OpenHouse24 | 27,806,971 | 499,442 | |
| Total | 3,289,962 | ||
| EgoN (Navigation) | Ego4D | 418,578,043 | 2,790,520 |
| OpenHouse24 | 27,806,971 | 499,442 | |
| RealEstate10K | 10,000,000 | 303,087 | |
| Total | 3,289,962 | ||
| EgoMN (Manipulation, Navigation) | Ego4D+M | 3,538,291 | 3,538,291 |
| OpenHouse24 | 27,806,971 | 499,442 | |
| RealEstate10K | 10,000,000 | 303,087 | |
| Total | 4,340,820 | ||
| EgoMNI (Manipulation, Navigation, ImageNet) | Ego4D+MN | 4,340,820 | 4,340,820 |
| ImageNet | 1,281,167 | 1,281,167 | |
| Total | 5,621,987 |
The VC-1 models were trained on EgoMNI (Manipulation, Navigation, ImageNet) assembled dataset.
Evaluation data (see also section Evaluation Results below): The mode was evaluated on CortexBench that includes 17 tasks from 7 benchmarks and described below:
| Benchmark | Tasks |
|---|---|
| Adroit | Relocate, Reorient-Pen |
| MetaWorld | Assembly, Bin-Picking, Button-Press, Drawer-Open, Hammer |
| DeepMind Control | Finger-Spin, Reacher-Hard, Cheetah-Run, Walker-Stand, Walker-Walk |
| TriFinger | Reach-Cube, Push-Cube |
| Habitat | Image-Goal Navigation (ImageNav), Object-Goal Navigation (ObjectNav) |
| Habitat 2.0 | Mobile Pick |
Model Creation & Maintenance
The VC-1 model was created by pre-training ViT-B and ViT-L on a combination of egocentric videos and ImageNet using Masked Auto-Encoding (MAE). The model is maintained by the authors and is available for open-source use.
Model Usage
The VC-1 model is intended for EmbodiedAI tasks, such as object manipulation and indoor navigation.. The model outputs embeddings for image frame, which can be used as features for downstream tasks:
from vc_models.models.vit import model_utils
model,embd_size,model_transforms,model_info = model_utils.load_model(model_utils.VC1_BASE_NAME)
#the img loaded should be Bx3x250x250
img = your_function_here ...
#output will be of size Bx3x224x224
transformed_img = model_transforms(img)
#img will be 1x768
embedding = model(transformed_img)
Performance
The performance of the models on the CortexBench:
| Model | Adroit | Meta-World | DMControl | Trifinger | ObjectNav | ImageNav | Mobile Pick | Mean Rank | Mean Success |
|---|---|---|---|---|---|---|---|---|---|
| Ego4D (VIT-B) | 48.7 ± 1.3 | 86.1 ± 2.1 | 64.1 ± 2.3 | 68.3 ± 1.1 | 46.8 ± 1.1 | 64.0 ± 0.7 | 57.4 ± 2.2 | 8.6 | 62.2 |
| Ego4D (VIT-L) | 50.0 ± 1.2 | 92.9 ± 2.4 | 60.8 ± 3.3 | 69.7 ± 0.5 | 47.6 ± 1.1 | 55.8 ± 0.8 | 67.6 ± 2.1 | 5.9 | 63.5 |
| Ego4D+N (VIT-B) | 50.0 ± 2.4 | 86.4 ± 2.9 | 59.5 ± 2.4 | 67.8 ± 1.3 | 54.7 ± 1.1 | 68.7 ± 0.7 | 59.4 ± 2.2 | 7.2 | 63.8 |
| Ego4D+N (VIT-L) | 54.0 ± 1.2 | 89.1 ± 2.9 | 66.4 ± 1.7 | 66.9 ± 0.4 | 57.4 ± 1.1 | 70.5 ± 0.7 | 65.2 ± 2.1 | 3.5 | 67.1 |
| Ego4D+M (VIT-B) | 51.3 ± 2.4 | 83.5 ± 2.6 | 64.3 ± 1.8 | 69.1 ± 0.4 | 47.3 ± 1.1 | 65.8 ± 0.7 | 59.8 ± 2.2 | 7.0 | 63.0 |
| Ego4D+M (VIT-L) | 52.0 ± 1.3 | 88.3 ± 3.2 | 64.7 ± 2.4 | 64.7 ± 0.9 | 47.3 ± 1.1 | 65.5 ± 0.7 | 68.6 ± 2.1 | 6.0 | 64.4 |
| VC-1: Ego4D+MN (VIT-B) | 48.7 ± 2.4 | 85.3 ± 5.2 | 64.2 ± 1.9 | 70.3 ± 0.5 | 52.8 ± 1.1 | 68.9 ± 0.7 | 58.6 ± 2.2 | 6.9 | 64.1 |
| VC-1: Ego4D + MNI (VIT-L) | 59.3 ± 5.2 | 88.8 ± 2.2 | 66.9 ± 1.4 | 71.7 ± 0.4 | 60.3 ± 1.1 | 70.3 ± 0.7 | 63.2 ± 2.2 | 2.4 | 68.7 |
Limitations
The VC-1 model has been evaluated on a limited set of benchmarks and may not perform as well on other tasks. While we have focused on masked auto-encoders as the pre-training objective and ViT as the architecture in our study, there may be other SSL algorithms that exhibit different scaling behaviors or superior performance on the proposed datasets in our benchmark.
Additionally, the VC-1 model is computationally expensive to train and may not be practical for all use cases. The large size of the model may also pose challenges for deployment on resource-constrained devices.
It is important to note that although we utilize real-world images and videos for pre-training our visual representation models (PVRs), the evaluation benchmarks used in this study serve as proxies for actual robotic tasks. Therefore, the performance of the PVR models on real robots may differ from the rankings established in this study. Further research is necessary to fully evaluate the effectiveness of these models in real-world scenarios.
Finally, while we have made efforts to ensure fairness and avoid bias in our benchmark selection, it is possible that certain demographics or use cases may not be adequately represented in our evaluation tasks. Future work could explore additional benchmarks that address a wider range of scenarios and demographics.