Init

LICENSE

@@ -0,0 +1,10 @@
+Sample Code License
+Version: 1.1
+ 
+IMPORTANT:  This software is supplied to you by École Polytechnique Fédérale de Lausanne (“EPFL”) and Apple Inc. ("Apple") in consideration of your agreement to the following terms, and your use, installation, modification or redistribution of this software constitutes acceptance of these terms.  If you do not agree with these terms, please do not use, install, modify or redistribute this software.
+ 
+In consideration of your agreement to abide by the following terms, and subject to these terms, EPFL and Apple (collectively, “Licensor”) grant you a personal, non-exclusive license, under Licensor’s copyrights in this original software (the " Software"), to use, reproduce, modify and redistribute the Software, with or without modifications, in source and/or binary forms for non-commercial use; provided that if you redistribute the Software in its entirety and without modifications, you must retain this notice and the following text and disclaimers in all such redistributions of the Software. Neither the name, trademarks, service marks or logos of Licensor may be used to endorse or promote products derived from the Software without specific prior written permission from Licensor.  Except as expressly stated in this notice, no other rights or licenses, express or implied, are granted by Licensor herein, including but not limited to any patent rights that may be infringed by your derivative works or by other works in which the Software may be incorporated.
+ 
+The Software is provided by Licensor on an "AS IS" basis.  LICENSOR MAKES NO WARRANTIES, EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, REGARDING THE SOFTWARE OR ITS USE AND OPERATION ALONE OR IN COMBINATION WITH YOUR PRODUCTS. IN NO EVENT SHALL LICENSOR BE LIABLE FOR ANY SPECIAL, INDIRECT, INCIDENTAL OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) ARISING IN ANY WAY OUT OF THE USE, REPRODUCTION, MODIFICATION AND/OR DISTRIBUTION OF THE SOFTWARE, HOWEVER CAUSED AND WHETHER UNDER THEORY OF CONTRACT, TORT (INCLUDING NEGLIGENCE), STRICT LIABILITY OR OTHERWISE, EVEN IF LICENSOR HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ 
+Copyright (C) 2024. All Rights Reserved.

README.md

@@ -0,0 +1,64 @@
+---
+license: other
+license_name: sample-code-license
+license_link: LICENSE
+library_name: ml-4m
+---
+
+# 4M: Massively Multimodal Masked Modeling
+
+*A framework for training any-to-any multimodal foundation models. <br>Scalable. Open-sourced. Across tens of modalities and tasks.*
+
+[`Website`](https://4m.epfl.ch) | [`GitHub`](https://github.com/apple/ml-4m) | [`BibTeX`](#citation)  
+
+Official implementation and pre-trained models for :
+
+[**4M: Massively Multimodal Masked Modeling**](https://arxiv.org/abs/2312.06647), NeurIPS 2023 (Spotlight) <br>
+*[David Mizrahi](https://dmizrahi.com/)\*, [Roman Bachmann](https://roman-bachmann.github.io/)\*, [Oğuzhan Fatih Kar](https://ofkar.github.io/), [Teresa Yeo](https://aserety.github.io/), [Mingfei Gao](https://fly6464.github.io/), [Afshin Dehghan](https://www.afshindehghan.com/), [Amir Zamir](https://vilab.epfl.ch/zamir/)*
+
+**4M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalities**, arXiv 2024 <br>
+*[Roman Bachmann](https://roman-bachmann.github.io/)\*, [Oğuzhan Fatih Kar](https://ofkar.github.io/)\*, [David Mizrahi](https://dmizrahi.com/)\*, [Ali Garjani](https://garjania.github.io/), [Mingfei Gao](https://fly6464.github.io/), [David Griffiths](https://www.dgriffiths.uk/), [Jiaming Hu](https://scholar.google.com/citations?user=vm3imKsAAAAJ&hl=en), [Afshin Dehghan](https://www.afshindehghan.com/), [Amir Zamir](https://vilab.epfl.ch/zamir/)*
+
+4M is a framework for training "any-to-any" foundation models, using tokenization and masking to scale to many diverse modalities. 
+Models trained using 4M can perform a wide range of vision tasks, transfer well to unseen tasks and modalities, and are flexible and steerable multimodal generative models. 
+We are releasing code and models for "4M: Massively Multimodal Masked Modeling" (here denoted 4M-7), as well as "4M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalities" (here denoted 4M-21).
+
+
+## Installation
+For install instructions, please see https://github.com/apple/ml-4m. 
+
+
+## Usage
+
+This model can be loaded from Hugging Face Hub as follows:
+```python
+from fourm.models.fm import FM
+fm = FM.from_pretrained('EPFL-VILAB/4M-21_L')
+```
+
+Please see [README_GENERATION.md](https://github.com/apple/ml-4m/blob/main/README_GENERATION.md) for more detailed instructions and https://github.com/apple/ml-4m for other 4M model and tokenizer checkpoints.
+
+Safetensors checkpoints are hosted under https://huggingface.co/EPFL-VILAB/4M.
+
+## Citation
+
+If you find this repository helpful, please consider citing our work:
+```
+@inproceedings{4m,
+    title={{4M}: Massively Multimodal Masked Modeling},
+    author={David Mizrahi and Roman Bachmann and O{\u{g}}uzhan Fatih Kar and Teresa Yeo and Mingfei Gao and Afshin Dehghan and Amir Zamir},
+    booktitle={Thirty-seventh Conference on Neural Information Processing Systems},
+    year={2023},
+}
+
+@article{4m21,
+    title={{4M-21}: An Any-to-Any Vision Model for Tens of Tasks and Modalities},
+    author={Roman Bachmann and O{\u{g}}uzhan Fatih Kar and David Mizrahi and Ali Garjani and Mingfei Gao and David Griffiths and Jiaming Hu and Afshin Dehghan and Amir Zamir},
+    journal={arXiv 2024},
+    year={2024},
+}
+```
+
+## License
+
+The model weights in this repository are released under the Sample Code license as found in the [LICENSE](LICENSE) file.

config.json

@@ -0,0 +1,57 @@
+{
+  "act_layer": "SiLU",
+  "decoder_depth": 24,
+  "dim": 1024,
+  "domains_in": [
+    "caption",
+    "t5_caption",
+    "det",
+    "metadata",
+    "human_poses",
+    "color_palette",
+    "sam_instance",
+    "rgb@224",
+    "tok_rgb@224",
+    "tok_normal@224",
+    "tok_depth@224",
+    "tok_semseg@224",
+    "tok_clip@224",
+    "tok_dinov2@224",
+    "tok_dinov2_global",
+    "tok_imagebind@224",
+    "tok_imagebind_global",
+    "tok_sam_edge@224",
+    "tok_canny_edge@224"
+  ],
+  "domains_out": [
+    "caption",
+    "t5_caption",
+    "det",
+    "metadata",
+    "human_poses",
+    "color_palette",
+    "sam_instance",
+    "tok_rgb@224",
+    "tok_normal@224",
+    "tok_depth@224",
+    "tok_semseg@224",
+    "tok_clip@224",
+    "tok_dinov2@224",
+    "tok_dinov2_global",
+    "tok_imagebind@224",
+    "tok_imagebind_global",
+    "tok_sam_edge@224",
+    "tok_canny_edge@224"
+  ],
+  "encoder_depth": 24,
+  "gated_mlp": true,
+  "image_size": 224,
+  "mlp_bias": false,
+  "mlp_ratio": 4,
+  "norm_bias": false,
+  "num_heads": 16,
+  "patch_size": 16,
+  "proj_bias": false,
+  "qkv_bias": false,
+  "share_modality_embeddings": false
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f6a0d9f552522b4a96cb2037c1ceeb59c8bb657d39dd8bb1bf5b607e74cef4c7
+size 6256957760