license: apache-2.0
title: MagicTime
sdk: gradio
emoji: 🚀
colorFrom: green
colorTo: gray
short_description: 'MagicTime: Time-lapse Video Generation Models as Metamorphic'
MagicTime: Time-lapse Video Generation Models
If you like our project, please give us a star ⭐ on GitHub for the latest update.
📣 News
- ⏳⏳⏳ Training a stronger model with the support of Open-Sora-Plan (e.g 257 x 512 × 512).
- ⏳⏳⏳ Release the training code of MagicTime.
- [2024.04.10] 🔥 We release the inference code, huggingface space and model weight of MagicTime.
- [2024.04.09] 🔥 We release the arXiv paper for MagicTime, and you can click here to see more details.
- [2024.04.08] 🔥 We released the subset of ChronoMagic dataset used to train MagicTime. The dataset includes 2,265 metamorphic video-text pairs and can be downloaded at Google Drive.
- [2024.04.08] 🔥 All codes & datasets are coming soon! Stay tuned 👀!
😮 Highlights
MagicTime shows excellent performance in metamorphic video generation.
Metamorphic Videos vs. General Videos
Compared to general videos, metamorphic videos contain physical knowledge, long persistence, and strong variation, making them difficult to generate. We show compressed .gif on github, which loses some quality. The general videos are generated by the Animatediff and MagicTime.
| General Videos |  |
 |
 |
 |
| Metamorphic Videos |  |
 |
 |
 |
Gallery
We showcase some metamorphic videos generated by MagicTime, MakeLongVideo, ModelScopeT2V, VideoCrafter, ZeroScope, LaVie, T2V-Zero, Latte and Animatediff below.
| MakeLongVideo |  |
 |
 |
 |
| ModelScopeT2V |  |
 |
 |
 |
| VideoCrafter |  |
 |
 |
 |
| ZeroScope |  |
 |
 |
 |
| LaVie |  |
 |
 |
 |
| T2V-Zero |  |
 |
 |
 |
| Latte |  |
 |
 |
 |
| Animatediff |  |
 |
 |
 |
| Ours |  |
 |
 |
 |
We show more metamorphic videos generated by MagicTime with the help of Realistic, ToonYou and RcnzCartoon.
 |
 |
 |
 |
 |
 |
 |
 |
 |
Prompts are trimmed for display, see here for full prompts.
Integrate into DiT-based Architecture
The mission of this project is to help reproduce Sora and provide high-quality video-text data and data annotation pipelines, to support Open-Sora-Plan or other DiT-based T2V models. To this end, we take an initial step to integrate our MagicTime scheme into the DiT-based Framework. Specifically, our method supports the Open-Sora-Plan v1.0.0 for fine-tuning. We first scale up with additional metamorphic landscape time-lapse videos in the same annotation framework to get the ChronoMagic-Landscape dataset. Then, we fine-tune the Open-Sora-Plan v1.0.0 with the ChronoMagic-Landscape dataset to get the MagicTime-DiT model. The results are as follows (257×512×512 (10s)):
 |
 |
 |
 |
 |
 |
 |
 |
Prompts are trimmed for display, see here for full prompts.
🤗 Demo
Gradio Web UI
Highly recommend trying out our web demo by the following command, which incorporates all features currently supported by MagicTime. We also provide online demo in Huggingface Spaces.
python app.py
⚙️ Requirements and Installation
We recommend the requirements as follows.
git clone https://github.com/PKU-YuanGroup/MagicTime.git
cd MagicTime
conda env create -f environment.yml
conda activate magictime
🗝️ Training & Inference
The training code is coming soon! For inference, some example are shown below:
# For [Realistic](https://civitai.com/models/4201/realistic-vision-v20)
python inference_magictime.py --config sample_configs/RealisticVision.yaml
# For [ToonYou](https://civitai.com/models/30240/toonyou)
python inference_magictime.py --config sample_configs/ToonYou.yaml
# For [RcnzCartoon](https://civitai.com/models/66347/rcnz-cartoon-3d)
python inference_magictime.py --config sample_configs/RcnzCartoon.yaml
# or you can directly run the .sh
sh inference.sh
🐳 ChronoMagic Dataset
ChronoMagic with 2265 metamorphic time-lapse videos, each accompanied by a detailed caption. We released the subset of ChronoMagic used to train MagicTime. The dataset can be downloaded at Google Drive. Some samples can be found on our Project Page.
👍 Acknowledgement
Animatediff The codebase we built upon and it is a strong U-Net-based text-to-video generation model.
Open-Sora-Plan The codebase we built upon and it is a simple and scalable DiT-based text-to-video generation repo, to reproduce Sora.
🔒 License
- The majority of this project is released under the Apache 2.0 license as found in the LICENSE file.
- The service is a research preview intended for non-commercial use only. Please contact us if you find any potential violations.
✏️ Citation
If you find our paper and code useful in your research, please consider giving a star :star: and citation :pencil:.
@misc{yuan2024magictime,
title={MagicTime: Time-lapse Video Generation Models as Metamorphic Simulators},
author={Shenghai Yuan and Jinfa Huang and Yujun Shi and Yongqi Xu and Ruijie Zhu and Bin Lin and Xinhua Cheng and Li Yuan and Jiebo Luo},
year={2024},
eprint={2404.05014},
archivePrefix={arXiv},
primaryClass={cs.CV}
}