File size: 4,836 Bytes
3d00365 b704780 3d00365 b704780 3e8d2a6 b704780 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 |
---
license: apache-2.0
language:
- en
---
# MotionCtrl Model Card
<div align="center">
[**Project Page**](https://wzhouxiff.github.io/projects/MotionCtrl/) **|** [**Paper (ArXiv)**](https://arxiv.org/pdf/2312.03641.pdf) **|** [**Code**](https://github.com/TencentARC/MotionCtrl)
[🤗 **Gradio demo (MotionCtrl+VideoCrafter)**](https://huggingface.co/spaces/TencentARC/MotionCtrl) **|** [🤗 **Gradio demo (MotionCtrl+SVD)**](https://huggingface.co/spaces/TencentARC/MotionCtrl_SVD)
</div>
## Introduction
MotionCtrl, when provided with specific camera movements indicated by camera poses or several trajectories,
can independently control both the **camera** and **object** motion in a generated video.
It can be deployed on both Text-to-Video generation models, such as **LVDM/VideoCrafter** and **AnimateDiff**,
as well as Image-to-Video generation models like **SVD**.
## MotionCtrl + SVD
- Model: [motionctrl.pth](https://huggingface.co/TencentARC/MotionCtrl/blob/main/motionctrl.pth)
- Results:
<div align="center">
<img src="https://raw.githubusercontent.com/TencentARC/MotionCtrl/main/assets/svd/00_ibzz5-dxv2h.gif", width="49.7%", height="100%", style="float: left">
<img src="https://raw.githubusercontent.com/TencentARC/MotionCtrl/main/assets/svd/01_5guvn-0x6v2.gif", width="49.7%", height="100%", style="float: right">
</div>
 
## MotionCtrl + LVDM/VideoCrafter
- Model: [motionctrl_svd.ckpt](https://huggingface.co/TencentARC/MotionCtrl/blob/main/motionctrl_svd.ckpt)
- Results:
<div align="center">
<img src="https://raw.githubusercontent.com/TencentARC/MotionCtrl/main/assets/hpxvu-3d8ym.gif", width="100%", height="100%", style="float: left">
<img src="https://raw.githubusercontent.com/TencentARC/MotionCtrl/main/assets/w3nb7-9vz5t.gif", width="100%", height="100%", style="float: right">
<img src="https://raw.githubusercontent.com/TencentARC/MotionCtrl/main/assets/62n2a-wuvsw.gif", width="100%", height="100%", style="float: left">
<img src="https://raw.githubusercontent.com/TencentARC/MotionCtrl/main/assets/ilw96-ak827.gif", width="100%", height="100%", style="float: right">
</div>
 
## MotionCtrl + AnimateDiff
- Model: [Coming Soon]()
- Results:
<div align="center">
<img src="https://raw.githubusercontent.com/wzhouxiff/wzhouxiff.github.io/master/projects/MotionCtrl/assets/videos/ours_animatediff/complex/1424acd0007d40b5.png", width="25%", height="100%", style="float: left">
<img src="https://raw.githubusercontent.com/wzhouxiff/wzhouxiff.github.io/master/projects/MotionCtrl/assets/videos/ours_animatediff/complex/1424acd0007d40b5.gif", width="25%", height="100%", style="float: left">
<img src="https://raw.githubusercontent.com/wzhouxiff/wzhouxiff.github.io/master/projects/MotionCtrl/assets/videos/ours_animatediff/complex/camera_Round-R_ZoomIn.png", width="25%", height="100%", style="float: left">
<img src="https://raw.githubusercontent.com/wzhouxiff/wzhouxiff.github.io/master/projects/MotionCtrl/assets/videos/ours_animatediff/complex/round_in.gif", width="25%", height="100%", style="float: left">
</div>
<div align="center">
<img src="https://raw.githubusercontent.com/wzhouxiff/wzhouxiff.github.io/master/projects/MotionCtrl/assets/videos/ours_animatediff/traj/traj_curve_1.gif", width="25%", height="100%", style="float: left">
<img src="https://raw.githubusercontent.com/wzhouxiff/wzhouxiff.github.io/master/projects/MotionCtrl/assets/videos/ours_animatediff/traj/curve_1_1.gif", width="25%", height="100%", style="float: left">
<img src="https://raw.githubusercontent.com/wzhouxiff/wzhouxiff.github.io/master/projects/MotionCtrl/assets/videos/ours_animatediff/traj/curve_1_2.gif", width="25%", height="100%", style="float: left">
<img src="https://raw.githubusercontent.com/wzhouxiff/wzhouxiff.github.io/master/projects/MotionCtrl/assets/videos/ours_animatediff/traj/curve_1.gif", width="25%", height="100%", style="float: left">
</div>
 
## Usage
- Download directly in this repository OR Download with python script
```python
from huggingface_hub import hf_hub_download
motionctrl_lvdm_ckpt = hf_hub_download(repo_id="TencentARC/MotionCtrl", filename="motionctrl.ckpt", repo_type="model")
motionctrl_svd_ckpt = hf_hub_download(repo_id="TencentARC/MotionCtrl", filename="motionctrl_svd.ckpt", repo_type="model")
```
- Then generate the controllable video following the instructions in our [GitHub repository](https://github.com/TencentARC/MotionCtrl/).
## Citation
```bibtex
@inproceedings{wang2023motionctrl,
title={MotionCtrl: A Unified and Flexible Motion Controller for Video Generation},
author={Wang, Zhouxia and Yuan, Ziyang and Wang, Xintao and Chen, Tianshui and Xia, Menghan and Luo, Ping and Shan, Yin},
booktitle={arXiv preprint arXiv:2312.03641},
year={2023}
}
```
|