|
--- |
|
license: apache-2.0 |
|
language: |
|
- en |
|
--- |
|
|
|
# MotionCtrl Model Card |
|
|
|
<div align="center"> |
|
|
|
[**Project Page**](https://wzhouxiff.github.io/projects/MotionCtrl/) **|** [**Paper (ArXiv)**](https://arxiv.org/pdf/2312.03641.pdf) **|** [**Code**](https://github.com/TencentARC/MotionCtrl) |
|
|
|
[🤗 **Gradio demo (MotionCtrl+VideoCrafter)**](https://huggingface.co/spaces/TencentARC/MotionCtrl) **|** [🤗 **Gradio demo (MotionCtrl+SVD)**](https://huggingface.co/spaces/TencentARC/MotionCtrl_SVD) |
|
|
|
</div> |
|
|
|
## Introduction |
|
|
|
MotionCtrl, when provided with specific camera movements indicated by camera poses or several trajectories, |
|
can independently control both the **camera** and **object** motion in a generated video. |
|
It can be deployed on both Text-to-Video generation models, such as **LVDM/VideoCrafter** and **AnimateDiff**, |
|
as well as Image-to-Video generation models like **SVD**. |
|
|
|
## MotionCtrl + SVD |
|
|
|
- Model: [motionctrl.pth](https://huggingface.co/TencentARC/MotionCtrl/blob/main/motionctrl.pth) |
|
- Results: |
|
|
|
<div align="center"> |
|
<img src="https://raw.githubusercontent.com/TencentARC/MotionCtrl/main/assets/svd/00_ibzz5-dxv2h.gif", width="49.7%", height="100%", style="float: left"> |
|
<img src="https://raw.githubusercontent.com/TencentARC/MotionCtrl/main/assets/svd/01_5guvn-0x6v2.gif", width="49.7%", height="100%", style="float: right"> |
|
</div> |
|
|
|
  |
|
|
|
## MotionCtrl + LVDM/VideoCrafter |
|
|
|
- Model: [motionctrl_svd.ckpt](https://huggingface.co/TencentARC/MotionCtrl/blob/main/motionctrl_svd.ckpt) |
|
- Results: |
|
|
|
<div align="center"> |
|
<img src="https://raw.githubusercontent.com/TencentARC/MotionCtrl/main/assets/hpxvu-3d8ym.gif", width="49.7%", height="100%", style="float: left"> |
|
<img src="https://raw.githubusercontent.com/TencentARC/MotionCtrl/main/assets/w3nb7-9vz5t.gif", width="49.7%", height="100%", style="float: right"> |
|
<img src="https://raw.githubusercontent.com/TencentARC/MotionCtrl/main/assets/62n2a-wuvsw.gif", width="49.7%", height="100%", style="float: left"> |
|
<img src="https://raw.githubusercontent.com/TencentARC/MotionCtrl/main/assets/ilw96-ak827.gif", width="49.7%", height="100%", style="float: right"> |
|
</div> |
|
|
|
  |
|
|
|
## MotionCtrl + AnimateDiff |
|
|
|
- Model: [Coming Soon]() |
|
- Results: |
|
|
|
<div align="center"> |
|
<img src="https://raw.githubusercontent.com/wzhouxiff/wzhouxiff.github.io/master/projects/MotionCtrl/assets/videos/ours_animatediff/complex/1424acd0007d40b5.png", width="25%", height="100%", style="float: left"> |
|
<img src="https://raw.githubusercontent.com/wzhouxiff/wzhouxiff.github.io/master/projects/MotionCtrl/assets/videos/ours_animatediff/complex/1424acd0007d40b5.gif", width="25%", height="100%", style="float: left"> |
|
<img src="https://raw.githubusercontent.com/wzhouxiff/wzhouxiff.github.io/master/projects/MotionCtrl/assets/videos/ours_animatediff/complex/camera_Round-R_ZoomIn.png", width="25%", height="100%", style="float: left"> |
|
<img src="https://raw.githubusercontent.com/wzhouxiff/wzhouxiff.github.io/master/projects/MotionCtrl/assets/videos/ours_animatediff/complex/round_in.gif", width="25%", height="100%", style="float: left"> |
|
</div> |
|
|
|
<div align="center"> |
|
<img src="https://raw.githubusercontent.com/wzhouxiff/wzhouxiff.github.io/master/projects/MotionCtrl/assets/videos/ours_animatediff/traj/traj_curve_1.gif", width="25%", height="100%", style="float: left"> |
|
<img src="https://raw.githubusercontent.com/wzhouxiff/wzhouxiff.github.io/master/projects/MotionCtrl/assets/videos/ours_animatediff/traj/curve_1_1.gif", width="25%", height="100%", style="float: left"> |
|
<img src="https://raw.githubusercontent.com/wzhouxiff/wzhouxiff.github.io/master/projects/MotionCtrl/assets/videos/ours_animatediff/traj/curve_1_2.gif", width="25%", height="100%", style="float: left"> |
|
<img src="https://raw.githubusercontent.com/wzhouxiff/wzhouxiff.github.io/master/projects/MotionCtrl/assets/videos/ours_animatediff/traj/curve_1.gif", width="25%", height="100%", style="float: left"> |
|
</div> |
|
|
|
  |
|
|
|
## Usage |
|
|
|
- Download directly in this repository OR Download with python script |
|
|
|
```python |
|
from huggingface_hub import hf_hub_download |
|
motionctrl_lvdm_ckpt = hf_hub_download(repo_id="TencentARC/MotionCtrl", filename="motionctrl.ckpt", repo_type="model") |
|
motionctrl_svd_ckpt = hf_hub_download(repo_id="TencentARC/MotionCtrl", filename="motionctrl_svd.ckpt", repo_type="model") |
|
``` |
|
|
|
- Then generate the controllable video following the instructions in our [GitHub repository](https://github.com/TencentARC/MotionCtrl/). |
|
|
|
## Citation |
|
|
|
```bibtex |
|
@inproceedings{wang2023motionctrl, |
|
title={MotionCtrl: A Unified and Flexible Motion Controller for Video Generation}, |
|
author={Wang, Zhouxia and Yuan, Ziyang and Wang, Xintao and Chen, Tianshui and Xia, Menghan and Luo, Ping and Shan, Yin}, |
|
booktitle={arXiv preprint arXiv:2312.03641}, |
|
year={2023} |
|
} |
|
``` |
|
|