README.md

MotionCtrl: A Unified and Flexible Motion Controller for Video Generation

     

Zhouxia Wang^1,2, Ziyang Yuan^1,4, Xintao Wang^1,3, Tianshui Chen⁶, Menghan Xia³, Ping Luo^2,5, Ying Shan^1,3

¹ ARC Lab, Tencent PCG, ² The University of Hong Kong, ³ Tencent AI Lab, ⁴ Tsinghua University, ⁵ Shanghai AI Laboratory, ⁶ Guangdong University of Technology

Our proposed MotionCtrl is capable of independently controlling the complex camera motion and object motion of the generated videos, with only a unified model. There are some results attained with MotionCtrl and more results are showcased in our Project Page.

Updating

• Release MotionCtrl depolyed on LVDM/VideoCrafter
• Gradio Demo Available

Inference

1. Download the weights of MotionCtrl from weipan to ./checkpoints.
2. Go into configs/inference/run.sh and set condtype as 'camera_motion', 'object_motion', or 'both'.

• condtype=camera_motion means only control the camera motion in the generated video.
• condtype=object_motion means only control the object motion in the generated video.
• condtype=both means control the camera motion and object motion in the generated video simultaneously.

3. sh configs/inference/run.sh

Citation

If you make use of our work, please cite our paper.

@inproceedings{wang2023motionctrl,
  title={MotionCtrl: A Unified and Flexible Motion Controller for Video Generation},
  author={Wang, Zhouxia and Yuan, Ziyang and Wang, Xintao and Chen, Tianshui and Xia, Menghan and Luo, Ping and Shan, Yin},
  booktitle={arXiv preprint arXiv:2312.03641},
  year={2023}
}