MotionCtrl: A Unified and Flexible
Motion Controller
for Video Generation
[](https://wzhouxiff.github.io/projects/MotionCtrl/assets/paper/MotionCtrl.pdf) [](https://arxiv.org/pdf/2312.03641.pdf) [
](https://wzhouxiff.github.io/projects/MotionCtrl/) []()
[Zhouxia Wang](https://vvictoryuki.github.io/website/)1,2, [Ziyang Yuan](https://github.com/jiangyzy)1,4, [Xintao Wang](https://xinntao.github.io/)1,3, [Tianshui Chen](http://tianshuichen.com/)6, [Menghan Xia](https://menghanxia.github.io/)3, [Ping Luo](http://luoping.me/)2,5, [Ying Shan](https://scholar.google.com/citations?hl=zh-CN&user=4oXBp9UAAAAJ)1,3
1 ARC Lab, Tencent PCG, 2 The University of Hong Kong, 3 Tencent AI Lab, 4 Tsinghua University, 5 Shanghai AI Laboratory, 6 Guangdong University of Technology
Our proposed MotionCtrl is capable of independently controlling the complex camera motion and object motion of the generated videos, with only a unified model.
There are some results attained with MotionCtrl and more results are showcased in our [Project Page](https://wzhouxiff.github.io/projects/MotionCtrl/).
## Updating
- [x] Release MotionCtrl depolyed on *LVDM/VideoCrafter*
- [ ] Gradio Demo Available
## Inference
1. Download the weights of MotionCtrl from [weipan](https://drive.weixin.qq.com/s?k=AJEAIQdfAAogLtIAPh) to `./checkpoints`.
2. Go into `configs/inference/run.sh` and set `condtype` as 'camera_motion', 'object_motion', or 'both'.
- `condtype=camera_motion` means only control the **camera motion** in the generated video.
- `condtype=object_motion` means only control the **object motion** in the generated video.
- `condtype=both` means control the camera motion and object motion in the generated video **simultaneously**.
3. sh configs/inference/run.sh
## Citation
If you make use of our work, please cite our paper.
```bibtex
@inproceedings{wang2023motionctrl,
title={MotionCtrl: A Unified and Flexible Motion Controller for Video Generation},
author={Wang, Zhouxia and Yuan, Ziyang and Wang, Xintao and Chen, Tianshui and Xia, Menghan and Luo, Ping and Shan, Yin},
booktitle={arXiv preprint arXiv:2312.03641},
year={2023}
}
```
