|
--- |
|
license: apache-2.0 |
|
pipeline_tag: image-to-video |
|
--- |
|
|
|
## Cinemo: Consistent and Controllable Image Animation with Motion Diffusion Models |
|
|
|
This repo contains pre-trained weights for our paper exploring image animation with motion diffusion models (Cinemo). You can find more visualizations on our [project page](https://maxin-cn.github.io/cinemo_project/). |
|
|
|
In this project, we propose a novel method called Cinemo, which can perform motion-controllable image animation with strong consistency and smoothness. To improve motion smoothness, Cinemo learns the distribution of motion residuals, rather than directly generating subsequent frames. Additionally, a structural similarity index-based method is proposed to control the motion intensity. Furthermore, we propose a noise refinement technique based on discrete cosine transformation to ensure temporal consistency. These three methods help Cinemo generate highly consistent, smooth, and motion-controlled image animation results. Compared to previous methods, Cinemo offers simpler and more precise user control and better generative performance. |
|
|
|
|
|
## News |
|
- (🔥 New) Jun. 2, 2024. 💥 The inference code is released. The checkpoint can be found [here](https://huggingface.co/maxin-cn/Cinemo/tree/main). |
|
|
|
|
|
## Setup |
|
|
|
First, download and set up the repo: |
|
|
|
```bash |
|
git clone https://github.com/maxin-cn/Cinemo |
|
cd Cinemo |
|
``` |
|
|
|
We provide an [`environment.yml`](environment.yml) file that can be used to create a Conda environment. If you only want |
|
to run pre-trained models locally on CPU, you can remove the `cudatoolkit` and `pytorch-cuda` requirements from the file. |
|
|
|
```bash |
|
conda env create -f environment.yml |
|
conda activate cinemo |
|
``` |
|
|
|
|
|
## Animation |
|
|
|
You can sample from our **pre-trained Cinemo models** with [`animation.py`](pipelines/animation.py). Weights for our pre-trained Cinemo model can be found [here](https://huggingface.co/maxin-cn/Cinemo/tree/main). The script has various arguments to adjust sampling steps, change the classifier-free guidance scale, etc: |
|
|
|
```bash |
|
bash pipelines/animation.sh |
|
``` |
|
|
|
## Other Applications |
|
|
|
You can also utilize Cinemo for other applications, such as motion transfer and video editing: |
|
|
|
```bash |
|
bash pipelines/video_editing.sh |
|
``` |
|
|
|
## Acknowledgments |
|
Cinemo has been greatly inspired by the following amazing works and teams: [LaVie](https://github.com/Vchitect/LaVie) and [SEINE](https://github.com/Vchitect/SEINE), we thank all the contributors for open-sourcing. |
|
|
|
## Bibtex citation |
|
|
|
```bibtex |
|
@misc{ma2024cinemoconsistentcontrollableimage, |
|
title={Cinemo: Consistent and Controllable Image Animation with Motion Diffusion Models}, |
|
author={Xin Ma and Yaohui Wang and Gengyun Jia and Xinyuan Chen and Yuan-Fang Li and Cunjian Chen and Yu Qiao}, |
|
year={2024}, |
|
eprint={2407.15642}, |
|
archivePrefix={arXiv}, |
|
primaryClass={cs.CV}, |
|
url={https://arxiv.org/abs/2407.15642}, |
|
} |
|
``` |
