Cinemo: Consistent and Controllable Image Animation with Motion Diffusion Models
_{Official PyTorch Implementation}

Cinemo: Consistent and Controllable Image Animation with Motion Diffusion Models
Xin Ma, Yaohui Wang*†, Gengyun Jia, Xinyuan Chen, Yuan-Fang Li, Cunjian Chen*, Yu Qiao
(*Corresponding authors, †Project Lead)

This repo contains pre-trained weights, and sampling code of Cinemo. Please visit our project page for more results.

News

• (🔥 New) Jul. 29, 2024. 💥 HuggingFace space is added, you can also launch gradio interface locally.

• (🔥 New) Jul. 23, 2024. 💥 Our paper is released on arxiv.

• (🔥 New) Jun. 2, 2024. 💥 The inference code is released. The checkpoint can be found here.

Setup

Download and set up the repo:

git clone https://github.com/maxin-cn/Cinemo
cd Cinemo
conda env create -f environment.yml
conda activate cinemo

Animation

You can sample from our pre-trained Cinemo models with animation.py. Weights for our pre-trained Cinemo model can be found here. The script has various arguments for adjusting sampling steps, changing the classifier-free guidance scale, etc:

bash pipelines/animation.sh

Related model weights will be downloaded automatically and following results can be obtained,

Input image	Output video	Input image	Output video
"People Walking"		"Sea Swell"
"Girl Dancing under the Stars"		"Dragon Glowing Eyes"
"Bubbles Floating upwards"		"Snowman Waving his Hand"

Gradio interface

We also provide a local gradio interface, just run:

python app.py

You can specify the --share and --server_name arguments to meet your requirement!

Other Applications

You can also utilize Cinemo for other applications, such as motion transfer and video editing:

bash pipelines/video_editing.sh

Related checkpoints will be downloaded automatically and following results will be obtained,

Input video	First frame	Edited first frame	Output video

or motion transfer,

Input video	First frame	Edited first frame	Output video

Contact Us

Xin Ma: [email protected], Yaohui Wang: [email protected]

Citation

If you find this work useful for your research, please consider citing it.

@article{ma2024cinemo,
  title={Cinemo: Latent Diffusion Transformer for Video Generation},
  author={Ma, Xin and Wang, Yaohui and Jia, Gengyun and Chen, Xinyuan and Li, Yuan-Fang and Chen, Cunjian and Qiao, Yu},
  journal={arXiv preprint arXiv:2407.15642},
  year={2024}
}

Acknowledgments

Cinemo has been greatly inspired by the following amazing works and teams: LaVie and SEINE, we thank all the contributors for open-sourcing.

License

The code and model weights are licensed under LICENSE.