benjamin-paine/champ

This repository contains a pruned and partially reorganized version of CHAMP.

@misc{zhu2024champ,
      title={Champ: Controllable and Consistent Human Image Animation with 3D Parametric Guidance}, 
      author={Shenhao Zhu and Junming Leo Chen and Zuozhuo Dai and Yinghui Xu and Xun Cao and Yao Yao and Hao Zhu and Siyu Zhu},
      year={2024},
      eprint={2403.14781},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Video credit: Polina Tankilevitch, Pexels

Image credit: Andrea Piacquadio, Pexels

Usage

First, install the CHAMP package into your python environment. If you're creating a new environment for CHAMP, be sure you also specify the version of torch you want with CUDA support, or else this will try to run only on CPU.

pip install git+https://github.com/painebenjamin/champ.git

Now, you can create the pipeline, automatically pulling the weights from this repository, either as individual models:

from champ import CHAMPPipeline
pipeline = CHAMPPipeline.from_pretrained(
  "benjamin-paine/champ",
  torch_dtype=torch.float16,
  variant="fp16",
  device="cuda"
).to("cuda", dtype=torch.float16)

Or, as a single file:

from champ import CHAMPPipeline
pipeline = CHAMPPipeline.from_single_file(
  "benjamin-paine/champ",
  torch_dtype=torch.float16,
  variant="fp16",
  device="cuda"
).to("cuda", dtype=torch.float16)

Follow this format for execution:

result = pipeline(
  reference: PIL.Image.Image,
  guidance: Dict[str, List[PIL.Image.Image]],
  width: int,
  height: int,
  video_length: int,
  num_inference_steps: int,
  guidance_scale: float
).videos
# Result is a list of PIL Images

Starting values for num_inference_steps and guidance_scale are 20 and 3.5, respectively.

Guidance keys include depth, normal, dwpose and semantic_map (densepose.) This guide does not provide details on how to obtain those samples, but examples are available in the git repository.