Hugging Face's logo

Spaces:
nev
/
CoNR
Runtime error

App Files Community
1
CoNR / README.md
p2oileen's picture
p2oileen
Merge branch 'main' of https://github.com/megvii-research/CoNR into main
42db4d8
preview code
|
raw
history blame
3.9 kB

English | 中文

Collaborative Neural Rendering using Anime Character Sheets

Homepage | Colab English/中文 | arXiv

image

Introduction

This project is the official implement of Collaborative Neural Rendering using Anime Character Sheets, which aims to genarate vivid dancing videos from hand-drawn anime character sheets(ACS). Watch more demos in our HomePage.

Contributors: @transpchan, @P2Oileen, @hzwer

Usage

Prerequisites

  • NVIDIA GPU + CUDA + CUDNN
  • Python 3.6

Installation

  • Clone this repository
git clone https://github.com/megvii-research/CoNR
  • Dependencies

To install all the dependencies, please run the following commands.

cd CoNR
pip install -r requirements.txt
  • Download Weights Download weights from Google Drive. Alternatively, you can download from Baidu Netdisk (password:RDxc).
mkdir weights && cd weights
gdown https://drive.google.com/uc?id=1M1LEpx70tJ72AIV2TQKr6NE_7mJ7tLYx
gdown https://drive.google.com/uc?id=1YvZy3NHkJ6gC3pq_j8agcbEJymHCwJy0
gdown https://drive.google.com/uc?id=1AOWZxBvTo9nUf2_9Y7Xe27ZFQuPrnx9i
gdown https://drive.google.com/uc?id=19jM1-GcqgGoE1bjmQycQw_vqD9C5e-Jm

Prepare Inputs

We provide two Ultra-Dense Pose sequences for two characters. You can generate more UDPs via 3D models and motions refers to our paper. Baidu Netdisk (password:RDxc) 

# for short hair girl
gdown https://drive.google.com/uc?id=11HMSaEkN__QiAZSnCuaM6GI143xo62KO
unzip short_hair.zip
mv short_hair/ poses/

# for double ponytail girl
gdown https://drive.google.com/uc?id=1WNnGVuU0ZLyEn04HzRKzITXqib1wwM4Q
unzip double_ponytail.zip
mv double_ponytail/ poses/

We provide sample inputs of anime character sheets. You can also draw more by yourself. Character sheets need to be cut out from the background and in png format. Baidu Netdisk (password:RDxc) 

# for short hair girl
gdown https://drive.google.com/uc?id=1r-3hUlENSWj81ve2IUPkRKNB81o9WrwT
unzip short_hair_images.zip
mv short_hair_images/ character_sheet/

# for double ponytail girl
gdown https://drive.google.com/uc?id=1XMrJf9Lk_dWgXyTJhbEK2LZIXL9G3MWc
unzip double_ponytail_images.zip
mv double_ponytail_images/ character_sheet/

RUN! 

streamlit run streamlit.py --server.port=8501

then open your browser and visit localhost:8501, follow the instructions to genarate video.

  • via terminal
mkdir {dir_to_save_result}

python -m torch.distributed.launch \
--nproc_per_node=1 train.py --mode=test \
--world_size=1 --dataloaders=2 \
--test_input_poses_images={dir_to_poses} \
--test_input_person_images={dir_to_character_sheet} \
--test_output_dir={dir_to_save_result} \
--test_checkpoint_dir={dir_to_weights}

ffmpeg -r 30 -y -i {dir_to_save_result}/%d.png -r 30 -c:v libx264 output.mp4 -r 30

Citation 

@article{lin2022conr,
  title={Collaborative Neural Rendering using Anime Character Sheets},
  author={Lin, Zuzeng and Huang, Ailin and Huang, Zhewei and Hu, Chen and Zhou, Shuchang},
  journal={arXiv preprint arXiv:2207.05378},
  year={2022}
}