Hugging Face's logo

svjack
/
ToonCrafter_with_SketchGuidance_fp16

Model card Files Community
ToonCrafter_with_SketchGuidance_fp16
File size: 2,793 Bytes 
3b3fc8d
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2eb03e6
 
3b3fc8d
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
 
## ___***ToonCrafter_with_SketchGuidance***___
This repository is an implementation that recreates the SketchGuidance feature of "ToonCrafter".

- https://github.com/ToonCrafter/ToonCrafter
- https://arxiv.org/pdf/2405.17933

https://github.com/user-attachments/assets/f72f287d-f848-4982-8f91-43c49d037007

# ToonCrafter with Sketch Guidance

## Overview
This repository contains the ToonCrafter model with additional sketch guidance functionality. The model is hosted on GitHub and can be accessed via the following link:

- **Repository:** [https://github.com/svjack/ToonCrafter_with_SketchGuidance](https://github.com/svjack/ToonCrafter_with_SketchGuidance)

## Installation

### Clone the Repository
To clone the repository and set up the environment, follow these steps:

```bash
git clone https://github.com/svjack/ToonCrafter_with_SketchGuidance && cd ToonCrafter_with_SketchGuidance
pip install -r requirements.txt
pip install --upgrade einops
```

### List Control Models
After cloning the repository, you can list the control models available:

```bash
ls control_models
```

### Download Sketch Encoder
Download the `sketch_encoder.ckpt` file from the Hugging Face repository:

```bash
wget https://huggingface.co/Doubiiu/ToonCrafter/resolve/main/sketch_encoder.ckpt
```

### Copy Sketch Encoder to Control Models
Move the downloaded `sketch_encoder.ckpt` file to the `control_models` directory:

```bash
cp sketch_encoder.ckpt control_models
```

## Usage

### Run the Gradio App
Once the repository is cloned and the necessary files are in place, you can run the Gradio app using the following command:

```bash
python gradio_app.py
```

This will start the Gradio interface, allowing you to interact with the ToonCrafter model with sketch guidance.

## 🧰 Models

|Model|Resolution|GPU Mem. & Inference Time (A100, ddim 50steps)|Checkpoint|
|:---------|:---------|:--------|:--------|
|ToonCrafter_512|320x512| TBD (`perframe_ae=True`)|[Hugging Face](https://huggingface.co/Doubiiu/ToonCrafter/blob/main/model.ckpt)|
|SketchEncoder|TBD| TBD |[Hugging Face](https://huggingface.co/Doubiiu/ToonCrafter/blob/main/sketch_encoder.ckpt)|


Currently, ToonCrafter can support generating videos of up to 16 frames with a resolution of 512x320. The inference time can be reduced by using fewer DDIM steps.



## ⚙️ Setup

### Install Environment via Anaconda (Recommended)
```bash
conda create -n tooncrafter python=3.8.5
conda activate tooncrafter
pip install -r requirements.txt
```


## 💫 Inference

### 1. Local Gradio demo
1. Download pretrained ToonCrafter_512 and put the model.ckpt in checkpoints/tooncrafter_512_interp_v1/model.ckpt.
2. Download pretrained SketchEncoder and put the model.ckpt in control_models/sketch_encoder.ckpt.

```bash
  python gradio_app.py 
```