README.md

---
license: apache-2.0
datasets:
- laion/laion-art
language:
- en
library_name: diffusers
pipeline_tag: text-to-image
tags:
- jax-diffusers-event
---

# Color-Canny CantrolNet

These are controlnet checkpoints trained on runwayml/stable-diffusion-v1-5, using fused color and canny edge as conditioning. 

You can find some example images in the following. 

**prompt**: a concept art of by Makoto Shinkai, a girl is standing in the middle of the sea

**negative prompt**: text, bad anatomy, blurry, (low quality, blurry)
![images_1)](./1.png)

**prompt**: a concept art of by Makoto Shinkai, a girl is standing in the middle of the sea

**negative prompt**: text, bad anatomy, blurry, (low quality, blurry)
![images_2)](./2.png)

**prompt**: a concept art of by Makoto Shinkai, a girl is standing in the middle of the grass

**negative prompt**: text, bad anatomy, blurry, (low quality, blurry)
![images_3)](./3.png)


## Limitations and Bias

- No strict control by input color
- Sometimes generate image with confusion When color description in prompt

## Training

**Dataset**
We train this model on [laion-art](https://huggingface.co/datasets/laion/laion-art) dataset with 2.6m images, the processed dataset can be found in [ghoskno/laion-art-en-colorcanny](https://huggingface.co/datasets/ghoskno/laion-art-en-colorcanny).


**Training Details**

- **Hardware**: Google Cloud TPUv4-8 VM

- **Optimizer**: AdamW

- **Train Batch Size**: 4 x 4 = 16

- **Learning rate**: 0.00001 constant

- **Gradient Accumulation Steps**: 4

- **Resolution**: 512

- **Train Steps**: 36000