license: apache-2.0
datasets:
- laion/laion-art
language:
- en
library_name: diffusers
pipeline_tag: text-to-image
tags:
- jax-diffusers-event
Color-Canny CantrolNet
These are controlnet checkpoints trained on runwayml/stable-diffusion-v1-5, using fused color and canny edge as conditioning.
You can find some example images in the following.
prompt: a concept art of by Makoto Shinkai, a girl is standing in the middle of the sea
negative prompt: text, bad anatomy, blurry, (low quality, blurry)
prompt: a concept art of by Makoto Shinkai, a girl is standing in the middle of the sea
negative prompt: text, bad anatomy, blurry, (low quality, blurry)
prompt: a concept art of by Makoto Shinkai, a girl is standing in the middle of the grass
negative prompt: text, bad anatomy, blurry, (low quality, blurry)
Limitations and Bias
- No strict control by input color
- Sometimes generate image with confusion When color description in prompt
Training
Dataset We train this model on laion-art dataset with 2.6m images, the processed dataset can be found in ghoskno/laion-art-en-colorcanny.
Training Details
Hardware: Google Cloud TPUv4-8 VM
Optimizer: AdamW
Train Batch Size: 4 x 4 = 16
Learning rate: 0.00001 constant
Gradient Accumulation Steps: 4
Resolution: 512
Train Steps: 36000