Restore missing RGB channels

Restore a missing channel of a RGB image by using ControlNet to guide image generation of Stable Diffusion to infer missing channel from the other two channels.

See accompanying discussion at github.com - Channels RGB with detailed report and evaluations.
To restore images with missing channels you can use this space.
For evaluation images see the corresponding .zip's at "files".
To run your own evaluations you can use this script at gitlab.com.

Training

accelerate launch train_controlnet.py \
  --pretrained_model_name_or_path="runwayml/stable-diffusion-v1-5" \
  --train_batch_size=4 \
  --gradient_accumulation_steps=8 \
  --proportion_empty_prompts=0.5
  --mixed_precision="fp16" \
  --learning_rate=1e-5 \
  --enable_xformers_memory_efficient_attention \
  --use_8bit_adam \
  --set_grads_to_none \
  --seed=0 \
  --num_train_epochs=2

Image dataset

laion2B-en aesthetics>=6.5 dataset
--min_image_size 512 --max_aspect_ratio 2 --resize_mode="center_crop" --image_size 512
Cleaned with fastdup default settings
Data augmented with right-left flipped images
Resulting in 214244 images
Set whole channel to 0 by alternating between R-G-B channels