File size: 4,468 Bytes
960617b 0c9b27d 960617b 0c9b27d 960617b 0c9b27d 960617b 0c9b27d 960617b 0c9b27d 960617b 0c9b27d 960617b 4460f9b e951fec 4460f9b e951fec 960617b cd2e780 562e3f6 960617b 22feceb 960617b 0c9b27d 960617b 36ecf12 0c9b27d |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 |
---
base_model: stabilityai/stable-diffusion-xl-base-1.0
library_name: diffusers
tags:
- stable-diffusion-xl
- stable-diffusion-xl-diffusers
- text-to-image
- diffusers
- diffusers-training
- lora
inference: true
datasets:
- AdamLucek/oldbookillustrations-small
language:
- en
---
# LoRA Weights on Old Book Illustrations for Stable Diffusion XL Base 1.0
These are LoRA adaption weights for [stabilityai/stable-diffusion-xl-base-1.0](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0). The weights were fine-tuned on the [AdamLucek/oldbookillustrations-small dataset](https://huggingface.co/datasets/AdamLucek/oldbookillustrations-small).
LoRA for the text encoder was enabled: **True**.
Special VAE used for training: [madebyollin/sdxl-vae-fp16-fix](https://huggingface.co/madebyollin/sdxl-vae-fp16-fix).
## Example Images
<!-- *"An Apple"*
<img src="https://cdn-uploads.huggingface.co/production/uploads/65ba68a15d2ef0a4b2c892b4/w82UDqXeWNK0WRPpCk0Wk.png" width=300>
*"A Flower Wreath"*
<img src="https://cdn-uploads.huggingface.co/production/uploads/65ba68a15d2ef0a4b2c892b4/PT9tb-2WDz9eb7Rv8vMsu.png" width=300>
*"A view down an alley in New York"*
<img src="https://cdn-uploads.huggingface.co/production/uploads/65ba68a15d2ef0a4b2c892b4/Sr9cHjKpXFRPFH4ODW6Sz.png" width=300>
*"An office setting with a desk and papers on it, with a view out the window above the desk into the town"*
<img src="https://cdn-uploads.huggingface.co/production/uploads/65ba68a15d2ef0a4b2c892b4/A8SETPQoBMs2oay7X6iwp.png" width=300> -->
<img src="https://cdn-uploads.huggingface.co/production/uploads/65ba68a15d2ef0a4b2c892b4/DIxEBANVrf1JSGdm_ldx-.jpeg" width=500>
*"An Apple"*
*"A Flower Wreath"*
*"A view down an alley in New York"*
*"An office setting with a desk and papers on it, with a view out the window above the desk into the town"*
## Intended uses & limitations
#### How to use
<div style="display: flex; align-items: center;">
<img src="https://upload.wikimedia.org/wikipedia/commons/d/d0/Google_Colaboratory_SVG_Logo.svg" width="100">
<a href="https://colab.research.google.com/drive/1SMLhqfW2xjOK8Ep3R2FGT-xUyX50DgZh?usp=sharing" style="margin-left: 10px;">COLAB Notebook Here</a>
</div>
```python
from diffusers import DiffusionPipeline
import torch
# Load Stable Diffusion XL Base1.0
pipe = DiffusionPipeline.from_pretrained(
"stabilityai/stable-diffusion-xl-base-1.0",
torch_dtype=torch.float16,
variant="fp16",
use_safetensors=True
).to("cuda")
# Optional CPU offloading to save some GPU Memory
pipe.enable_model_cpu_offload()
# Loading Trained Old Book Illustrations LoRA Weights
pipe.load_lora_weights("AdamLucek/sdxl-base-1.0-oldbookillustrations-lora")
# Generate an Image
prompt = "An Apple"
image = pipe(
prompt = prompt,
num_inference_steps=50,
height=1024,
width=1024,
).images[0]
# Save the image
image.save("SDXL_OldBookIllustrations.png")
```
#### Limitations and bias
**Note**: See original Stable Diffusion XL Base 1.0 page for additional limitations and biases
**Note**: First try with tuning hyperparameters
## Training details
**Video Overview**
<a href="https://youtu.be/v89kB4OScOA">
<img src="https://i.imgur.com/fW6hHu2.png" width="350">
</a>
Trained on a single a100 using the [Diffuser's package](https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image_lora_sdxl.py), documentation available [here](https://huggingface.co/docs/diffusers/main/en/training/lora).
Training script used:
```
accelerate launch train_text_to_image_lora_sdxl.py \
--pretrained_model_name_or_path=stabilityai/stable-diffusion-xl-base-1.0 \
--pretrained_vae_model_name_or_path="madebyollin/sdxl-vae-fp16-fix" \
--dataset_name="AdamLucek/oldbookillustrations-small" \
--validation_prompt="An inventor tinkers with a complex machine in his workshop, oblivious to the setting sun outside" \
--num_validation_images=4 \
--validation_epochs=1 \
--output_dir="output/sdxl-base-1.0-oldbookillustrations-lora" \
--resolution=1024 \
--center_crop \
--random_flip \
--train_text_encoder \
--train_batch_size=1 \
--num_train_epochs=10 \
--checkpointing_steps=500 \
--gradient_accumulation_steps=4 \
--learning_rate=1e-04 \
--lr_warmup_steps=0 \
--report_to="wandb" \
--dataloader_num_workers=8 \
--allow_tf32 \
--mixed_precision="fp16" \
--push_to_hub \
--hub_model_id="sdxl-base-1.0-oldbookillustrations-lora"
``` |