---
datasets:
- nielsgl/dreambooth-ace
library_name: tf-keras
license: creativeml-openrail-m
pipeline_tag: text-to-image
tags:
- diffusers
- stable-diffusion
- text-to-image
- diffusion-models-class
- keras-dreambooth
- nature
inference: true
widget:
- text: a photo of puggieace dog on the beach, sunset in background
emoji: 🐶
---

# KerasCV Stable Diffusion in Diffusers 🧨🤗

DreamBooth model for the `puggieace` concept trained by nielsgl on the `nielsgl/dreambooth-ace` dataset.
It can be used by modifying the `instance_prompt`: **a photo of puggieace**.

The examples are from 2 different Keras CV models (`StableDiffusion` and `StableDiffusionV2`, corresponding to Stable Diffusion V1.4 and V2.1, respectively) trained on the same dataset (`nielsgl/dreambooth-ace`).

## Description

The Stable Diffusion V2 pipeline contained in the corresponding repository (`nielsgl/dreambooth-keras-pug-ace-sd2.1`) was created using a modified version of [this Space](https://huggingface.co/spaces/sayakpaul/convert-kerascv-sd-diffusers) for StableDiffusionV2 from KerasCV. The purpose is to convert the KerasCV Stable Diffusion weights in a way that is compatible with [Diffusers](https://github.com/huggingface/diffusers). This allows users to fine-tune using KerasCV and use the fine-tuned weights in Diffusers taking advantage of its nifty features (like [schedulers](https://huggingface.co/docs/diffusers/main/en/using-diffusers/schedulers), [fast attention](https://huggingface.co/docs/diffusers/optimization/fp16), etc.).
This model was created as part of the Keras DreamBooth Sprint 🔥. Visit the [organisation page](https://huggingface.co/keras-dreambooth) for instructions on how to take part!

## Examples

### Stable Diffusion V1.4

> Portrait of puggieace dog as a Roman Emperor, city in background

![Portrait of puggieace dog as a Roman Emperor, city in background, ultra realistic, intricate details, eerie, highly detailed, photorealistic, octane render, 8 k, unreal engine. art by artgerm and greg rutkowski and charlie bowater and magali villeneuve and alphonse mucha](examples/emperor-1.4.jpeg)

> Photo of puggieace dog wearing sunglasses on the beach, sunset in background, golden hour

![Photo of puggieace dog wearing sunglasses on the beach, sunset in background, golden hour](examples/beach-1.4.jpg)

> Photo of cute puggieace dog as an astronaut, planet and spaceship in background

![Photo of cute puggieace dog as an astronaut, planet and spaceship in background, ultra realistic, intricate details, highly detailed, photorealistic, octane render, 8 k, unreal engine. trending on artstation](examples/astronaut-1.4.jpg)

### Stable Diffusion V2.1

> Portrait painting of a cute puggieace dog as a samurai

![Portrait painting of a cute puggieace dog as a samurai, ultra realistic, concept art, intricate details, eerie, highly detailed, photorealistic, octane render, 8 k, unreal engine. art by artgerm and greg rutkowski and charlie bowater and magali villeneuve and alphonse mucha](examples/samurai-2.1.jpg)

> Photo of cute puggieace dog as an astronaut, space and planet in background

![Photo of cute puggieace dog as an astronaut, space and planet in background, ultra realistic, concept art, intricate details, highly detailed, photorealistic, octane render, 8 k, unreal engine. art by artgerm and greg rutkowski and charlie bowater, trending on artstation](examples/astronaut-2.1.jpg)

> A photo of a cute puggieace dog getting a haircut in a barbershop

![A photo of a cute puggieace dog getting a haircut in a barbershop, ultra realistic, intricate details, highly detailed, photorealistic, octane render, 8 k, unreal engine. art by artgerm and greg rutkowski and charlie bowater and magali villeneuve and alphonse mucha](examples/haircut-2.1.jpg)

> Portrait photo of puggieace dog in New York

![Portrait photo of puggieace dog in New York, city and skyscrapers in background, highly detailed, photorealistic, hdr, 4k](examples/ny-2.1.jpg)

> Portrait of puggieace dog as a Roman Emperor, city in background

![Portrait of puggieace dog as a Roman Emperor, city in background, ultra realistic, intricate details, eerie, highly detailed, photorealistic, octane render, 8 k, unreal engine. art by artgerm and greg rutkowski and charlie bowater and magali villeneuve and alphonse mucha](examples/emperor-2.1.jpg)


## Usage with Stable Diffusion V1.4

```python
from huggingface_hub import from_pretrained_keras
import keras_cv
import matplotlib.pyplot as plt


model = keras_cv.models.StableDiffusion(img_width=512, img_height=512, jit_compile=True)
model._diffusion_model = from_pretrained_keras("nielsgl/dreambooth-pug-ace")
model._text_encoder = from_pretrained_keras("nielsgl/dreambooth-pug-ace-text-encoder")

images = model.text_to_image("a photo of puggieace dog on the beach", batch_size=3)
plt.imshow(image[0])
```

## Usage with Stable Diffusion V2.1

```python
from diffusers import StableDiffusionPipeline

pipeline = StableDiffusionPipeline.from_pretrained('nielsgl/dreambooth-keras-pug-ace-sd2.1')
image = pipeline().images[0]
image
```

### Training hyperparameters

The following hyperparameters were used during training for Stable Diffusion v1.4:

| Hyperparameters | Value |
| :-- | :-- |
| name | RMSprop |
| weight_decay | None |
| clipnorm | None |
| global_clipnorm | None |
| clipvalue | None |
| use_ema | False |
| ema_momentum | 0.99 |
| ema_overwrite_frequency | 100 |
| jit_compile | True |
| is_legacy_optimizer | False |
| learning_rate | 0.0010000000474974513 |
| rho | 0.9 |
| momentum | 0.0 |
| epsilon | 1e-07 |
| centered | False |
| training_precision | float32 |