metadata

base_model:
  - CompVis/stable-diffusion-v1-4
pipeline_tag: text-to-image
tags:
  - art

The Superposition of Diffusion Models Using the Itô Density Estimator: Pipeline

This pipeline shows how to superimpose different text prompts from Stable Diffusion v1-4 based the paper The Superposition of Diffusion Models Using the Itô Density Estimator.

drawing

Example usage

from PIL import Image
from diffusers import DiffusionPipeline

pipeline = DiffusionPipeline.from_pretrained("superdiff/superdiff-sd-v1-4", custom_pipeline='pipeline', trust_remote_code=True)
image = pipeline("a flamingo", "a candy cane", seed=1, num_inference_steps=1000, batch_size=1)

image = Image.fromarray(image.cpu().numpy())
image.save("superdiff_output.png")

Arguments that can be set by user in pipeline():

prompt_1: text prompt describing first concept to superimpose (e.g. "a flamingo")
prompt_2: text prompt describing second concept to superimpose (e.g. "a candy cane")
seed: seed for random noise generator for reproducibility; for non-deterministic outputs, do not provide value
num_inference_steps: number of denoising steps (we recommend 1000!)
batch_size: batch size
lift: bias value that favours generation towards one prompt over the other
guidance_scale: scale for classifier-free guidance
height, width: height and width of generated images

To replicate images from Section 4.2 of the paper, you can use the following:

image = pipeline(prompt_1, prompt_2, seed=1, num_inference_steps=1000, batch_size=20, lift=0.0, guidance_scale=7.5)

Citation

BibTeX:

@article{skreta2024superposition,
  title={The Superposition of Diffusion Models Using the It$\backslash$\^{} o Density Estimator},
  author={Skreta, Marta and Atanackovic, Lazar and Bose, Avishek Joey and Tong, Alexander and Neklyudov, Kirill},
  journal={arXiv preprint arXiv:2412.17762},
  year={2024}
}