metadata

title: Erasing Concepts from Diffusion Models
emoji: 💡
colorFrom: indigo
colorTo: gray
sdk: gradio
sdk_version: 3.21.0
app_file: app.py
pinned: false
license: mit

A GUI with custom model support, validation, and sample generation for "Erasing Concepts from Diffusion Models"

Enables xformers, 8 bit AdamW via bitsandbytes, and AMP - editing SD1.5 models works with 16GB VRAM, and 2.5 models including the ESD-u training works with 24GB VRAM.

Quick start

To run on vast.ai, use eg pytorch/pytorch:2.0.1-cuda11.7-cudnn8-devel - you need -devel for 8bit AdamW to work.

On the dev machine:

pip install -r requirements.txt
python app.py

then use the Gradio interface at port 7860.

Erasing Concepts from Diffusion Models

Project Website https://erasing.baulab.info
Arxiv Preprint https://arxiv.org/pdf/2303.07345.pdf
Fine-tuned Weights https://erasing.baulab.info/weights/esd_models/

Motivated by recent advancements in text-to-image diffusion, we study erasure of specific concepts from the model's weights. While Stable Diffusion has shown promise in producing explicit or realistic artwork, it has raised concerns regarding its potential for misuse. We propose a fine-tuning method that can erase a visual concept from a pre-trained diffusion model, given only the name of the style and using negative guidance as a teacher. We benchmark our method against previous approaches that remove sexually explicit content and demonstrate its effectiveness, performing on par with Safe Latent Diffusion and censored training.

To evaluate artistic style removal, we conduct experiments erasing five modern artists from the network and conduct a user study to assess the human perception of the removed styles. Unlike previous methods, our approach can remove concepts from a diffusion model permanently rather than modifying the output at the inference time, so it cannot be circumvented even if a user has access to model weights

Given only a short text description of an undesired visual concept and no additional data, our method fine-tunes model weights to erase the targeted concept. Our method can avoid NSFW content, stop imitation of a specific artist's style, or even erase a whole object class from model output, while preserving the model's behavior and capabilities on other topics.

Demo vs github

This demo uses an updated implementation from the original Erasing codebase the publication is based from.

Running locally

1.) Create an environment using the packages included in the requirements.txt file

2.) Run python app.py

3.) Open the application in browser at http://127.0.0.1:7860/

4.) Train, evaluate, and save models using our method

Citing our work

The preprint can be cited as follows

@article{gandikota2023erasing,
  title={Erasing Concepts from Diffusion Models},
  author={Rohit Gandikota and Joanna Materzy\'nska and Jaden Fiotto-Kaufman and David Bau},
  journal={arXiv preprint arXiv:2303.07345},
  year={2023}
}