Overview
🤗 Diffusers provides a collection of training scripts for you to train your own diffusion models. You can find all of our training scripts in diffusers/examples.
Each training script is:
- Self-contained: the training script does not depend on any local files, and all packages required to run the script are installed from the
requirements.txtfile. - Easy-to-tweak: the training scripts are an example of how to train a diffusion model for a specific task and won't work out-of-the-box for every training scenario. You'll likely need to adapt the training script for your specific use-case. To help you with that, we've fully exposed the data preprocessing code and the training loop so you can modify it for your own use.
- Beginner-friendly: the training scripts are designed to be beginner-friendly and easy to understand, rather than including the latest state-of-the-art methods to get the best and most competitive results. Any training methods we consider too complex are purposefully left out.
- Single-purpose: each training script is expressly designed for only one task to keep it readable and understandable.
Our current collection of training scripts include:
| Training | SDXL-support | LoRA-support | Flax-support |
|---|---|---|---|
unconditional image generation  |
|||
| text-to-image | 👍 | 👍 | 👍 |
| textual inversion |
👍 | ||
| DreamBooth |
👍 | 👍 | 👍 |
| ControlNet | 👍 | 👍 | |
| InstructPix2Pix | 👍 | ||
| Custom Diffusion | |||
| T2I-Adapters | 👍 | ||
| Kandinsky 2.2 | 👍 | ||
| Wuerstchen | 👍 |
These examples are actively maintained, so please feel free to open an issue if they aren't working as expected. If you feel like another training example should be included, you're more than welcome to start a Feature Request to discuss your feature idea with us and whether it meets our criteria of being self-contained, easy-to-tweak, beginner-friendly, and single-purpose.
Install
Make sure you can successfully run the latest versions of the example scripts by installing the library from source in a new virtual environment:
git clone https://github.com/huggingface/diffusers
cd diffusers
pip install .
Then navigate to the folder of the training script (for example, DreamBooth) and install the requirements.txt file. Some training scripts have a specific requirement file for SDXL, LoRA or Flax. If you're using one of these scripts, make sure you install its corresponding requirements file.
cd examples/dreambooth
pip install -r requirements.txt
# to train SDXL with DreamBooth
pip install -r requirements_sdxl.txt
To speedup training and reduce memory-usage, we recommend:
- using PyTorch 2.0 or higher to automatically use scaled dot product attention during training (you don't need to make any changes to the training code)
- installing xFormers to enable memory-efficient attention