File size: 6,537 Bytes
2e77f41 9e84786 2e77f41 9e84786 2e77f41 9e84786 9fbdb4e 9e84786 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 |
---
library_name: diffusers
base_model: stabilityai/stable-diffusion-xl-base-1.0
tags:
- text-to-image
license: openrail++
inference: false
---
# What is different about this fork from the original (h1t/oms_b_openclip_xl)?
The code has been modified to work with the current final version (0.27.2) of diffusers.
The behavior remains the same. Enjoy.
<img src="kaeru-dev.png" width="600"/>
```diff
- OMSPipeline.from_pretrained('h1t/oms_b_openclip_xl', ...)
+ OMSPipeline.from_pretrained('kaeru-shigure/oms_b_openclip_xl', ...)
```
```diff
--- a/diffusers_patch/models/unet_2d_condition_woct.py
+++ b/diffusers_patch/models/unet_2d_condition_woct.py
@@ -35,7 +35,7 @@ from diffusers.models.embeddings import (
Timesteps,
)
from diffusers.models.modeling_utils import ModelMixin
-from diffusers.models.unet_2d_blocks import (
+from diffusers.models.unets.unet_2d_blocks import (
CrossAttnDownBlock2D,
CrossAttnUpBlock2D,
DownBlock2D,
@@ -159,6 +159,7 @@ class UNet2DConditionWoCTModel(ModelMixin, ConfigMixin, UNet2DConditionLoadersMi
conv_out_kernel: int = 3,
mid_block_only_cross_attention: Optional[bool] = None,
cross_attention_norm: Optional[str] = None,
+ subfolder: Optional[str] = None,
):
super().__init__()
```
```diff
--- a/diffusers_patch/pipelines/oms/pipeline_oms.py
+++ b/diffusers_patch/pipelines/oms/pipeline_oms.py
@@ -8,6 +8,7 @@ from transformers import CLIPTextModel, CLIPTextModelWithProjection, CLIPTokeniz
from diffusers.loaders import FromSingleFileMixin
+from huggingface_hub.constants import HF_HUB_CACHE, HF_HUB_OFFLINE
from diffusers.utils import (
USE_PEFT_BACKEND,
deprecate,
@@ -17,6 +18,7 @@ from diffusers.utils.torch_utils import randn_tensor
from diffusers.pipelines.pipeline_utils import DiffusionPipeline
from diffusers.pipelines.pipeline_utils import *
from diffusers.pipelines.pipeline_utils import _get_pipeline_class
+from diffusers.pipelines.pipeline_loading_utils import *
from diffusers.models.modeling_utils import _LOW_CPU_MEM_USAGE_DEFAULT
from diffusers_patch.models.unet_2d_condition_woct import UNet2DConditionWoCTModel
@@ -164,7 +166,8 @@ class OMSPipeline(DiffusionPipeline, FromSingleFileMixin):
sd_pipeline: DiffusionPipeline,
oms_text_encoder:Optional[Union[CLIPTextModel, SDXLTextEncoder]],
oms_tokenizer:Optional[Union[CLIPTokenizer, SDXLTokenizer]],
- sd_scheduler = None
+ sd_scheduler = None,
+ trust_remote_code: bool = False,
):
# assert sd_pipeline is not None
@@ -279,7 +282,7 @@ class OMSPipeline(DiffusionPipeline, FromSingleFileMixin):
@classmethod
os.PathLike]], **kwargs):
- cache_dir = kwargs.pop("cache_dir", DIFFUSERS_CACHE)
+ cache_dir = kwargs.pop("cache_dir", HF_HUB_CACHE)
resume_download = kwargs.pop("resume_download", False)
force_download = kwargs.pop("force_download", False)
proxies = kwargs.pop("proxies", None)
```
-----
# One More Step
One More Step (OMS) module was proposed in [One More Step: A Versatile Plug-and-Play Module for Rectifying Diffusion Schedule Flaws and Enhancing Low-Frequency Controls](https://github.com/mhh0318/OneMoreStep)
by *Minghui Hu, Jianbin Zheng, Chuanxia Zheng, Tat-Jen Cham et al.*
By **adding one small step** on the top the sampling process, we can address the issues caused by the current schedule flaws of diffusion models **without changing the original model parameters**. This also allows for some control over low-frequency information, such as color.
Our model is **versatile** and can be integrated into almost all widely-used Stable Diffusion frameworks. It's compatible with community favorites such as **LoRA, ControlNet, Adapter, and foundational models**.
## Usage
OMS now is supported 🤗 `diffusers` with a customized pipeline [github](https://github.com/mhh0318/OneMoreStep). To run the model (especially with `LCM` variant), first install the latest version of `diffusers` library as well as `accelerate` and `transformers`.
```bash
pip install --upgrade pip
pip install --upgrade diffusers transformers accelerate
```
And then we clone the repo
```bash
git clone https://github.com/mhh0318/OneMoreStep.git
cd OneMoreStep
```
### SDXL
The OMS module can be loaded with SDXL base model `stabilityai/stable-diffusion-xl-base-1.0`.
And all the SDXL based model and its LoRA can **share the same OMS** `h1t/oms_b_openclip_xl`.
Here is an example for SDXL with LCM-LoRA.
Firstly import the related packages and choose SDXL based backbone and LoRA:
```python
import torch
from diffusers import StableDiffusionXLPipeline, LCMScheduler
sd_pipe = StableDiffusionXLPipeline.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16, variant="fp16", add_watermarker=False).to('cuda')
sd_scheduler = LCMScheduler.from_config(sd_pipe.scheduler.config)
sd_pipe.load_lora_weights('latent-consistency/lcm-lora-sdxl', variant="fp16")
```
Following import the customized OMS pipeline to wrap the backbone and add OMS for sampling. We have uploaded the `.safetensors` to [HuggingFace Hub](https://huggingface.co/h1t/). There are 2 choices for SDXL backbone currently, one is base OMS module with OpenCLIP text encoder [h1t/oms_b_openclip_xl)](https://huggingface.co/h1t/oms_b_openclip_xl) and the other is large OMS module with two text encoder followed by SDXL architecture [h1t/oms_l_mixclip_xl)](https://huggingface.co/h1t/oms_b_mixclip_xl).
```python
from diffusers_patch import OMSPipeline
pipe = OMSPipeline.from_pretrained('h1t/oms_b_openclip_xl', sd_pipeline = sd_pipe, torch_dtype=torch.float16, variant="fp16", trust_remote_code=True, sd_scheduler=sd_scheduler)
pipe.to('cuda')
```
After setting a random seed, we can easily generate images with the OMS module.
```python
prompt = 'close-up photography of old man standing in the rain at night, in a street lit by lamps, leica 35mm summilux'
generator = torch.Generator(device=pipe.device).manual_seed(1024)
image = pipe(prompt, guidance_scale=1, num_inference_steps=4, generator=generator)
image['images'][0]
```

Or we can offload the OMS module and generate a image only using backbone
```python
image = pipe(prompt, guidance_scale=1, num_inference_steps=4, generator=generator, oms_flag=False)
image['images'][0]
```

For more models and more functions like diverse prompt, please refer to [OMS Repo](https://github.com/mhh0318/OneMoreStep).
|