LoRA adapters

.gitignore

@@ -1,2 +1,3 @@
 __pycache__/
 .venv/
+loras/

DOCS.md

@@ -10,7 +10,7 @@ Use `+` or `-` to increase the weight of a token. The weight grows exponentially
 
 For groups of tokens, wrap them in parentheses and multiply by a float between 0 and 2. For example, `a (birthday cake)1.3 on a table` will increase the weight of both `birthday` and `cake` by 1.3x. This also means the entire scene will be more birthday-like, not just the cake. To counteract this, you can use `-` inside the parentheses on specific tokens, e.g., `a (birthday-- cake)1.3`, to reduce the birthday aspect.
 
-Note that this is also the same syntax used in [InvokeAI](https://invoke-ai.github.io/InvokeAI/features/PROMPTS/) and it differs from AUTOMATIC1111:
+This is the same syntax used in [InvokeAI](https://invoke-ai.github.io/InvokeAI/features/PROMPTS/) and it differs from AUTOMATIC1111:
 
 | Compel      | AUTOMATIC1111 |
 | ----------- | ------------- |
@@ -21,25 +21,47 @@ Note that this is also the same syntax used in [InvokeAI](https://invoke-ai.gith
 
 #### Arrays
 
-Arrays allow you to generate multiple different images from a single prompt. For example, `a [[cute,adorable]] [[cat,corgi]]` will expand into `a cute cat` and `a cute corgi`.
+Arrays allow you to generate multiple different images from a single prompt. For example, `an adult [[blonde,brunette]] [[man,woman]]` will expand into **4** different prompts. This implementation was inspired by [Fooocus](https://github.com/lllyasviel/Fooocus/pull/1503).
 
-Before generating, make sure `Images` is set to the number of images you want and keep in mind that there is a max of 4. Note that arrays in the negative prompt are ignored. This implementation was inspired by [Fooocus](https://github.com/lllyasviel/Fooocus/pull/1503).
+> NB: Make sure to set `Images` to the number of images you want to generate. Otherwise, only the first prompt will be used.
+
+### Models
+
+Each model checkpoint has a different aesthetic:
+
+* [Comfy-Org/stable-diffusion-v1-5](https://huggingface.co/Comfy-Org/stable-diffusion-v1-5-archive): base
+* [cyberdelia/CyberRealistic_V5](https://huggingface.co/cyberdelia/CyberRealistic): realistic
+* [Lykon/dreamshaper-8](https://huggingface.co/Lykon/dreamshaper-8): general purpose (default)
+* [fluently/Fluently-v4](https://huggingface.co/fluently/Fluently-v4): general purpose stylized
+* [Linaqruf/anything-v3-1](https://huggingface.co/Linaqruf/anything-v3-1): anime
+* [prompthero/openjourney-v4](https://huggingface.co/prompthero/openjourney-v4): Midjourney art style
+* [SG161222/Realistic_Vision_V5](https://huggingface.co/SG161222/Realistic_Vision_V5.1_noVAE): realistic
+* [XpucT/Deliberate_v6](https://huggingface.co/XpucT/Deliberate): general purpose stylized
+
+### LoRA
+
+Apply up to 2 LoRA (low-rank adaptation) adapters with adjustable strength:
+
+* [Perfection Style](https://civitai.com/models/411088?modelVersionId=486099): attempts to improve aesthetics, use high strength
+* [Detailed Style](https://civitai.com/models/421162?modelVersionId=486110): attempts to improve details, use low strength
+
+> NB: The trigger words are automatically appended to the positive prompt for you.
 
 ### Embeddings
 
 Select one or more [textual inversion](https://huggingface.co/docs/diffusers/en/using-diffusers/textual_inversion_inference) embeddings:
 
 * [`fast_negative`](https://civitai.com/models/71961?modelVersionId=94057): all-purpose (default)
-* [`unrealistic_dream`](https://civitai.com/models/72437?modelVersionId=77173): realistic add-on (for RealisticVision)
 * [`cyberrealistic_negative`](https://civitai.com/models/77976?modelVersionId=82745): realistic add-on (for CyberRealistic)
+* [`unrealistic_dream`](https://civitai.com/models/72437?modelVersionId=77173): realistic add-on (for RealisticVision)
+
+> NB: The trigger token is automatically appended to the negative prompt for you.
 
 ### Styles
 
 [Styles](https://huggingface.co/spaces/adamelliotfields/diffusion/blob/main/data/styles.json) are prompt templates that wrap your positive and negative prompts. They were originally derived from the [twri/sdxl_prompt_styler](https://github.com/twri/sdxl_prompt_styler) Comfy node, but have since been entirely rewritten.
 
-Start by framing a simple subject like `portrait of a young adult woman` or `landscape of a mountain range`. Experiment with different styles and don't forget about the negative prompt.
-
-> NB: Most styles work best with the Dreamshaper model; however, the "Enhance" style is meant to be universal. The "Photography" styles work especially well with the realistic models.
+Start by framing a simple subject like `portrait of a young adult woman` or `landscape of a mountain range` and experiment.
 
 ### Scale
 
@@ -47,19 +69,6 @@ Rescale up to 4x using [Real-ESRGAN](https://github.com/xinntao/Real-ESRGAN) wit
 
 > NB: I find this Real-ESRGAN model to work well, so I do not use a _hi-res fix_.
 
-### Models
-
-Each model checkpoint has a different aesthetic:
-
-* [Comfy-Org/stable-diffusion-v1-5](https://huggingface.co/Comfy-Org/stable-diffusion-v1-5-archive): base
-* [cyberdelia/CyberRealistic_v5](https://huggingface.co/cyberdelia/CyberRealistic): photorealistic
-* [Lykon/dreamshaper-8](https://huggingface.co/Lykon/dreamshaper-8): general purpose (default)
-* [fluently/Fluently-v4](https://huggingface.co/fluently/Fluently-v4): general purpose
-* [Linaqruf/anything-v3-1](https://huggingface.co/Linaqruf/anything-v3-1): anime
-* [prompthero/openjourney-v4](https://huggingface.co/prompthero/openjourney-v4): Midjourney-like
-* [SG161222/Realistic_Vision_v5.1](https://huggingface.co/SG161222/Realistic_Vision_V5.1_noVAE): photorealistic
-* [XpucT/Deliberate_v6](https://huggingface.co/XpucT/Deliberate): general purpose
-
 ### Image-to-Image
 
 The `🖼️ Image` tab enables the image-to-image and IP-Adapter pipelines. Either use the image input or select a generation from the gallery. To disable, simply clear the image input (the `x` overlay button).

app.css

@@ -37,6 +37,10 @@
   max-height: none;
 }
 
+.gap-0, .gap-0 * {
+  gap: 0px;
+}
+
 .icon-button {
   max-width: 42px;
 }

app.py

@@ -1,10 +1,11 @@
 import argparse
 import json
+import os
 import random
 
 import gradio as gr
 
-from lib import Config, async_call, download_repo_files, generate, read_file
+from lib import Config, async_call, download_civit_file, download_repo_files, generate, read_file
 
 # the CSS `content` attribute expects a string so we need to wrap the number in quotes
 refresh_seed_js = """
@@ -130,9 +131,8 @@ with gr.Blocks(
             with gr.TabItem("⚙️ Settings"):
                 with gr.Group():
                     negative_prompt = gr.Textbox(
-                        value=None,
+                        value="nsfw+",
                         label="Negative Prompt",
-                        placeholder="ugly, bad",
                         lines=2,
                     )
 
@@ -159,6 +159,7 @@ with gr.Blocks(
                         style = gr.Dropdown(
                             value=Config.STYLE,
                             label="Style",
+                            min_width=240,
                             choices=[("None", "none")]
                             + [(styles[sid]["name"], sid) for sid in style_ids],
                         )
@@ -171,6 +172,44 @@ with gr.Blocks(
                             min_width=240,
                         )
 
+                    with gr.Row():
+                        with gr.Group(elem_classes=["gap-0"]):
+                            lora_1 = gr.Dropdown(
+                                min_width=240,
+                                label="LoRA #1",
+                                value="none",
+                                choices=[("None", "none")]
+                                + [
+                                    (lora["name"], lora_id)
+                                    for lora_id, lora in Config.CIVIT_LORAS.items()
+                                ],
+                            )
+                            lora_1_weight = gr.Slider(
+                                value=0.0,
+                                minimum=0.0,
+                                maximum=1.0,
+                                step=0.1,
+                                show_label=False,
+                            )
+                        with gr.Group(elem_classes=["gap-0"]):
+                            lora_2 = gr.Dropdown(
+                                min_width=240,
+                                label="LoRA #2",
+                                value="none",
+                                choices=[("None", "none")]
+                                + [
+                                    (lora["name"], lora_id)
+                                    for lora_id, lora in Config.CIVIT_LORAS.items()
+                                ],
+                            )
+                            lora_2_weight = gr.Slider(
+                                value=0.0,
+                                minimum=0.0,
+                                maximum=1.0,
+                                step=0.1,
+                                show_label=False,
+                            )
+
                     with gr.Row():
                         guidance_scale = gr.Slider(
                             value=Config.GUIDANCE_SCALE,
@@ -336,7 +375,7 @@ with gr.Blocks(
                 columns=2,
             )
             prompt = gr.Textbox(
-                placeholder="corgi, beach, 8k",
+                placeholder="What do you want to see?",
                 autoscroll=False,
                 show_label=False,
                 label="Prompt",
@@ -448,6 +487,10 @@ with gr.Blocks(
             image_prompt,
             ip_image,
             ip_face,
+            lora_1,
+            lora_1_weight,
+            lora_2,
+            lora_2_weight,
             embeddings,
             style,
             seed,
@@ -475,10 +518,19 @@ if __name__ == "__main__":
     args = parser.parse_args()
 
     # download to hub cache
-    for repo_id, allow_patterns in Config.DOWNLOAD_FILES.items():
-        print(f"Downloading {repo_id}...")
+    for repo_id, allow_patterns in Config.HF_MODELS.items():
         download_repo_files(repo_id, allow_patterns, token=Config.HF_TOKEN)
 
+    # download civit loras
+    for lora_id, lora in Config.CIVIT_LORAS.items():
+        file_path = os.path.join(os.path.dirname(__file__), "loras")
+        download_civit_file(
+            lora_id,
+            lora["model_version_id"],
+            file_path=file_path,
+            token=Config.CIVIT_TOKEN,
+        )
+
     # https://www.gradio.app/docs/gradio/interface#interface-queue
     demo.queue().launch(
         server_name=args.server,

cli.py

@@ -17,7 +17,7 @@ async def main():
     parser = argparse.ArgumentParser(add_help=False, allow_abbrev=False)
     parser.add_argument("prompt", type=str, metavar="PROMPT")
     parser.add_argument("-n", "--negative", type=str, metavar="STR", default="")
-    parser.add_argument("-e", "--embedding", type=str, metavar="STR", default=[], action="append")
+    parser.add_argument("-e", "--embeddings", type=str, metavar="STR", default="")
     parser.add_argument("-s", "--seed", type=int, metavar="INT", default=Config.SEED)
     parser.add_argument("-i", "--images", type=int, metavar="INT", default=1)
     parser.add_argument("-f", "--filename", type=str, metavar="STR", default="image.png")
@@ -25,12 +25,16 @@ async def main():
     parser.add_argument("-h", "--height", type=int, metavar="INT", default=Config.HEIGHT)
     parser.add_argument("-m", "--model", type=str, metavar="STR", default=Config.MODEL)
     parser.add_argument("-d", "--deepcache", type=int, metavar="INT", default=Config.DEEPCACHE_INTERVAL)
+    parser.add_argument("--lora-1", type=str, metavar="STR", default="")
+    parser.add_argument("--lora-1-weight", type=float, metavar="FLOAT", default=0.0)
+    parser.add_argument("--lora-2", type=str, metavar="STR", default="")
+    parser.add_argument("--lora-2-weight", type=float, metavar="FLOAT", default=0.0)
     parser.add_argument("--scale", type=int, metavar="INT", choices=Config.SCALES, default=Config.SCALE)
     parser.add_argument("--style", type=str, metavar="STR", default=Config.STYLE)
     parser.add_argument("--scheduler", type=str, metavar="STR", default=Config.SCHEDULER)
     parser.add_argument("--guidance", type=float, metavar="FLOAT", default=Config.GUIDANCE_SCALE)
     parser.add_argument("--steps", type=int, metavar="INT", default=Config.INFERENCE_STEPS)
-    parser.add_argument("--strength", type=float, metavar="FLOAT", default=Config.DENOISING_STRENGTH)
+    parser.add_argument("--image-strength", type=float, metavar="FLOAT", default=Config.DENOISING_STRENGTH)
     parser.add_argument("--image", type=str, metavar="STR")
     parser.add_argument("--ip-image", type=str, metavar="STR")
     parser.add_argument("--ip-face", action="store_true")
@@ -48,7 +52,11 @@ async def main():
         args.image,
         args.ip_image,
         args.ip_face,
-        args.embedding,
+        args.lora_1,
+        args.lora_1_weight,
+        args.lora_2,
+        args.lora_2_weight,
+        args.embeddings.split(",") if args.embeddings else [],
         args.style,
         args.seed,
         args.model,
@@ -57,7 +65,7 @@ async def main():
         args.height,
         args.guidance,
         args.steps,
-        args.strength,
+        args.image_strength,
         args.deepcache,
         args.scale,
         args.images,
@@ -66,7 +74,7 @@ async def main():
         args.freeu,
         args.clip_skip,
     )
-    await async_call(save_images, images, args.filename)
+    save_images(images, args.filename)
 
 
 if __name__ == "__main__":

lib/__init__.py

@@ -2,13 +2,14 @@ from .config import Config
 from .inference import generate
 from .loader import Loader
 from .upscaler import RealESRGAN
-from .utils import async_call, download_repo_files, load_json, read_file
+from .utils import async_call, download_civit_file, download_repo_files, load_json, read_file
 
 __all__ = [
     "Config",
     "Loader",
     "RealESRGAN",
     "async_call",
+    "download_civit_file",
     "download_repo_files",
     "generate",
     "load_json",

lib/config.py

@@ -14,7 +14,8 @@ from diffusers import (
 
 Config = SimpleNamespace(
     HF_TOKEN=os.environ.get("HF_TOKEN", None),
-    DOWNLOAD_FILES={
+    CIVIT_TOKEN=os.environ.get("CIVIT_TOKEN", None),
+    HF_MODELS={
         "Lykon/dreamshaper-8": [
             "feature_extractor/preprocessor_config.json",
             "safety_checker/config.json",
@@ -32,6 +33,22 @@ Config = SimpleNamespace(
             "model_index.json",
         ],
     },
+    CIVIT_LORAS={
+        # https://civitai.com/models/411088?modelVersionId=486099
+        "perfection_style": {
+            "model_id": "411088",
+            "model_version_id": "486099",
+            "name": "Perfection Style",
+            "trigger": "perfection style",
+        },
+        # https://civitai.com/models/421162?modelVersionId=486110
+        "detailed_style": {
+            "model_id": "421162",
+            "model_version_id": "486110",
+            "name": "Detailed Style",
+            "trigger": "detailed style",
+        },
+    },
     MONO_FONTS=["monospace"],
     SANS_FONTS=[
         "sans-serif",
@@ -81,8 +98,8 @@ Config = SimpleNamespace(
         "unrealistic_dream",
     ],
     STYLE="enhance",
-    WIDTH=448,
-    HEIGHT=576,
+    WIDTH=512,
+    HEIGHT=512,
     NUM_IMAGES=1,
     SEED=-1,
     GUIDANCE_SCALE=5,

lib/inference.py

@@ -13,6 +13,7 @@ from compel.prompt_parser import PromptParser
 from huggingface_hub.utils import HFValidationError, RepositoryNotFoundError
 from PIL import Image
 
+from .config import Config
 from .loader import Loader
 from .utils import load_json
 
@@ -39,7 +40,7 @@ def parse_prompt_with_arrays(prompt: str) -> list[str]:
     return prompts
 
 
-def apply_style(positive_prompt, negative_prompt, style_id):
+def apply_style(positive_prompt, negative_prompt, style_id="none"):
     if style_id.lower() == "none":
         return (positive_prompt, negative_prompt)
 
@@ -96,6 +97,10 @@ def generate(
     image_prompt=None,
     ip_image=None,
     ip_face=False,
+    lora_1=None,
+    lora_1_weight=0.0,
+    lora_2=None,
+    lora_2_weight=0.0,
     embeddings=[],
     style=None,
     seed=None,
@@ -176,7 +181,7 @@ def generate(
     )
 
     if loader.pipe is None:
-        raise Error(f"RuntimeError: Error loading {model}")
+        raise Error(f"Error loading {model}")
 
     pipe = loader.pipe
     upscaler = None
@@ -186,9 +191,36 @@ def generate(
     if scale == 4:
         upscaler = loader.upscaler_4x
 
-    embeddings_tokens = []
-    embeddings_dir = os.path.join(os.path.dirname(__file__), "..", "embeddings")
-    embeddings_dir = os.path.abspath(embeddings_dir)
+    # load loras
+    loras = []
+    weights = []
+    loras_and_weights = [(lora_1, lora_1_weight), (lora_2, lora_2_weight)]
+    loras_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "loras"))
+    for lora, weight in loras_and_weights:
+        if lora and lora.lower() != "none" and lora not in loras:
+            config = Config.CIVIT_LORAS.get(lora)
+            if config:
+                try:
+                    pipe.load_lora_weights(
+                        loras_dir,
+                        adapter_name=lora,
+                        weight_name=f"{lora}.{config['model_version_id']}.safetensors",
+                    )
+                    weights.append(weight)
+                    loras.append(lora)
+                except Exception:
+                    raise Error(f"Error loading {config['name']} LoRA")
+
+    # unload after generating or if there was an error
+    try:
+        if loras:
+            pipe.set_adapters(loras, adapter_weights=weights)
+    except Exception:
+        pipe.unload_lora_weights()
+        raise Error("Error setting LoRA weights")
+
+    # load embeddings
+    embeddings_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "embeddings"))
     for embedding in embeddings:
         try:
             # wrap embeddings in angle brackets
@@ -196,9 +228,8 @@ def generate(
                 pretrained_model_name_or_path=f"{embeddings_dir}/{embedding}.pt",
                 token=f"<{embedding}>",
             )
-            embeddings_tokens.append(f"<{embedding}>")
         except (EnvironmentError, HFValidationError, RepositoryNotFoundError):
-            raise Error(f"Invalid embedding: <{embedding}>")
+            raise Error(f"Invalid embedding: {embedding}")
 
     # prompt embeds
     compel = Compel(
@@ -212,7 +243,6 @@ def generate(
 
     images = []
     current_seed = seed
-
     for i in range(num_images):
         # seeded generator for each iteration
         generator = torch.Generator(device=pipe.device).manual_seed(current_seed)
@@ -229,14 +259,18 @@ def generate(
             if negative_styled.startswith("(), "):
                 negative_styled = negative_styled[4:]
 
-            if embeddings_tokens:
-                negative_styled += ", " + ", ".join(embeddings_tokens)
+            for lora in loras:
+                positive_styled += f", {Config.CIVIT_LORAS[lora]['trigger']}"
+
+            for embedding in embeddings:
+                negative_styled += f", <{embedding}>"
 
+            # print prompts
             positive_embeds, negative_embeds = compel.pad_conditioning_tensors_to_same_length(
                 [compel(positive_styled), compel(negative_styled)]
             )
         except PromptParser.ParsingException:
-            raise Error("ValueError: Invalid prompt")
+            raise Error("Invalid prompt")
 
         kwargs = {
             "width": width,
@@ -244,8 +278,8 @@ def generate(
             "generator": generator,
             "prompt_embeds": positive_embeds,
             "guidance_scale": guidance_scale,
-            "negative_prompt_embeds": negative_embeds,
             "num_inference_steps": inference_steps,
+            "negative_prompt_embeds": negative_embeds,
             "output_type": "np" if scale > 1 else "pil",
         }
 
@@ -257,7 +291,7 @@ def generate(
             kwargs["image"] = prepare_image(image_prompt, (width, height))
 
         if IP_ADAPTER:
-            # don't resize full-face images
+            # don't resize full-face images since they are usually square crops
             size = None if ip_face else (width, height)
             kwargs["ip_adapter_image"] = prepare_image(ip_image, size)
 
@@ -268,9 +302,12 @@ def generate(
             images.append((image, str(current_seed)))
             current_seed += 1
         except Exception as e:
-            raise Error(f"RuntimeError: {e}")
+            raise Error(f"{e}")
         finally:
-            pipe.unload_textual_inversion()
+            if embeddings:
+                pipe.unload_textual_inversion()
+            if loras:
+                pipe.unload_lora_weights()
             CURRENT_STEP = 0
             CURRENT_IMAGE += 1
 

lib/utils.py

@@ -1,9 +1,11 @@
 import functools
 import inspect
 import json
+import os
 from typing import Callable, TypeVar
 
 import anyio
+import httpx
 from anyio import Semaphore
 from huggingface_hub._snapshot_download import snapshot_download
 from typing_extensions import ParamSpec
@@ -38,6 +40,34 @@ def download_repo_files(repo_id, allow_patterns, token=None):
     )
 
 
+def download_civit_file(lora_id, version_id, file_path=".", token=None):
+    base_url = "https://civitai.com/api/download/models"
+    file = f"{file_path}/{lora_id}.{version_id}.safetensors"
+
+    if os.path.exists(file):
+        return
+
+    try:
+        params = {"token": token}
+        response = httpx.get(
+            f"{base_url}/{version_id}",
+            timeout=None,
+            params=params,
+            follow_redirects=True,
+        )
+
+        response.raise_for_status()
+        os.makedirs(file_path, exist_ok=True)
+
+        with open(file, "wb") as f:
+            f.write(response.content)
+    except httpx.HTTPStatusError as e:
+        print(e.request.url)
+        print(f"HTTPError: {e.response.status_code} {e.response.text}")
+    except httpx.RequestError as e:
+        print(f"RequestError: {e}")
+
+
 # like the original but supports args and kwargs instead of a dict
 # https://github.com/huggingface/huggingface-inference-toolkit/blob/0.2.0/src/huggingface_inference_toolkit/async_utils.py
 async def async_call(fn: Callable[P, T], *args: P.args, **kwargs: P.kwargs) -> T:

requirements.txt

@@ -1,11 +1,13 @@
-anyio==4.4.0
 accelerate
-einops==0.8.0
+anyio==4.4.0
 compel==2.0.3
 deepcache==0.1.1
 diffusers==0.30.2
-hf-transfer
+einops==0.8.0
 gradio==4.41.0
+h2
+hf-transfer
+httpx
 numpy==1.26.4
 ruff==0.5.7
 spaces