BRIA 2.3 ControlNet GenFill Model Card
Trained exclusively on the largest multi-source commercial-grade licensed dataset, BRIA 2.3 ControlNet GenFill guarantees best quality while safe for commercial use. The model provides full legal liability coverage for copyright and privacy infringement and harmful content mitigation, as our dataset does not represent copyrighted materials, such as fictional characters, logos or trademarks, public figures, harmful content or privacy infringing content.
BRIA 2.3 ControlNet GenFill is a model designed to fill masked regions in images based on user-provided textual prompts, specialised in the tasks object replacement, addition, and modification within an image.
What's New
BRIA 2.3 ControlNet GenFill should be applied on top of BRIA 2.3 Text-to-Image and therefore enable to use Fast-LORA. This results in an extremely fast inpainting model, requires only 6.3s using A10 GPU.
Model Description
- Developed by: BRIA AI
- Model type: Latent diffusion image-to-image model
- License: BRIA 2.3 ControlNet GenFill Licensing terms & conditions.
- Purchase is required to license and access the model.
- Model Description: BRIA 2.3 ControlNet GenFill was trained exclusively on a professional-grade, licensed dataset. It is designed for commercial use and includes full legal liability coverage.
- Resources for more information: BRIA AI
Get Access to the source code and pre-trained model
Interested in BRIA 2.3 ControlNet GenFill? Our Model is available for purchase.
Purchasing access to BRIA 2.3 ControlNet GenFill ensures royalty management and full liability for commercial use.
Are you a startup or a student? We encourage you to apply for our specialized Academia and Startup Programs to gain access. These programs are designed to support emerging businesses and academic pursuits with our cutting-edge technology.
Contact us today to unlock the potential of BRIA 2.3 ControlNet GenFill!
By submitting the form above, you agree to BRIA’s Privacy policy and Terms & Conditions.
How To Use
from diffusers import (
AutoencoderKL,
LCMScheduler,
)
from pipeline_controlnet_sd_xl import StableDiffusionXLControlNetPipeline
from controlnet import ControlNetModel, ControlNetConditioningEmbedding
import torch
import numpy as np
from PIL import Image
import requests
import PIL
from io import BytesIO
from torchvision import transforms
import pandas as pd
import os
def resize_image_to_retain_ratio(image):
pixel_number = 1024*1024
granularity_val = 8
ratio = image.size[0] / image.size[1]
width = int((pixel_number * ratio) ** 0.5)
width = width - (width % granularity_val)
height = int(pixel_number / width)
height = height - (height % granularity_val)
image = image.resize((width, height))
return image
def download_image(url):
response = requests.get(url)
return PIL.Image.open(BytesIO(response.content)).convert("RGB")
def get_masked_image(image, image_mask, width, height):
image_mask = image_mask # inpaint area is white
image_mask = image_mask.resize((width, height)) # object to remove is white (1)
image_mask_pil = image_mask
image = np.array(image.convert("RGB")).astype(np.float32) / 255.0
image_mask = np.array(image_mask_pil.convert("L")).astype(np.float32) / 255.0
assert image.shape[0:1] == image_mask.shape[0:1], "image and image_mask must have the same image size"
masked_image_to_present = image.copy()
masked_image_to_present[image_mask > 0.5] = (0.5,0.5,0.5) # set as masked pixel
image[image_mask > 0.5] = 0.5 # set as masked pixel - s.t. will be grey
image = Image.fromarray((image * 255.0).astype(np.uint8))
masked_image_to_present = Image.fromarray((masked_image_to_present * 255.0).astype(np.uint8))
return image, image_mask_pil, masked_image_to_present
image_transforms = transforms.Compose(
[
transforms.ToTensor(),
]
)
default_negative_prompt = "Logo,Watermark,Text,Ugly,Morbid,Extra fingers,Poorly drawn hands,Mutation,Blurry,Extra limbs,Gross proportions,Missing arms,Mutated hands,Long neck,Duplicate,Mutilated,Mutilated hands,Poorly drawn face,Deformed,Bad anatomy,Cloned face,Malformed limbs,Missing legs,Too many fingers"
img_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo.png"
mask_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo_mask.png"
init_image = download_image(img_url).resize((1024, 1024))
mask_image = download_image(mask_url).resize((1024, 1024))
init_image = resize_image_to_retain_ratio(init_image)
width, height = init_image.size
mask_image = mask_image.convert("L").resize(init_image.size)
width, height = init_image.size
# Load, init model
controlnet = ControlNetModel().from_pretrained("briaai/BRIA-2.3-ControlNet-GenFill", torch_dtype=torch.float16)
vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16)
pipe = StableDiffusionXLControlNetPipeline.from_pretrained("briaai/BRIA-2.3", controlnet=controlnet.to(dtype=torch.float16), torch_dtype=torch.float16, vae=vae) #force_zeros_for_empty_prompt=False, # vae=vae)
pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)
pipe.load_lora_weights("briaai/BRIA-2.3-FAST-LORA")
pipe.fuse_lora()
pipe = pipe.to(device="cuda")
# pipe.enable_xformers_memory_efficient_attention()
generator = torch.Generator(device="cuda").manual_seed(123456)
vae = pipe.vae
masked_image, image_mask, masked_image_to_present = get_masked_image(init_image, mask_image, width, height)
masked_image_tensor = image_transforms(masked_image)
masked_image_tensor = (masked_image_tensor - 0.5) / 0.5
masked_image_tensor = masked_image_tensor.unsqueeze(0).to(device="cuda")
control_latents = vae.encode(
masked_image_tensor[:, :3, :, :].to(vae.dtype)
).latent_dist.sample()
control_latents = control_latents * vae.config.scaling_factor
image_mask = np.array(image_mask)[:,:]
mask_tensor = torch.tensor(image_mask, dtype=torch.float32)[None, ...]
# binarize the mask
mask_tensor = torch.where(mask_tensor > 128.0, 255.0, 0)
mask_tensor = mask_tensor / 255.0
mask_tensor = mask_tensor.to(device="cuda")
mask_resized = torch.nn.functional.interpolate(mask_tensor[None, ...], size=(control_latents.shape[2], control_latents.shape[3]), mode='nearest')
masked_image = torch.cat([control_latents, mask_resized], dim=1)
prompt = ""
gen_img = pipe(negative_prompt=default_negative_prompt, prompt=prompt,
controlnet_conditioning_scale=1.0,
num_inference_steps=12,
height=height, width=width,
image = masked_image, # control image
init_image = init_image,
mask_image = mask_tensor,
guidance_scale = 1.2,
generator=generator).images[0]
- Downloads last month
- 11