aaronb

/

dreamshaper-8-dmd-48k

like 0

Hey there -- I stumbled upon this DMD weight not too long ago and it's quite fast, and looks fantastic. Only problem is I cannot for the life of me get ControlNet working with it. Seems like ControlNet Openpose barely effects it (the image just wiggles the character a bit in the realtime stream I am rendering instead of following the pose like it needs to). I am using diffusers and the regular StableDiffusionControlNetPipeline, and loading the model like so:

    controlnet = ControlNetModel.from_pretrained("lllyasviel/control_v11p_sd15_openpose", torch_dtype=torch.float16, use_safetensors=True)
    unet = UNet2DConditionModel.from_pretrained('local_huggingface_checkpoints/dreamshaper-8-dmd-48k', torch_dtype=torch.float16)
    
    self.pipe = StableDiffusionControlNetPipeline.from_pretrained(
        "XCLiu/instaflow_0_9B_from_sd_1_5", 
        unet=unet,
        safety_checker=None, 
        requires_safety_checker=None, 
        torch_dtype=torch.float16,
        controlnet=controlnet
        )
    

I am running it at 1 step with instaflow and using the PerFlow scheduler. If I could just get the controlnet to fully take effect this would be incredibly useful. Excellent work on this model btw, and thanks in advance.