Freepik

/

flux.1-lite-8B-alpha

like 382

Follow

freepik 136

Yes, some lora's don't work well with this model. Seems to either have little effect or none. Especially when using anatomy or art style loras.

ah makes sense, the loras that werent having an effect were indeed style like loras (loras for realism). Hopefully someone finds a way to fix because this model is exactly what i needed but without good lora workage doesnt have much use for me

It might be that due to parameter difference and how this model was trained other flux loras may not be compatible. I would say, if true, this would be a huge downside of this model as it voids all other flux loras.

@swwww55 @Vigilence Can you provide me any lora model that is not working to check if something can be done?

@dverdu-freepik Here is one of the anatomy Lora’s that I tested, and noticed it not having the same effect as it would the regular dev or similar dev derived models.
https://huggingface.co/enhanceaiteam/Flux-Uncensored-V2

The other lora I was testing was a personal impressionist painting lora. It has a major effect on every model I tried as it’s derived from a finetune, but not so much here, Like if I turned down the strength to 0.25 or something. I tried increasing the lora strength but that does nothing but fry the image.

@dverdu-freepik https://civitai.com/models/631986/xlabs-flux-realism-lora
https://civitai.com/models/738556/iphone-photo-fluxsd35-realism-booster

These were two that didnt work at all for me

And yes @Vigilence I really hope that to not be the case and/or that it can actually be fixed ! :(

The distillation recipe involves pruning blocks 5-15 and fine-tuning block 4 to produce the output of block 15. The rest of the original model's blocks remain identical. This means that LoRAs cannot be applied as-is because 11 MMDiT blocks are just gone, and the ones after that need to be renumbered. The block ids mapping from Flux Dev to Lite would roughly be:

Original MMDiT 0-4 -> 0-4 in Lite
Original MMDiT 5-15 -> can't map to Lite as they were pruned! best option is probably throw them away
Original MMDiT 16-18 -> now 5-7 in Lite
Original DiT 0-37 -> 0-37 in Lite.

Best option overall is to retrain the LoRAs.

However, here is a short script that makes a best-effort attempt to convert a Flux LoRA to Flux Lite following the mapping above:

import re
import torch
from safetensors.torch import load_file, save_file

d = load_file("YOUR_FLUX_DEV_LORA.safetensors")
d_lite = {}
for k in d.keys():
    if (match := re.match(r'(transformer\.transformer_blocks\.)(\d+)(.*)', k)):
        idx = int(match.groups()[1])
        if 5 <= idx <= 15:
            continue
        if idx >= 16:
            idx -= 11
        d_lite[f"{match.groups()[0]}{idx}{match.groups()[2]}"] = d[k]
    else:
        d_lite[k] = d[k]
save_file(d_lite, "YOUR_FLUX_LITE_LORA.safetensors")

Please try it out and let us know if it helps!

The distillation recipe involves pruning blocks 5-15 and fine-tuning block 4 to produce the output of block 15. The rest of the original model's blocks remain identical. This means that LoRAs cannot be applied as-is because 11 MMDiT blocks are just gone, and the ones after that need to be renumbered. The block ids mapping from Flux Dev to Lite would roughly be:

Original MMDiT 0-4 -> 0-4 in Lite
Original MMDiT 5-15 -> can't map to Lite as they were pruned! best option is probably throw them away
Original MMDiT 16-18 -> now 5-7 in Lite
Original DiT 0-37 -> 0-37 in Lite.

Best option overall is to retrain the LoRAs.

However, here is a short script that makes a best-effort attempt to convert a Flux LoRA to Flux Lite following the mapping above:

import re
import torch
from safetensors.torch import load_file, save_file

d = load_file("YOUR_FLUX_DEV_LORA.safetensors")
d_lite = {}
for k in d.keys():
    if (match := re.match(r'(transformer\.transformer_blocks\.)(\d+)(.*)', k)):
        idx = int(match.groups()[1])
        if 5 <= idx <= 15:
            continue
        if idx >= 16:
            idx -= 11
        d_lite[f"{match.groups()[0]}{idx}{match.groups()[2]}"] = d[k]
    else:
        d_lite[k] = d[k]
save_file(d_lite, "YOUR_FLUX_LITE_LORA.safetensors")

Please try it out and let us know if it helps!

Can this also be used to convert a regular flux model to a flux lite model? I would like to experiment with merging different flux lite models together, or merge flux lite with different loras.

I tried the script and did notice a big improvement over the original lora. However, it seems the script doesn't work on loras extracted from finetunes. Can anyone confirm if this is correct?

Can this also be used to convert a regular flux model to a flux lite model?

There is a noticeable quality drop if you just drop 5-15, which the fine-tuning of 4 attempts to bridge to a good extent.

But for merging purposes, yes: dropping 5-15 before merging should be better than not doing it.

Can this also be used to convert a regular flux model to a flux lite model?

There is a noticeable quality drop if you just drop 5-15, which the fine-tuning of 4 attempts to bridge to a good extent.

But for merging purposes, yes: dropping 5-15 before merging should be better than not doing it.

I tried using the script on the flux dev model and the file size remained the same. Does that mean the script didn't work?

I also tried on a rank 256 finetune extracted lora, and the file size also remained the same.

The anatomy lora I tried however, was successful and reduced by around 20%.

Does that mean the script didn't work?

Right. Full checkpoint keys appear to follow a different naming convention, try this one instead:

d = load_file("YOUR_FLUX.safetensors")
d_lite = {}
for k in d.keys():
    if (match := re.match(r'(double_blocks\.)(\d+)(.*)', k)):
        idx = int(match.groups()[1])
        if 5 <= idx <= 15:
            continue
        if idx >= 16:
            idx -= 11
        d_lite[f"{match.groups()[0]}{idx}{match.groups()[2]}"] = d[k]
    else:
        d_lite[k] = d[k]
save_file(d_lite, "PRUNED_FLUX.safetensors")

In general, take a look at d.keys() and tweak the regex to match the MMDiT blocks (aka double_blocks or transformer_blocks)...

Would it be possible to convert dev controlnets to work with this model?
For example, this one. https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha

Would it be possible to convert dev controlnets to work with this model?
For example, this one. https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha

I got really bad results with the alimama controlnet, i was devestated :(