Quick questions
I haven't had a chance to test this yet, but assuming it works, so nice work :)
My question is, is it only the Unet that needs replacement, or were there any censorship mechanisms in-place in the text encoders or vae as well?
Just wondering if we can still use the existing clip models and vae, or if any of them need replacement along with the unet.
Thanks.
No specific subarchitecture within Flux was targeted. Rather, we had a vague idea of the residual streams from the architecture posted on X. We mostly focused on which parameters had to undergo permanent weight orthogonalization by inferencing on a test dataset of harmless and harmful prompts. Regarding which parameters, I can't tell as of now. It seems some parameters across the entire model underwent change.
So to answer your question, no we didn't tamper with the Unet or any specific encoder/decoder.
You would have to use the entire model as a whole, not just its modular components (Unet, Enc or Dec).
Hope that clears your doubt.