Create koikatsu_animation_workflow.md

koikatsu_animation_workflow.md

@@ -0,0 +1,47 @@
+This is my second attempt in my quest for consistent animation optimization that I thought it was worth to share this time.
+
+It directly uses computed depth frames from a 3D motion here, which means clean depth, allowing qualitative character swap. This approach is different from real-to-anime img2img chick videos. So there is no video reference. Good thing is it avoids the EBSynth hassle. Also VERY few manual aberration correction.
+
+The workflow is a bit special since it uses the Koikatsu h-game studio. I guess Blender works too. But this "studio" is perfect for 3D character and pose/scene customization with awesome community and plugins (like depth). The truth is I have more skills in Koikatsu than in Blender (shhh~).
+
+Here is the workflow, and I probably need some advice from you to optimize it:
+
+KOIKATSU STUDIO
+
+Once satisfied with the motion (can be MMD), extract the depth sequence, 15fps, 544x960
+
+STABLE DIFFUSION
+
+2. Use an anime consistent model and LorA
+
+3. t2i : Generate the reference picture with one of the first depth frame
+
+4. i2i : Using Multi-Control Net a. Batch depth with no pre-processor b. Reference with the reference pic generated in 2. c. TemporalKit starting with the reference pic generated in 2.
+
+POST PROCESS
+
+5. FILM interpolation (x2 frames)
+
+6. Optionnal : Upscale x2 (Anime6B)
+
+7. FFMPEG to build the video (30fps)
+
+8. Optionnal : Deflicker with Adobe
+
+NB :
+
+Well known animes are usually rendered at low fps, so I wouldn't overkill it at 60fps to keep the same anime feeling (+ it would take ages to process each step, and also randomly supported by socials apps like TikTok)
+
+Short hair + tight clothes are our friends
+
+Good consistency even without Deflicker
+
+Depth is better than Openpose to keep hair/clothes physics
+
+TO IMPROVE :
+
+- Hands gestures are still awful even with the TI negatives
+
+- Background consistency by processing the character separately and efficiently
+
+Hope you enjoy it, all this gives me new ideas...