license: apache-2.0
pipeline_tag: text-generation
base_model_relation: quantized
quantized_by: Quant-Cartel
base_model: InferenceIllusionist/MilkDropLM-7b-v0.3
tags:
- gguf
- Visualizations
- MilkDrop
⚠️ Epilepsy Warning: This model may generate visuals with flashing lights. Viewer discretion is advised for those with photosensitive epilepsy.
MilkDropLM-7b-v0.3
Update 12/10: NEW! No GPU? No problem! You can now try MilkDropLM Lite on a colab notebook! 🗒️
Background
MilkDropLM-7b-v0.3 is a state-of-the-art language model designed to revolutionize the process of creating visually stunning MilkDrop presets. This model has been trained on a carefully curated dataset of 10,000+ examples encompassing the most exceptional MilkDrop presets available.
What is MilkDrop? It’s a music visualization engine which reacts in real time to the music. MilkDrop has a long 23 year history, has been ported to numerous systems, and a robust community of users that have made over 52,000 original presets. Today, there is a new generation of artists using the NestDrop app within their live concert performances.
What sets MilkDropLM apart is its unique ability to understand and generate the intricate, domain-specific scripting language used in MilkDrop presets. These scripts are frequently a blend of mathematical expressions and equations which traditional programming languages cannot easily replicate. MilkDropLM has overcome this challenge and can produce presets that are both mathematically precise and artistically captivating.
The result is a tool that empowers users to create visuals without having to necessarily know how to write shaders from scratch. Whether you're a programmer or preset author, newcomer or seasoned pro, MilkdropLM offers an intuitive and powerful way to bring your visions to life.
We’ve been grappling with an interesting challenge while working on this project. How do we teach an LLM the various concepts that exist beyond our linguistic frameworks? Since MilkdropLM has been trained on a human curated dataset which connects presets to categories, the code within the presets is in theory linked to artistic concepts that it would otherwise not make by itself.
MilkdropLM has made hidden connections to the code within the presets and likely understands the overall latent space. And yet MilkdropLM is a text based model and therefore cannot experience the resulting rendered visuals. So we’re left with an interesting middle ground where the model has a deep understanding of the preset code and the human can see the thematic connections between the various presets in the dataset, but will the model be able to make these same connections? Can the model be given a complex text description of a scene and create a wholly original preset? How far can the model interpolate and extrapolate within this domain? Can the model convert Shadertoy, Processing, or GLSL code into the MilkDrop format? But this isn’t just a thought experiment, since the results are directly applicable to the VJ community which already widely uses NestDrop for concerts worldwide.
Explore the potential of our early alpha language model and shape its development with your feedback. The variety of potential presets to explore is boundless since there are quadrillions of possible permutations, so jump in and see your experiments visualized in MilkDrop!
Training Methodology
The training process for MilkDropLM v0.3 was thorough, involving a carefully hand curated dataset of 10,000+ high-quality MilkDrop presets. We have invested significant effort in creating a comprehensive taxonomy of MilkDrop presets, which served as the foundation for the training dataset.
To ensure the quality and diversity of the training examples, we applied rigorous criteria to select only the most exceptional presets and also categorize them. The scripts were then transposed into ShareGPT format and paired with prompts which included a category and sub-category, allowing for a more nuanced understanding of the relationships between different elements within the presets.
One of the key challenges in training MilkDropLM was handling the unique syntax and structure of the MilkDrop scripting language. To address this, we trained on top of the nimble yet powerful Qwen-Coder-7b-Instuct model. This allowed us to build from the strong foundation of Qwen’s powerful language and programming capabilities.
Furthermore, to maximize the utility of MilkDropLM, we ensured that the model’s output follows the correct formatting. The presets generated by MilkDropLM are presented in markdown code blocks, making them easy to read, edit, and integrate into various projects. This aspect ensures that users can seamlessly incorporate MilkDropLM into their workflows without any hassle.
Original Model Weights
- BF16 - Full Precision, not required if running inference only
Suggested Workflow
Use your favorite LLM front-end app for inference (such as text-gen-web-ui, llama.cpp, kobold.cpp, LM Studio, etc). Here are the recommended settings for this model:
- Temperature: 0.3 (lower further to improve output at the cost of creativity)
- Top K Sampling: 20
- Top P Sampling: 0.8
- Min P Sampling: 0.05
- Repetition Penalty: 1.05
- Context Length: 32768 (Mininum 16384)
After you've generated a MilkDrop preset, copy-and-paste it into a text file, and then save it using the .milk file format.
Move the .milk file into the presets folder of your Milkdrop app. We recommend using the NestDrop Classic app, which is freely available. (Make sure to close and reopen NestDrop to see your newly added presets.)
Text Prompt Suggestions
Since this model has an understanding of what a MilkDrop preset is, there are numerous ways to ask it to make or edit a preset. One powerful technique is to ask the model to create a preset in one of the categories or subcategories that we’ve trained it to understand. Such as the following text prompt: “Give me a Glowsticks milkdrop preset”. Below is a full list of the available categories and their subcategories:
Dancer /// Aurora, Blobby Mirror, Blobby, Comet Mirror, Comet, Glowsticks Fast, Glowsticks Mirror, Glowsticks, Hatches Mirror, Hatches, Infect Mirror, Infect, Jello Mirror, Lasers, Murky Mirror, Murky, Nexus Mirror, Orbit Mirror, Orbit, Petals, Serpent, Shapes, Spinner Mirror, Spinner, Streamers, Swarm, Tree Branch, Wake Mirror, Wake, Whirl Mirror, Whirl, Wire
Drawing /// Dunes Mirror, Dunes, Explosions Mirror, Explosions, Feedback, Fractal Mirror, Fractal, Glimmer Mirror, Growth Mirror, Growth, Lasers, Liquid Mirror, Liquid, Maze, Rorschach Mirror, Rorschach, Trails Mirror, Trails, Viscera, Whirlpools Mirror, Whirlpools
Fractal /// Blobby, Core Mirror, Core Tunnel, Core, Grid, Horizon Mirror, Lattice Mirror, Lattice, Loops, Mandelbox, Nested Circle, Nested Dancer, Nested Ellipse, Nested Hexagon, Nested Pyramid, Nested Spiral Multiple, Nested Spiral, Nested Square, Nested Triangle, Pointy, Radial, Shine, Sierpinski, Trees, Wave Interference, Wings, Womb
Geometric /// Cathedral, Circles Nested, City, Cube Array, Cube Fly, Cube, Dots, Gears, Honeycomb, Landscape, Monster, Pyramids, Snowflakes, Sphere Array, Sphere Particles, Sphere Wild, Spiral Bounce, Squares Glass, Squares, Stripes Circle, Stripes Dance, Stripes Liquid, Stripes, Symbols, Torus Interior, Triangles Solo, Tunnel Fans, Tunnel Morph, Tunnel Spheres, Wire Circles, Wire Cube Trace, Wire Flower, Wire Grid, Wire Morph, Wire Orbits, Wire Parallel, Wire Sphere, Wire Spiral Flower, Wire Spiral, Wire Torus, Wire Trace Mirror, Wire Trace
Hypnotic /// Illusion Radiate, Illusion, Polar Closeup, Polar Mirror, Polar Rolling, Polar Static, Polar Warp, Radial Warp
Particles /// Blobby, Crystal, Grid, Orbit, Points Fast, Points Trails, Points, Swarm
Reaction /// Automata, Cloudy, Contagion, Crystalize, Dunes, Feedback, Growth, Liquid Blobby, Liquid Closeup, Liquid Gradient, Liquid Ripples, Liquid Simmering, Liquid Windy, Luma Mirror, Maze, Mountains, Rorschach, Viscera, Whirlpools, Windy
Sparkle /// Explosions, Glimmer Mirror, Glimmer Tunnel, Glimmer, Jewel, Mass Circles, Mass Squares, Mass Stars, Mass Triangles, Squares
Supernova /// Burst, Gas, Lasers, Orbits, Radiate, Shimmer, Stars
Waveform /// Spectrum, Wire Circular, Wire Flat Double, Wire Flat, Wire Flower, Wire Mirror, Wire Rising, Wire Spirograph, Wire Tangle, Wire Tunnel
Examples of Generated Presets
Here are some curated examples that we generated using MilkDropLM v0.3. We used vLLM to do a batch render and output 3,176 presets. This was then hand curated to the best 130 presets. While this may seem like a large ratio (1 out of every 24) we argue that it’s a very good result. Additionally, since ISOSCELES curated the original dataset and has therefore previously viewed every preset within the training data, he can attest that each of these examples exhibit a unique aspect.
Some of the examples feel like a classic preset mashup and a few feel original. We have only scratched the surface of what’s possible.
Version History
- v0.01: Closed alpha release. Basic ability to draw shapes and waves.
- v0.02: Closed alpha release. Improved grasp of geometric elements, better variety and creativity in output.
- v0.3: Public alpha release. Optimized training hyperparameters & dataset, even more variety and creativity, better formatting. MilkDropLM can now use warp & comp shaders.
Alpha Release Notice
MilkDropLM v0.3 represents an exciting step forward in breathing fresh life into the 23 year history of MilkDrop. We are thrilled to share this powerful tool with you, but we want to emphasize that it is currently in the alpha stage of development. Hence this model likely has some rough edges and limitations, but we are committed to continuous improvement and welcome your feedback to help refine and enhance this model. With each iteration we aim to make MilkDropLM more user-friendly, reliable, and capable of producing previously unseen visuals.
We encourage users to approach MilkDropLM with a spirit of experimentation and discovery. While not all presets may be perfect, the potential for creating new visuals is immense. At this stage we recommend batch rendering out a few hundred presets so that you can get a better sense of what’s possible with this model.
Please don't hesitate to reach out with your thoughts and ideas. You can submit any feedback through community posts. Or, if you’ve created some interesting presets that you’d like to share then please head over to the Reddit community.
Acknowledgements
This project is the result of a collaboration between ISOSCELES and InferenceIllusionist. This was a unique meeting of minds since ISOSCELES brought his MilkDrop preset knowledge and experience in helping develop NestDrop for the VJ community, and InferenceIllusionist brought his vital experience in fine-tuning and quantizing LLMs. We stand on the shoulders of the many Milkdrop authors which have freely released their original presets for everyone to enjoy. Much respect!