Riffusion-Melodiff is simple, but interesting idea, (that I have not seen anywhere else) how to modify your music.
Riffusion-Melodiff is built on a top of Riffusion model, which is fine-tuned Stable Diffusion model to generate Mel Spectrograms. (Spectrogram is kind of visual representation of music by dividing waveforms into frequencies.) Riffusion-Melodiff does not contain new model, there was no new training, nor fine-tunig. It uses the same model as Riffusion only in a different way.
Riffusion-Melodiff uses Img2Img pipeline from Diffusers library to modify images of Mel Spectrograms to produce cover versions of music. Just upload your audio in wav format (if you have audio in a different format, transfer it first to wav by online converter). Then you may use Img2img pipeline from the Diffusers library with your prompt, seed and strength. Stregth parameter decides, how much will modified audio relate to initial audio and how much it will relate to the prompt. When strength is too low the spectrogram is too similar with original one and we do not receive new modification. When strength is too high, then spectrogram is too close to the new promopt, which may cause loss of melody and/or tempo from the base image. Good values of strength are usually about 0,4-0,5.
Good modifications are possible for proper prompt, seed and strength values. Those modifications will keep the tempo and melody from the initial audio, but they will change eg. instrument, playing that melody. Also with this pipeline longer than 5s music modifications are possible. If you cut your audio into 5s pieces and use the same prompt, seed and strength for each modification, generated samples will be somewhat consistent. So if you concatenate them together, you will have longer audio modified.
Quality of the generated music is not amazing, (mediocre, I would say) and it needs a bit of prompt and seed engineering. But it shows one way, how to make cover versions of music in the future.
Colab notebook is included, where you can find step by step, how to do it. Melodiff_v1.
Here are some examples of audio generated by modifying underlying song:
Amazing Grace, originally played by flute, modified to be played by violin