Starting with a music video in mind, we worked backwards to the music, considering what style would match each scene. We love the challenge of multi-genre music. The dense and varied production requires much nuance, particularly when crafting meaningful transitions. Layers of audio/MIDI assets were created using generative models, then arranged in Ableton Live. Some models were pre-trained and some were trained on new custom datasets. Most samples were generated using a new multi-genre audio diffusion model we trained at Harmonai. It is exceptionally good at primal drumming which opens the song. 

For this model I built a softmask infilling algorithm that helps me craft smooth transitions between two different genres. I designed several different algorithms and eventually settled on this one as best. You can hear it at [1:20] nicely transitioning from Stride piano and into Children’s xylophone music. For chanting we trained three custom SVC (singing voice conversion) models. Each of us (CJ, Zack, and PortraitXO) recorded 30-60 minute datasets to train them. We then ran the diffusion drumming through the SVC model. Even though SVC wasn’t built for that purpose, it works really well at adding chants that follow the syncopation of the drums.

The lyrics come in [1:30 - 1:45] over blast beats. I sing this part, but the words come from the ChatGPT prompt “Write metal lyrics about bands from multiple music genres being grinded up and fed into a giant mouth” whose output I cherry-picked from and edited. As the MEGABRAIN is worshipped, we hear four-part harmonies generated with Coconet trained on Bach MIDI, synthesized using Kontakt’s All Saints Choir. For the ending, we trained a custom Dance Diffusion model on Exjaynine’s library of hardcore bass kicks she designs. At [2:35] you hear our favorite artifact from this model: a screaming squelchy bass. Layering this with metal breakdowns from MusicGen, with muddiness filtered out, then adding in a pure subbass sine wave, makes for a powerful sound. 

Finally, our Diffusion model isn’t just for music, it does sound effects too. The song closes out with the sound of six layers of glass shattering. In conclusion, the rapid inference speed of the Diffusion generator enabled us to stay in flow state as we were crafting and curating the best samples for each section. It is gratifying to make multi-genre music this way.