Participants 2022
Wavy weights and bassy biases
AI Song Contest 2022 / participantsTEAM / Wavy weights and bassy biases
SONG / Upfall
TEAM MEMBERS / Lamtharn “Hanoi” Hantrakul
About the TEAM
Wavy weights and bassy biases consist of eight members: six AI PhD students, an AI BSc student and singer-songwriter Maya Shanti. Although all located in the Netherlands, we originally come from five different countries: Nigeria, Turkey, Italy, Germany and the Netherlands. We are all somehow involved with music, as producers, practitioners, DJs or simply music lovers. This competition brought us together as we share a passion for innovation, creation and tough challenges. For this contest, we combined our expertise - from Deep Learning, Neuroscience and Engineering to Music, Literature and Visual Arts - with cutting-edge AI technology into our song “upfall”. Altogether, we show how AI can be integrated into the creative process of making music, enhancing the artist’s creativity and capabilities by creating something beautiful together, the humans and the machines.
About the SONG
A heart beats fast: mechanically, based on some incontestable rules, which even emotions have to follow. Through active motion, the heart tires itself to energize its dear community of fellow organs. It runs towards its own death, energised with joy and fun.
This, we call an "upfall". It reminds us of the drive to always move up like bubbles enjoying their movement in the air while showcasing the colours around them, despite leading to their own fall that finalizes with their explosion.
We are driven by this kind of upward motion in the world: bringing life to AI through our human interaction with it. The human movement determines where the AI creation goes. Our co-creativity breathes life into AI while leading us humans to our eventual fall by handing in terms of control in the music creation to our fellow contributor AI. This way, we let AI speak for us.
About the HUMAN-AI PROCESS
When starting out, we agreed that we wanted to enhance the creative process and capabilities of the musical artists rather than simply use the “press-one-button” AI solution. After establishing a (preliminary) topic and genre, the team explored the wide range of models and tools with an ultimate preference for those in Magenta Studio VST. The song lyrics were created by singer-songwriter Maya Shanti and GPT-3, a general-purpose text-to-text algorithm. Specifically, she wrote the first verse herself and let GPT-3 complete the second verse. The latter was used in the final song.
To finalize the song, we used an innovation that we are the proudest of that we call “the Conductor”. It started when we wanted to create tools that allow for alternative ways of making music. As such, we created an interface between Ableton Live and Python, and use the Mediapipe collection of pre-trained computer vision algorithms. The Conductor is a specific instance of this framework in which we use 3D hand pose estimation to link parts of our hands to specific controls in the DAW. It takes the name from the fact that, when using it, we swing our hands around directly affecting the music that plays, as if we were conductors leading a digital orchestra. We used the Conductor to apply filters and effects to our song and to adjust mixing and mastering.
Finally, the cover image was created with VQGAN and CLIP and we applied style transfer to our team image.
Lyrics
Maya Shanti:
I’m hanging upside down
And see the world is changing
I don’t know what to think of now
But it feels like I’m fading
So many questions but I know it has always been there
I know, does she know, does he know
I know that I’m upfalling
GPT-3:
I close my eyes and dream
Of a world that's brand new
I hope that one day you'll see
That changes can be good
I can be good, I can be good