ABOUT THE TEAM

Everything started when I, Tomas Nihlén, a Swede living in Catalonia, with a background in tech + creativity, received a question if I wanted to start a Catalan team to participate in the AI Song Contest. I wrote a tweet and started an intensive search for the right people to join the project. I have been learning Catalan using Twitter as a way to get help, meet friendly people and in so building a very personal relationship with the Catalan community. The response blew my mind and everything was moving really from the start. Software engineers, writers, AI evangelists, philosophers, visual artists, 3D animators, song writers, machine learning engineers, psychologists, composers and singers, journalists, all sort of profiles were eager to join this project. Neither of us really knew each other and most us had never met before but we got to know each other over Slack and Zoom during the following 7 weeks.

ABOUT THE SONG

The song is a hopeful dialogue (not a love story) between a human and an AI based on admiration and respect. We wanted to avoid crossed readings about gender, sex, romanticism, and other clichés. The story begins in a phase of curiosity between the human and AI, transitions into a dialogue and ends in a symbiosis. The song features two female human voices and two synthetic voices (the latter appears in two different places, like a mix between voices and instruments). We have made four different language versions (English, Catalan, English-Catalan, Catalan-English), in the main version for the competition the lyrics are mainly in English with the two final choruses in Catalan.

ABOUT THE HUMAN-AI CO-CREATION PROCESS

As we didn’t have time to develop on-purpose AI tools, we decided to develop the scripts for managing MIDI files and text files to interact via API with ML models. We took the approach of the dialogue and we applied it to all four areas:

Lyrics. We opened a challenge to collectively create stanzas of 4 verses. We fed them into the GPT-3 model to generate new stanzas. GPT-3 is a very powerful model, but very far from human linguistic capacity. So we needed to apply filters to the outputs and undergo a cherry picking of the model outputs to build the final lyrics.

Music. We used AI-tools at different points: Generation, Continuation and Accompaniment. The tools used: Magenta Music Transformer, Magenta RNN and Magenta MusicVAE; we even implemented a genetic algorithm for the generation but, for a matter of time, was not included into the final version. We used these tools for dialogue. Starting with an initial suggestion by the musicians we explored the outputs of the models and played around in iterative loops.

Visuals. We used DeepMotion’s Animate 3D to reconstruct the best possible 3D motion perceived from 2D video. For enlarging the resolution of the final shot, we used the services of www.deep-image.ai, and also to add some extra effects to the dancers, we used EbSynth App. Synthetic voices. We generated four voice layers that, combined, recreated the voice effect. This was done 100% digitally with AI algorithms and text embeddings (using the library midi2voice).