Sonic Mutations

Sydney Opera House | 2023

Sonic Mutations is a personalized AI instrument showcased by musicians Alexis Weaver and Salllvage at the Sydney Opera House. I researched and developed the AI models and built the UI to integrate this technology into live performances.

Choosing from pre-selected prompts in the interface.

Adjusting the level of noise in the interface.

A panel of clips generating in the interface.

Project Details

Team

Kopi Su Studio

Role

Creative Technologist

AI Researcher

Full Stack Developer

Supporting Creativity with Generative AI

Sonic Mutations was a project commissioned by the Sydney Opera House to explore how artists could use generative-AI in live performances. Researching the landscape of music generation, I sought models that would allow the musicians to explore. Riffusion, a diffusion-based image model that transforms spectrograms into audio, emerged as the optimal choice by testing multiple open-source options. By modifying the pipeline of Riffusion, we could, in real-time, combine input sounds, prompts, and degrees of transformation into new audio that the artists were excited to play with.

The process to take recorded input audio and generate new audio.

Crafting Personalized Models

To truly support artists in their music production, we understood the tool needed to generate audio that aligned with each artist's style. Collaborating closely with the artists, we focused on transforming it into a personalized creative tool by finetuning the model using LoRA. We experimented with training the model on a diverse range of sounds, from their previous music and singing to recordings of Australian wildlife and rocks. This iterative and exploratory process resulted in bespoke models for each artist, allowing them to express their unique musical vision through a truly customized instrument.

The "Skype" sound effect before and after being run through a model trained on clips of Alexis Weaver's voice harmonies.

From Experimentation to Live Performance

With many options for interaction, we explored ways the artists could incorporate generating music into live performances. After testing multiple iterations, the final interface enabled artists to upload or record audio, experiment with various prompts, save their preferred settings, generate multiple sounds in a row, and seamlessly integrate the generated audio into their performance software through batch downloading and live playback/looping features. We designed these approaches to make it as fast and intuitive as possible for the artists to incorporate the tool into their workflow.

The resulting tool was used in performances at the Sydney Opera House "Outlines Festival", and SXSW Sydney, and is now being developed further.

Two performers standing in front of a colorful stage ready to play music