Concert Creator

Concert Creator turns audio recordings into virtual concert performances and piano lessons. It analyzes an uploaded audio file and produces a hyper-realistic video performance synced to the notes in that recording, intended for people learning to play piano or creating piano animation content from their own audio, or for musicians who want a video rendition of a performance to share.

The system is trained on recordings of professional musicians so that the generated motion looks like a real performance rather than a mechanical animation. Users can adjust fingering technique, hand separation, and the amount of force applied to each note, and Concert Creator regenerates the performance to reflect those preferences. Camera angles, key colors and lighting effects can also be changed.

Concert Creator lets users choose and customize avatars to represent themselves in the generated video, including an avatar import feature. Generated performances can be adjusted across camera angles, key colors and lighting effects, and separately through fingering technique, hand separation and the force applied to each note, with the video regenerated to match the updated settings.

Use cases

• Learning to play piano through generated animations
• Creating virtual concert performances and music lessons from audio