Inspired by a couple of singers, NVIDIA Founder and CEO Jensen Huang tasked his team to produce a video of his avatar Toy Jensen (TJ) singing Jingle Bells — using NVIDIA technology, of course.
Huang chanced upon Lah Yileh Lee and Xinting Lee, who often stream their performances online, performing in a public square in Taipei during his recent trip to Taipei.
He then got his creative team to develop a video using core technologies that are part of the NVIDIA Omniverse Avatar Cloud Engine (ACE) platform. Omniverse ACE is a collection of cloud-native AI microservices and workflows for developers to easily build, customise and deploy engaging and interactive avatars.
NVIDIA’s team used a recently developed voice conversion model to extract the voice of a professional singer from a sample provided by them and turn it into TJ’s voice. They used the musical notes from that sample and applied them to the digital voice of TJ to make the avatar sing the same notes and with the same rhythm as the original singer.
Omniverse ACE along with Omniverse Audio2Face and Audio2Gesture technologies are used to generate realistic facial expressions and body movements for the animated performance based on TJ’s audio alone.
They even collaborated with Jochem van der Saag, a composer and producer who has worked with Michael Bublé and David Foster, to create the perfect backing track for TJ to sing along to.
The entire process of turning the music track into a polished video took just hours.
The result is a unique festive video by Huang’s avatar. Catch it here.
Entelechy Asia 