A.I. Researchers of Facebook Figured out How to Make Avatars Appear Like They Are Playing Music

    Photo by Brian Solis/Flickr

    The researchers on artificial intelligence have come up with a novel way of making cartoonish avatars appear like they are actually playing musical instruments. The work could lead to fascinating virtual reality or augmented reality experiences in the future.

    In its new paper, Facebook research scientists Ira Kemelmacher-Shlizerman and Eli Shlizerman and collaborators Hayden Schoen and Lucio Dery discussed about how they trained AI systems using YouTube videos of piano and violin recitals. They then used the trained systems in order to make avatars move their fingers and hands with fake instruments based solely on audio recordings, with the help of the ARKit AR software for developers of Apple.

    Training machines to understand how humans move is an active area of research in AI, and it is one that has been explored by various researchers at other technology companies, including Microsoft and Google. Generally, however, that needs video feeds. The achievement here was to attempt to act on audio alone — even if the results were not perfectly realistic.

    The researchers wrote: “We believe the correlation between audio to the human body is very promising for a variety of applications in VR/AR and recognition.”

    Facebook proceeds to push VR through its Oculus branch, including with the Spaces app that renders users with avatars. Earlier this month, the company offered developers new tools in order to build AR features for the apps of Facebook.

    The researchers think that they could improve the accuracy of the avatars in the future by tapping MIDI files from people that are playing music, or data from sensors that are hooked up to musicians.