Professor Greg Mori from SFU's School of Computing Science is part of a research team working with Disney Research to develop a new way to assess and predict the facial expressions of movie-goers. Credit: SFU Researchers in SFU's School of Computing Science have been working with Disney Research to develop a new way to assess and predict the facial expressions of movie goers. This method could help to make artificial data created in animation look more realistic.
The method uses factorized variational autoencoders—or FVAEs, computer algorithms that automatically analyze patterns in data. The researchers found they demonstrate "a surprising ability" to reliably predict viewers' facial expressions for the remainder of the movie after just a few minutes of observation.
SFU computing science professor Greg Mori has collaborated for nearly a decade with Disney Research, where six of his students have also had internships. Disney has provided nearly $300,000 in funding towards his research.
In this latest project, SFU researchers, working with others from the California Institute of Technology and Disney Research, applied FVAEs to as many as 150 showings of nine mainstream movies, including Star Wars: The Force Awakens, The Jungle Book and Big Hero 6. The films were shown in a 400-seat theatre housed with infrared cameras to monitor audience faces.
The data collected from 3,179 audience members resulted in 16 million facial 'landmarks' to be evaluated.
"Our focus is on specific landmarks such as the corner of the mouth, or the blink of an eye," says Mori, who specializes in computer vision in SFU's Vision and Media Lab. His interest is in developing algorithms that interpret images and videos, particularly those containing people.
"So we are tracking facial expressions such as frowns or smiles to get a range of structures, based on their reactions."
According to colleagues at Disney Research, FVAEs sought out audience members who exhibited similar facial expressions through the entire movie. They work much like systems used for online shopping that suggest new products based on previous purchases.
FVAEs are capable of "learning" a set of stereotypical reactions from the entire audience, and from the various facial expressions, and can then determine how audience members should react to a particular movie. These attributes help FVAEs learn more effectively than other systems and enable them to predict facial expressions over an entire movie after just a few minutes of observing the audience.
Disney Research aims to pursue scientific and technological innovation that will advance the company's media and entertainment efforts. Mori and PhD student Zhiwei Deng shared their research at a recent IEEE Conference on Computer Vision and Pattern Recognition in Hawaii.
Explore further: Neural nets model audience reactions to movies
