A team led by doctoral student Xin (Cindy) Guo developed an algorithm that can classify the emotions of subjects in photos. Credit: University of Delaware The saying goes that a picture is worth a thousand words. But what if you can't tell what the picture shows? From awkward family photos to class photographs, sometimes it's tricky to tell what the people in the pictures are thinking.
Using machine learning and deep learning with neural networks, a team from the University of Delaware is figuring that out. A team led by doctoral student Xin (Cindy) Guo scored first place in the Group-level Emotion Recognition sub-challenge, one of three sub-challenges in the 6th Emotion Recognition in the Wild (EmotiW 2018) Challenge. Winners were announced at the ACM International Conference on Multimodal Interaction 2018, which was held in October 2018.
Teams were given a set of images picturing a group of people and tasked with developing an algorithm that could classify the people in the photos as happy, neutral or negative. Teams had a month and a half and seven attempts to produce the most accurate algorithm possible. The UD team's winning solution, titled "Group-Level Emotion Recognition using Hybrid Deep Models based on Faces, Scenes, Skeletons and Visual Attentions," will be published by ACM. The group fused eight different models together to develop their winning solution, which works on photographs at a variety of resolutions, blurry to clear.
The goal of such work? To automatically classify images uploaded to websites.
"When people search, they would see the images they are looking for because the algorithm would run and label whether people are happy or not," said Guo. "It could be used to analyze the emotions of a group of people pictured at a protest, a party, a wedding, or a meeting, for example. This technology could also be developed to determine what kind of event a given image shows."
Explore further: A light-weight and accurate deep learning model for audiovisual emotion recognition
More information: Xin Guo et al. Group-Level Emotion Recognition Using Hybrid Deep Models Based on Faces, Scenes, Skeletons and Visual Attentions, Proceedings of the 2018 on International Conference on Multimodal Interaction - ICMI '18 (2018). DOI: 10.1145/3242969.3264990
