Combining modality specific deep neural networks for emotion recognition in video
Combining modality specific deep neural networks for emotion recognition in video
Kahou, Samira Ebrahimi and Pal, Christopher and Bouthillier, Xavier and Froumenty, Pierre and Gülçehre, Çaǧlar and Memisevic, Roland and Vincent, Pascal and Courville, Aaron and Bengio, Yoshua and Ferrari, Raul Chandias and Mirza, Mehdi and Jean, Sébastien and Carrier, Pierre Luc and Dauphin, Yann and Boulanger-Lewandowski, Nicolas and Aggarwal, Abhishek and Zumer, Jeremie and Lamblin, Pascal and Raymond, Jean Philippe and Desjardins, Guillaume and Pascanu, Razvan and Warde-Farley, David and Torabi, Atousa and Sharma, Arjun and Bengio, Emmanuel and Konda, Kishore Reddy and Wu, Zhenzhou
ICMI 2013 – Proceedings of the 2013 ACM International Conference on Multimodal Interaction 2013
Abstract : In this paper we present the techniques used for the University of Montréal’s team submissions to the 2013 Emotion Recognition in the Wild Challenge. The challenge is to classify the emotions expressed by the primary human subject in short video clips extracted from feature length movies. This involves the analysis of video clips of acted scenes lasting approximately one-two seconds, including the audio track which may contain human voices as well as background music. Our approach combines multiple deep neural networks for different data modalities, including: (1) a deep convolutional neural network for the analysis of facial expressions within video frames; (2) a deep belief net to capture audio information; (3) a deep autoencoder to model the spatio-temporal information produced by the human actions depicted within the entire scene; and (4) a shallow network architecture focused on extracted features of the mouth of the primary human subject in the scene. We discuss each of these techniques, their performance characteristics and different strategies to aggregate their predictions. Our best single model was a convolutional neural network trained to predict emotions from static frames using two large data sets, the Toronto Face Database and our own set of faces images harvested from Google image search, followed by a per frame aggregation strategy that used the challenge training data. This yielded a test set accuracy of 35.58%. Using our best strategy for aggregating our top performing models into a single predictor we were able to produce an accuracy of 41.03% on the challenge test set. These compare favorably to the challenge baseline test set accuracy of 27.56%. © 2013 ACM.
