Deep multimodal fusion: Combining discrete events and continuous signals

ICMI 2014 - Proceedings of the 2014 International Conference on Multimodal Interaction

Volumn , Issue , 2014, Pages 34-41

  Martínez, Héctor P ^a   Yannakakis, Georgios N ^a  

^a UNIVERSITY OF MALTA   (Malta)

Author keywords

Auto encoders; Behaviour; Convolutional neural networks; Deep learning; Multimodal fusion; Physiology; Pooling method; Sequence classification; Sequence fusion

Indexed keywords

CONVOLUTION; INTERACTIVE COMPUTER SYSTEMS; NEURAL NETWORKS; PHYSIOLOGY; VIDEO RECORDING;

AUTO ENCODERS; BEHAVIOUR; CONVOLUTIONAL NEURAL NETWORK; DEEP LEARNING; MULTI-MODAL FUSION; POOLING METHOD; SEQUENCE CLASSIFICATION;

BEHAVIORAL RESEARCH;

EID: 84939496406     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2663204.2663236     Document Type: Conference Paper

References (21)

1. L. Barrett, B. Mesquita, K. Ochsner, and J. Gross. The experience of emotion. Annual review of psychology, 58:373, 2007.
2. V. E. Farrugia, H. P. Martinez, and G. N. Yannakakis. The preference learning toolbox. Technical Report IDG-2014-01, Institute of Digital Games, University of Malta, 2014.
3. V. Furnkranz and E. Hullermeier. Preference learning. Springer, 2010.
4. G. Hinton and R. Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 313(5786):504-507, 2006.
5. G. E. Hinton and R. S. Zemel. Autoencoders, minimum description length, and helmholtz free energy. In Advances in neural information processing systems (NIPS), 1994.
6. A. Jain and D. Zongker. Feature selection: Evaluation, application, and small sample performance. IEEE transactions on pattern analysis and machine intelligence, 19(2):153-158, 1997.
7. W. Krzanowski. The performance of fisher's linear discriminant function under non-optimal conditions. Technometrics, 19(2):191-200, 1977.
8. Y. LeCun and Y. Bengio. Convolutional networks for images, speech, and time series. In The handbook of brain theory and neural networks, volume 3361. Cambridge, MA: MIT Press, 1995.
9. H. P. Martinez, Y. Bengio, and G. N. Yannakakis. Learning deep physiological models of affect. Computational Intelligence Magazine, IEEE, 9(1):20-33, 2013.
10. H. P. Martinez and G. N. Yannakakis. Genetic search feature selection for affective modeling: A case study on reported preferences. In Proceedings of international workshop on Affective interaction in natural environments (AFFINE), pages 15-20. ACM, 2010.
11. H. P. Martinez and G. N. Yannakakis. Mining multimodal sequential patterns: A case study on affect detection. In Proceedings of International Conference on Multimodal Interfaces (ICMI), pages 3-10. ACM, 2011.
12. S. Mcquiggan, B. Mott, and J. Lester. Modeling self-efficacy in intelligent tutoring systems: An inductive approach. User Modeling and User-Adapted Interaction, 18(1):81-123, 2008.
13. J. Ngiam, A. Khosla, M. Kim, J. Nam, H. Lee, and A. Y. Ng. Multimodal deep learning. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pages 689-696, 2011.
14. P. O'Connor, D. Neil, S.-C. Liu, T. Delbruck, and M. Pfeiffer. Real-time classification and sensor fusion with a spiking deep belief network. Frontiers in neuroscience, 7, 2013.
15. T. Pahikkala, E. Tsivtsivadze, A. Airola, J. Jarvinen, and J. Boberg. An efficient algorithm for learning to rank from preference graphs. Machine Learning, 75(1):129-165, 2009.
16. M. Pantic and L. Rothkrantz. Toward an affect-sensitive multimodal human-computer interaction. Proceedings of the IEEE, 91(9):1370-1390, 2003.
17. S. Rifai, Y. Bengio, A. Courville, P. Vincent, and M. Mirza. Disentangling factors of variation for facial expression recognition. In Proceedings of European Conference on Computer Vision (ECCV), 2012.
18. D. Rumelhart. Backpropagation: Theory, architectures, and applications. Lawrence Erlbaum, 1995.
19. N. Srivastava and R. Salakhutdinov. Multimodal learning with deep boltzmann machines. In F. Pereira, C. Burges, L. Bottou, and K. Weinberger, editors, Advances in Neural Information Processing Systems 25, pages 2222-2230. Curran Associates, Inc., 2012.
20. S. Wold, K. Esbensen, and P. Geladi. Principal component analysis. Chemometrics and intelligent laboratory systems, 2(1):37-52, 1987.
21. G. Yannakakis, H. Martinez, and A. Jhala. Towards affective camera control in games. User Modeling and User-Adapted Interaction, 20(4):313-340, 2010.