Comprehensive approach for solving multimodal data analysis problems based on integration of evolutionary, neural and deep neural network algorithms

[摘要] In this work we propose the comprehensive approach for solving multimodal data analysis problems. This approach involves multimodal data fusion techniques, multi-objective approach to feature selection and neural network ensemble optimization, as well as convolutional neural networks trained with hybrid learning algorithm that includes consecutive use of the genetic optimization algorithm and the back-propagation algorithm. This approach is aimed at using different available channels of information and fusing them at data-level and decision-level. The proposed approach was tested on the emotion recognition problem. SAVEE database was used as the raw input data, containing visual markers dataset, audio features dataset, and the combined audio-visual dataset. The best emotion recognition accuracy achieved with the proposed approach on visual markers data is 65.8%, on audio features data - 52.3%, on audio-visual data - 71%.

[发布机构] Reshetnev Siberian State University of Science and Technology, 31 Krasnoyarsky Rabochy ave., Krasnoyarsk; 660037, Russia^1

[效力级别] 工业技术

[关键词] Convolutional neural network;Emotion recognition;Genetic optimization algorithm;Hybrid learning algorithm;Multimodal data analysis;Multimodal data fusion;Neural network algorithm;Neural network ensembles