Analysis of Deep-Learning-Based Emotion Predictors with Multi-Channel Multi-Label EEG Signals and SHAP XAI

Abstract

Human emotion recognition is a peculiar task. Humans express emotions or provide emotional responses via facial gestures, body temperature, and brain activity. Interestingly, brain activities can be observed via EEG recordings. Analysis of an individual's emotional state to stimulations such as video, music, or activity is vital to their behaviour. Deep learning (DL) models are popular and influential enough to predict emotions from EEG signals. Mapping predictions to different EEG channels or features would be critical to further understanding human behaviour. The study in this paper presents an analysis of various deep-learning model performances in predictions of emotions with multi-channel, multi-label EEG signals. In the context of DL, convolutional and recurrent deep neural network models are utilized for emotion recognition. The synergistic use of CNNs and RNNs to extract temporal, spatial, and spectral features from multi-modal physiological data is especially considered here. The DEAP dataset, being a rich source of multi-modal physiological signal representation data, is therefore used in this study. The DEAP data encapsulates a range of stimulated human emotions and is suitable for this study. Most importantly, emotion predictions from proposed DL models on the DEAP dataset are analyzed within an XAI framework. SHAP XAI framework is used to interpret the predictions from DL models and its mapping onto input different physiological signals within the DEAP dataset. Results from DL models indicate improved emotion recognition permeance and SHAP values from the XAI framework indicate the significance of the DL model architecture and its features in achieving this performance.