Multimodal AI-Based Frustration and Escalation Risk Prediction in Banking Call Centres Using Voice Prosody, Text Sentiment, and Interaction Metadata

Abstract

This study proposes a multimodal approach to the prediction of customer frustration and escalation risk in call centres of financial institutions based on voice (prosody) and text (sentiment) as well as interaction (metadata). Low, medium and high frustration class es were generated using the IEMOCAP dataset, and various machine-learning models were tested on it. These results indicated that the XGBoost model with Audio, Text and Metadata performed best with 85.7% accuracy, 85.7% F1-score and 0.961 ROC-AUC. There was generally a large amount of predictive information compared with audio and metadata features. Results indicate that multimodal fusion could significantly contribute to the earlier and more accurate identification of the customers' dissatisfaction and thus to the proactive customer service intervention.