Temporal Multimodal Fusion Network for Real-Time Patient Monitoring in Rehabilitation

Abstract

With an increased prevalence of neurological and musculoskeletal disorders, the real-time monitoring of patients with a significant degree of accuracy and in a fully automated way in a rehabilitation setting is very urgently needed. Existing approaches make use of single-modal data or employ simplistic fusion methods, which do not capture the complex interrelationship between motor function parameters. Most methods are not developed in terms of sufficient temporal resolution to support continuous monitoring for dynamic real-time applications. In this direction, we are presenting a novel deep learning-based framework that integrates multimodal sensor data, such as motion capture, force sensors, and EMG signals, to give real-time insights into the patient's motor function and rehabilitation progress. TMFN will employ LSTM layers with attention mechanisms in order to capture temporal dependencies across sensor modalities and be able to provide precise critical assessments during rehabilitation. This is done by utilizing a temporal fusion mechanism that enables the model to merge short-term performance metrics with long-term trends, thereby providing comprehensive insights into range of motion, joint coordination, and muscle activations. The system offers real-time feedback with a latency of less than 1 second, allowing timely adjustments to therapy protocols. Our approach provides substantial accuracy and efficiency gains and is expected to have assessment accuracy of motor function above 90% compared to clinical evaluations. This work enhances patient outcomes through the facilitation of personalized and adaptive rehabilitation strategies, while at the same time reducing the burden on clinicians by automating complex assessments. TMFN is the first of its kind in establishing a new benchmark for intelligent, multimodal rehabilitation monitoring and is a gateway to more effective and scalable healthcare solutions.