Occlusion-Robust Pose Sequence Aware GAN based Classification using CNN for Early Detection of Cerebral Palsy in Infants

Abstract

Early detection and diagnosis of Cerebral Palsy (CP) are crucial for minimizing its impact. Previous studies have used pose estimation techniques called the OpenPose for CP detection, but these methods have limitations in annotating large infant movement datasets. To address this, a new method called Pose Sequence-aware Generative Adversarial Network (PS-GAN) has been developed to create high-quality skeleton images, followed by the Convolutional Neural Network (CNN) with softmax classifier for CP detection. However, pose estimation techniques can still have recognition errors in the left upper limbs since the left upper limbs have extra movements and a few movements were rigorously obstructed with the torso segment. This results in missing values in the Feature Matrix (FM) used for CP classification. To solve this, this article proposes an Occlusion-Robust PS-GAN-CNN (OR-PSGAN-CNN) model for detecting CP in infants from video. First, it uses an OpenPose model to estimate infant skeletal joint positions, which are then augmented through the PS-GAN. Then, the coordinates of the infant's joints are extracted into FM based on the matrix encoding, along with the extraction of joint motion complexity and joint motion correlation features. These extracted FM are fed to the CNN with a softmax classifier to detect CP. Thus, this model handles occlusion by transforming the skeletons into the FM, substituting the missing coordinates with zeros, resulting in high accuracy. Finally, experiments results show that the OR-PSGAN-CNN model achieves 93.7%, 93.3%, and 93.2% accuracy on the MINI-RGBD, babyPose, and MIA datasets, respectively, outperforming existing CP detection models.