Predicting Sentiments of Users about Medical Treatments using Pre-trained Large Language Models

Abstract

Users of social media write their opinions about products, services, market, social life, health and any facet of the life in the form of texts in web-based or mobile applications. The other users use these comments to select the best services and products. In the area of the medication, the production of user generated texts has been increased, because of information explosion and technological advancements. Manual extraction of useful knowledge using the tremendous amount of textual data is impossible. Opinion mining and Sentiment Analysis (SA) is a crucial mechanism for extracting useful knowledge, including users’ opinions about medical systems, to help physicians with this information. Physicians will use the extracted information to know how patients feel about the course of treatment and other health related topics. This paper investigates the application of Large Language Models (LLMs) to predict polarity of patients' opinions. This study uses a dataset that includes patient reviews regarding their opinions about medications, prescriptions, and treatment. Three scenarios are considered in this paper: scenarios of two classes (positive, negative), three classes (positive, neutral, negative), and five classes (negative, slightly negative, neutral, slightly positive, positive). BERT and DistilBERT tokenization methods are used for word embedding. For training and fine tuning in clinical domains, one traditional ML based method, One Boosting based method, and three BERT-based methods, are utilized in model development. We found the best hyper-parameters for all models using Grid-CV method. The results reveal that the fine-tuned BERT model with corresponding word embedding representation, achieved the best results, with accuracy and F1-Score of 97.71% and 98% in two classes, 97.24% and 97% in three classes, and 80.35% and 80% in five classes, respectively. Due to the high accuracy, the proposed models can be used as an auxiliary tool in clinics and medical centers.