Designing a Performance Metrics Evaluation Framework for NLP-Driven Chatbots in Local Government Unit

Abstract

The integration of Natural Language Processing (NLP) in AI-driven chatbots offers a transformative solution to enhance service delivery and citizen engagement within the Local Government Units (LGUs). The primary goal of this study is to identify and evaluate relevant performance metrics for NLP-driven chatbot and proposes a comprehensive performance metrics evaluation framework tailored to LGUs. A systematic review of relevant academic literature was conducted to identify key performance aspects which were then organized into a multi-perspective framework. The proposed framework includes five perspectives: User Experience, Information Retrieval, Linguistic Quality, Technology Efficiency, and Public Service. These perspectives address critical aspects of chatbot performance, including task completion rates and response accuracy to linguistic coherence and public trust. The multi-perspective approach of the framework specifically addresses LGU challenges by incorporating bilingual support and inclusivity that ensures alignment with the unique needs of diverse citizens. Future work includes pilot testing with LGU-specific datasets to empirically validate the framework, refine its metrics and enhance its practical applicability.