An Empirical Analysis of Reliability Challenges in Multi-Step Agentic AI Systems

Abstract

The application of Agentic Artificial Intelligence (AI) systems attracts a lot of interest to undertake complex tasks through planning, reasoning, memory usage, tool interaction, and validation processes. It is common knowledge that while the applications are widely used in various fields like software engineering, healthcare, and business automation, reliability issues are still one of the primary challenges. The aim of this study is to highlight possible reliability issues in the multi-step agentic AI system, employing an experimental framework. A quantitative research design is used in which the AgentBench benchmark dataset with 5,000 task instances and 12 variables related to the workflow is used. The data pre-processing, reliability modelling and performance evaluation are performed using Python libraries and agent simulation frameworks. Task success rate, failure rate, time to execute task, task recovery rate, consistency of output and error propagation rate are used as measures of reliability. Results indicated that the reliability reduces as difficulty of the task escalates and reasoning and tool related errors are the leading causes of failure. In addition, it is found that the propagation of error over extended workflows could drastically escalate resulting in domino effects. It extends an established reliability evaluation framework, and offers ideas for developing more robust, transparent, and dependable autonomous systems that incorporate agentic AI in complex settings.