Event-Driven Architecture for Real-Time Enterprise Data Processing Using Apache Kafka
Main Article Content
Abstract
The amount of data being generated by companies is growing extremely fast, thanks to the increasing number of connected systems, IoT devices, and digital services. This has created a need for systems that can handle large amounts of data in real-time, without failing or losing any information. To address this challenge, we've been exploring the use of Event-Driven Architecture (EDA) with Apache Kafka, a popular tool for processing large amounts of data in realtime. In this paper, we propose a new framework called KEDA-Enterprise, which combines Kafka's distributed log mechanism with microservices, stream processing engines, and enterprise integration patterns. We tested our framework on a large cluster of 24 nodes, processing over 10 million events per second, and found that it reduced the time it takes to process data by 73% compared to traditional batch-processing systems. Additionally, our framework guarantees that 99.99% of messages will be delivered without being lost. We also tested our framework in three different industries: financial services, e-commerce, and healthcare. The results showed that our framework can process events in under 10 milliseconds, even when handling a large volume of data. It can also scale up to 48 nodes without any issues, and allows for changes to be made to the system without disrupting service. Our work provides a clear blueprint for building scalable and fault-tolerant systems, as well as a benchmark for measuring performance and a model for ensuring fault-tolerance in mission-critical deployments. Our goal is to provide a solution that can handle the increasingly large amounts of data being generated by companies, and to do so in a way that is fast, reliable, and scalable. We believe that our framework has the potential to make a significant impact in a variety of industries, and we're excited to share our findings with the community. By providing a reproducible architectural blueprint, a quantitative performance benchmark, and a formal fault-tolerance model, we hope to contribute to the development of more efficient and effective systems for processing large amounts of data in real-time.