Unified Frameworks for Automatic Parallelization and Multi-Tenancy in Machine Learning Pipelines

Abstract

The increasing complexity and scale of machine learning (ML) workflows demand advanced frameworks capable of optimizing computational performance and resource utilization. This paper introduces a unified framework that automates the parallelization of ML algorithms while supporting multi-tenancy, enabling concurrent execution of diverse pipelines across multiple users. The proposed system addresses key challenges including scalability, workload balancing, and dynamic resource allocation, offering an adaptive, end-to-end solution for large-scale ML environments. By integrating a multi-tenant architecture, the framework ensures fair resource sharing and sustained system efficiency, even under heterogeneous workloads. Extensive experimentation on real-world datasets demonstrates significant improvements in training time, scalability, and throughput when compared to conventional single-tenant and manually parallelized approaches. The results validate the framework’s potential as a robust and scalable foundation for modern machine learning deployment and automation.