Implementing a Petabyte-Scale Data Lakehouse for India’s Public Financial Management System: A High-Throughput Ingestion and Processing Framework

Main Article Content

Syed Khundmir Azmi
Suresh Vethachalam
Gopalakrishna Karamchand
Anoop Gopi

Abstract

The high-growth rate of the Public Financial Management System (PFMS) and its Direct Benefit Transfer (DBT) ecosystem in India has created massive amounts of data that has put the bottlenecks of traditional warehousing and antique infrastructure to the test. This research will be a design and implementation of a petabyte Data Lakehouse infrastructure designed specifically to deliver PFMS 2.0 with its ability to ingest actionable data, process billions of open transactions, and perform advanced analytics on them. Using a multi-layered Bronze-SilverGold data model and cloud-native applications, including ADLS Gen2, Azure Databricks, and Synapse Analytics, the suggested framework eliminates the long-standing scalability, reliability and heterogeneous data integration bottlenecks. A gradual national implementation shows notable gains in ingestion rate, query latency and capability of managing data, and PFMS 2.0 is a stable platform with the ability to provide the extensive financial transparency and accountability requirements of India. The findings demonstrate the potential to transform the large-scale provision of public finance landscape through the Lakehouse architectures and give a reproducible example of the same to other national digital governance systems.

Article Details

Section
Articles