Data-First Application Architecture: Unifying Transactional and Analytical Workloads on Modern Data Platforms

Abstract

Customarily, enterprise data architectures have been understood to be built on the separation of transactional systems and analytical systems, leading to separate repository data stores, extract-transform-load (ETL) processes, and latency to business-critical decision-making. Modern cloud-native data platforms are beginning to challenge this model with a new architecture that breaks down the boundary between operational systems and analytical systems. In a data-first application architecture, the data platform is the system of record, and applications pull from it. A hub-and-spoke model is often used with medallion data layers to maintain multiple representations of business-ready data as it passes through various levels of refinement from raw ingested data to business-ready data structures. Hybrid transactional-analytical processing allows a mixture of OLTP and OLAP workloads with dual-format data representations, query optimizers, and related techniques. Streaming ingestion and change data capture enable streaming data with sub-second data freshness and ACID transaction semantics using isolation levels such as serializable snapshot isolation. Event-driven architectures complement this pattern by propagating data in the form of immutably logged events to distributed consumers. They can respond to business event flows through choreography without tight coupling. Practical implementation aspects of workload-optimized querying, multi-tier caching, elastic resource management, and infrastructure consolidation yield economic benefits and extremely low-latency performance for business events consumed by users. Those organizations using unified transactional-analytical architectures report dramatic improvements in time-to-understand, reduced infrastructure costs, accelerated development lifecycles, and improved support for machine and artificial intelligence-based systems that require a fresh and consistent view of data on which prescriptive models can act. Operational-analytical workload convergence is an architectural step change to make organizations more agile in data-rich environments where speed of business understanding and decisions are critical to competitive differentiation.