Index Lifecycle and Shard Allocation Optimization in LargeScale Elasticsearch Clusters: A Performance–Cost Trade-Off Analysis

Main Article Content

Guruprasad Raghothama Rao

Abstract

Modern search solutions rely on Elasticsearch clusters to scale to lots of data with a low latency of the search. Shard under sizing and ineffective index lifecycle policy may however augment latency and infrastructure cost. In this paper, a quantitative analysis of the allocation of shards and lifecycle management is introduced in largescale clusters. The experiment results indicate that the medium size of a shard (35 GB) decreased the mean query latency to 118 ms, as compared to 142 ms (15 GB) and 156 ms (60 GB). The optimized configuration had an indexing throughput of 20, 200 docs/sec. Lifecycle tiering (hot-warm-cold) cut the storage overhead by 1.80 to 1.45 and cut the total cost each month down to $19,200, which is a savings of 20%. Equal shard distribution also cut down on the latency by 134 ms to 121 ms and shard variance was also reduced by 64 percent. The results indicate a combination of appropriate sizing of shard, lifecycle policy that is tier-sensitive, and a balanced distribution can achieve better performance and a significant reduction in the cost of infrastructure.

Article Details

Section
Articles