Operator parallel optimization strategy for distributed databases

面向分布式数据库的算子并行优化策略

School of Computer Science, Northwestern Polytechnical University, Xi'an 710072, China

Received: 30 May 2023

Abstract

With the continuous development of network technology, the scale of data has shown explosive growth, which leads gradually to replacing traditional single machine databases with distributed databases. Distributed databases solve large-scale data storage problems through collaborative work among nodes, but due to increased communication costs between nodes, its query efficiency is not as good as a standalone database. In a distributed architecture, the data of storage nodes is only used as redundancy for multiple backups, providing data recovery in case of system failure, and it is not utilized to improve query efficiency. In response to the above issues, this article proposes an operator parallel optimization strategy for distributed databases. By splitting key physical operators, the split sub requests are evenly distributed to multiple nodes in the storage layer, which are processed in parallel by multiple nodes, thereby reducing query response time. The above strategy has been applied on distributed database CBase, and experiments have shown that the parallel optimization strategy proposed in this paper can significantly shorten SQL request query time and improve system resource utilization.

摘要

随着网络技术的不断发展, 数据规模呈现爆发式增长, 使得传统的单机数据库逐步被分布式数据库所取代。分布式数据库采用节点协同工作方式解决了大规模数据存储问题, 但由于增加了节点间通信开销, 查询效率却不如单机数据库。分布式架构下, 存储节点的数据仅用作多备份的冗余, 为系统故障时提供数据恢复, 并未被利用起来改善查询效率。针对上述问题, 提出了一种面向分布式数据库的算子并行优化策略, 通过对关键物理算子进行拆分, 将拆分后的子请求均匀分配到存储层多个节点, 由多个节点并行处理, 从而减少查询响应时间。上述策略已经在分布式数据库CBase上进行了应用, 实验表明, 提出的并行优化策略可显著缩短SQL请求查询时间, 并提高系统资源利用率。