Fine-Grained Allocation Algorithm for Sharing Heterogeneous Resources in Data Center

数据中心上异构资源的细粒度分配算法研究

School of Computer Science and Technology, Northwestern Polytechnical University, Xi'an 710072, China

Received: 11 September 2019

Abstract

Data in a data center are stored dispersively. The data-oriented task computing disperses big data analysis tasks to different computing nodes. The extensive use of graphics processing unit (GPU) makes it urgent and important to study how to reasonably assign heterogeneous resources to different computing frameworks. We investigate the existing big data computing framework and the GPU computing. Based on the existing cluster resource management model and the GPU management model, we propose a hybrid heterogeneous resource management model that combines CPU resources with GPU resources. The computing nodes manage local resources and implement tasks; the resource management center concertedly manage various computing frameworks. We design and implement a hybrid domain resource sharing and allocation algorithm, which allocates the hybrid domain resources to computing frameworks according to the coordinated use of them so as to fairly share the hybrid domain resources among various computing frameworks and prevent the CPU from too many tasks but the GPU or CPU from resource "hunger". The experimental results show that the allocation algorithm can increase the use of heterogeneous resources and the number of completed tasks by around 15%.

摘要

数据中心的出现，使得大数据分析任务被分散到不同的计算节点。随着GPU计算的广泛应用，如何为不同的计算框架合理分配异构计算资源是目前的研究热点。研究了传统大数据计算框架和GPU计算的特点，针对现有的集群资源管理和GPU管理模式，提出了一种集中式异构资源管理模型，计算节点负责本地资源管理和任务的执行和管理，资源管理中心统一管理各个计算框架。对于不同的计算框架，根据其使用CPU以及GPU资源的不同，设计并实现了一种混合主资源共享分配算法，通过计算不同框架对主资源的使用，优先从可用资源中为主资源使用率最小的框架分配资源，实现主资源在各个框架的公平共享，防止CPU任务过多而导致GPU资源"饥饿"，或者反过来导致CPU资源"饥饿"的现象发生。通过实验验证，该分配算法在异构资源使用效率以及任务完成数量方面能提高15%左右。