VPCH : a consistent hashing algorithm for better load balancing in a hadoop environment

Liu, Q, Cai, W, Shen, J, Wang, B, Fu, Z and Linge, N 2015, 'VPCH : a consistent hashing algorithm for better load balancing in a hadoop environment' , in: 2015 Third International Conference on Advanced Cloud and Big Data , IEEE, pp. 69-72.

Full text not available from this repository.

Abstract

MapReduce (MR) is a popular programming model for the purposes of processing large data sets among data clusters or grids, e.g. a Hadoop environment. Load balancing as a key factor affecting the performance of map resource distribution, has recently gained high concerns to optimize. Current MR processes in the realization of distributing tasks to clusters use hashing with random modulo operations, which can lead to uneven data distribution and inclined loads, thereby obstruct the performance of the entire distribution system. In this paper, a virtual partition consistent hashing (VPCH) algorithm is proposed for the reduce stage of MR processes, in order to achieve such a trade-off on job allocation. According to the results, using our method can reduce task execution time with or without MJR (mapreduce.job.reduce.slowstart.completedmaps) parameter set.

Item Type: Book Section
Schools: Schools > School of Computing, Science and Engineering
Publisher: IEEE
ISBN: 9781467385374
Depositing User: WM Taylor
Date Deposited: 15 Dec 2016 12:46
Last Modified: 15 Dec 2016 12:46
URI: http://usir.salford.ac.uk/id/eprint/41026

Actions (login required)

Edit record (repository staff only) Edit record (repository staff only)