Liu, Q, Cai, W, Shen, J, Wang, B, Fu, Z and Linge, N 
ORCID: https://orcid.org/0000-0002-4318-8782
2015,
'VPCH : a consistent hashing algorithm for better load balancing in a hadoop environment'
, in:
2015 Third International Conference on Advanced Cloud and Big Data
, IEEE, pp. 69-72.
Abstract
MapReduce (MR) is a popular programming model for the purposes of processing large data sets among data clusters or grids, e.g. a Hadoop environment. Load balancing as a key factor affecting the performance of map resource distribution, has recently gained high concerns to optimize. Current MR processes in the realization of distributing tasks to clusters use hashing with random modulo operations, which can lead to uneven data distribution and inclined loads, thereby obstruct the performance of the entire distribution system. In this paper, a virtual partition consistent hashing (VPCH) algorithm is proposed for the reduce stage of MR processes, in order to achieve such a trade-off on job allocation. According to the results, using our method can reduce task execution time with or without MJR (mapreduce.job.reduce.slowstart.completedmaps) parameter set.
| Item Type: | Book Section |
|---|---|
| Schools: | Schools > School of Computing, Science and Engineering |
| Publisher: | IEEE |
| ISBN: | 9781467385374 |
| Depositing User: | USIR Admin |
| Date Deposited: | 15 Dec 2016 12:46 |
| Last Modified: | 17 Aug 2020 13:40 |
| URI: | http://usir.salford.ac.uk/id/eprint/41026 |
Actions (login required)
 |
Edit record (repository staff only) |
Tools
Tools