Volume & Issue no: Volume 6, Issue 4, July - August 2017
____________________________________________________________________________________________________
Title: |
AIMS: Adaptive Improved MapReduce Scheduling in Hadoop |
Author Name: |
D.Vasudha, K.Ishthaq Ahamed |
Abstract: |
Abstract
Hadoop is a complete eco-system of open source projects that
provide us the framework to deal with big data. Let’s start by
brainstorming the possible challenges of dealing with big data
(on traditional systems) and then look at the capability of
Hadoop solution. This approach lowers the risk of catastrophic
system failure and unexpected data loss, even if a significant
number of nodes become inoperative. Consequently, Hadoop
quickly emerged as a foundation for big data processing tasks,
such as scientific analytics, business and sales planning, and
processing enormous volumes of sensor data, including from
internet of things sensors. As an important extension of
Hadoop, SAMR and ESAMR MapReduce scheduling
algorithms take heterogeneous environment into consideration.
However, it falls short of solving the crucial problem – poor
performance due to the large data sets in which it computes
progress of tasks. Consequently, neither Hadoop nor ESAMR
schedulers are desirable in heterogeneous environment. To this
end, we propose AIMS: an Adaptive Improved Scheduling
Algorithm, which calculates progressof tasks dynamically and
splits the files into multiple chunks, based on user inputs and
adapts to the continuously varying environment automatically.
When a job is committed, AIMS splits the input file into
multiple chunks so that the process will be completed easily,
then assigns them to a series of nodes. Meanwhile, it reads
historical information which stored on every node and updated
after every execution. Then, AIMSadjusts time weight of each
stage of map and reducetasks according to the historical
information respectively. Thus, it gets the progress of each task
accurately and finds which tasks need backup tasks. What’s
more, it identifies slow nodes and classifies them into the sets of
slow nodes dynamically. According to the information of these
slow nodes, AIMS will not launch backup tasks on them,
ensuring the backup tasks will not be slow tasks any more. It
gets the final results of the fine-grained tasks when either slow
tasks or backup tasks finish first. The proposed algorithm is
evaluated by extensive experiments over various heterogeneous
environment. Experimental results show that AIMS
significantly decreases the time of execution up to 19%
compared with Hadoop’s scheduler and up to 10% compared
with ESAMR scheduler..
Keywords: Map Reduce, Scheduling algorithm, He
terogeneous environment, Self-adaptive, aims |
Cite this article: |
D.Vasudha, K.Ishthaq Ahamed , "
AIMS: Adaptive Improved MapReduce Scheduling in Hadoop" , International Journal of Emerging Trends & Technology in Computer Science (IJETTCS) ,
Volume 6, Issue 4, July - August 2017 , pp.
249-256 , ISSN 2278-6856.
|
| Full Text [PDF] Back to Current Issue | |
NOTE: Authors note that paper cannot be withdrawn at any condition once it is accepted. The Team of IJETTCS advise you, do not submit same article to the multiple journals simultaneously. This may create a problem for you. Please wait for review report which will take maximum 01 to 02 week.