[ https://issues.apache.org/jira/browse
/HADOOP-1900?page=com.atlassian.jira.plugin.system.issuetabp
anels:comment-tabpanel#action_12535894 ]
Devaraj Das commented on HADOOP-1900:
-------------------------------------
bq. So, one way to take this into account might be to
maintain an average time-to-complete for all tasks in the
system (of current jobs) and factor that into the scaling of
the intervals.
The TaskTracker currently pings the JobTracker asking for a
task as soon as it finishes executing a task. I think that
should be the behavior to keep the utilization of the
tasktrackers optimal (of course, in general we could do
better by sending it a bunch of tasks every time it asks for
a new task, but that's the subject of another jira).
bq. Also, while we are at this, I say we should start to
consider busy-ness of JobTracker too, along with the
cluster-size. So, for e.g., if the individual tasks are
taking in the order of minutes, then it might not matter
much if we send one every 20s or so, in some cases it might.
I know that the sort's map tasks take around 40s each...
I propose a change to the status message in the heartbeat -
the tasktracker can compare the current task status with the
previous one and if it finds the status to be the same, it
doesn't send the complete status object to the JobTracker,
but just a flag saying it is a duplicate or something to
that effect. That will reduce the data per RPC considerably
for long running tasks whose statuses don't change
frequently and also reduce the processing load on the
JobTracker.
Thoughts?
> the heartbeat and task event queries interval should be
set dynamically by the JobTracker
>
------------------------------------------------------------
-----------------------------
>
> Key: HADOOP-1900
> URL: htt
ps://issues.apache.org/jira/browse/HADOOP-1900
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Reporter: Owen O'Malley
> Assignee: Amareshwari Sri Ramadasu
>
> The JobTracker should scale the intervals that the
TaskTrackers use to contact it dynamically, based on how the
busy it is and the size of the cluster.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue
online.
|
