Created: (HADOOP-1985) Abstract node to switch mapping into a topology service class used by

Abstract node to switch mapping into a topology service
class used by namenode and jobtracker
------------------------------------------------------------
---------------------------------

                 Key: HADOOP-1985
                 URL: htt
ps://issues.apache.org/jira/browse/HADOOP-1985
             Project: Hadoop
          Issue Type: New Feature
            Reporter: eric baldeschwieler


In order to implement switch locality in MapReduce, we need
to have switch location in both the namenode and job
tracker.  Currently the namenode asks the data nodes for
this info and they run a local script to answer this
question.  In our environment and others that I know of
there is no reason to push this to each node.  It is easier
to maintain a centralized script that maps node DNS names to
switch strings.

I propose that we build a new class that caches known DNS
name to switch mappings and invokes a loadable class or a
configurable system call to resolve unknown DNS to switch
mappings.  We can then add this to the namenode to support
the current block to switch mapping needs and simplify the
data nodes.  We can also add this same callout to the job
tracker and then implement rack locality logic there without
needing to chane the filesystem API or the split planning
API.

Not only is this the least intrusive path to building
racklocal MR I can ID, it is also future compatible to
future infrastructures that may derive topology on the fly,
etc, etc...

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue
online.

    [ https://issues.apache.org/jira/browse
/HADOOP-1985?page=com.atlassian.jira.plugin.system.issuetabp
anels:comment-tabpanel#action_12533350 ] 

Runping Qi commented on HADOOP-1985:
------------------------------------


yes, DNS name (hostname) to switch id mapping should be
managed just like the hostname to IP mapping. 
The info should be available to the DFS  namenode, datanodes
and applications in the same way. 
Job tracker uses this info for task assignment. In general,
DFS client should also use this info to decide where to
fetch a needed block.


> Abstract node to switch mapping into a topology service
class used by namenode and jobtracker
>
------------------------------------------------------------
---------------------------------
>
>                 Key: HADOOP-1985
>                 URL: htt
ps://issues.apache.org/jira/browse/HADOOP-1985
>             Project: Hadoop
>          Issue Type: New Feature
>            Reporter: eric baldeschwieler
>
> In order to implement switch locality in MapReduce, we
need to have switch location in both the namenode and job
tracker.  Currently the namenode asks the data nodes for
this info and they run a local script to answer this
question.  In our environment and others that I know of
there is no reason to push this to each node.  It is easier
to maintain a centralized script that maps node DNS names to
switch strings.
> I propose that we build a new class that caches known
DNS name to switch mappings and invokes a loadable class or
a configurable system call to resolve unknown DNS to switch
mappings.  We can then add this to the namenode to support
the current block to switch mapping needs and simplify the
data nodes.  We can also add this same callout to the job
tracker and then implement rack locality logic there without
needing to chane the filesystem API or the split planning
API.
> Not only is this the least intrusive path to building
racklocal MR I can ID, it is also future compatible to
future infrastructures that may derive topology on the fly,
etc, etc...

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue
online.

    [ https://issues.apache.org/jira/browse
/HADOOP-1985?page=com.atlassian.jira.plugin.system.issuetabp
anels:comment-tabpanel#action_12540233 ] 

Hairong Kuang commented on HADOOP-1985:
---------------------------------------

Currently in dfs a datanode can get its network location
either from the command line or by running a pluggable
script at the startup time. The property is defined in the
default configuration file as below.

<property>
  <name>dfs.network.script</name>
  <value></value>
  <description>
        Specifies a script name that print the network
location path
        of the current machine.
  </description>
</property>


> Abstract node to switch mapping into a topology service
class used by namenode and jobtracker
>
------------------------------------------------------------
---------------------------------
>
>                 Key: HADOOP-1985
>                 URL: htt
ps://issues.apache.org/jira/browse/HADOOP-1985
>             Project: Hadoop
>          Issue Type: New Feature
>            Reporter: eric baldeschwieler
>            Assignee: Devaraj Das
>
> In order to implement switch locality in MapReduce, we
need to have switch location in both the namenode and job
tracker.  Currently the namenode asks the data nodes for
this info and they run a local script to answer this
question.  In our environment and others that I know of
there is no reason to push this to each node.  It is easier
to maintain a centralized script that maps node DNS names to
switch strings.
> I propose that we build a new class that caches known
DNS name to switch mappings and invokes a loadable class or
a configurable system call to resolve unknown DNS to switch
mappings.  We can then add this to the namenode to support
the current block to switch mapping needs and simplify the
data nodes.  We can also add this same callout to the job
tracker and then implement rack locality logic there without
needing to chane the filesystem API or the split planning
API.
> Not only is this the least intrusive path to building
racklocal MR I can ID, it is also future compatible to
future infrastructures that may derive topology on the fly,
etc, etc...

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue
online.