Executing hadoop binded on localhost

Hi, is there a way to start hadoop with all its socket
(namenode, 
datanode, tracker...) binded to localhost ?

cheers

-- 
Philippe GASSMANN
Solutions & Technologies
ANYWARE TECHNOLOGIES
Tel : +33 (0)561 000 655 
Fax : +33 (0)561 005 146
http://www.anyware-tech.
com/

Check out this url

http://lucene.apache.org/hadoop/docs/api/overview-
summary.html

--Jugs

-----Original Message-----
From: Philippe Gassmann [mailto:philippe.gassmannanyware-tech.com] 
Sent: Friday, September 08, 2006 2:37 PM
To: hadoop-userlucene.apache.org
Subject: Executing hadoop binded on localhost

Hi, is there a way to start hadoop with all its socket
(namenode, 
datanode, tracker...) binded to localhost ?

cheers

-- 
Philippe GASSMANN
Solutions & Technologies
ANYWARE TECHNOLOGIES
Tel : +33 (0)561 000 655 
Fax : +33 (0)561 005 146
http://www.anyware-tech.
com/

Jagadeesh wrote:
> Check out this url
>
> http://lucene.apache.org/hadoop/docs/api/overview-
summary.html
>
>   
That does not solve my issue : when you specify localhost in

hadoop-site.xml, hadoop does not bind to localhost but to
0.0.0.0.

In the source tree I can see : new ServerSocket(port) to
create a server 
socket (by default without specifying the bind address, java
will bind 
on 0.0.0.0)
or some code like that :  address = new 
InetSocketAddress(port);acceptChannel.socket().bind(address)
;


> --Jugs
>
>   


-- 
Philippe GASSMANN
Solutions & Technologies
ANYWARE TECHNOLOGIES
Tel : +33 (0)561 000 655 
Fax : +33 (0)561 005 146
http://www.anyware-tech.
com/

Perhaps you need to add an entry for 'localhost' in the
hosts file on 
your machine?  My linux /etc/hosts has an entry like:

127.0.0.1       localhost

Alternately you could specify '127.0.0.1' as the host
instead of 
'localhost'.  That should work, since 127.0.0.1 always
refers to the 
local machine.

Doug

Philippe Gassmann wrote:
> 
> 
> Jagadeesh wrote:
>> Check out this url
>>
>> http://lucene.apache.org/hadoop/docs/api/overview-
summary.html
>>
>>   
> That does not solve my issue : when you specify
localhost in 
> hadoop-site.xml, hadoop does not bind to localhost but
to 0.0.0.0.
> 
> In the source tree I can see : new ServerSocket(port)
to create a server 
> socket (by default without specifying the bind address,
java will bind 
> on 0.0.0.0)
> or some code like that :  address = new 
>
InetSocketAddress(port);acceptChannel.socket().bind(address)
;
> 
> 
>> --Jugs
>>
>>   
> 
> 

Doug Cutting wrote:
> Perhaps you need to add an entry for 'localhost' in
the hosts file on
> your machine?  My linux /etc/hosts has an entry like:
>
> 127.0.0.1       localhost
>
> Alternately you could specify '127.0.0.1' as the host
instead of
> 'localhost'.  That should work, since 127.0.0.1
always refers to the
> local machine.


Philippe's question is related to machines with multiples
interfaces
(e.g. one public-facing interface and another one for a
private
network). We'd like to bind Hadoop's sockets to the
private interface,
so that only machines on the private network can access it.

Sylvain

-- 
Sylvain Wallez - http://bluxte.net

Sylvain Wallez wrote:
> Philippe's question is related to machines with
multiples interfaces
> (e.g. one public-facing interface and another one for a
private
> network). We'd like to bind Hadoop's sockets to the
private interface,
> so that only machines on the private network can access
it.

Does the following address your needs?

http://svn.apache.org/viewvc?view=rev&revision=4412
04

http:
//issues.apache.org/jira/browse/HADOOP-497

This is in the 0.6.0 release, appearing on mirrors soon.

Doug

Doug Cutting wrote:
> Sylvain Wallez wrote:
>> Philippe's question is related to machines with
multiples interfaces
>> (e.g. one public-facing interface and another one
for a private
>> network). We'd like to bind Hadoop's sockets to
the private interface,
>> so that only machines on the private network can
access it.
>
> Does the following address your needs?
>
> http://svn.apache.org/viewvc?view=rev&revision=4412
04
>
> http:
//issues.apache.org/jira/browse/HADOOP-497

Don't think so (although I have to understand the exact
implication of
this).

That should be in o.a.h.ipc.Server$Listener that says:

      address = new InetSocketAddress(port);
      // Create a new server socket and set to non blocking
mode
      acceptChannel = ServerSocketChannel.open();
      acceptChannel.configureBlocking(false);

      // Bind the server socket to the local host and port
      acceptChannel.socket().bind(address, backlogLength);

The socket is bound to the wildcard address and therefore
will accept
connections from any interface.

I don't know Hadoop's internals well, but it seems to me
that an
additional configuration could do the trick, e.g.

    String itfAddr =
conf.getString("ipc.server.listen.address")
    address = (itfAddr == null) ? new
InetSocketAddress(port) : new
InetSocketAddress(itfAddr, port);

How does that sound?

Sylvain

-- 
Sylvain Wallez - http://bluxte.net

Sylvain Wallez wrote:
> I don't know Hadoop's internals well, but it seems to
me that an
> additional configuration could do the trick, e.g.
> 
>     String itfAddr =
conf.getString("ipc.server.listen.address")
>     address = (itfAddr == null) ? new
InetSocketAddress(port) : new
> InetSocketAddress(itfAddr, port);
> 
> How does that sound?

I'm not sure why you'd want this.  The previously
mentioned patch 
addresses the datanode and tasktracker, which must provide
an address 
for peers to contact.  The namenode and jobtracker's public
address is 
already determined by a config setting.  So, while it is
true that these 
daemons listen on all interfaces, they're only contacted on
the single 
interface specified in the configuration.

The only reason I can see for the change you suggest
(restricting 
daemons to only listen on a single interface) is security:
you'd like 
these daemons to not be visible over the private address. 
Is that 
indeed your concern?  If so, can this instead be managed
with a firewall?

Doug

Doug Cutting wrote:
> Sylvain Wallez wrote:
>> I don't know Hadoop's internals well, but it
seems to me that an
>> additional configuration could do the trick, e.g.
>>
>>     String itfAddr =
conf.getString("ipc.server.listen.address")
>>     address = (itfAddr == null) ? new
InetSocketAddress(port) : new
>> InetSocketAddress(itfAddr, port);
>>
>> How does that sound?
>
> I'm not sure why you'd want this.  The previously
mentioned patch
> addresses the datanode and tasktracker, which must
provide an address
> for peers to contact.  The namenode and jobtracker's
public address is
> already determined by a config setting.  So, while it
is true that
> these daemons listen on all interfaces, they're only
contacted on the
> single interface specified in the configuration.
>
> The only reason I can see for the change you suggest
(restricting
> daemons to only listen on a single interface) is
security: you'd like
> these daemons to not be visible over the private
address.  Is that
> indeed your concern?

Exactly!

> If so, can this instead be managed with a firewall?

Unfortunately not, because this is an environment where we
want to avoid
firewalls, which are the first thing that will break in case
of a DoS
attack, hence the need to bind to a particular address.

I agree that we normally should have the HDFS running
entirely on a
private network, but in some restricted test/demo
configurations we run
the HDFS on the same machines than some of the publically
accessible
services, hence this need.

Sylvain

-- 
Sylvain Wallez - http://bluxte.net

Doug Cutting wrote:
> Sylvain Wallez wrote:
>> I don't know Hadoop's internals well, but it
seems to me that an
>> additional configuration could do the trick, e.g.
>>
>>     String itfAddr =
conf.getString("ipc.server.listen.address")
>>     address = (itfAddr == null) ? new
InetSocketAddress(port) : new
>> InetSocketAddress(itfAddr, port);
>>
>> How does that sound?
>
> I'm not sure why you'd want this.  The previously
mentioned patch
> addresses the datanode and tasktracker, which must
provide an address
> for peers to contact.  The namenode and jobtracker's
public address is
> already determined by a config setting.  So, while it
is true that
> these daemons listen on all interfaces, they're only
contacted on the
> single interface specified in the configuration.
>
> The only reason I can see for the change you suggest
(restricting
> daemons to only listen on a single interface) is
security: you'd like
> these daemons to not be visible over the private
address.  Is that
> indeed your concern?

Exactly!

> If so, can this instead be managed with a firewall?

Unfortunately not, because this is an environment where we
want to avoid
firewalls, which are the first thing that will break in case
of a DoS
attack, hence the need to bind to a particular address.

I agree that we normally should have the HDFS running
entirely on a
private network, but in some restricted test/demo
configurations we run
the HDFS on the same machines than some of the publically
accessible
services, hence this need.

Sylvain

-- 
Sylvain Wallez - http://bluxte.net