Re: clarification on running a job locally

It is definitely easier to build a jar and use the hadoop
script.  You can
do it yourself, though.  Just duplicate the line in
bin/hadoop that runs
java and prefix it with "echo" to see what is
happening.


On 11/1/07 1:37 PM, "Jim the Standing Bear"
<standingbeargmail.com> wrote:

> Hi Ted,
> 
> It is funny how I am having so much difficulties with
hadoop... since
> i am on this subject, let me ask another stupid
question:
> 
> In order to run a hadoop job, I must zip it into a jar,
and then use
> $HADOOP_HOME/bin/hadoop to launch it, correct?  I
cannot simply use
> "java" to directly launch it, right?
> 
> best wishes
> 
> Yiming
> 
> 
> On 11/1/07, Ted Dunning <tdunningveoh.com> wrote:
>> 
>> When running a job standalone (typically for
debugging), it suffices for me
>> to simply add -Dmapred.job.tracker=local to the
program command line.
>> 
>> If you simply want to run the program on a single
node, then you can just
>> stop the other node.  Using local storage with
distributed programs is not
>> recommended.
>> 
>> 
>> On 11/1/07 12:35 PM, "Jim the Standing
Bear" <standingbeargmail.com> wrote:
>> 
>>> Hello,
>>> 
>>> I am in need of some clarifications on how to
run a hadoop job locally.
>>> 
>>> The cluster was originally set up to have two
nodes, where one of them
>>> also acts as the master node and job tracker.
>>> 
>>> According to the wiki, I can run a job locally
by altering
>>> "mapred.job.tracker" and
"fs.default.name" properties to "local"
in
>>> hadoop-site.xml.  But when I start the server,
it stack dumped:
>>> 
>>> localhost: starting secondarynamenode, logging
to /home/blahblahblah
>>> localhost: Exception in thread "main"
java.lang.RuntimeException: Not
>>> a host:port pair: local
>>> 
>>> Apparently it didn't like the value
"local"?
>>> 
>>> Also, the wiki noted that all these XML
configuration files should be
>>> included somewhere in the class path to the
job, does it mean I need
>>> to include the XMLs as I do jars?
>>> 
>>> Thank
>>> 
>>> -- Jim
>> 
>> 
> 

Thanks Ted.  just as I thought.

On 11/1/07, Ted Dunning <tdunningveoh.com> wrote:
>
> It is definitely easier to build a jar and use the
hadoop script.  You can
> do it yourself, though.  Just duplicate the line in
bin/hadoop that runs
> java and prefix it with "echo" to see what is
happening.
>
>
> On 11/1/07 1:37 PM, "Jim the Standing Bear"
<standingbeargmail.com> wrote:
>
> > Hi Ted,
> >
> > It is funny how I am having so much difficulties
with hadoop... since
> > i am on this subject, let me ask another stupid
question:
> >
> > In order to run a hadoop job, I must zip it into a
jar, and then use
> > $HADOOP_HOME/bin/hadoop to launch it, correct?  I
cannot simply use
> > "java" to directly launch it, right?
> >
> > best wishes
> >
> > Yiming
> >
> >
> > On 11/1/07, Ted Dunning <tdunningveoh.com> wrote:
> >>
> >> When running a job standalone (typically for
debugging), it suffices for me
> >> to simply add -Dmapred.job.tracker=local to
the program command line.
> >>
> >> If you simply want to run the program on a
single node, then you can just
> >> stop the other node.  Using local storage with
distributed programs is not
> >> recommended.
> >>
> >>
> >> On 11/1/07 12:35 PM, "Jim the Standing
Bear" <standingbeargmail.com> wrote:
> >>
> >>> Hello,
> >>>
> >>> I am in need of some clarifications on how
to run a hadoop job locally.
> >>>
> >>> The cluster was originally set up to have
two nodes, where one of them
> >>> also acts as the master node and job
tracker.
> >>>
> >>> According to the wiki, I can run a job
locally by altering
> >>> "mapred.job.tracker" and
"fs.default.name" properties to "local"
in
> >>> hadoop-site.xml.  But when I start the
server, it stack dumped:
> >>>
> >>> localhost: starting secondarynamenode,
logging to /home/blahblahblah
> >>> localhost: Exception in thread
"main" java.lang.RuntimeException: Not
> >>> a host:port pair: local
> >>>
> >>> Apparently it didn't like the value
"local"?
> >>>
> >>> Also, the wiki noted that all these XML
configuration files should be
> >>> included somewhere in the class path to
the job, does it mean I need
> >>> to include the XMLs as I do jars?
> >>>
> >>> Thank
> >>>
> >>> -- Jim
> >>
> >>
> >
>
>


-- 
--------------------------------------
Standing Bear Has Spoken
--------------------------------------