Any plans to move to build Nutchusing Maven?

(This thread moved from the User List.)

OK Lukas, lets open it up to the dev list! 

Particularly, does the group feel moving to Maven would be
_a good thing_ ?

Even if so, what are the problems?

There are currently two versions of Lucene in the Maven
repos, but Hadoop
would have to be added manually, I think.

All thoughts gratefully received.
Cheers
Steven

On 8/16/06, Lukas Vlcek <lukas.vlcekgmail.com> wrote:
>
> Hi,
>
> I would like to help. But first of all I would suggest
to start wider
> discussion in dev list to get more
feedback/suggestions. I think one
> problem can be that Nutch depends on both Lucene and
Hadoop libraries
> and it won't be easy to maintain these dependencies if
recent versions
> are not yet committed into some maven accesible repo.
>
> Regards,
> Lukas
>
> On 8/16/06, steven shingler <shinglergmail.com> wrote:
> > Well I'm up for giving it a try. My current work
has me looking at both
> > Nutch and Maven, so what better way to understand
both projects 
> >
> > I agree it is far from trivial - so if anyone here
would like to
> collaborate
> > on it, that would be great.
> > Cheers,
> > Steven
> >
> >
> > On 8/15/06, Lukas Vlcek <lukas.vlcekgmail.com> wrote:
> > >
> > > Hi,
> > >
> > > I would warmly appreciate this activity. At
least it would help more
> > > people to understand/join this great project.
But I don't think this
> > > will be an easy step (this reminds me what
N.Armstrong said on moon:
> > > That's one small step for [a] man, one giant
leap for mankind.)
> > > 
> > >
> > > Regards,
> > > Lukas
> > >
> > > On 8/15/06, Sami Siren <ssirengmail.com> wrote:
> > > > steven shingler wrote:
> > > > > Hi all,
> > > > >
> > > > > I know this has come up at least
once before, but I just thought
> I'd
> > > raise
> > > > > the question again:
> > > > >
> > > > > Are there any plans to move to
building Nutch using Maven?
> > > >
> > > > Haven't heard of such activities,
however if you or somebody else
> > > > can put such thing together and it
proves to be a good thing to do
> then
> > > > I certainly don't have anything against
it.
> > > >
> > > > --
> > > >   Sami Siren
> > > >
> > >
> >
> >
>

Hi Steven,


On 8/16/06 7:36 AM, "steven shingler"
<shinglergmail.com> wrote:

> (This thread moved from the User List.)
> 
> OK Lukas, lets open it up to the dev list! 
> 
> Particularly, does the group feel moving to Maven would
be _a good thing_ ?

+1

I suggested this (however did not make any progress on
realizing it ;) ) a
while back. I think it makes a * lot of sense *. Maven's
dependency system
would significantly reduce the size of the CM'ed Nutch
source code, as all
the jars required by Nutch could be referenced externally
(plugins are a
different beast, but we're working on that). Additionally,
Maven would allow
automatic generation of a soft of "nightly
build" Nutch site, showing recent
commits, unit test results and more.

> 
> Even if so, what are the problems?

The main problem I see is the plugin system, and how to
appropriate
represent plugin dependencies in Maven (or just neglect to
elegantly handle
them, and treat them like invididual projects, like nutch,
which requires
CM'ing jar files). Additionally, I think it will probably
require writing
some custom Jelly scripts to do all the neat ant build stuff
that Nutch does
on the side (e.g., unpack Hadoop, etc.).

> 
> There are currently two versions of Lucene in the Maven
repos, but Hadoop
> would have to be added manually, I think.

It would probably make most sense to run a Maven repo
explicitly for Nutch
off of the Lucene Nutch site. Something like
(http://lucene.a
pache.org/nutch/maven/) might be sensible.

Just my 2 cents.

Cheers,
  Chris

> 
> All thoughts gratefully received.
> Cheers
> Steven
> 
> On 8/16/06, Lukas Vlcek <lukas.vlcekgmail.com> wrote:
>> 
>> Hi,
>> 
>> I would like to help. But first of all I would
suggest to start wider
>> discussion in dev list to get more
feedback/suggestions. I think one
>> problem can be that Nutch depends on both Lucene
and Hadoop libraries
>> and it won't be easy to maintain these
dependencies if recent versions
>> are not yet committed into some maven accesible
repo.
>> 
>> Regards,
>> Lukas
>> 
>> On 8/16/06, steven shingler <shinglergmail.com> wrote:
>>> Well I'm up for giving it a try. My current
work has me looking at both
>>> Nutch and Maven, so what better way to
understand both projects 
>>> 
>>> I agree it is far from trivial - so if anyone
here would like to
>> collaborate
>>> on it, that would be great.
>>> Cheers,
>>> Steven
>>> 
>>> 
>>> On 8/15/06, Lukas Vlcek <lukas.vlcekgmail.com> wrote:
>>>> 
>>>> Hi,
>>>> 
>>>> I would warmly appreciate this activity. At
least it would help more
>>>> people to understand/join this great
project. But I don't think this
>>>> will be an easy step (this reminds me what
N.Armstrong said on moon:
>>>> That's one small step for [a] man, one
giant leap for mankind.)
>>>> 
>>>> 
>>>> Regards,
>>>> Lukas
>>>> 
>>>> On 8/15/06, Sami Siren <ssirengmail.com> wrote:
>>>>> steven shingler wrote:
>>>>>> Hi all,
>>>>>> 
>>>>>> I know this has come up at least
once before, but I just thought
>> I'd
>>>> raise
>>>>>> the question again:
>>>>>> 
>>>>>> Are there any plans to move to
building Nutch using Maven?
>>>>> 
>>>>> Haven't heard of such activities,
however if you or somebody else
>>>>> can put such thing together and it
proves to be a good thing to do
>> then
>>>>> I certainly don't have anything
against it.
>>>>> 
>>>>> --
>>>>>   Sami Siren
>>>>> 
>>>> 
>>> 
>>> 
>> 

I am an outsider, I hope my comments do not cause grief.

AFAIK, Maven is transitioning to version 2. The newer
version is more
attractive, but noticeably different than version 1 in
implementation
but, closer to home, in configuration.

It might be wise to explore some of the comments from other,
open
sourced, projects regarding maven (about a month ago I was
researching a
maven problem and bumped into some negative comments that
were unexpected.)

This is a big step. Due diligence before commitment might be
beneficial.

With all those negatives, I want maven, especially 2, to be
successful.
I believe it will solve many tedious problems.

Bill

Hi,

I have almost no experience with maven subprojects but
somehow I feel
this could help us with Nutch plugins. Am I correct?
In maven we can always call ant goals as well and Jelly is a
fun to
use. With maven one of the biggest benefit would be that
eclipse (or
other IDE) classpath settings hell will be over which could
help naive
users (like me) a lot.

(and silently I hope that what I have said now holds for M2
too)

Regards,
Lukas

On 8/16/06, William Surowiec <wsurowiecgmail.com> wrote:
> I am an outsider, I hope my comments do not cause
grief.
>
> AFAIK, Maven is transitioning to version 2. The newer
version is more
> attractive, but noticeably different than version 1 in
implementation
> but, closer to home, in configuration.
>
> It might be wise to explore some of the comments from
other, open
> sourced, projects regarding maven (about a month ago I
was researching a
> maven problem and bumped into some negative comments
that were unexpected.)
>
> This is a big step. Due diligence before commitment
might be beneficial.
>
> With all those negatives, I want maven, especially 2,
to be successful.
> I believe it will solve many tedious problems.
>
> Bill
>

Lukas Vlcek wrote:
> Hi,
> 
> I have almost no experience with maven subprojects but
somehow I feel
> this could help us with Nutch plugins. Am I correct?
> In maven we can always call ant goals as well and Jelly
is a fun to
> use. With maven one of the biggest benefit would be
that eclipse (or
> other IDE) classpath settings hell will be over which
could help naive
> users (like me) a lot.
> 
> (and silently I hope that what I have said now holds
for M2 too)
> 
Just for clarification when I was talking about maven I
meant maven 2, 
maven 1 is a dead end.

--
  Sami Siren

Le Mercredi 16 Août 2006 17:18, Sami Siren a écrit :
> Lukas Vlcek wrote:
> > Hi,
> >
> > I have almost no experience with maven subprojects
but somehow I feel
> > this could help us with Nutch plugins. Am I
correct?
> > In maven we can always call ant goals as well and
Jelly is a fun to
> > use. With maven one of the biggest benefit would
be that eclipse (or
> > other IDE) classpath settings hell will be over
which could help naive
> > users (like me) a lot.
> >
> > (and silently I hope that what I have said now
holds for M2 too)
>
> Just for clarification when I was talking about maven I
meant maven 2,
> maven 1 is a dead end.
Hi,

I am an outsider too, but my experience with maven 2 was not
as awful as 
expected. The problem with maven is that you would depend
enormously of the 
repository service. If it's down, just hope that you have
download everything 
before, otherwise you have to wait until it's up again. Two
or three month 
ago, the main repositories were down, and even more, one was
corrupted 
(because of a disk failure or something like that, I don't
remember). So our 
team of 5 people lost 3 days...
The ASF Cocoon project has mooved to maven and now the
README explains that :
> Build times will vary dramatically depending on how
good your connectivity
> to the Maven repository/mirror is, and how well it
performs, and on whether
> you already have a loaded local Maven repository. A
full build can take
> from 5 minutes to maybe 4 hours...
http://svn.apache.org/viewvc/cocoon/trunk/README
.txt?view=markup

Now, for our ex-maven2ish project, we use a combinaison of
ant tasks and ivy 
for the dependencies (http://jayasoft.org/ivy).
The difference between maven 
and ivy is that ivy runs completely offline, and maven
download only on 
demand. So if you do a svn up on your project, you probably
immediately want 
to do a mvn install to download every dependencies. But you
also should run 
every maven task before getting offline : maven jetty6:run,
maven source:jar, 
maven deploy...

As I said, I am a completely outsider, I am not a Nutch
user, just a Lucene 
user that read sometimes this dev list to read some
interesting discussion 
about "search". I just wanted to add my unlucky
maven user experience.

Here are some notes about Sylvain, one of my coworker that
had some trouble 
with maven :
http://b
luxte.net/blog/2006-05/21-25-04.html
http://b
luxte.net/blog/2006-06/30-32-27.html

Nicolas

Hi,

I have just noticed that there is going on some activity
called Tika
(http://code.google.com
/p/tika/) and these guys are starting directly
with Maven2 (am I right?).
The more Nutch/Lucene/Hadoop/[Tika]/[?] thing grows the more
sophisticated project management tool will be needed I
think.

Lukas

On 8/16/06, Nicolas Lalevée <nicolas.laleveeanyware-tech.com> wrote:
> Le Mercredi 16 Août 2006 17:18, Sami Siren a écrit:
> > Lukas Vlcek wrote:
> > > Hi,
> > >
> > > I have almost no experience with maven
subprojects but somehow I feel
> > > this could help us with Nutch plugins. Am I
correct?
> > > In maven we can always call ant goals as well
and Jelly is a fun to
> > > use. With maven one of the biggest benefit
would be that eclipse (or
> > > other IDE) classpath settings hell will be
over which could help naive
> > > users (like me) a lot.
> > >
> > > (and silently I hope that what I have said
now holds for M2 too)
> >
> > Just for clarification when I was talking about
maven I meant maven 2,
> > maven 1 is a dead end.
> Hi,
>
> I am an outsider too, but my experience with maven 2
was not as awful as
> expected. The problem with maven is that you would
depend enormously of the
> repository service. If it's down, just hope that you
have download everything
> before, otherwise you have to wait until it's up
again. Two or three month
> ago, the main repositories were down, and even more,
one was corrupted
> (because of a disk failure or something like that, I
don't remember). So our
> team of 5 people lost 3 days...
> The ASF Cocoon project has mooved to maven and now the
README explains that :
> > Build times will vary dramatically depending on
how good your connectivity
> > to the Maven repository/mirror is, and how well it
performs, and on whether
> > you already have a loaded local Maven repository.
A full build can take
> > from 5 minutes to maybe 4 hours...
> http://svn.apache.org/viewvc/cocoon/trunk/README
.txt?view=markup
>
> Now, for our ex-maven2ish project, we use a combinaison
of ant tasks and ivy
> for the dependencies (http://jayasoft.org/ivy).
The difference between maven
> and ivy is that ivy runs completely offline, and maven
download only on
> demand. So if you do a svn up on your project, you
probably immediately want
> to do a mvn install to download every dependencies. But
you also should run
> every maven task before getting offline : maven
jetty6:run, maven source:jar,
> maven deploy...
>
> As I said, I am a completely outsider, I am not a Nutch
user, just a Lucene
> user that read sometimes this dev list to read some
interesting discussion
> about "search". I just wanted to add my
unlucky maven user experience.
>
> Here are some notes about Sylvain, one of my coworker
that had some trouble
> with maven :
> http://b
luxte.net/blog/2006-05/21-25-04.html
> http://b
luxte.net/blog/2006-06/30-32-27.html
>
> Nicolas
>

Hi,

On 8/16/06, Lukas Vlcek <lukas.vlcekgmail.com> wrote:
> I have just noticed that there is going on some
activity called Tika
> (http://code.google.com
/p/tika/) and these guys are starting directly
> with Maven2 (am I right?).

The Maven 2 choise for the Tika staging ground was purely my
personal
choise. I have no problems migrating to a different build
system if
needed.

That said, I don't think an Ant build could ever come close
to the
descriptive power of the Maven POM like the one in
http://
tika.googlecode.com/svn/trunk/pom.xml. Combined with
sticking
with the Maven conventions that gives me an instant build
environment,
IDE integration, automatic site generation, and all sorts of
nice
extra features like automatic DOAP file generation for
projects.apache.org.

BR,

Jukka Zitting

-- 
Yukatan - http://yukatan.fi/ - infoyukatan.fi
Software craftsmanship, JCR consulting, and Java
development

Old issue.  I don't think there were any conclusions.  Not
sure if Maven2 would be a good thing, because I haven't
used Maven in 2+ years, and I understand it's changed a lot
since then.

The best way to move anything forward is to contribute the
solution/fix/patch and then persuade others to give it a try
and hope they like it.  When that happens, your contribution
typically goes in.

Otis

----- Original Message ----
From: steven shingler <shinglergmail.com>
To: nutch-devlucene.apache.org
Sent: Wednesday, August 16, 2006 10:36:17 AM
Subject: Re: Any plans to move to build Nutchusing Maven?

(This thread moved from the User List.)

OK Lukas, lets open it up to the dev list! 

Particularly, does the group feel moving to Maven would be
_a good thing_ ?

Even if so, what are the problems?

There are currently two versions of Lucene in the Maven
repos, but Hadoop
would have to be added manually, I think.

All thoughts gratefully received.
Cheers
Steven

On 8/16/06, Lukas Vlcek <lukas.vlcekgmail.com> wrote:
>
> Hi,
>
> I would like to help. But first of all I would suggest
to start wider
> discussion in dev list to get more
feedback/suggestions. I think one
> problem can be that Nutch depends on both Lucene and
Hadoop libraries
> and it won't be easy to maintain these dependencies if
recent versions
> are not yet committed into some maven accesible repo.
>
> Regards,
> Lukas
>
> On 8/16/06, steven shingler <shinglergmail.com> wrote:
> > Well I'm up for giving it a try. My current work
has me looking at both
> > Nutch and Maven, so what better way to understand
both projects 
> >
> > I agree it is far from trivial - so if anyone here
would like to
> collaborate
> > on it, that would be great.
> > Cheers,
> > Steven
> >
> >
> > On 8/15/06, Lukas Vlcek <lukas.vlcekgmail.com> wrote:
> > >
> > > Hi,
> > >
> > > I would warmly appreciate this activity. At
least it would help more
> > > people to understand/join this great project.
But I don't think this
> > > will be an easy step (this reminds me what
N.Armstrong said on moon:
> > > That's one small step for [a] man, one giant
leap for mankind.)
> > > 
> > >
> > > Regards,
> > > Lukas
> > >
> > > On 8/15/06, Sami Siren <ssirengmail.com> wrote:
> > > > steven shingler wrote:
> > > > > Hi all,
> > > > >
> > > > > I know this has come up at least
once before, but I just thought
> I'd
> > > raise
> > > > > the question again:
> > > > >
> > > > > Are there any plans to move to
building Nutch using Maven?
> > > >
> > > > Haven't heard of such activities,
however if you or somebody else
> > > > can put such thing together and it
proves to be a good thing to do
> then
> > > > I certainly don't have anything against
it.
> > > >
> > > > --
> > > >   Sami Siren
> > > >
> > >
> >
> >
>