Created: (NUTCH-442) Integrate Solr/Nutch

Integrate Solr/Nutch
--------------------

                 Key: NUTCH-442
                 URL: https
://issues.apache.org/jira/browse/NUTCH-442
             Project: Nutch
          Issue Type: New Feature
         Environment: Ubuntu linux
            Reporter: rubdabadub


Hi:

After trying out Sami's patch regarding Solr/Nutch. Can be
found here (http://blog.foofactory.fi/2007/02/o
nline-indexing-integrating-nutch-with.html) and I can
confirm it worked  And that
lead me to request the following :

I would be very very great full if this could be included in
nutch 0.9 as I am trying to eliminate my python based
crawler which post documents to solr. As I am in the
corporate enviornment I can't install trunk version in the
production enviornment thus I am asking this to be included
in 0.9 release. I hope my wish would be granted.

I look forward to get some feedback.

Thank you.



-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue
online.

     [
HTTPS://ISSUES.APACHE.ORG/JIRA/BROWSE/NUTCH-442?PAGE=COM.ATL
ASSIAN.JIRA.PLUGIN.SYSTEM.ISSUETABPANELS:ALL-TABPANEL ]

DO?ACAN GüNEY UPDATED NUTCH-442:
--------------------------------

    ATTACHMENT: RFC_MULTIPLE_SEARCH_BACKENDS.PATCH

HERE IS MY (VERY LARGE - SORRY) PATCH FOR THIS ISSUE:

PATCH CONSISTS OF TWO PARTS:

1) SUPPORT FOR MULTIPLE INDEXING BACKENDS: SEE NUTCH-520 FOR
DETAILS. THIS PATCH INCLUDES THE LATEST PATCH FROM
NUTCH-520.

2) SUPPORT FOR MULTIPLE SEARCH BACKENDS:

* DISTRIBUTEDSEARCH.CLIENT IS REMOVED.

* SEARCH IS DIVIDED INTO THREE MAIN PARTS:
    - SEARCHBEAN: IMPLEMENTS SEARCHER AND HITDETAILER
    - SEGMENTBEAN: IMPLEMENTS HITCONTENT AND HITSUMMARIZER
    - HITINLINKS: SAME AS OLD

    THIS DIVISION MAY SEEM ARBITRARY (AND IT ACTUALLY IS),
HOWEVER THESE ABSTRACTIONS ARE USEFUL ENOUGH THAT SOLR AND
NUTCH'S SEARCH SERVER CAN WORK. IF LATER FURTHER
ABSTRACTIONS ARE NEEDED FOR NEW SEARCH BACKENDS, THEY CAN BE
ADDED.

     THIS DIVISION ALSO HAS A NICE SIDE EFFECT: CURRENTLY,
AN SEARCH SERVER SEARCHES LUCENE INDEXES _AND_ GENERATE
SUMMARIES FOR RESULTS. AFTER THIS PATCH, IT IS NOW POSSIBLE
TO START A SEARCH SERVER THAT SEARCHES AN INDEX AND A
'SEGMENT SERVER' (THAT RETURNS CACHED CONTENT OF PAGES,
GENERATES SUMMARIES, ETC.) SEPERATELY.
DISTRIBUTEDSEARCH$INDEXSERVER (USES LUCENESEARCHBEAN) AND
DISTRIBUTEDSEARCH$SEGMENTSERVER (USES FETCHEDSEGMENTS)
CLASSES ARE ADDED FOR THIS.

*  SEARCHBEAN HIERARCHY IS LIKE THIS:

   SEARCHBEAN (EXTENDS SEARCHER, HITDETAILER)

       RPCSEARCHBEAN (EXTENDS SEARCHBEAN,
VERSIONEDPROTOCOL)

           LUCENESEARCHBEAN (IMPLEMENTS RPCSEARCHBEAN,
SEARCHES LUCENE INDEXES (MAY BE LOCAL OR ON DFS), CAN ALSO
RESPOND TO RPC REQUESTS)

       SOLRSEARCHBEAN (IMPLEMENTS SEARCHBEAN, PROCESSES
RESPONSES FROM A SOLR SERVER)

       DISTRIBUTEDSEARCHBEAN (IMPLEMENTS SEARCHBEAN, IS ALSO
A CONTAINER OF SEARCHBEANS. THIS CLASS IMPLEMENTS THE
SEARCHING PART OF DISTRIBUTEDSEARCH$CLIENT. SENDS PARALLEL
CONNECTIONS TO MULTIPLE BEANS AND MERGES THEIR RESULTS. DOES
NOT USE RPC.CALL API (SINCE NOT ALL BEANS SUPPORT HADOOP'S
RPC), INSTEAD USES A MODERN THREADING POOL FOR PARALLEL
REQUESTS.

* LOCATION OF REMOTE NUTCH/LUCENE SERVERS ARE STILL READ
FROM CRAWL/SEARCH-SERVERS.TXT. LOCATION OF SOLR SERVERS ARE
READ FROM CRAWL/SOLR-SERVERS.TXT (YES, IT SUPPORTS SEARCHING
FROM MORE THAN 1 SOLR SERVERS).

* DISTRIBUTEDSEARCHBEAN ROUTINELY SENDS PINGS TO ITS BEANS.
IF A BEAN FAILS TO RESPOND, IT IS REMOVED FROM ACTIVE LIST
OF SEARCH SERVERS (SO THAT IT DOESN'T BLOCK SEARCHING). FOR
EXAMPLE, IF SOLR SERVER DIES, DISTRIBUTEDSEARCHBEAN REALIZES
THIS AND STOPS SENDING SEARCH REQUESTS TO SOLR SERVER. LATER
WHEN SOLR COMES BACK UP, DISTRIBUTEDSEARCHBEAN RE-ADDS IT TO
ACTIVE SEARCH SERVER LIST.

* SEGMENTBEAN IS SIMILAR:
    
    SEGMENTBEAN (EXTENDS HITCONTENTS, HITSUMMARIZER)

        RPCSEGMENTBEAN (EXTENDS SEGMENTBEAN,
VERSIONEDPROTOCOL),

            FETCHEDSEGMENTS (IS SIMILAR TO OLDER VERSION)

* DISTRIBUTEDSEARCH$SEGMENTSERVER (WHICH USES
FETCHEDSEGMENTS INTERNALLY) READS ITS CONFIG FROM
CRAWL/SEGMENT-SERVERS.TXT .

* I ALSO ADDED A COUPLE OF UTILITY CLASSES FOR SENDING
REQUESTS TO SOLR AND PROCESSING RESPONSES (UNDER
O.A.N.UTIL.SOLR)

SORRY, IF THE DESCRIPTION IS A BIT COMPLEX (HOWEVER, CODE
ITSEF SHOULD BE EASY TO UNDERSTAND) . COMMENTS, SUGGESTIONS,
REVIEWS AND ALL OTHER SORTS OF FEEDBACK ARE WELCOME.


> INTEGRATE SOLR/NUTCH
> --------------------
>
>                 KEY: NUTCH-442
>                 URL:
HTTPS://ISSUES.APACHE.ORG/JIRA/BROWSE/NUTCH-442
>             PROJECT: NUTCH
>          ISSUE TYPE: NEW FEATURE
>         ENVIRONMENT: UBUNTU LINUX
>            REPORTER: RUBDABADUB
>         ATTACHMENTS:
RFC_MULTIPLE_SEARCH_BACKENDS.PATCH
>
>
> HI:
> AFTER TRYING OUT SAMI'S PATCH REGARDING SOLR/NUTCH. CAN
BE FOUND HERE
(HTTP://BLOG.FOOFACTORY.FI/2007/02/ONLINE-INDEXING-INTEGRATI
NG-NUTCH-WITH.HTML) AND I CAN CONFIRM IT WORKED  AND THAT
LEAD ME TO REQUEST THE FOLLOWING :
> I WOULD BE VERY VERY GREAT FULL IF THIS COULD BE
INCLUDED IN NUTCH 0.9 AS I AM TRYING TO ELIMINATE MY PYTHON
BASED CRAWLER WHICH POST DOCUMENTS TO SOLR. AS I AM IN THE
CORPORATE ENVIORNMENT I CAN'T INSTALL TRUNK VERSION IN THE
PRODUCTION ENVIORNMENT THUS I AM ASKING THIS TO BE INCLUDED
IN 0.9 RELEASE. I HOPE MY WISH WOULD BE GRANTED.
> I LOOK FORWARD TO GET SOME FEEDBACK.
> THANK YOU.

-- 
THIS MESSAGE IS AUTOMATICALLY GENERATED BY JIRA.
-
YOU CAN REPLY TO THIS EMAIL TO ADD A COMMENT TO THE ISSUE
ONLINE.

     [
HTTPS://ISSUES.APACHE.ORG/JIRA/BROWSE/NUTCH-442?PAGE=COM.ATL
ASSIAN.JIRA.PLUGIN.SYSTEM.ISSUETABPANELS:ALL-TABPANEL ]

DO?ACAN GüNEY UPDATED NUTCH-442:
--------------------------------

    ATTACHMENT: SCHEMA.XML

THIS SIMPLE SCHEMA CAN BE USED TO TEST SOLR INTEGRATION.

> INTEGRATE SOLR/NUTCH
> --------------------
>
>                 KEY: NUTCH-442
>                 URL:
HTTPS://ISSUES.APACHE.ORG/JIRA/BROWSE/NUTCH-442
>             PROJECT: NUTCH
>          ISSUE TYPE: NEW FEATURE
>         ENVIRONMENT: UBUNTU LINUX
>            REPORTER: RUBDABADUB
>         ATTACHMENTS:
RFC_MULTIPLE_SEARCH_BACKENDS.PATCH, SCHEMA.XML
>
>
> HI:
> AFTER TRYING OUT SAMI'S PATCH REGARDING SOLR/NUTCH. CAN
BE FOUND HERE
(HTTP://BLOG.FOOFACTORY.FI/2007/02/ONLINE-INDEXING-INTEGRATI
NG-NUTCH-WITH.HTML) AND I CAN CONFIRM IT WORKED  AND THAT
LEAD ME TO REQUEST THE FOLLOWING :
> I WOULD BE VERY VERY GREAT FULL IF THIS COULD BE
INCLUDED IN NUTCH 0.9 AS I AM TRYING TO ELIMINATE MY PYTHON
BASED CRAWLER WHICH POST DOCUMENTS TO SOLR. AS I AM IN THE
CORPORATE ENVIORNMENT I CAN'T INSTALL TRUNK VERSION IN THE
PRODUCTION ENVIORNMENT THUS I AM ASKING THIS TO BE INCLUDED
IN 0.9 RELEASE. I HOPE MY WISH WOULD BE GRANTED.
> I LOOK FORWARD TO GET SOME FEEDBACK.
> THANK YOU.

-- 
THIS MESSAGE IS AUTOMATICALLY GENERATED BY JIRA.
-
YOU CAN REPLY TO THIS EMAIL TO ADD A COMMENT TO THE ISSUE
ONLINE.

    [
HTTPS://ISSUES.APACHE.ORG/JIRA/BROWSE/NUTCH-442?PAGE=COM.ATL
ASSIAN.JIRA.PLUGIN.SYSTEM.ISSUETABPANELS:COMMENT-TABPANEL#AC
TION_12517221 ] 

DO?ACAN GüNEY COMMENTED ON NUTCH-442:
-------------------------------------

BTW, AFTER THIS PATCH IT IS POSSIBLE TO MIX DIFFERENT TYPE
OF SERVERS IN A SEARCH APPLICATION. FOR EXAMPLE, A(NY NUMBER
OF) SOLR SEARCH SERVER(S), AN OLD INDEX/SUMMARY SERVER, AN
INDEX SERVER AND A SUMMARY SERVER CAN WORK SIDE-BY-SIDE IN A
SINGLE APPLICATION.

> INTEGRATE SOLR/NUTCH
> --------------------
>
>                 KEY: NUTCH-442
>                 URL:
HTTPS://ISSUES.APACHE.ORG/JIRA/BROWSE/NUTCH-442
>             PROJECT: NUTCH
>          ISSUE TYPE: NEW FEATURE
>         ENVIRONMENT: UBUNTU LINUX
>            REPORTER: RUBDABADUB
>         ATTACHMENTS:
RFC_MULTIPLE_SEARCH_BACKENDS.PATCH, SCHEMA.XML
>
>
> HI:
> AFTER TRYING OUT SAMI'S PATCH REGARDING SOLR/NUTCH. CAN
BE FOUND HERE
(HTTP://BLOG.FOOFACTORY.FI/2007/02/ONLINE-INDEXING-INTEGRATI
NG-NUTCH-WITH.HTML) AND I CAN CONFIRM IT WORKED  AND THAT
LEAD ME TO REQUEST THE FOLLOWING :
> I WOULD BE VERY VERY GREAT FULL IF THIS COULD BE
INCLUDED IN NUTCH 0.9 AS I AM TRYING TO ELIMINATE MY PYTHON
BASED CRAWLER WHICH POST DOCUMENTS TO SOLR. AS I AM IN THE
CORPORATE ENVIORNMENT I CAN'T INSTALL TRUNK VERSION IN THE
PRODUCTION ENVIORNMENT THUS I AM ASKING THIS TO BE INCLUDED
IN 0.9 RELEASE. I HOPE MY WISH WOULD BE GRANTED.
> I LOOK FORWARD TO GET SOME FEEDBACK.
> THANK YOU.

-- 
THIS MESSAGE IS AUTOMATICALLY GENERATED BY JIRA.
-
YOU CAN REPLY TO THIS EMAIL TO ADD A COMMENT TO THE ISSUE
ONLINE.

     [
HTTPS://ISSUES.APACHE.ORG/JIRA/BROWSE/NUTCH-442?PAGE=COM.ATL
ASSIAN.JIRA.PLUGIN.SYSTEM.ISSUETABPANELS:ALL-TABPANEL ]

DO?ACAN GüNEY UPDATED NUTCH-442:
--------------------------------

    ATTACHMENT: NUTCH_442_V2.PATCH

NEW VERSION.

* PROPERLY RESET INTERNAL STATE OF SOLRRESPONSEHANDLER
BETWEEN PARSES.

* REMOVE UNUSED IMPORTS IN SCORINGFILTERS.

* REDUCE NUMBER OF WARNINGS IN O.A.N.SEARCHER.
    
    INDEXSEARCHER: REMOVE DEPRECATED FSDIRECTORY API USAGE.
REMOVE DEPRECATED FILESYSTEM API USAGE.
    LUCENEQUERYOPTIMIZER: REMOVE USAGE OF LUCENE'S
QUERYFILTER.
    EVERYTHING: SUPPRESS SERIAL UID WARNINGS AND USE
GENERICS.


* BETTER HANDLING OF HIT RESULTS IN DISTRIBUTEDSEARCHBEAN.


> INTEGRATE SOLR/NUTCH
> --------------------
>
>                 KEY: NUTCH-442
>                 URL:
HTTPS://ISSUES.APACHE.ORG/JIRA/BROWSE/NUTCH-442
>             PROJECT: NUTCH
>          ISSUE TYPE: NEW FEATURE
>         ENVIRONMENT: UBUNTU LINUX
>            REPORTER: RUBDABADUB
>         ATTACHMENTS: NUTCH_442_V2.PATCH,
RFC_MULTIPLE_SEARCH_BACKENDS.PATCH, SCHEMA.XML
>
>
> HI:
> AFTER TRYING OUT SAMI'S PATCH REGARDING SOLR/NUTCH. CAN
BE FOUND HERE
(HTTP://BLOG.FOOFACTORY.FI/2007/02/ONLINE-INDEXING-INTEGRATI
NG-NUTCH-WITH.HTML) AND I CAN CONFIRM IT WORKED  AND THAT
LEAD ME TO REQUEST THE FOLLOWING :
> I WOULD BE VERY VERY GREAT FULL IF THIS COULD BE
INCLUDED IN NUTCH 0.9 AS I AM TRYING TO ELIMINATE MY PYTHON
BASED CRAWLER WHICH POST DOCUMENTS TO SOLR. AS I AM IN THE
CORPORATE ENVIORNMENT I CAN'T INSTALL TRUNK VERSION IN THE
PRODUCTION ENVIORNMENT THUS I AM ASKING THIS TO BE INCLUDED
IN 0.9 RELEASE. I HOPE MY WISH WOULD BE GRANTED.
> I LOOK FORWARD TO GET SOME FEEDBACK.
> THANK YOU.

-- 
THIS MESSAGE IS AUTOMATICALLY GENERATED BY JIRA.
-
YOU CAN REPLY TO THIS EMAIL TO ADD A COMMENT TO THE ISSUE
ONLINE.

     [
HTTPS://ISSUES.APACHE.ORG/JIRA/BROWSE/NUTCH-442?PAGE=COM.ATL
ASSIAN.JIRA.PLUGIN.SYSTEM.ISSUETABPANELS:ALL-TABPANEL ]

DO?ACAN GüNEY UPDATED NUTCH-442:
--------------------------------

    ATTACHMENT: NUTCH_442_V3.PATCH

OOPS, ATTACHED WRONG FILE. HERE IS THE CORRECT ONE.

> INTEGRATE SOLR/NUTCH
> --------------------
>
>                 KEY: NUTCH-442
>                 URL:
HTTPS://ISSUES.APACHE.ORG/JIRA/BROWSE/NUTCH-442
>             PROJECT: NUTCH
>          ISSUE TYPE: NEW FEATURE
>         ENVIRONMENT: UBUNTU LINUX
>            REPORTER: RUBDABADUB
>         ATTACHMENTS: NUTCH_442_V2.PATCH,
NUTCH_442_V3.PATCH, RFC_MULTIPLE_SEARCH_BACKENDS.PATCH,
SCHEMA.XML
>
>
> HI:
> AFTER TRYING OUT SAMI'S PATCH REGARDING SOLR/NUTCH. CAN
BE FOUND HERE
(HTTP://BLOG.FOOFACTORY.FI/2007/02/ONLINE-INDEXING-INTEGRATI
NG-NUTCH-WITH.HTML) AND I CAN CONFIRM IT WORKED  AND THAT
LEAD ME TO REQUEST THE FOLLOWING :
> I WOULD BE VERY VERY GREAT FULL IF THIS COULD BE
INCLUDED IN NUTCH 0.9 AS I AM TRYING TO ELIMINATE MY PYTHON
BASED CRAWLER WHICH POST DOCUMENTS TO SOLR. AS I AM IN THE
CORPORATE ENVIORNMENT I CAN'T INSTALL TRUNK VERSION IN THE
PRODUCTION ENVIORNMENT THUS I AM ASKING THIS TO BE INCLUDED
IN 0.9 RELEASE. I HOPE MY WISH WOULD BE GRANTED.
> I LOOK FORWARD TO GET SOME FEEDBACK.
> THANK YOU.

-- 
THIS MESSAGE IS AUTOMATICALLY GENERATED BY JIRA.
-
YOU CAN REPLY TO THIS EMAIL TO ADD A COMMENT TO THE ISSUE
ONLINE.

     [
HTTPS://ISSUES.APACHE.ORG/JIRA/BROWSE/NUTCH-442?PAGE=COM.ATL
ASSIAN.JIRA.PLUGIN.SYSTEM.ISSUETABPANELS:ALL-TABPANEL ]

DO?ACAN GüNEY UPDATED NUTCH-442:
--------------------------------

    ATTACHMENT:     (WAS: NUTCH_442_V2.PATCH)

> INTEGRATE SOLR/NUTCH
> --------------------
>
>                 KEY: NUTCH-442
>                 URL:
HTTPS://ISSUES.APACHE.ORG/JIRA/BROWSE/NUTCH-442
>             PROJECT: NUTCH
>          ISSUE TYPE: NEW FEATURE
>         ENVIRONMENT: UBUNTU LINUX
>            REPORTER: RUBDABADUB
>         ATTACHMENTS: NUTCH_442_V3.PATCH,
RFC_MULTIPLE_SEARCH_BACKENDS.PATCH, SCHEMA.XML
>
>
> HI:
> AFTER TRYING OUT SAMI'S PATCH REGARDING SOLR/NUTCH. CAN
BE FOUND HERE
(HTTP://BLOG.FOOFACTORY.FI/2007/02/ONLINE-INDEXING-INTEGRATI
NG-NUTCH-WITH.HTML) AND I CAN CONFIRM IT WORKED  AND THAT
LEAD ME TO REQUEST THE FOLLOWING :
> I WOULD BE VERY VERY GREAT FULL IF THIS COULD BE
INCLUDED IN NUTCH 0.9 AS I AM TRYING TO ELIMINATE MY PYTHON
BASED CRAWLER WHICH POST DOCUMENTS TO SOLR. AS I AM IN THE
CORPORATE ENVIORNMENT I CAN'T INSTALL TRUNK VERSION IN THE
PRODUCTION ENVIORNMENT THUS I AM ASKING THIS TO BE INCLUDED
IN 0.9 RELEASE. I HOPE MY WISH WOULD BE GRANTED.
> I LOOK FORWARD TO GET SOME FEEDBACK.
> THANK YOU.

-- 
THIS MESSAGE IS AUTOMATICALLY GENERATED BY JIRA.
-
YOU CAN REPLY TO THIS EMAIL TO ADD A COMMENT TO THE ISSUE
ONLINE.

    [ https://issues.apache.org/jira/browse/N
UTCH-442?page=com.atlassian.jira.plugin.system.issuetabpanel
s:comment-tabpanel#action_12534869 ] 

Enis Soztutar commented on NUTCH-442:
-------------------------------------

Using nutch with solr has been a very demanding request, so
it will be very useful when this makes into trunk. I have
spend some time reviewing the patch, which I find quite
elegant. 

Some improvements to the patch would be 
- make NutchDocument implement VersionedWritable instead of
writable, and delegate version checking to superclass
- refactor getDetails() methods in HitDetailer to Searcher
(it is not likely that a class would implement Searcher but
not HitDetailer)
- use Searcher, delete HitDetailer and SearchBean 
- Rename XXXBean classes so that they do not include
"bean". (I think it is confusing to have bean
objects that have non-trivial functionality)
- refactor LuceneSearchBean.VERSION to RPCSearchBean
- remove unrelated changes from the patch.(the changes in
NGramProfile, HTMLLanguageParser,LanguageIdentifier,...
correct me if i'm wrong)

As far as i can see, we do not need any metadata for Solr
backend, and only need Store,Index and Vector options for
lucene backend, so i think we can simplify
NutchDocument#metadata. We may implement :  

class FieldMeta {
o.a.l.document.Field.Store store;
o.a.l.document.Field.Index index;
o.a.l.document.Field.TermVector tv;
}

FieldMeta[] IndexingFilter.getFields();

class NutchDocument {
...
private ArrayList<Field> fieldMeta;
...
}



Or alternatively we may wish to keep add methods of
NutchDocument compatible with o.a.l.document.Document,
keeping the metadata up-to-date as we add new fields, using
this info at LuceneWriter, but ignoring in SolrWriter. This
will be slightly slower but the API will be much more
intuitive. 

> Integrate Solr/Nutch
> --------------------
>
>                 Key: NUTCH-442
>                 URL: https
://issues.apache.org/jira/browse/NUTCH-442
>             Project: Nutch
>          Issue Type: New Feature
>         Environment: Ubuntu linux
>            Reporter: rubdabadub
>         Attachments: NUTCH_442_v3.patch,
RFC_multiple_search_backends.patch, schema.xml
>
>
> Hi:
> After trying out Sami's patch regarding Solr/Nutch. Can
be found here (http://blog.foofactory.fi/2007/02/o
nline-indexing-integrating-nutch-with.html) and I can
confirm it worked  And that
lead me to request the following :
> I would be very very great full if this could be
included in nutch 0.9 as I am trying to eliminate my python
based crawler which post documents to solr. As I am in the
corporate enviornment I can't install trunk version in the
production enviornment thus I am asking this to be included
in 0.9 release. I hope my wish would be granted.
> I look forward to get some feedback.
> Thank you.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue
online.