Index HotSwap

Hi all,
 I'm wondering what's the best way to completely change a
big index
without loosing any requests.
 That's how I do at the moment:

 solr index is a soft link to a directory dir.
 When I want to install a new index (in dir.new), I do a

 mv dir dir.old ; mv dir.new dir
 Then I ask for a reload of the solr application (within
tomcat).

 I can see two problems with this method:

  - Between the two mv's, the directory dir does not exists,
which can
cause some solr failure.

  - Apparently It's not that safe to reload a webapp within
tomcat.
   I thought it was the equivalent of the apache graceful
reloading
(completing current requests and putting incoming ones into
a queue
while the application restarts), but it's   apparently not.
I noticed
we have a couple of query lost when it happens.
 One is a 503 This application is not currently available,
and the one
just after is
 a 404 /solr//select/ - The requested resource
(/solr//select/) is not
available.

 Does anybody know how to avoid this behaviour, and
eventually what is
the best way to swap between two big indexes.

Thanks for any help !

Jerome.





-- 
Jerome Eteve.
jeromeeteve.net
http://jerome.eteve.free
.fr/

I guess the first question is why you have to swap in a big
index, instead of rsyc'ng or another method.  I've
entertained the idea of putting a load balancer in front of
two solr instances.  In this scenario take one off-line swap
in the index, bring it back on and then bring down the
other.  Not being fluent in load balancing I believe in
theory this could work.


-Andrew

-----Original Message-----
From: jerome.etevegmail.com [mailto:jerome.etevegmail.com] On Behalf Of Jérôme Etévé
Sent: Tuesday, August 21, 2007 1:52 PM
To: solr-userlucene.apache.org
Subject: Index HotSwap

Hi all,
 I'm wondering what's the best way to completely change a
big index
without loosing any requests.
 That's how I do at the moment:

 solr index is a soft link to a directory dir.
 When I want to install a new index (in dir.new), I do a

 mv dir dir.old ; mv dir.new dir
 Then I ask for a reload of the solr application (within
tomcat).

 I can see two problems with this method:

  - Between the two mv's, the directory dir does not exists,
which can
cause some solr failure.

  - Apparently It's not that safe to reload a webapp within
tomcat.
   I thought it was the equivalent of the apache graceful
reloading
(completing current requests and putting incoming ones into
a queue
while the application restarts), but it's   apparently not.
I noticed
we have a couple of query lost when it happens.
 One is a 503 This application is not currently available,
and the one
just after is
 a 404 /solr//select/ - The requested resource
(/solr//select/) is not
available.

 Does anybody know how to avoid this behaviour, and
eventually what is
the best way to swap between two big indexes.

Thanks for any help !

Jerome.





-- 
Jerome Eteve.
jeromeeteve.net
http://jerome.eteve.free
.fr/

:  I'm wondering what's the best way to completely change a
big index
: without loosing any requests.

use the snapinstaller script -- or adopt the same atomic
copying approach
it uses.

:   - Between the two mv's, the directory dir does not
exists, which can
: cause some solr failure.

this shouldn't cause any failure unless you tell Solr to try
and reload
turing the move (ie: you send it a commit) ... either way an
atomic copy
in place of a mv should work much better.


-Hoss

On 8/21/07, Chris Hostetter <hossman_lucenefucit.org> wrote:
>
> :  I'm wondering what's the best way to completely
change a big index
> : without loosing any requests.
>
> use the snapinstaller script -- or adopt the same
atomic copying approach
> it uses.

I'm having a look 

> :   - Between the two mv's, the directory dir does not
exists, which can
> : cause some solr failure.
>
> this shouldn't cause any failure unless you tell Solr
to try and reload
> turing the move (ie: you send it a commit) ... either
way an atomic copy
> in place of a mv should work much better.

Why, does the reloading of the searcher triggers a re
loading of the
files from disk ?
Thx !

>
> -Hoss
>
>


-- 
Jerome Eteve.
jeromeeteve.net
http://jerome.eteve.free
.fr/