Rsync

I was thinking, would it not be possible to avoid using
rsync and record a list of all new segment files added (from
within Lucene), and simply use HTTP to sync down the newest
ones?  Perhaps only using rsync after an optimize?  Seems
like if I understand Lucene correctly only new files are
created?

I think the segments file will also change if documents are
deleted from the
index.

Other ways to distribute the index will works as long as:

1) it makes a copy of the index that is in a consistent
state

2) it keeps track of files that have changed (normally only
a small amount)
and transfter them to the slave

Lucene can certainly record a list of all new segment files
added.  I think
the tricky part
is to ensure that a consistent copy of the index is being
distributed.

Bill

On 3/27/06, jason rutherglen <jasonhusongyahoo.com> wrote:
>
> I was thinking, would it not be possible to avoid using
rsync and record a
> list of all new segment files added (from within
Lucene), and simply use
> HTTP to sync down the newest ones?  Perhaps only using
rsync after an
> optimize?  Seems like if I understand Lucene correctly
only new files are
> created?
>
>
>

The segments file will change when new segments are created.
 But what I
really
meant before was that the file deletable also changes when
document are
deleted from
the index.

Bill

On 3/28/06, Bill Au <bill.w.augmail.com> wrote:
>
> I think the segments file will also change if documents
are deleted from
> the index.
>
> Other ways to distribute the index will works as long
as:
>
> 1) it makes a copy of the index that is in a consistent
state
>
> 2) it keeps track of files that have changed (normally
only a small
> amount)
> and transfter them to the slave
>
> Lucene can certainly record a list of all new segment
files added.  I
> think the tricky part
> is to ensure that a consistent copy of the index is
being distributed.
>
> Bill
>
>
> On 3/27/06, jason rutherglen <jasonhusongyahoo.com> wrote:
> >
> > I was thinking, would it not be possible to avoid
using rsync and record
> > a list of all new segment files added (from within
Lucene), and simply use
> > HTTP to sync down the newest ones?  Perhaps only
using rsync after an
> > optimize?  Seems like if I understand Lucene
correctly only new files are
> > created?
> >
> >
> >
>

Perhaps a future project to increase the speed of the
syncing to sub-minute times.  Sounds like two files will
change, in addition to segment files being added.  Is this
correct?  Or maybe other pieces such as cache reloading
would make this more difficult.  

----- Original Message ----
From: Bill Au <bill.w.augmail.com>
To: solr-userlucene.apache.org; jason rutherglen
<jasonhusongyahoo.com>
Sent: Wednesday, March 29, 2006 6:07:35 AM
Subject: Re: Rsync

The segments file will change when new segments are created.
 But what I
really
meant before was that the file deletable also changes when
document are
deleted from
the index.

Bill

On 3/28/06, Bill Au <bill.w.augmail.com> wrote:
>
> I think the segments file will also change if documents
are deleted from
> the index.
>
> Other ways to distribute the index will works as long
as:
>
> 1) it makes a copy of the index that is in a consistent
state
>
> 2) it keeps track of files that have changed (normally
only a small
> amount)
> and transfter them to the slave
>
> Lucene can certainly record a list of all new segment
files added.  I
> think the tricky part
> is to ensure that a consistent copy of the index is
being distributed.
>
> Bill
>
>
> On 3/27/06, jason rutherglen <jasonhusongyahoo.com> wrote:
> >
> > I was thinking, would it not be possible to avoid
using rsync and record
> > a list of all new segment files added (from within
Lucene), and simply use
> > HTTP to sync down the newest ones?  Perhaps only
using rsync after an
> > optimize?  Seems like if I understand Lucene
correctly only new files are
> > created?
> >
> >
> >
>

On 3/29/06, jason rutherglen <jasonhusongyahoo.com> wrote:
> Perhaps a future project to increase the speed of the
syncing to sub-minute times.  Sounds like two files will
change, in addition to segment files being added.  Is this
correct?  Or maybe other pieces such as cache reloading
would make this more difficult.

rsync will only copy over the changed index files, not the
whole index
each time.

-Yonik