changes in Lucene not visible through Solr

hello,

I'm pulling data into Lucene several times an hour,
approaching a  
total document count of  ~2 million. Sometimes I pull in
brand new  
data, other times I replace an existing document with an
updated  
copy. The number of documents that I update in Lucene will
pretty  
much never be more than a thousand or so.

I have a Solr interface exposed to another part of our
system, and  
it's basically sitting on top of Lucene as a read-only view
into the  
index. I can perform updates and optimizations in Lucene and
Solr  
will keep searching just fine, but I've discovered that
changes in  
Lucene are not visible through Solr.

For example, say there is no matching document in Lucene for
id=1000,  
so if I query Solr using id:1000, I will correctly find 0
matches.  
But then I import new data into Lucene, pulling in a new
document  
where id=1000. At that point, the query for id:1000 should
find one  
match, but it doesn't (0 still). If I bounce Solr, I can see
the  
results just fine.

I thought it might be a caching issue, but I have all of the
cache  
options disabled in solfconfig.xml and the problem persists.
I also  
ran Lucene optimization while Solr was running, but again no
fix. If  
anyone has any suggestions for configuring / poking Solr
somehow so  
that it will see new changes in Lucene, please let me know.

cheers,
Kaan

On 11/28/06, Kaan Erdener <kerdenerrackspace.com> wrote:

> I thought it might be a caching issue, but I have all
of the cache
> options disabled in solfconfig.xml and the problem
persists. I also
> ran Lucene optimization while Solr was running, but
again no fix. If
> anyone has any suggestions for configuring / poking
Solr somehow so
> that it will see new changes in Lucene, please let me
know.

Changes to the lucene index are not visible until you
perform
'<commit/>'.  This is true regardless of whether you
are modifying the
index directly or through solr's xml interface.

regards,
-MIke

I'm glad I asked. I probably wouldn't have discovered that
on my  
own... 

This worked great:
	curl http://localhost:89
83/solr/update --data-binary '<commit/>'

thanks,
Kaan

On Nov 29, 2006, at 12:31 AM, Mike Klaas wrote:

> On 11/28/06, Kaan Erdener <kerdenerrackspace.com> wrote:
>
>> I thought it might be a caching issue, but I have
all of the cache
>> options disabled in solfconfig.xml and the problem
persists. I also
>> ran Lucene optimization while Solr was running, but
again no fix. If
>> anyone has any suggestions for configuring / poking
Solr somehow so
>> that it will see new changes in Lucene, please let
me know.
>
> Changes to the lucene index are not visible until you
perform
> '<commit/>'.  This is true regardless of whether
you are modifying the
> index directly or through solr's xml interface.
>
> regards,
> -MIke

: This worked great:
: 	curl http://localhost:89
83/solr/update --data-binary '<commit/>'

you should go turn your caching options back on ... they'll
help out a lot
if you tend to get any repetition in your queries, and they
are "smart"
caches ... entries will expire on commit, and with
autoWarming configured
Solr will even prefill the new caches on each commit based
on the "best"
keys from the old cache.

: >> I thought it might be a caching issue, but I have
all of the cache
: >> options disabled in solfconfig.xml and the
problem persists. I also


-Hoss