Reviving a dead index

Michael McCandless wrote:

/This means the segments files is referencing a segment
named _1j8s and 
in trying to load that segment, the first thing Lucene does
is load the 
"field infos" (_1j8s.fnm).  It tries to do so
from a compound file (if 
you have it turned on & it exists), else from the
filesystem directly.

/Michael,
For a moment I wondered what exactly do you mean by
"compound file"?
Then I read h
ttp://lucene.apache.org/java/docs/fileformats.html and
got 
the idea.
I do not have access to that specific machine that all this
is happening at.
It is a 80x86 machine running Win 2003 server;
Sorry, but they neglected my question about the index is
stored on a 
Local FS or on a NFS.
I was only able to obtain a directory listing of the index
dir and guess 
what - there's no a /*_1j8s.cfs * /file at all!
Pitty, I can't have a look at segments file, but I guess it
lists the _1j8s
Given these scarce resources, can you give me some further
advise about 
what has happened and what can be done to prevent it from
happening again?

Regards,
Stanislav


> Stanislav Jordanov wrote:
>> What might be the possible reason for an
IndexReader failing to open 
>> properly,
>> because it can not find a .fnm file that is
expected to be there:
>
> This means the segments files is referencing a segment
named _1j8s and 
> in trying to load that segment, the first thing Lucene
does is load 
> the "field infos" (_1j8s.fnm).  It tries to
do so from a compound file 
> (if you have it turned on & it exists), else from
the filesystem 
> directly.
>
> Which version of Lucene are you using?  And which OS
are you running on?
>
> Is this error easily repeated (not a transient error)? 
Ie, 
> instantiating an IndexSearcher against your index
always causes this 
> exception?  Because, this sort of exception is
certainly possible when 
> Lucene's locking is not working correctly (for exmple
over NFS), but 
> in that case it's typically very intermittant.
>
> Could you send a list of the files in your index?
>
>> The only thing that comes to my mind is that last
time the indexing 
>> process was not shut down properly.
>> Is there a way to revive the index or everything
should be reindexed 
>> from scratch?
>
> Hmmm.  It's surprising that an improper shutdown
caused this because 
> when the IndexWriter commits its change, it first
writes all files for 
> the new segment and only when that's successful does
it write a new 
> segments file referencing the newly written segment. 
Could you 
> provide some more detail about your setup and how the
improper 
> shutdown happened?
>
> Mike
>
>
------------------------------------------------------------
---------
> To unsubscribe, e-mail: java-user-unsubscribelucene.apache.org
> For additional commands, e-mail: java-user-helplucene.apache.org
>
>


------------------------------------------------------------
---------
To unsubscribe, e-mail: java-user-unsubscribelucene.apache.org
For additional commands, e-mail: java-user-helplucene.apache.org

I missed something that may be very important:
I find it really strange, that the exception log reads:

java.io.FileNotFoundException:
F:\Indexes\index1\_16f6.fnm (The system 
cannot find the file specified)
    at java.io.RandomAccessFile.open(Native 
Method)                                                     
    at 
java.io.RandomAccessFile.<init>(RandomAccessFile.java:
212)                                        

    at 
org.apache.lucene.store.FSIndexInput$Descriptor.<init>
(FSDirectory.java:425)                      

    at 
org.apache.lucene.store.FSIndexInput.<init>(FSDirector
y.java:434)                                 

    at 
org.apache.lucene.store.FSDirectory.openInput(FSDirectory.ja
va:324)                               

    at 
org.apache.lucene.index.FieldInfos.<init>(FieldInfos.j
ava:56)                                     

    at 
org.apache.lucene.index.SegmentReader.initialize(SegmentRead
er.java:144)                          

    at 
org.apache.lucene.index.SegmentReader.get(SegmentReader.java
:129)                                 

    at 
org.apache.lucene.index.SegmentReader.get(SegmentReader.java
:110)                                 

    at 
org.apache.lucene.index.IndexReader$1.doBody(IndexReader.jav
a:154)                                

    at 
org.apache.lucene.store.Lock$With.run(Lock.java:109)        
                                     

    at 
org.apache.lucene.index.IndexReader.open(IndexReader.java:14
3)                                    

    at 
org.apache.lucene.index.IndexReader.open(IndexReader.java:12
7)                                    

    at 
org.apache.lucene.search.IndexSearcher.<init>(IndexSea
rcher.java:42)                              



After all, the Lucene's CFS format is abstraction over the
OS's native 
FS and the App should not be trying to open a native FS file
named *.fnm
when it is supposed to open the corresponding *.cfs file and
"manually" 
extract the *.fnm file from it.
Right?


------------------------------------------------------------
---------
To unsubscribe, e-mail: java-user-unsubscribelucene.apache.org
For additional commands, e-mail: java-user-helplucene.apache.org

Stanislav Jordanov wrote:

> After all, the Lucene's CFS format is abstraction over
the OS's native 
> FS and the App should not be trying to open a native FS
file named *.fnm
> when it is supposed to open the corresponding *.cfs
file and "manually" 
> extract the *.fnm file from it.
> Right?

Yes, good catch 

This always confuses people, but it's actually
"normal" (when a segments 
file is missing) because Lucene first checks whether the
compound-file 
exists and if it does it will use that.  If it does not, it
falls back 
to trying to directly open the individual files against the
filesystem.

So, when there is a problem and a given segment is
referenced but does 
not exist, you will see this [confusing] exception making it
look like 
Lucene "forgot" that it's using the compound
file format.

[Still intending to respond to your previous email but a bit
busy right 
now...]

Mike

------------------------------------------------------------
---------
To unsubscribe, e-mail: java-user-unsubscribelucene.apache.org
For additional commands, e-mail: java-user-helplucene.apache.org

Stanislav Jordanov wrote:

> For a moment I wondered what exactly do you mean by
"compound file"?
> Then I read h
ttp://lucene.apache.org/java/docs/fileformats.html and
got 
> the idea.
> I do not have access to that specific machine that all
this is happening 
> at.
> It is a 80x86 machine running Win 2003 server;
> Sorry, but they neglected my question about the index
is stored on a 
> Local FS or on a NFS.
> I was only able to obtain a directory listing of the
index dir and guess 
> what - there's no a /*_1j8s.cfs * /file at all!
> Pitty, I can't have a look at segments file, but I
guess it lists the _1j8s
> Given these scarce resources, can you give me some
further advise about 
> what has happened and what can be done to prevent it
from happening again?

I'm assuming this is easily repeated (question from my last
email) and
not a transient error?  If it's transient, this could be
explained by
locking not working properly.

If it's not transient (ie, happens every time you open this
index),
it sounds like indeed the segments file is referencing a
segment that
does not exist.

But, how the index got into this state is a mystery.  I
don't know of
any existing Lucene bugs that can do this.  Furthermore,
crashing
an indexing process should not lead to this (it can lead to
other things
like only have a segments.new file and no segments file).

Were there any earlier exceptions (before indexing hit an
"improper
shutdown") in your indexing process that could give a
clue as to root
cause?  Or for example was the machine rebooted and did
windows to run
a "filesystem check" on rebooting this box
(which can remove corrupt
files)?

Mike

------------------------------------------------------------
---------
To unsubscribe, e-mail: java-user-unsubscribelucene.apache.org
For additional commands, e-mail: java-user-helplucene.apache.org