Commented: (LUCENE-665) temporary file access denied on Windows

While what you say is true about indexing should be
disabled, that
really doesn't solve the actual issue. Administrators of
applications
using lucene often do not have control over the actual
machine and thus
cannot determine what is and is not installed. Besides that,
many of us
do development on Windows machines and don't want the
hassle of being
forced to run the application either remotely on a unix box
or in a VM
just to work around this issue. I've hit this exact issue
during
development - it's truly an annoying issue that a simple
sleep/retry
loop should resolve.


Regards,

Bruce Ritchie

 
-----Original Message-----
From: robert engels [mailto:rengelsix.netcom.com] 
Sent: Wednesday, September 13, 2006 3:41 PM
To: java-devlucene.apache.org
Subject: Re: [jira] Commented: (LUCENE-665) temporary file
access denied
on Windows

This is a server application. Windows indexing service
should be disable
on the directories that contain Lucene locks and files.

This is the same procedure that would be required for any
database.

On Sep 13, 2006, at 2:27 PM, Michael McCandless (JIRA)
wrote:

>     [ http
://issues.apache.org/jira/browse/LUCENE-665? 
> page=comments#action_12434527 ]
>
> Michael McCandless commented on LUCENE-665:
> -------------------------------------------
>
> I do think we should make Lucene robust to
"windows change log"
> software.
>
> We could take the position that you have to uninstall
such software 
> because they "conflict" with Lucene, but I
don't think that's 
> realistic.  Apparently many packages use this
convenient API and that 
> will only get worse with time.
>
> I would put this under the "Lucene should assume
the least common 
> denominator of filesystem's capabilities"
umbrella.  Meaning, Lucene 
> now assumes it can rename files right after closing
them, but on 
> Windows this isn't a safe assumption so if possible we
should change 
> the index format to not require this.
>
> I will try to reproduce this bug with my [upcoming]
changes for 
> lockless commits (numbered segments files) -- the
lockless commits 
> changes do much less file renaming, so the issue should
be rarer (but 
> could still occur).
>
>
>> temporary file access denied on Windows
>> ---------------------------------------
>>
>>                 Key: LUCENE-665
>>                 URL: http:
//issues.apache.org/jira/browse/LUCENE-665
>>             Project: Lucene - Java
>>          Issue Type: Bug
>>          Components: Store
>>    Affects Versions: 2.0.0
>>         Environment: Windows
>>            Reporter: Doron Cohen
>>         Attachments: FSDirectory_Retry_Logic.patch,

>> FSDirs_Retry_Logic_3.patch, Test_Output.txt, 
>> TestInterleavedAddAndRemoves.java
>>
>>
>> When interleaving adds and removes there is
frequent opening/ closing

>> of readers and writers.
>> I tried to measure performance in such a scenario
(for issue 565), 
>> but the performance test failed  - the indexing
process crashed 
>> consistently with file "access denied"
errors - "cannot create a lock

>> file" in
"lockFile.createNewFile()" and "cannot
rename file".
>> This is related to:
>> - issue 516 (a closed issue: "TestFSDirectory
fails on Windows") -
>> http:
//issues.apache.org/jira/browse/LUCENE-516
>> - user list questions due to file errors:
>>   - http://www.nabble.com/OutOfMemory-and-IOException-Acces
s-
>> Denied-errors-tf1649795.html
>>   - http://www.nabble.com/running-a-lucene-indexing-app-as-
a-
>> windows-service-on-xp%2C-crashing-tf2053536.html
>> - discussion on lock-less commits http://www.nabble.co
m/Lock-less- 
>> commits-tf2126935.html My test setup is: XP (SP1),
JAVA 1.5 - both 
>> SUN and IBM SDKs.
>> I noticed that the problem is more frequent when
locks are created on

>> one disk and the index on another. Both are NTFS
with Windows 
>> indexing service enabled. I suspect this indexing
service might be 
>> related - keeping files busy for a while, but
don't know for sure.
>> After experimenting with it I conclude that these
problems - at least

>> in my scenario - are due to a temporary situation -
the FS, or the 
>> OS, is *temporarily* holding references to files or
folders, 
>> preventing from renaming them, deleting them, or
creating new files 
>> in certain directories.
>> So I added to FSDirectory a retry logic in cases
the error was 
>> related to "Access Denied". This is the
same approach brought in
>> http://www.nabble.com/running-a-lucene-indexing
-app-as-a-windows-
>> service-on-xp%2C-crashing-tf2053536.html - there,
in addition to the 
>> retry, gc() is invoked (I did not gc()). This is
based on the
>> *hope* that a access-denied situation would vanish
after a small 
>> delay, and the retry would succeed.
>> I modified FSDirectory this way for "Access
Denied" errors during 
>> creating a new files, renaming a file.
>> This worked fine for me. The performance test that
failed before, now

>> managed to complete. There should be no performance
implications due 
>> to this modification, because only the cases that
would otherwise 
>> wrongly fail are now delaying some extra millis and
retry.
>> I am attaching here a patch -
FSDirectory_Retry_Logic.patch - that 
>> has these changes to FSDirectory.
>> All "ant test" tests pass with this
patch.
>> Also attaching a test case that demostrates the
problem - at least on

>> my machine. There two tests cases in that test file
- one that works 
>> in system temp (like most Lucene tests) and one
that creates the 
>> index in a different disk. The latter case can only
run if the path 
>> ("D:" , "tmp") is valid.
>> It would be great if people that experienced these
problems could try

>> out this patch and comment whether it made any
difference for them.
>> If it turns out useful for others as well,
including this patch in 
>> the code might help to relieve some of those
"frustration" user 
>> cases.
>> A comment on state of proposed patch:
>> - It is not a "ready to deploy" code -
it has some debug printing, 
>> showing the cases that the "retry
logic" actually took place.
>> - I am not sure if current 30ms is the right
delay... why not 50ms? 
>> 10ms? This is currently defined by a constant.
>> - Should a call to gc() be added? (I think not.)
>> - Should the retry be attempted also on "non
access-denied"  
>> exceptions? (I think not).
>> - I feel it is somewhat "woodoo
programming", but though I don't like

>> it, it seems to work...
>> Attached files:
>> 1. TestInterleavedAddAndRemoves.java - the LONG
test that fails on XP

>> without the patch and passes with the patch.
>> 2. FSDirectory_Retry_Logic.patch
>> 3. Test_Output.txt- output of the test with the
patch, on my XP.  
>> Only the createNewFile() case had to be bypassed in
this test, but 
>> for another program I also saw the renameFile()
being bypassed.
>> - Doron
>
> --
> This message is automatically generated by JIRA.
> -
> If you think it was sent incorrectly contact one of the
> administrators: http://issues.a
pache.org/jira/secure/
> Administrators.jspa
> -
> For more information on JIRA, see: http://www.atlassian.com/ 
> software/jira
>
>
>
>
------------------------------------------------------------
---------
> To unsubscribe, e-mail: java-dev-unsubscribelucene.apache.org
> For additional commands, e-mail: java-dev-helplucene.apache.org
>


------------------------------------------------------------
---------
To unsubscribe, e-mail: java-dev-unsubscribelucene.apache.org
For additional commands, e-mail: java-dev-helplucene.apache.org


------------------------------------------------------------
---------
To unsubscribe, e-mail: java-dev-unsubscribelucene.apache.org
For additional commands, e-mail: java-dev-helplucene.apache.org

I do extensive Lucene on Windows machines - never had this
problem.

On Sep 13, 2006, at 2:52 PM, Bruce Ritchie wrote:

> While what you say is true about indexing should be
disabled, that
> really doesn't solve the actual issue. Administrators
of applications
> using lucene often do not have control over the actual
machine and  
> thus
> cannot determine what is and is not installed. Besides
that, many  
> of us
> do development on Windows machines and don't want the
hassle of being
> forced to run the application either remotely on a unix
box or in a VM
> just to work around this issue. I've hit this exact
issue during
> development - it's truly an annoying issue that a
simple sleep/retry
> loop should resolve.
>
>
> Regards,
>
> Bruce Ritchie
>
>
> -----Original Message-----
> From: robert engels [mailto:rengelsix.netcom.com]
> Sent: Wednesday, September 13, 2006 3:41 PM
> To: java-devlucene.apache.org
> Subject: Re: [jira] Commented: (LUCENE-665) temporary
file access  
> denied
> on Windows
>
> This is a server application. Windows indexing service
should be  
> disable
> on the directories that contain Lucene locks and files.
>
> This is the same procedure that would be required for
any database.
>
> On Sep 13, 2006, at 2:27 PM, Michael McCandless (JIRA)
wrote:
>
>>     [ http
://issues.apache.org/jira/browse/LUCENE-665?
>> page=comments#action_12434527 ]
>>
>> Michael McCandless commented on LUCENE-665:
>> -------------------------------------------
>>
>> I do think we should make Lucene robust to
"windows change log"
>> software.
>>
>> We could take the position that you have to
uninstall such software
>> because they "conflict" with Lucene,
but I don't think that's
>> realistic.  Apparently many packages use this
convenient API and that
>> will only get worse with time.
>>
>> I would put this under the "Lucene should
assume the least common
>> denominator of filesystem's capabilities"
umbrella.  Meaning, Lucene
>> now assumes it can rename files right after closing
them, but on
>> Windows this isn't a safe assumption so if
possible we should change
>> the index format to not require this.
>>
>> I will try to reproduce this bug with my [upcoming]
changes for
>> lockless commits (numbered segments files) -- the
lockless commits
>> changes do much less file renaming, so the issue
should be rarer (but
>> could still occur).
>>
>>
>>> temporary file access denied on Windows
>>> ---------------------------------------
>>>
>>>                 Key: LUCENE-665
>>>                 URL: http:
//issues.apache.org/jira/browse/LUCENE-665
>>>             Project: Lucene - Java
>>>          Issue Type: Bug
>>>          Components: Store
>>>    Affects Versions: 2.0.0
>>>         Environment: Windows
>>>            Reporter: Doron Cohen
>>>         Attachments:
FSDirectory_Retry_Logic.patch,
>>> FSDirs_Retry_Logic_3.patch, Test_Output.txt,
>>> TestInterleavedAddAndRemoves.java
>>>
>>>
>>> When interleaving adds and removes there is
frequent opening/  
>>> closing
>
>>> of readers and writers.
>>> I tried to measure performance in such a
scenario (for issue 565),
>>> but the performance test failed  - the indexing
process crashed
>>> consistently with file "access
denied" errors - "cannot create a  
>>> lock
>
>>> file" in
"lockFile.createNewFile()" and "cannot
rename file".
>>> This is related to:
>>> - issue 516 (a closed issue:
"TestFSDirectory fails on Windows") -
>>> http:
//issues.apache.org/jira/browse/LUCENE-516
>>> - user list questions due to file errors:
>>>   - http://www.nabble.com/OutOfMemory-and-IOException-Acces
s-
>>> Denied-errors-tf1649795.html
>>>   - http://www.nabble.com/running-a-lucene-indexing-app-as-
a-
>>>
windows-service-on-xp%2C-crashing-tf2053536.html
>>> - discussion on lock-less commits http://www.nabble.co
m/Lock-less-
>>> commits-tf2126935.html My test setup is: XP
(SP1), JAVA 1.5 - both
>>> SUN and IBM SDKs.
>>> I noticed that the problem is more frequent
when locks are  
>>> created on
>
>>> one disk and the index on another. Both are
NTFS with Windows
>>> indexing service enabled. I suspect this
indexing service might be
>>> related - keeping files busy for a while, but
don't know for sure.
>>> After experimenting with it I conclude that
these problems - at  
>>> least
>
>>> in my scenario - are due to a temporary
situation - the FS, or the
>>> OS, is *temporarily* holding references to
files or folders,
>>> preventing from renaming them, deleting them,
or creating new files
>>> in certain directories.
>>> So I added to FSDirectory a retry logic in
cases the error was
>>> related to "Access Denied". This is
the same approach brought in
>>> http://www.nabble.com/running-a-lucene-indexing
-app-as-a-windows-
>>> service-on-xp%2C-crashing-tf2053536.html -
there, in addition to the
>>> retry, gc() is invoked (I did not gc()). This
is based on the
>>> *hope* that a access-denied situation would
vanish after a small
>>> delay, and the retry would succeed.
>>> I modified FSDirectory this way for
"Access Denied" errors during
>>> creating a new files, renaming a file.
>>> This worked fine for me. The performance test
that failed before,  
>>> now
>
>>> managed to complete. There should be no
performance implications due
>>> to this modification, because only the cases
that would otherwise
>>> wrongly fail are now delaying some extra millis
and retry.
>>> I am attaching here a patch -
FSDirectory_Retry_Logic.patch - that
>>> has these changes to FSDirectory.
>>> All "ant test" tests pass with this
patch.
>>> Also attaching a test case that demostrates the
problem - at  
>>> least on
>
>>> my machine. There two tests cases in that test
file - one that works
>>> in system temp (like most Lucene tests) and one
that creates the
>>> index in a different disk. The latter case can
only run if the path
>>> ("D:" , "tmp") is
valid.
>>> It would be great if people that experienced
these problems could  
>>> try
>
>>> out this patch and comment whether it made any
difference for them.
>>> If it turns out useful for others as well,
including this patch in
>>> the code might help to relieve some of those
"frustration" user
>>> cases.
>>> A comment on state of proposed patch:
>>> - It is not a "ready to deploy"
code - it has some debug printing,
>>> showing the cases that the "retry
logic" actually took place.
>>> - I am not sure if current 30ms is the right
delay... why not 50ms?
>>> 10ms? This is currently defined by a constant.
>>> - Should a call to gc() be added? (I think
not.)
>>> - Should the retry be attempted also on
"non access-denied"
>>> exceptions? (I think not).
>>> - I feel it is somewhat "woodoo
programming", but though I don't  
>>> like
>
>>> it, it seems to work...
>>> Attached files:
>>> 1. TestInterleavedAddAndRemoves.java - the LONG
test that fails  
>>> on XP
>
>>> without the patch and passes with the patch.
>>> 2. FSDirectory_Retry_Logic.patch
>>> 3. Test_Output.txt- output of the test with the
patch, on my XP.
>>> Only the createNewFile() case had to be
bypassed in this test, but
>>> for another program I also saw the renameFile()
being bypassed.
>>> - Doron
>>
>> --
>> This message is automatically generated by JIRA.
>> -
>> If you think it was sent incorrectly contact one of
the
>> administrators: http://issues.a
pache.org/jira/secure/
>> Administrators.jspa
>> -
>> For more information on JIRA, see: http://www.atlassian.com/
>> software/jira
>>
>>
>>
>>
------------------------------------------------------------
---------
>> To unsubscribe, e-mail: java-dev-unsubscribelucene.apache.org
>> For additional commands, e-mail: java-dev-helplucene.apache.org
>>
>
>
>
------------------------------------------------------------
---------
> To unsubscribe, e-mail: java-dev-unsubscribelucene.apache.org
> For additional commands, e-mail: java-dev-helplucene.apache.org
>
>
>
------------------------------------------------------------
---------
> To unsubscribe, e-mail: java-dev-unsubscribelucene.apache.org
> For additional commands, e-mail: java-dev-helplucene.apache.org
>


------------------------------------------------------------
---------
To unsubscribe, e-mail: java-dev-unsubscribelucene.apache.org
For additional commands, e-mail: java-dev-helplucene.apache.org

Also, what was stated is that "this is not a safe
assumption" is  
ludicrous. It happens to not be safe because X software
package that  
was installed is holding locks on files.... If could be that
Y  
software package (virus scanner !) deletes files
automatically when  
it doesn't recognize an extension - or possibly one of the
lucene  
index files ends up matching some virus signature.

You always must be aware of the environment in which you
install  
anyway software. Having indexing software or virus scanning
software  
run on the directories that contains Lucene files is just a
bad  
(incorrect) deployment - will perform badly as best, and not
at all  
at worst.


On Sep 13, 2006, at 2:52 PM, Bruce Ritchie wrote:

> While what you say is true about indexing should be
disabled, that
> really doesn't solve the actual issue. Administrators
of applications
> using lucene often do not have control over the actual
machine and  
> thus
> cannot determine what is and is not installed. Besides
that, many  
> of us
> do development on Windows machines and don't want the
hassle of being
> forced to run the application either remotely on a unix
box or in a VM
> just to work around this issue. I've hit this exact
issue during
> development - it's truly an annoying issue that a
simple sleep/retry
> loop should resolve.
>
>
> Regards,
>
> Bruce Ritchie
>
>
> -----Original Message-----
> From: robert engels [mailto:rengelsix.netcom.com]
> Sent: Wednesday, September 13, 2006 3:41 PM
> To: java-devlucene.apache.org
> Subject: Re: [jira] Commented: (LUCENE-665) temporary
file access  
> denied
> on Windows
>
> This is a server application. Windows indexing service
should be  
> disable
> on the directories that contain Lucene locks and files.
>
> This is the same procedure that would be required for
any database.
>
> On Sep 13, 2006, at 2:27 PM, Michael McCandless (JIRA)
wrote:
>
>>     [ http
://issues.apache.org/jira/browse/LUCENE-665?
>> page=comments#action_12434527 ]
>>
>> Michael McCandless commented on LUCENE-665:
>> -------------------------------------------
>>
>> I do think we should make Lucene robust to
"windows change log"
>> software.
>>
>> We could take the position that you have to
uninstall such software
>> because they "conflict" with Lucene,
but I don't think that's
>> realistic.  Apparently many packages use this
convenient API and that
>> will only get worse with time.
>>
>> I would put this under the "Lucene should
assume the least common
>> denominator of filesystem's capabilities"
umbrella.  Meaning, Lucene
>> now assumes it can rename files right after closing
them, but on
>> Windows this isn't a safe assumption so if
possible we should change
>> the index format to not require this.
>>
>> I will try to reproduce this bug with my [upcoming]
changes for
>> lockless commits (numbered segments files) -- the
lockless commits
>> changes do much less file renaming, so the issue
should be rarer (but
>> could still occur).
>>
>>
>>> temporary file access denied on Windows
>>> ---------------------------------------
>>>
>>>                 Key: LUCENE-665
>>>                 URL: http:
//issues.apache.org/jira/browse/LUCENE-665
>>>             Project: Lucene - Java
>>>          Issue Type: Bug
>>>          Components: Store
>>>    Affects Versions: 2.0.0
>>>         Environment: Windows
>>>            Reporter: Doron Cohen
>>>         Attachments:
FSDirectory_Retry_Logic.patch,
>>> FSDirs_Retry_Logic_3.patch, Test_Output.txt,
>>> TestInterleavedAddAndRemoves.java
>>>
>>>
>>> When interleaving adds and removes there is
frequent opening/  
>>> closing
>
>>> of readers and writers.
>>> I tried to measure performance in such a
scenario (for issue 565),
>>> but the performance test failed  - the indexing
process crashed
>>> consistently with file "access
denied" errors - "cannot create a  
>>> lock
>
>>> file" in
"lockFile.createNewFile()" and "cannot
rename file".
>>> This is related to:
>>> - issue 516 (a closed issue:
"TestFSDirectory fails on Windows") -
>>> http:
//issues.apache.org/jira/browse/LUCENE-516
>>> - user list questions due to file errors:
>>>   - http://www.nabble.com/OutOfMemory-and-IOException-Acces
s-
>>> Denied-errors-tf1649795.html
>>>   - http://www.nabble.com/running-a-lucene-indexing-app-as-
a-
>>>
windows-service-on-xp%2C-crashing-tf2053536.html
>>> - discussion on lock-less commits http://www.nabble.co
m/Lock-less-
>>> commits-tf2126935.html My test setup is: XP
(SP1), JAVA 1.5 - both
>>> SUN and IBM SDKs.
>>> I noticed that the problem is more frequent
when locks are  
>>> created on
>
>>> one disk and the index on another. Both are
NTFS with Windows
>>> indexing service enabled. I suspect this
indexing service might be
>>> related - keeping files busy for a while, but
don't know for sure.
>>> After experimenting with it I conclude that
these problems - at  
>>> least
>
>>> in my scenario - are due to a temporary
situation - the FS, or the
>>> OS, is *temporarily* holding references to
files or folders,
>>> preventing from renaming them, deleting them,
or creating new files
>>> in certain directories.
>>> So I added to FSDirectory a retry logic in
cases the error was
>>> related to "Access Denied". This is
the same approach brought in
>>> http://www.nabble.com/running-a-lucene-indexing
-app-as-a-windows-
>>> service-on-xp%2C-crashing-tf2053536.html -
there, in addition to the
>>> retry, gc() is invoked (I did not gc()). This
is based on the
>>> *hope* that a access-denied situation would
vanish after a small
>>> delay, and the retry would succeed.
>>> I modified FSDirectory this way for
"Access Denied" errors during
>>> creating a new files, renaming a file.
>>> This worked fine for me. The performance test
that failed before,  
>>> now
>
>>> managed to complete. There should be no
performance implications due
>>> to this modification, because only the cases
that would otherwise
>>> wrongly fail are now delaying some extra millis
and retry.
>>> I am attaching here a patch -
FSDirectory_Retry_Logic.patch - that
>>> has these changes to FSDirectory.
>>> All "ant test" tests pass with this
patch.
>>> Also attaching a test case that demostrates the
problem - at  
>>> least on
>
>>> my machine. There two tests cases in that test
file - one that works
>>> in system temp (like most Lucene tests) and one
that creates the
>>> index in a different disk. The latter case can
only run if the path
>>> ("D:" , "tmp") is
valid.
>>> It would be great if people that experienced
these problems could  
>>> try
>
>>> out this patch and comment whether it made any
difference for them.
>>> If it turns out useful for others as well,
including this patch in
>>> the code might help to relieve some of those
"frustration" user
>>> cases.
>>> A comment on state of proposed patch:
>>> - It is not a "ready to deploy"
code - it has some debug printing,
>>> showing the cases that the "retry
logic" actually took place.
>>> - I am not sure if current 30ms is the right
delay... why not 50ms?
>>> 10ms? This is currently defined by a constant.
>>> - Should a call to gc() be added? (I think
not.)
>>> - Should the retry be attempted also on
"non access-denied"
>>> exceptions? (I think not).
>>> - I feel it is somewhat "woodoo
programming", but though I don't  
>>> like
>
>>> it, it seems to work...
>>> Attached files:
>>> 1. TestInterleavedAddAndRemoves.java - the LONG
test that fails  
>>> on XP
>
>>> without the patch and passes with the patch.
>>> 2. FSDirectory_Retry_Logic.patch
>>> 3. Test_Output.txt- output of the test with the
patch, on my XP.
>>> Only the createNewFile() case had to be
bypassed in this test, but
>>> for another program I also saw the renameFile()
being bypassed.
>>> - Doron
>>
>> --
>> This message is automatically generated by JIRA.
>> -
>> If you think it was sent incorrectly contact one of
the
>> administrators: http://issues.a
pache.org/jira/secure/
>> Administrators.jspa
>> -
>> For more information on JIRA, see: http://www.atlassian.com/
>> software/jira
>>
>>
>>
>>
------------------------------------------------------------
---------
>> To unsubscribe, e-mail: java-dev-unsubscribelucene.apache.org
>> For additional commands, e-mail: java-dev-helplucene.apache.org
>>
>
>
>
------------------------------------------------------------
---------
> To unsubscribe, e-mail: java-dev-unsubscribelucene.apache.org
> For additional commands, e-mail: java-dev-helplucene.apache.org
>
>
>
------------------------------------------------------------
---------
> To unsubscribe, e-mail: java-dev-unsubscribelucene.apache.org
> For additional commands, e-mail: java-dev-helplucene.apache.org
>


------------------------------------------------------------
---------
To unsubscribe, e-mail: java-dev-unsubscribelucene.apache.org
For additional commands, e-mail: java-dev-helplucene.apache.org

we hit it a few times, and we use dedicated servers, but
unfortunatelly someone else is hosting our app, well, if it
does not hurt someone else, would be nice to patch it
somehow (like ant did it for example)

----- Original Message ----
From: robert engels <rengelsix.netcom.com>
To: java-devlucene.apache.org
Sent: Wednesday, 13 September, 2006 9:52:29 PM
Subject: Re: [jira] Commented: (LUCENE-665) temporary file
access denied on Windows

I do extensive Lucene on Windows machines - never had this
problem.

On Sep 13, 2006, at 2:52 PM, Bruce Ritchie wrote:

> While what you say is true about indexing should be
disabled, that
> really doesn't solve the actual issue. Administrators
of applications
> using lucene often do not have control over the actual
machine and  
> thus
> cannot determine what is and is not installed. Besides
that, many  
> of us
> do development on Windows machines and don't want the
hassle of being
> forced to run the application either remotely on a unix
box or in a VM
> just to work around this issue. I've hit this exact
issue during
> development - it's truly an annoying issue that a
simple sleep/retry
> loop should resolve.
>
>
> Regards,
>
> Bruce Ritchie
>
>
> -----Original Message-----
> From: robert engels [mailto:rengelsix.netcom.com]
> Sent: Wednesday, September 13, 2006 3:41 PM
> To: java-devlucene.apache.org
> Subject: Re: [jira] Commented: (LUCENE-665) temporary
file access  
> denied
> on Windows
>
> This is a server application. Windows indexing service
should be  
> disable
> on the directories that contain Lucene locks and files.
>
> This is the same procedure that would be required for
any database.
>
> On Sep 13, 2006, at 2:27 PM, Michael McCandless (JIRA)
wrote:
>
>>     [ http
://issues.apache.org/jira/browse/LUCENE-665?
>> page=comments#action_12434527 ]
>>
>> Michael McCandless commented on LUCENE-665:
>> -------------------------------------------
>>
>> I do think we should make Lucene robust to
"windows change log"
>> software.
>>
>> We could take the position that you have to
uninstall such software
>> because they "conflict" with Lucene,
but I don't think that's
>> realistic.  Apparently many packages use this
convenient API and that
>> will only get worse with time.
>>
>> I would put this under the "Lucene should
assume the least common
>> denominator of filesystem's capabilities"
umbrella.  Meaning, Lucene
>> now assumes it can rename files right after closing
them, but on
>> Windows this isn't a safe assumption so if
possible we should change
>> the index format to not require this.
>>
>> I will try to reproduce this bug with my [upcoming]
changes for
>> lockless commits (numbered segments files) -- the
lockless commits
>> changes do much less file renaming, so the issue
should be rarer (but
>> could still occur).
>>
>>
>>> temporary file access denied on Windows
>>> ---------------------------------------
>>>
>>>                 Key: LUCENE-665
>>>                 URL: http:
//issues.apache.org/jira/browse/LUCENE-665
>>>             Project: Lucene - Java
>>>          Issue Type: Bug
>>>          Components: Store
>>>    Affects Versions: 2.0.0
>>>         Environment: Windows
>>>            Reporter: Doron Cohen
>>>         Attachments:
FSDirectory_Retry_Logic.patch,
>>> FSDirs_Retry_Logic_3.patch, Test_Output.txt,
>>> TestInterleavedAddAndRemoves.java
>>>
>>>
>>> When interleaving adds and removes there is
frequent opening/  
>>> closing
>
>>> of readers and writers.
>>> I tried to measure performance in such a
scenario (for issue 565),
>>> but the performance test failed  - the indexing
process crashed
>>> consistently with file "access
denied" errors - "cannot create a  
>>> lock
>
>>> file" in
"lockFile.createNewFile()" and "cannot
rename file".
>>> This is related to:
>>> - issue 516 (a closed issue:
"TestFSDirectory fails on Windows") -
>>> http:
//issues.apache.org/jira/browse/LUCENE-516
>>> - user list questions due to file errors:
>>>   - http://www.nabble.com/OutOfMemory-and-IOException-Acces
s-
>>> Denied-errors-tf1649795.html
>>>   - http://www.nabble.com/running-a-lucene-indexing-app-as-
a-
>>>
windows-service-on-xp%2C-crashing-tf2053536.html
>>> - discussion on lock-less commits http://www.nabble.co
m/Lock-less-
>>> commits-tf2126935.html My test setup is: XP
(SP1), JAVA 1.5 - both
>>> SUN and IBM SDKs.
>>> I noticed that the problem is more frequent
when locks are  
>>> created on
>
>>> one disk and the index on another. Both are
NTFS with Windows
>>> indexing service enabled. I suspect this
indexing service might be
>>> related - keeping files busy for a while, but
don't know for sure.
>>> After experimenting with it I conclude that
these problems - at  
>>> least
>
>>> in my scenario - are due to a temporary
situation - the FS, or the
>>> OS, is *temporarily* holding references to
files or folders,
>>> preventing from renaming them, deleting them,
or creating new files
>>> in certain directories.
>>> So I added to FSDirectory a retry logic in
cases the error was
>>> related to "Access Denied". This is
the same approach brought in
>>> http://www.nabble.com/running-a-lucene-indexing
-app-as-a-windows-
>>> service-on-xp%2C-crashing-tf2053536.html -
there, in addition to the
>>> retry, gc() is invoked (I did not gc()). This
is based on the
>>> *hope* that a access-denied situation would
vanish after a small
>>> delay, and the retry would succeed.
>>> I modified FSDirectory this way for
"Access Denied" errors during
>>> creating a new files, renaming a file.
>>> This worked fine for me. The performance test
that failed before,  
>>> now
>
>>> managed to complete. There should be no
performance implications due
>>> to this modification, because only the cases
that would otherwise
>>> wrongly fail are now delaying some extra millis
and retry.
>>> I am attaching here a patch -
FSDirectory_Retry_Logic.patch - that
>>> has these changes to FSDirectory.
>>> All "ant test" tests pass with this
patch.
>>> Also attaching a test case that demostrates the
problem - at  
>>> least on
>
>>> my machine. There two tests cases in that test
file - one that works
>>> in system temp (like most Lucene tests) and one
that creates the
>>> index in a different disk. The latter case can
only run if the path
>>> ("D:" , "tmp") is
valid.
>>> It would be great if people that experienced
these problems could  
>>> try
>
>>> out this patch and comment whether it made any
difference for them.
>>> If it turns out useful for others as well,
including this patch in
>>> the code might help to relieve some of those
"frustration" user
>>> cases.
>>> A comment on state of proposed patch:
>>> - It is not a "ready to deploy"
code - it has some debug printing,
>>> showing the cases that the "retry
logic" actually took place.
>>> - I am not sure if current 30ms is the right
delay... why not 50ms?
>>> 10ms? This is currently defined by a constant.
>>> - Should a call to gc() be added? (I think
not.)
>>> - Should the retry be attempted also on
"non access-denied"
>>> exceptions? (I think not).
>>> - I feel it is somewhat "woodoo
programming", but though I don't  
>>> like
>
>>> it, it seems to work...
>>> Attached files:
>>> 1. TestInterleavedAddAndRemoves.java - the LONG
test that fails  
>>> on XP
>
>>> without the patch and passes with the patch.
>>> 2. FSDirectory_Retry_Logic.patch
>>> 3. Test_Output.txt- output of the test with the
patch, on my XP.
>>> Only the createNewFile() case had to be
bypassed in this test, but
>>> for another program I also saw the renameFile()
being bypassed.
>>> - Doron
>>
>> --
>> This message is automatically generated by JIRA.
>> -
>> If you think it was sent incorrectly contact one of
the
>> administrators: http://issues.a
pache.org/jira/secure/
>> Administrators.jspa
>> -
>> For more information on JIRA, see: http://www.atlassian.com/
>> software/jira
>>
>>
>>
>>
------------------------------------------------------------
---------
>> To unsubscribe, e-mail: java-dev-unsubscribelucene.apache.org
>> For additional commands, e-mail: java-dev-helplucene.apache.org
>>
>
>
>
------------------------------------------------------------
---------
> To unsubscribe, e-mail: java-dev-unsubscribelucene.apache.org
> For additional commands, e-mail: java-dev-helplucene.apache.org
>
>
>
------------------------------------------------------------
---------
> To unsubscribe, e-mail: java-dev-unsubscribelucene.apache.org
> For additional commands, e-mail: java-dev-helplucene.apache.org
>


------------------------------------------------------------
---------
To unsubscribe, e-mail: java-dev-unsubscribelucene.apache.org
For additional commands, e-mail: java-dev-helplucene.apache.org





------------------------------------------------------------
---------
To unsubscribe, e-mail: java-dev-unsubscribelucene.apache.org
For additional commands, e-mail: java-dev-helplucene.apache.org

Also, folks write desktop apps with lucene... and users of
desktop search are not sys admins ... 

----- Original Message ----
From: robert engels <rengelsix.netcom.com>
To: java-devlucene.apache.org
Sent: Wednesday, 13 September, 2006 9:58:06 PM
Subject: Re: [jira] Commented: (LUCENE-665) temporary file
access denied on Windows

Also, what was stated is that "this is not a safe
assumption" is  
ludicrous. It happens to not be safe because X software
package that  
was installed is holding locks on files.... If could be that
Y  
software package (virus scanner !) deletes files
automatically when  
it doesn't recognize an extension - or possibly one of the
lucene  
index files ends up matching some virus signature.

You always must be aware of the environment in which you
install  
anyway software. Having indexing software or virus scanning
software  
run on the directories that contains Lucene files is just a
bad  
(incorrect) deployment - will perform badly as best, and not
at all  
at worst.


On Sep 13, 2006, at 2:52 PM, Bruce Ritchie wrote:

> While what you say is true about indexing should be
disabled, that
> really doesn't solve the actual issue. Administrators
of applications
> using lucene often do not have control over the actual
machine and  
> thus
> cannot determine what is and is not installed. Besides
that, many  
> of us
> do development on Windows machines and don't want the
hassle of being
> forced to run the application either remotely on a unix
box or in a VM
> just to work around this issue. I've hit this exact
issue during
> development - it's truly an annoying issue that a
simple sleep/retry
> loop should resolve.
>
>
> Regards,
>
> Bruce Ritchie
>
>
> -----Original Message-----
> From: robert engels [mailto:rengelsix.netcom.com]
> Sent: Wednesday, September 13, 2006 3:41 PM
> To: java-devlucene.apache.org
> Subject: Re: [jira] Commented: (LUCENE-665) temporary
file access  
> denied
> on Windows
>
> This is a server application. Windows indexing service
should be  
> disable
> on the directories that contain Lucene locks and files.
>
> This is the same procedure that would be required for
any database.
>
> On Sep 13, 2006, at 2:27 PM, Michael McCandless (JIRA)
wrote:
>
>>     [ http
://issues.apache.org/jira/browse/LUCENE-665?
>> page=comments#action_12434527 ]
>>
>> Michael McCandless commented on LUCENE-665:
>> -------------------------------------------
>>
>> I do think we should make Lucene robust to
"windows change log"
>> software.
>>
>> We could take the position that you have to
uninstall such software
>> because they "conflict" with Lucene,
but I don't think that's
>> realistic.  Apparently many packages use this
convenient API and that
>> will only get worse with time.
>>
>> I would put this under the "Lucene should
assume the least common
>> denominator of filesystem's capabilities"
umbrella.  Meaning, Lucene
>> now assumes it can rename files right after closing
them, but on
>> Windows this isn't a safe assumption so if
possible we should change
>> the index format to not require this.
>>
>> I will try to reproduce this bug with my [upcoming]
changes for
>> lockless commits (numbered segments files) -- the
lockless commits
>> changes do much less file renaming, so the issue
should be rarer (but
>> could still occur).
>>
>>
>>> temporary file access denied on Windows
>>> ---------------------------------------
>>>
>>>                 Key: LUCENE-665
>>>                 URL: http:
//issues.apache.org/jira/browse/LUCENE-665
>>>             Project: Lucene - Java
>>>          Issue Type: Bug
>>>          Components: Store
>>>    Affects Versions: 2.0.0
>>>         Environment: Windows
>>>            Reporter: Doron Cohen
>>>         Attachments:
FSDirectory_Retry_Logic.patch,
>>> FSDirs_Retry_Logic_3.patch, Test_Output.txt,
>>> TestInterleavedAddAndRemoves.java
>>>
>>>
>>> When interleaving adds and removes there is
frequent opening/  
>>> closing
>
>>> of readers and writers.
>>> I tried to measure performance in such a
scenario (for issue 565),
>>> but the performance test failed  - the indexing
process crashed
>>> consistently with file "access
denied" errors - "cannot create a  
>>> lock
>
>>> file" in
"lockFile.createNewFile()" and "cannot
rename file".
>>> This is related to:
>>> - issue 516 (a closed issue:
"TestFSDirectory fails on Windows") -
>>> http:
//issues.apache.org/jira/browse/LUCENE-516
>>> - user list questions due to file errors:
>>>   - http://www.nabble.com/OutOfMemory-and-IOException-Acces
s-
>>> Denied-errors-tf1649795.html
>>>   - http://www.nabble.com/running-a-lucene-indexing-app-as-
a-
>>>
windows-service-on-xp%2C-crashing-tf2053536.html
>>> - discussion on lock-less commits http://www.nabble.co
m/Lock-less-
>>> commits-tf2126935.html My test setup is: XP
(SP1), JAVA 1.5 - both
>>> SUN and IBM SDKs.
>>> I noticed that the problem is more frequent
when locks are  
>>> created on
>
>>> one disk and the index on another. Both are
NTFS with Windows
>>> indexing service enabled. I suspect this
indexing service might be
>>> related - keeping files busy for a while, but
don't know for sure.
>>> After experimenting with it I conclude that
these problems - at  
>>> least
>
>>> in my scenario - are due to a temporary
situation - the FS, or the
>>> OS, is *temporarily* holding references to
files or folders,
>>> preventing from renaming them, deleting them,
or creating new files
>>> in certain directories.
>>> So I added to FSDirectory a retry logic in
cases the error was
>>> related to "Access Denied". This is
the same approach brought in
>>> http://www.nabble.com/running-a-lucene-indexing
-app-as-a-windows-
>>> service-on-xp%2C-crashing-tf2053536.html -
there, in addition to the
>>> retry, gc() is invoked (I did not gc()). This
is based on the
>>> *hope* that a access-denied situation would
vanish after a small
>>> delay, and the retry would succeed.
>>> I modified FSDirectory this way for
"Access Denied" errors during
>>> creating a new files, renaming a file.
>>> This worked fine for me. The performance test
that failed before,  
>>> now
>
>>> managed to complete. There should be no
performance implications due
>>> to this modification, because only the cases
that would otherwise
>>> wrongly fail are now delaying some extra millis
and retry.
>>> I am attaching here a patch -
FSDirectory_Retry_Logic.patch - that
>>> has these changes to FSDirectory.
>>> All "ant test" tests pass with this
patch.
>>> Also attaching a test case that demostrates the
problem - at  
>>> least on
>
>>> my machine. There two tests cases in that test
file - one that works
>>> in system temp (like most Lucene tests) and one
that creates the
>>> index in a different disk. The latter case can
only run if the path
>>> ("D:" , "tmp") is
valid.
>>> It would be great if people that experienced
these problems could  
>>> try
>
>>> out this patch and comment whether it made any
difference for them.
>>> If it turns out useful for others as well,
including this patch in
>>> the code might help to relieve some of those
"frustration" user
>>> cases.
>>> A comment on state of proposed patch:
>>> - It is not a "ready to deploy"
code - it has some debug printing,
>>> showing the cases that the "retry
logic" actually took place.
>>> - I am not sure if current 30ms is the right
delay... why not 50ms?
>>> 10ms? This is currently defined by a constant.
>>> - Should a call to gc() be added? (I think
not.)
>>> - Should the retry be attempted also on
"non access-denied"
>>> exceptions? (I think not).
>>> - I feel it is somewhat "woodoo
programming", but though I don't  
>>> like
>
>>> it, it seems to work...
>>> Attached files:
>>> 1. TestInterleavedAddAndRemoves.java - the LONG
test that fails  
>>> on XP
>
>>> without the patch and passes with the patch.
>>> 2. FSDirectory_Retry_Logic.patch
>>> 3. Test_Output.txt- output of the test with the
patch, on my XP.
>>> Only the createNewFile() case had to be
bypassed in this test, but
>>> for another program I also saw the renameFile()
being bypassed.
>>> - Doron
>>
>> --
>> This message is automatically generated by JIRA.
>> -
>> If you think it was sent incorrectly contact one of
the
>> administrators: http://issues.a
pache.org/jira/secure/
>> Administrators.jspa
>> -
>> For more information on JIRA, see: http://www.atlassian.com/
>> software/jira
>>
>>
>>
>>
------------------------------------------------------------
---------
>> To unsubscribe, e-mail: java-dev-unsubscribelucene.apache.org
>> For additional commands, e-mail: java-dev-helplucene.apache.org
>>
>
>
>
------------------------------------------------------------
---------
> To unsubscribe, e-mail: java-dev-unsubscribelucene.apache.org
> For additional commands, e-mail: java-dev-helplucene.apache.org
>
>
>
------------------------------------------------------------
---------
> To unsubscribe, e-mail: java-dev-unsubscribelucene.apache.org
> For additional commands, e-mail: java-dev-helplucene.apache.org
>


------------------------------------------------------------
---------
To unsubscribe, e-mail: java-dev-unsubscribelucene.apache.org
For additional commands, e-mail: java-dev-helplucene.apache.org





------------------------------------------------------------
---------
To unsubscribe, e-mail: java-dev-unsubscribelucene.apache.org
For additional commands, e-mail: java-dev-helplucene.apache.org

: While what you say is true about indexing should be
disabled, that
: really doesn't solve the actual issue. Administrators of
applications
: using lucene often do not have control over the actual
machine and thus
: cannot determine what is and is not installed. Besides
that, many of us

no, but they can st the operating params of the software
they support --
as far as i can tell based on what i've read about the
issue, the problem
stems from software that does what seems to be the
equivilent of trying to
open any file that has recently been closed by any other app
... as
a java library Lucene shouldn't be expected to guard
against that any more
then it should be expected to guard against the possibility
of people
randmoly renaming or deleting files in the index directory
-- Lucene makes
some files, and it expects to manage them without
interference from
anything else -- as long as that expectation is documented
(and i'll
admit, those expectations could be documented more
thoroughly) then i
think our work here is done.

Dealing withthis problem becomes the topic of a FAQ about
the cause with a
pointer to a Wiki explaining the details and (*hopefully*) a
list of crazy
registry hacks you can use to disable it on a per directory
basis.

(DISCLAIMER: I am not saying such hacks exist, merely that i
hope some
method of disabling this "feature" exists and if
it does pointeres on how
to do so would be useful)



-Hoss


------------------------------------------------------------
---------
To unsubscribe, e-mail: java-dev-unsubscribelucene.apache.org
For additional commands, e-mail: java-dev-helplucene.apache.org

not promoting, "let lucene fix all Winblows
problems", just saying, if someone has cool, simple
trick in patch form, that hurts nobody, would be nice to
accept it. Enough people burned their fingers on this one

----- Original Message ----
From: Chris Hostetter <hossman_lucenefucit.org>
To: java-devlucene.apache.org
Sent: Wednesday, 13 September, 2006 10:04:05 PM
Subject: RE: [jira] Commented: (LUCENE-665) temporary file
access denied on Windows

: While what you say is true about indexing should be
disabled, that
: really doesn't solve the actual issue. Administrators of
applications
: using lucene often do not have control over the actual
machine and thus
: cannot determine what is and is not installed. Besides
that, many of us

no, but they can st the operating params of the software
they support --
as far as i can tell based on what i've read about the
issue, the problem
stems from software that does what seems to be the
equivilent of trying to
open any file that has recently been closed by any other app
... as
a java library Lucene shouldn't be expected to guard
against that any more
then it should be expected to guard against the possibility
of people
randmoly renaming or deleting files in the index directory
-- Lucene makes
some files, and it expects to manage them without
interference from
anything else -- as long as that expectation is documented
(and i'll
admit, those expectations could be documented more
thoroughly) then i
think our work here is done.

Dealing withthis problem becomes the topic of a FAQ about
the cause with a
pointer to a Wiki explaining the details and (*hopefully*) a
list of crazy
registry hacks you can use to disable it on a per directory
basis.

(DISCLAIMER: I am not saying such hacks exist, merely that i
hope some
method of disabling this "feature" exists and if
it does pointeres on how
to do so would be useful)



-Hoss


------------------------------------------------------------
---------
To unsubscribe, e-mail: java-dev-unsubscribelucene.apache.org
For additional commands, e-mail: java-dev-helplucene.apache.org





------------------------------------------------------------
---------
To unsubscribe, e-mail: java-dev-unsubscribelucene.apache.org
For additional commands, e-mail: java-dev-helplucene.apache.org

Start -> Administrative Tools -> Windows Indexing
Service -> Disable

On Sep 13, 2006, at 3:09 PM, eks dev wrote:

> not promoting, "let lucene fix all Winblows
problems", just saying,  
> if someone has cool, simple trick in patch form, that
hurts nobody,  
> would be nice to accept it. Enough people burned their
fingers on  
> this one
>
> ----- Original Message ----
> From: Chris Hostetter <hossman_lucenefucit.org>
> To: java-devlucene.apache.org
> Sent: Wednesday, 13 September, 2006 10:04:05 PM
> Subject: RE: [jira] Commented: (LUCENE-665) temporary
file access  
> denied on Windows
>
> : While what you say is true about indexing should be
disabled, that
> : really doesn't solve the actual issue.
Administrators of  
> applications
> : using lucene often do not have control over the
actual machine  
> and thus
> : cannot determine what is and is not installed.
Besides that, many  
> of us
>
> no, but they can st the operating params of the
software they  
> support --
> as far as i can tell based on what i've read about the
issue, the  
> problem
> stems from software that does what seems to be the
equivilent of  
> trying to
> open any file that has recently been closed by any
other app ... as
> a java library Lucene shouldn't be expected to guard
against that  
> any more
> then it should be expected to guard against the
possibility of people
> randmoly renaming or deleting files in the index
directory --  
> Lucene makes
> some files, and it expects to manage them without
interference from
> anything else -- as long as that expectation is
documented (and i'll
> admit, those expectations could be documented more
thoroughly) then i
> think our work here is done.
>
> Dealing withthis problem becomes the topic of a FAQ
about the cause  
> with a
> pointer to a Wiki explaining the details and
(*hopefully*) a list  
> of crazy
> registry hacks you can use to disable it on a per
directory basis.
>
> (DISCLAIMER: I am not saying such hacks exist, merely
that i hope some
> method of disabling this "feature" exists
and if it does pointeres  
> on how
> to do so would be useful)
>
>
>
> -Hoss
>
>
>
------------------------------------------------------------
---------
> To unsubscribe, e-mail: java-dev-unsubscribelucene.apache.org
> For additional commands, e-mail: java-dev-helplucene.apache.org
>
>
>
>
>
>
------------------------------------------------------------
---------
> To unsubscribe, e-mail: java-dev-unsubscribelucene.apache.org
> For additional commands, e-mail: java-dev-helplucene.apache.org
>


------------------------------------------------------------
---------
To unsubscribe, e-mail: java-dev-unsubscribelucene.apache.org
For additional commands, e-mail: java-dev-helplucene.apache.org

> Start -> Administrative Tools -> Windows Indexing
Service -> Disable

Yes, tried that of course, just to discover that it was not
the cause in my
case...


------------------------------------------------------------
---------
To unsubscribe, e-mail: java-dev-unsubscribelucene.apache.org
For additional commands, e-mail: java-dev-helplucene.apache.org

> eks dev <eksdevyahoo.co.uk> wrote on 13/09/2006
13:09:42:
> would be nice to accept it. Enough people burned
> their fingers on this one

> > <brucejivesoftware.com> wrote on 13/09/2006
12:52:00:
> > I've hit this exact issue during development -
it's truly
> > an annoying issue that a simple sleep/retry
> > loop should resolve.

Actually this is what the current patch attempts for
(FSDirs_Retry_Logic_3)
- did you get a chance to try it?


------------------------------------------------------------
---------
To unsubscribe, e-mail: java-dev-unsubscribelucene.apache.org
For additional commands, e-mail: java-dev-helplucene.apache.org