Stale NFS handle

Hi,

We are puzzled by an apache2 hanging in an 'Uninterrubtable
sleep'. The
error.log tells us about a 'Stale NFS handle'.
Unfortunately the log
does not say which file it tried to access, only which
request it was
trying to serve.

I've already tried with hard/soft mounted NFS, I tried
disabling MMAP as
well as Sendfile. None of that helped. The LockFile remains
local
on /var/lock... Further googling did not reveal other hints.

We are on debian testing, linux kernel 2.6.11-1-k7, apache
2.0.55-4
(prefork) with mod_php 4.4.2-1.1. We are working on the php
files, which
might therefore be edited or (re)moved under apache, but
normally that
doesn't pose any problems, does it?

Help greatly appreciated.

Regards

Markus


------------------------------------------------------------
---------
The official User-To-User support forum of the Apache HTTP
Server Project.
See <URL:http://htt
pd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribehttpd.apache.org
   "   from the digest: users-digest-unsubscribehttpd.apache.org
For additional commands, e-mail: users-helphttpd.apache.org

	Hiyas,

Markus Schiltknecht escreveu:
> Hi,
> 
> We are puzzled by an apache2 hanging in an
'Uninterrubtable sleep'. The
> error.log tells us about a 'Stale NFS handle'.
Unfortunately the log
> does not say which file it tried to access, only which
request it was
> trying to serve.

	When reading content from a NFS filesystem, most
operational systems 
try to "cache" the data just read for an amount
of time. On Linux, for 
example, each time you read a block/inode from a NFS
mountpoint, it will 
cache its content for 30 seconds (value is aproximate, I'm
not sure of 
the exact value).

	Usually, apache will use this "inode cache",
when reading data from 
NFS. Now, lets say an apache "child" reads a
file for the first time 
(one request is made to the NFS server). During 30 seconds,
any 
subsequent read from this child to the same file will use
the "inode 
cache" (which means the NFS server will not be
contacted in any way). 
After this period, the cache is discarted, and the child
will try to 
read that same inode again. If something changed/deleted
this inode in 
the mean time, the SO will give a 'Stale NFS handle' error
to the child 
(after all, it tried to access an invalid file handler).

> I've already tried with hard/soft mounted NFS, I tried
disabling MMAP as
> well as Sendfile. None of that helped. The LockFile
remains local
> on /var/lock... Further googling did not reveal other
hints.

	In any case, you should always disable Sendfile and MMAP
when serving 
content from a NFS mounted filesystem. Also, your lockfile
should always 
point to a local filesystem (/var/lock usually is a good
choice), and 
the same rule is valid for your scoreboard file (either you
point it to 
a local file, or you comment the
"ScoreBoardFile" keyword, which will 
force apache to use a shared memory segment for the
scoreboard).

> We are on debian testing, linux kernel 2.6.11-1-k7,
apache 2.0.55-4
> (prefork) with mod_php 4.4.2-1.1. We are working on the
php files, which
> might therefore be edited or (re)moved under apache,
but normally that
> doesn't pose any problems, does it?

	Yes, it usually does. You gotta be really careful when
modifying files 
on an NFS partition, if something else is reading the same
data from it 
(your apache server, in this case).

	There is no real solution for this issue. The basic problem
lies in the 
fact that you "reuse" the inode, when modifying
a file. You can try one 
of the following:

- If you're serving just "static" content
(html, images and the such), 
try to always create new files, with new names. Do not try
to reuse 
filenames at all, if possible;

- If you're serving dynamic content (and in this case,
modifying PHP 
scripts), you must somehow modify the way you
"publish" your content on 
your NFS filesystem. You must always try to use
"atomic" operations. 
That could be done in the following way:
	1 - Lets say you're modifying a file called login.php.
First, you copy 
it to a new name (which in turn will create a new inode).
For example, 
".new.login.php.<pid>.<timestamp>";
	2 - Then, you modify the file as usual, but using the new
file (the old 
file is still valid);
	3 - After you finished modifying the new file, you must
find a way to 
not "invalidate" any inode in the
"move" process. This is done in 3 steps:
	3.1 - Create a hardlink of the old file to a temporary
directory (temp 
dir must be on the same NFS partition). Something like that:
"ln -f 
login.php
temp/login.php.<pid>.<timestamp>"
	3.2 - Overwrite the old file with the new file, using an
atomic 
operation. This is done like that: "ln -f 
.new.login.php.<pid>.<timestamp>
login.php"
	3.3 - After a few minutes (you _must_ wait at least 30
seconds), remove 
the temporary file
"temp/login.php.<pid>.<timestamp>".
It is safer to 
use a tool like "tmpwatch" (via cron) to clean
files automatically in 
the "temp" directory (for example, all files
created after 1 hour).

	I know this is not simple to understand, and most
important, it is 
really hard to implement. Its easier if you have your own
system do 
publish content, but could be done too if you use samba or
ftp to 
publish your files (look in the configuration files for
those two 
daemons, and see if you find a way to create temp files on
the upload 
process, and then move the temp file into the old one).

> Help greatly appreciated.
> 
> Regards

	Regards,

> Markus

Domingos.

------------------------------------------------------------
---------
The official User-To-User support forum of the Apache HTTP
Server Project.
See <URL:http://htt
pd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribehttpd.apache.org
   "   from the digest: users-digest-unsubscribehttpd.apache.org
For additional commands, e-mail: users-helphttpd.apache.org

	Hiyas,

	One or two "bugfixes" on my last message. 

Domingos Parra Novo escreveu:
> Markus Schiltknecht escreveu:
<snip>
>> I've already tried with hard/soft mounted NFS, I
tried disabling MMAP as
>> well as Sendfile. None of that helped. The LockFile
remains local
>> on /var/lock... Further googling did not reveal
other hints.
> 
>     In any case, you should always disable Sendfile and
MMAP when 
> serving content from a NFS mounted filesystem. Also,
your lockfile 
> should always point to a local filesystem (/var/lock
usually is a good 
> choice), and the same rule is valid for your scoreboard
file (either you 
> point it to a local file, or you comment the
"ScoreBoardFile" keyword, 
> which will force apache to use a shared memory segment
for the scoreboard).

	Just as a sidenote, none of my recomendations above will
solve your 
specific issue. It is just that you must follow them, to
certify that 
you're not going to find any other NFS related issues on
your apache 
server (eg, modifying a file, while apache continues to
"magically" 
deliver the old content, which does not exist anymore).

>> We are on debian testing, linux kernel 2.6.11-1-k7,
apache 2.0.55-4
>> (prefork) with mod_php 4.4.2-1.1. We are working on
the php files, which
>> might therefore be edited or (re)moved under
apache, but normally that
>> doesn't pose any problems, does it?
> 
>     Yes, it usually does. You gotta be really careful
when modifying 
> files on an NFS partition, if something else is reading
the same data 
> from it (your apache server, in this case).
> 
>     There is no real solution for this issue. The basic
problem lies in 
> the fact that you "reuse" the inode, when
modifying a file. You can try 
> one of the following:
> 
> - If you're serving just "static" content
(html, images and the such), 
> try to always create new files, with new names. Do not
try to reuse 
> filenames at all, if possible;
> 
> - If you're serving dynamic content (and in this case,
modifying PHP 
> scripts), you must somehow modify the way you
"publish" your content on 
> your NFS filesystem. You must always try to use
"atomic" operations. 
> That could be done in the following way:
>     1 - Lets say you're modifying a file called
login.php. First, you 
> copy it to a new name (which in turn will create a new
inode). For 
> example,
".new.login.php.<pid>.<timestamp>";
>     2 - Then, you modify the file as usual, but using
the new file (the 
> old file is still valid);
>     3 - After you finished modifying the new file, you
must find a way 
> to not "invalidate" any inode in the
"move" process. This is done in 3 
> steps:
>     3.1 - Create a hardlink of the old file to a
temporary directory 
> (temp dir must be on the same NFS partition). Something
like that: "ln 
> -f login.php
temp/login.php.<pid>.<timestamp>"
>     3.2 - Overwrite the old file with the new file,
using an atomic 
> operation. This is done like that: "ln -f 
> .new.login.php.<pid>.<timestamp>
login.php"

	Whoops. That would not work. The correct command would be
something 
like that: "mv -f
.new.login.php.<pid>.<timestamp>
login.php"

>     3.3 - After a few minutes (you _must_ wait at least
30 seconds), 
> remove the temporary file
"temp/login.php.<pid>.<timestamp>".
It is 
> safer to use a tool like "tmpwatch" (via
cron) to clean files 
> automatically in the "temp" directory (for
example, all files created 
> after 1 hour).
> 
>     I know this is not simple to understand, and most
important, it is 
> really hard to implement. Its easier if you have your
own system do 
> publish content, but could be done too if you use samba
or ftp to 
> publish your files (look in the configuration files for
those two 
> daemons, and see if you find a way to create temp files
on the upload 
> process, and then move the temp file into the old one).
> 
>> Help greatly appreciated.
>>
>> Regards
> 
>     Regards,
> 
>> Markus
> 
> Domingos.

Regards,

Domingos.


------------------------------------------------------------
---------
The official User-To-User support forum of the Apache HTTP
Server Project.
See <URL:http://htt
pd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribehttpd.apache.org
   "   from the digest: users-digest-unsubscribehttpd.apache.org
For additional commands, e-mail: users-helphttpd.apache.org

Hello Domingos,

thank you very much for this in depth answer. It helped me
to better
understand NFS in general. Should that probably be
documented in the
apache manual (or is it already?)

Regards

Markus

On Fri, 2006-07-28 at 14:19 -0300, Domingos Parra Novo
wrote: 
> 	Hiyas,
> 
> Markus Schiltknecht escreveu:
> > Hi,
> > 
> > We are puzzled by an apache2 hanging in an
'Uninterrubtable sleep'. The
> > error.log tells us about a 'Stale NFS handle'.
Unfortunately the log
> > does not say which file it tried to access, only
which request it was
> > trying to serve.
> 
> 	When reading content from a NFS filesystem, most
operational systems 
> try to "cache" the data just read for an
amount of time. On Linux, for 
> example, each time you read a block/inode from a NFS
mountpoint, it will 
> cache its content for 30 seconds (value is aproximate,
I'm not sure of 
> the exact value).
> 
> 	Usually, apache will use this "inode
cache", when reading data from 
> NFS. Now, lets say an apache "child" reads
a file for the first time 
> (one request is made to the NFS server). During 30
seconds, any 
> subsequent read from this child to the same file will
use the "inode 
> cache" (which means the NFS server will not be
contacted in any way). 
> After this period, the cache is discarted, and the
child will try to 
> read that same inode again. If something
changed/deleted this inode in 
> the mean time, the SO will give a 'Stale NFS handle'
error to the child 
> (after all, it tried to access an invalid file
handler).
> 
> > I've already tried with hard/soft mounted NFS, I
tried disabling MMAP as
> > well as Sendfile. None of that helped. The
LockFile remains local
> > on /var/lock... Further googling did not reveal
other hints.
> 
> 	In any case, you should always disable Sendfile and
MMAP when serving 
> content from a NFS mounted filesystem. Also, your
lockfile should always 
> point to a local filesystem (/var/lock usually is a
good choice), and 
> the same rule is valid for your scoreboard file (either
you point it to 
> a local file, or you comment the
"ScoreBoardFile" keyword, which will 
> force apache to use a shared memory segment for the
scoreboard).
> 
> > We are on debian testing, linux kernel
2.6.11-1-k7, apache 2.0.55-4
> > (prefork) with mod_php 4.4.2-1.1. We are working
on the php files, which
> > might therefore be edited or (re)moved under
apache, but normally that
> > doesn't pose any problems, does it?
> 
> 	Yes, it usually does. You gotta be really careful when
modifying files 
> on an NFS partition, if something else is reading the
same data from it 
> (your apache server, in this case).
> 
> 	There is no real solution for this issue. The basic
problem lies in the 
> fact that you "reuse" the inode, when
modifying a file. You can try one 
> of the following:
> 
> - If you're serving just "static" content
(html, images and the such), 
> try to always create new files, with new names. Do not
try to reuse 
> filenames at all, if possible;
> 
> - If you're serving dynamic content (and in this case,
modifying PHP 
> scripts), you must somehow modify the way you
"publish" your content on 
> your NFS filesystem. You must always try to use
"atomic" operations. 
> That could be done in the following way:
> 	1 - Lets say you're modifying a file called
login.php. First, you copy 
> it to a new name (which in turn will create a new
inode). For example, 
>
".new.login.php.<pid>.<timestamp>";
> 	2 - Then, you modify the file as usual, but using the
new file (the old 
> file is still valid);
> 	3 - After you finished modifying the new file, you
must find a way to 
> not "invalidate" any inode in the
"move" process. This is done in 3 steps:
> 	3.1 - Create a hardlink of the old file to a temporary
directory (temp 
> dir must be on the same NFS partition). Something like
that: "ln -f 
> login.php
temp/login.php.<pid>.<timestamp>"
> 	3.2 - Overwrite the old file with the new file, using
an atomic 
> operation. This is done like that: "ln -f 
> .new.login.php.<pid>.<timestamp>
login.php"
> 	3.3 - After a few minutes (you _must_ wait at least 30
seconds), remove 
> the temporary file
"temp/login.php.<pid>.<timestamp>".
It is safer to 
> use a tool like "tmpwatch" (via cron) to
clean files automatically in 
> the "temp" directory (for example, all
files created after 1 hour).
> 
> 	I know this is not simple to understand, and most
important, it is 
> really hard to implement. Its easier if you have your
own system do 
> publish content, but could be done too if you use samba
or ftp to 
> publish your files (look in the configuration files for
those two 
> daemons, and see if you find a way to create temp files
on the upload 
> process, and then move the temp file into the old one).
> 
> > Help greatly appreciated.
> > 
> > Regards
> 
> 	Regards,
> 
> > Markus
> 
> Domingos.
> 
>
------------------------------------------------------------
---------
> The official User-To-User support forum of the Apache
HTTP Server Project.
> See <URL:http://htt
pd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribehttpd.apache.org
>    "   from the digest:
users-digest-unsubscribehttpd.apache.org
> For additional commands, e-mail: users-helphttpd.apache.org
> 


------------------------------------------------------------
---------
The official User-To-User support forum of the Apache HTTP
Server Project.
See <URL:http://htt
pd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribehttpd.apache.org
   "   from the digest: users-digest-unsubscribehttpd.apache.org
For additional commands, e-mail: users-helphttpd.apache.org

	Hiyas Markus,

Markus Schiltknecht escreveu:
> Hello Domingos,
> 
> thank you very much for this in depth answer. It helped
me to better
> understand NFS in general. Should that probably be
documented in the
> apache manual (or is it already?)

	Not really in depth. Maybe a bit long (and boring). And
I'm positive 
I've mangled something, somewhere in my text. 

	Anyway, never found such kind of documentation anywhere
online, to tell 
you the truth. And I don't know if the Apache doc is the
best place to 
store it, as the issue is not really related to Apache, but,
to all 
applications in general, reading and writing to NFS.

> Regards

	Regards,

> Markus

Domingos.

> On Fri, 2006-07-28 at 14:19 -0300, Domingos Parra Novo
wrote: 
>> 	Hiyas,
>>
>> Markus Schiltknecht escreveu:
>>> Hi,
>>>
>>> We are puzzled by an apache2 hanging in an
'Uninterrubtable sleep'. The
>>> error.log tells us about a 'Stale NFS
handle'. Unfortunately the log
>>> does not say which file it tried to access,
only which request it was
>>> trying to serve.
>> 	When reading content from a NFS filesystem, most
operational systems 
>> try to "cache" the data just read for
an amount of time. On Linux, for 
>> example, each time you read a block/inode from a
NFS mountpoint, it will 
>> cache its content for 30 seconds (value is
aproximate, I'm not sure of 
>> the exact value).
>>
>> 	Usually, apache will use this "inode
cache", when reading data from 
>> NFS. Now, lets say an apache "child"
reads a file for the first time 
>> (one request is made to the NFS server). During 30
seconds, any 
>> subsequent read from this child to the same file
will use the "inode 
>> cache" (which means the NFS server will not
be contacted in any way). 
>> After this period, the cache is discarted, and the
child will try to 
>> read that same inode again. If something
changed/deleted this inode in 
>> the mean time, the SO will give a 'Stale NFS
handle' error to the child 
>> (after all, it tried to access an invalid file
handler).
>>
>>> I've already tried with hard/soft mounted NFS,
I tried disabling MMAP as
>>> well as Sendfile. None of that helped. The
LockFile remains local
>>> on /var/lock... Further googling did not reveal
other hints.
>> 	In any case, you should always disable Sendfile
and MMAP when serving 
>> content from a NFS mounted filesystem. Also, your
lockfile should always 
>> point to a local filesystem (/var/lock usually is a
good choice), and 
>> the same rule is valid for your scoreboard file
(either you point it to 
>> a local file, or you comment the
"ScoreBoardFile" keyword, which will 
>> force apache to use a shared memory segment for the
scoreboard).
>>
>>> We are on debian testing, linux kernel
2.6.11-1-k7, apache 2.0.55-4
>>> (prefork) with mod_php 4.4.2-1.1. We are
working on the php files, which
>>> might therefore be edited or (re)moved under
apache, but normally that
>>> doesn't pose any problems, does it?
>> 	Yes, it usually does. You gotta be really careful
when modifying files 
>> on an NFS partition, if something else is reading
the same data from it 
>> (your apache server, in this case).
>>
>> 	There is no real solution for this issue. The
basic problem lies in the 
>> fact that you "reuse" the inode, when
modifying a file. You can try one 
>> of the following:
>>
>> - If you're serving just "static"
content (html, images and the such), 
>> try to always create new files, with new names. Do
not try to reuse 
>> filenames at all, if possible;
>>
>> - If you're serving dynamic content (and in this
case, modifying PHP 
>> scripts), you must somehow modify the way you
"publish" your content on 
>> your NFS filesystem. You must always try to use
"atomic" operations. 
>> That could be done in the following way:
>> 	1 - Lets say you're modifying a file called
login.php. First, you copy 
>> it to a new name (which in turn will create a new
inode). For example, 
>>
".new.login.php.<pid>.<timestamp>";
>> 	2 - Then, you modify the file as usual, but using
the new file (the old 
>> file is still valid);
>> 	3 - After you finished modifying the new file, you
must find a way to 
>> not "invalidate" any inode in the
"move" process. This is done in 3 steps:
>> 	3.1 - Create a hardlink of the old file to a
temporary directory (temp 
>> dir must be on the same NFS partition). Something
like that: "ln -f 
>> login.php
temp/login.php.<pid>.<timestamp>"
>> 	3.2 - Overwrite the old file with the new file,
using an atomic 
>> operation. This is done like that: "ln -f 
>> .new.login.php.<pid>.<timestamp>
login.php"
>> 	3.3 - After a few minutes (you _must_ wait at
least 30 seconds), remove 
>> the temporary file
"temp/login.php.<pid>.<timestamp>".
It is safer to 
>> use a tool like "tmpwatch" (via cron)
to clean files automatically in 
>> the "temp" directory (for example, all
files created after 1 hour).
>>
>> 	I know this is not simple to understand, and most
important, it is 
>> really hard to implement. Its easier if you have
your own system do 
>> publish content, but could be done too if you use
samba or ftp to 
>> publish your files (look in the configuration files
for those two 
>> daemons, and see if you find a way to create temp
files on the upload 
>> process, and then move the temp file into the old
one).
>>
>>> Help greatly appreciated.
>>>
>>> Regards
>> 	Regards,
>>
>>> Markus
>> Domingos.
>>
>>
------------------------------------------------------------
---------
>> The official User-To-User support forum of the
Apache HTTP Server Project.
>> See <URL:http://htt
pd.apache.org/userslist.html> for more info.
>> To unsubscribe, e-mail: users-unsubscribehttpd.apache.org
>>    "   from the digest:
users-digest-unsubscribehttpd.apache.org
>> For additional commands, e-mail: users-helphttpd.apache.org
>>
> 
> 
>
------------------------------------------------------------
---------
> The official User-To-User support forum of the Apache
HTTP Server Project.
> See <URL:http://htt
pd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribehttpd.apache.org
>    "   from the digest:
users-digest-unsubscribehttpd.apache.org
> For additional commands, e-mail: users-helphttpd.apache.org
> 

------------------------------------------------------------
---------
The official User-To-User support forum of the Apache HTTP
Server Project.
See <URL:http://htt
pd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribehttpd.apache.org
   "   from the digest: users-digest-unsubscribehttpd.apache.org
For additional commands, e-mail: users-helphttpd.apache.org