Mystery Problem with directories on Tiger

Recently I have been running into a weird problem with
OpenAFS-1.5.15,
which I'm running on MacOS 10.4.8.  I'm pretty sure I
started seeing
this before I installed 1.5.15, but I'm not sure what
version I was
running when I first noticed it.  I am running with
afs.options of
     -afsdb -stat 2000 -dcache 800 -daemons 3 -volumes 70
-dynroot -fakestat

Short description: I have cd'ed into some directory in AFS
space, and
am typing in unix commands in a terminal window.  One minute
things
will be fine, and the next minute a few specific unix
commands will
start to fail consistently, with error msgs such as:
      open[19187] No such file: (null)
And then, at some point later on, everything starts working
again.
It might be 5 minutes later, it might be 15 minutes, I can't
really
say for sure.  The amount of time probably varies.

Longer:
I usually notice the problem first when doing an 'open'
command, because
the way I edit source files is via an alias that does
'/usr/bin/open -a
/Developer/Applications/Xcode.app <filename>'.  Most
unix commands
continue to work fine, but another very simple one which
fails is
/bin/pwd.  The 'pwd' function which is built-in to bash does
not get
any errors, but separate /bin/pwd program will say:
      pwd: : No such file or directory

Iirc, all 'pwd' does is get the current working-directory,
and then
does repeated stat("..")'s until it makes it back
to the root ("/").
I'm pretty sure that all the commands which fail are doing
the same
kind of cycle through ".." links.

I've been hitting this a lot more recently, and I think it's
because
we're starting to permit things as 'system:authuser rl'
instead of
'system:anyuser rl'.  I am authenticated when this problem
pops up,
so it's not like my tokens expired.  And things will start
working
again without me doing another 'klog'.  I've tried doing
things like
'fs checkservers', 'fs checkvolumes' and 'fs flushvolume
...', and
none of those seem to have any effect on the problem.

I know the release page for 1.5.15 says that one known issue
is
"When authentication state changes, Finder may cache
stale data
(files appear unaccessible or do not appear)", but this
problem isn't
happening in the finder.  In fact, I usually do not have
*any* finder
windows open (none at all, not even ones looking directories
outside
of AFS).  I'm also pretty sure that I had seen this when I
was still
running version 1.4.1.

I don't know if it makes any difference, but I'm logged into
MacOS
as a local userid 'gad', while my AFS authentication is for
userid
'drosehn'.  When I'm working in AFS, I'm always
authenticating via
the 'klog' command.

I'm still trying to collect more information on what exactly
is
happening, but I thought I'd mention what I've learned so
far and
see if this is an issue which people already know about.  Or
if anyone
has suggestions on things I should try during the periods
where the
problem is active.

-- 
Garance Alistair Drosehn            =   gadgilead.netel.rpi.edu
Senior Systems Programmer           or  gadfreebsd.org
Rensselaer Polytechnic Institute    or  drosihrpi.edu
_______________________________________________
port-darwin mailing list
port-darwinopenafs.org
https://lists.openafs.org/mailman/listinfo/port-darwin

1) what's in syslog.log?
2) does it happen with 1.4.2?
3) does cmdebug have anything to say?

On Sat, 10 Mar 2007, Garance A Drosihn wrote:

> Recently I have been running into a weird problem with
OpenAFS-1.5.15,
> which I'm running on MacOS 10.4.8.  I'm pretty sure I
started seeing
> this before I installed 1.5.15, but I'm not sure what
version I was
> running when I first noticed it.  I am running with
afs.options of
>    -afsdb -stat 2000 -dcache 800 -daemons 3 -volumes 70
-dynroot -fakestat
>
> Short description: I have cd'ed into some directory in
AFS space, and
> am typing in unix commands in a terminal window.  One
minute things
> will be fine, and the next minute a few specific unix
commands will
> start to fail consistently, with error msgs such as:
>     open[19187] No such file: (null)
> And then, at some point later on, everything starts
working again.
> It might be 5 minutes later, it might be 15 minutes, I
can't really
> say for sure.  The amount of time probably varies.
>
> Longer:
> I usually notice the problem first when doing an 'open'
command, because
> the way I edit source files is via an alias that does
'/usr/bin/open -a
> /Developer/Applications/Xcode.app <filename>'. 
Most unix commands
> continue to work fine, but another very simple one
which fails is
> /bin/pwd.  The 'pwd' function which is built-in to bash
does not get
> any errors, but separate /bin/pwd program will say:
>     pwd: : No such file or directory
>
> Iirc, all 'pwd' does is get the current
working-directory, and then
> does repeated stat("..")'s until it makes it
back to the root ("/").
> I'm pretty sure that all the commands which fail are
doing the same
> kind of cycle through ".." links.
>
> I've been hitting this a lot more recently, and I think
it's because
> we're starting to permit things as 'system:authuser rl'
instead of
> 'system:anyuser rl'.  I am authenticated when this
problem pops up,
> so it's not like my tokens expired.  And things will
start working
> again without me doing another 'klog'.  I've tried
doing things like
> 'fs checkservers', 'fs checkvolumes' and 'fs
flushvolume ...', and
> none of those seem to have any effect on the problem.
>
> I know the release page for 1.5.15 says that one known
issue is
> "When authentication state changes, Finder may
cache stale data
> (files appear unaccessible or do not appear)", but
this problem isn't
> happening in the finder.  In fact, I usually do not
have *any* finder
> windows open (none at all, not even ones looking
directories outside
> of AFS).  I'm also pretty sure that I had seen this
when I was still
> running version 1.4.1.
>
> I don't know if it makes any difference, but I'm logged
into MacOS
> as a local userid 'gad', while my AFS authentication is
for userid
> 'drosehn'.  When I'm working in AFS, I'm always
authenticating via
> the 'klog' command.
>
> I'm still trying to collect more information on what
exactly is
> happening, but I thought I'd mention what I've learned
so far and
> see if this is an issue which people already know
about.  Or if anyone
> has suggestions on things I should try during the
periods where the
> problem is active.
>
> -- 
> Garance Alistair Drosehn            =   gadgilead.netel.rpi.edu
> Senior Systems Programmer           or  gadfreebsd.org
> Rensselaer Polytechnic Institute    or  drosihrpi.edu
> _______________________________________________
> port-darwin mailing list
> port-darwinopenafs.org
> https://lists.openafs.org/mailman/listinfo/port-darwin

>
_______________________________________________
port-darwin mailing list
port-darwinopenafs.org
https://lists.openafs.org/mailman/listinfo/port-darwin

At 1:32 PM -0500 3/10/07, Derrick J Brashear wrote:
>1) what's in syslog.log?

if you mean /var/log/system.log, then nothing shows up in
there
when the problem is happening.

>2) does it happen with 1.4.2?

I have a lot of things open on my main machine right now,
and I
don't want to risk a reboot or have to restart everything
I'm in
the middle of.  I'll see if the same problem shows up on one
of
my mini-Macs, and if so then I'll try 1.4.2 on that.  (the
Mac-
mini's are here in the same office, so that's easy enough to
do).

>3) does cmdebug have anything to say?

Hmm.  Nothing at the moment.  I haven't tried it when I'm in
the
middle of the problem though.  Right now the only
interesting
thing from a -long output is:
** Cache entry  0xf8b71000 for 134.537094275.1.1
[rpi.edu]
             8192 bytes  DV         1467  refcnt     1
     callback 00000000   expires 1173524556
     0 opens     0 writers
     volume root
     states (0x4), read-only

I haven't used cmdebug much, so I'll have to play around
with it
some more and see what comes up.

I should also mention that the problem does not effect all
of AFS
when it does happen.  I can 'cd ..' a few times, and I'll
get to
a directory where /bin/pwd doesn't fail.  That directory
which
works is still in AFS, but if I 'cd' back into the
subdirectory
I had been in, the problem is still happening in the
subdirectory.

-- 
Garance Alistair Drosehn            =   gadgilead.netel.rpi.edu
Senior Systems Programmer           or  gadfreebsd.org
Rensselaer Polytechnic Institute    or  drosihrpi.edu
_______________________________________________
port-darwin mailing list
port-darwinopenafs.org
https://lists.openafs.org/mailman/listinfo/port-darwin

At 1:32 PM -0500 3/10/07, Derrick J Brashear wrote:
>1) what's in syslog.log?
>2) does it happen with 1.4.2?

I have not been able to reproduce the problem on my Mac-mini
yet,
but I did have some other fun adventures.

Right now I seem to have an invocation of bash where the
problem
does not go away, which made it easier to investigate.  I
found
that what fails is the system routine getcwd().  The code:

	errno = 0;
	cwdres = getcwd(bigbuf, BIGBUFLEN);
	if (cwdres == 0) {
		perror("call to getcwd() failed");
		return (1);
	}
	printf("cur dir = %sn", cwdres);

will print out:
      call to getcwd() failed: No such file or directory

if I run it from an AFS directory where I'm having the
problem.
It will print out the correct "cur dir = ..." if I
run it after
cd-ing to some other directory.

So then I wrote a ruby script which would take some
starting
directory, and find each AFS volume underneath that
directory by
doing a lot of 'fs lq' commands.  For each AFS volume found,
the
script does a Dir.chdir to that directory, tries a
Dir.getwd,
and catches any errors that come up.  This shows me which
AFS
volumes are having the problem.

I ran the script on some reasonably-large tree of
directories in
our AFS cell (1817 directories), and sure enough it
discovers that
Dir.getwd is failing in the two AFS volumes I'm working in. 
But
it also fails in four other AFS volumes which I'm sure I
haven't
touched since the last time I rebooted.  Out of the 43
volumes
which were checked by that run of the script, the problem is
only
seen on those six volumes.

And then I notice something even more bizarre...  While
writing up
this script, I opened up a few more Terminal windows to test
various
things.  It turns out that the script finds *no* problem
volumes if
I run it with the exact same parameters from a different
session of
bash!  I can even do the simple test of cd'ing to the same
directory
in two different windows.  /bin/pwd works fine in one
window, and
still fails in my original session!

It could be that the problem I'm seeing this time is
different than
problems I've seen in the past.  This problem bash-session
has
remained screwed-up for several hours now, and I've never
seen that
before.  And for at least some times this has come up, I
know I've
tried the same 'cd ... ; open ...' combination in multiple
windows,
and either it works in all windows or it fails in all
windows.

I had installed the OpenAFS-1.5.15 about 3 weeks ago, and
have not
rebooted my machine in the past two weeks.  For my next
step, I'll
install 1.4.2, reboot, and see if the problem shows up
again.  If it
does, I can continue investigating with the simple getcwd()
program
and the more elaborate ruby script.

>3) does cmdebug have anything to say?

cmdebug `hostname`

has nothing to say.


-- 
Garance Alistair Drosehn            =   gadgilead.netel.rpi.edu
Senior Systems Programmer           or  gadfreebsd.org
Rensselaer Polytechnic Institute    or  drosihrpi.edu
_______________________________________________
port-darwin mailing list
port-darwinopenafs.org
https://lists.openafs.org/mailman/listinfo/port-darwin

At 10:49 PM -0500 3/10/07, Garance A Drosihn wrote:
>At 1:32 PM -0500 3/10/07, Derrick J Brashear wrote:
>>1) what's in syslog.log?
>>2) does it happen with 1.4.2?
>
>Right now I seem to have an invocation of bash where the
problem
>does not go away, which made it easier to investigate. 
I found
>that what fails is the system routine getcwd().  The
code:
>
>	errno = 0;
>	cwdres = getcwd(bigbuf, BIGBUFLEN);
>	if (cwdres == 0) {
>		perror("call to getcwd() failed");
>		return (1);
>	}
>	printf("cur dir = %sn", cwdres);
>
>will print out:
>      call to getcwd() failed: No such file or
directory


>I had installed the OpenAFS-1.5.15 about 3 weeks ago,
and have not
>rebooted my machine in the past two weeks.  For my next
step, I'll
>install 1.4.2, reboot, and see if the problem shows up
again.  If it
>does, I can continue investigating with the simple
getcwd() program
>and the more elaborate ruby script.

Okay, I installed 1.4.2, and the problem still shows up.

First I installed 1.4.2.  As mentioned in documentation, I
hit the
issue where the installer says "You cannot continue.
There is nothing
to install." So, I removed the OpenAFS package receipt,
and I think I
also ran the un-installer.  I was then able to install okay,
but the
machine hung when I rebooted.  On a hunch, I booted into
single-user
mode and blew away the entire /var/db/openafs/cache
directory, and
then re-created the directory.  After doing that, I had no
trouble
booting up.  The nice side-effect of that is I know I
started with a
clean AFS cache.

At first I didn't seem to have any trouble, but then I
didn't do much
work in AFS for a few days.  On Friday I was ready to do
some serious
work, and the two AFS volumes that I wanted to get at were
showing
this problem with getcwd().  This time I opened several
different
Terminal windows, and they were all showing the problem on
the same
set of AFS volumes.

When I run my ruby script, I generally run it on a set of
directories
which include 43 different AFS volumes.  Fairly often it is
the same
set of 6 AFS volumes which have the problem with getcwd(). 
But at one
point on Friday the problem went away for the two AFS
volumes that I
wanted to work in, even though it stayed around for the
other four
volumes which I don't care about and haven't touched.

I then thought it'd be interesting to run the script on a
much larger
set of directories.  That run found a few more AFS volumes
which were
having the getcwd() problem.  Unfortunately, my main Mac
also locked up
while running the script on that larger portion of our AFS
cell.  Most
running processes were okay, but it seemed that anything new
that I
started up would hang.  I was able to close all my active
apps (and save
away any documents I had open), but then the machine
completely froze
up when I tried out log out of that userid.  I had to do a
forced reboot.

I usually have a lot going on, and lose a lot of context if
I have to
reboot.  So, I brought up my intel-based Mac-Mini.  (My main
desktop is
a dual-CPU G5).  It is also running MacOS 10.4.9, and is
still running
OpenAFS 1.5.15.  I'm running my checking-script on the same
large section
of our AFS cell, and so far it hasn't hit the getcwd problem
on any AFS
volume.  The script does seem to run slower on the Mac-mini,
but it has
already searched through 175,000 directories without hitting
a problem.
I'm not sure how many AFS volumes that is, but I'm pretty
sure it has
already checked about 50,000 more directories than had been
checked
during the run which locked-up my desktop.

The mini-mac is setup with the same set of AFSD options as
my desktop,
namely:
  -afsdb -stat 2000 -dcache 800 -daemons 3 -volumes 70
-dynroot -fakestat
The mini-mac is on a different network than my desktop,
which might be
significant because the desktop is on a 10-Mbit network,
while the
mini-mac is on 100-Mbit.

So, the problem still shows up, and it still doesn't make a
whole lot of
sense.  It is only the getcwd() call which fails.  I found
that I can
work around the problem with 'open' and 'opendiff' commands
if I just
fully-specify the filenames I want to edit or compare.  I
can do this by
using the PWD variable that bash is keeping track of.  So,
if the command
     opendiff afile bfile
fails, then I can simply use
     opendiff $PWD/afile $PWD/bfile
and it will work fine.

For the AFS volumes where I've seen the getcwd() problem,
they don't seem
to have a lot in common.  They are spread out across a few
different AFS
fileservers, for instance.

Obviously the problem is very repeatable on my desktop, but
I'd like to
have it repeatable on some machine where I won't lose so
much work if I
need to reboot.  So, I'll keep trying various combinations
of things, to
see if I can pin down the problem some more.  I think this
is a problem
at the OS-level, and not at the AFS-locking level.  I can
always get to
the files in question, and can do anything I want to do with
them just
as long as I don't need to call getcwd().

-- 
Garance Alistair Drosehn            =   gadgilead.netel.rpi.edu
Senior Systems Programmer           or  gadfreebsd.org
Rensselaer Polytechnic Institute    or  drosihrpi.edu
_______________________________________________
port-darwin mailing list
port-darwinopenafs.org
https://lists.openafs.org/mailman/listinfo/port-darwin

On Sun, 18 Mar 2007, Garance A Drosihn wrote:

> First I installed 1.4.2.  As mentioned in
documentation, I hit the
> issue where the installer says "You cannot
continue. There is nothing
> to install." So, I removed the OpenAFS package
receipt, and I think I
> also ran the un-installer.  I was then able to install
okay, but the

The uninstaller uses the receipt...

> Obviously the problem is very repeatable on my desktop,
but I'd like to
> have it repeatable on some machine where I won't lose
so much work if I
> need to reboot.  So, I'll keep trying various
combinations of things, to
> see if I can pin down the problem some more.  I think
this is a problem
> at the OS-level, and not at the AFS-locking level.  I
can always get to

Ok,. Thank you.
_______________________________________________
port-darwin mailing list
port-darwinopenafs.org
https://lists.openafs.org/mailman/listinfo/port-darwin

At 11:33 PM -0400 3/18/07, Derrick J Brashear wrote:
>On Sun, 18 Mar 2007, Garance A Drosihn wrote:
>
>>First I installed 1.4.2.  As mentioned in
documentation, I hit the
>>issue where the installer says "You cannot
continue. There is nothing
>>to install." So, I removed the OpenAFS package
receipt, and I think I
>>also ran the un-installer.  I was then able to
install okay, but the
>
>The uninstaller uses the receipt...

Well, what I should have said was that *either* I removed
the packages,
*or* I ran the uninstaller.  I think the documentation
mentioned both
options, and I don't remember which one I did.

Hmm.  Thinking about it some more, I suspect that I ran the
uninstaller.
One thing I noticed was that I lost everything in
/var/db/openafs/etc,
for instance.  I have a custom CellServDB and the slightly
customized
afsd.options, and after installing 1.4.2 I had to recreate
those.  I
had to recreate ThisCell, too.  I assume that would not have
been
necessary if I had only removed the receipts.

The run of my ruby script finished on my intel-based
Mac-mini.  It
checked almost 200,000 directories, found 2067 AFS volumes,
and all of
those AFS volumes worked fine.  No getcwd() problems.

-- 
Garance Alistair Drosehn            =   gadgilead.netel.rpi.edu
Senior Systems Programmer           or  gadfreebsd.org
Rensselaer Polytechnic Institute    or  drosihrpi.edu
_______________________________________________
port-darwin mailing list
port-darwinopenafs.org
https://lists.openafs.org/mailman/listinfo/port-darwin

On Mon, 19 Mar 2007, Garance A Drosihn wrote:

> At 11:33 PM -0400 3/18/07, Derrick J Brashear wrote:
>> On Sun, 18 Mar 2007, Garance A Drosihn wrote:
>> 
>>> First I installed 1.4.2.  As mentioned in
documentation, I hit the
>>> issue where the installer says "You cannot
continue. There is nothing
>>> to install." So, I removed the OpenAFS
package receipt, and I think I
>>> also ran the un-installer.  I was then able to
install okay, but the
>> 
>> The uninstaller uses the receipt...
>
> Well, what I should have said was that *either* I
removed the packages,
> *or* I ran the uninstaller.  I think the documentation
mentioned both
> options, and I don't remember which one I did.

Ok.

> Hmm.  Thinking about it some more, I suspect that I ran
the uninstaller.
> One thing I noticed was that I lost everything in
/var/db/openafs/etc,
> for instance.  I have a custom CellServDB and the
slightly customized
> afsd.options, and after installing 1.4.2 I had to
recreate those.  I
> had to recreate ThisCell, too.  I assume that would not
have been
> necessary if I had only removed the receipts.

True. I suppose we should look at making the uninstaller
able to keep 
customization, but right now it's a "scorched
earth" uninstaller.

> The run of my ruby script finished on my intel-based
Mac-mini.  It
> checked almost 200,000 directories, found 2067 AFS
volumes, and all of
> those AFS volumes worked fine.  No getcwd() problems.

Ok. Can you produce the getcwd problem with 1.5 on that
machine?


_______________________________________________
port-darwin mailing list
port-darwinopenafs.org
https://lists.openafs.org/mailman/listinfo/port-darwin

At 1:38 PM -0400 3/19/07, Derrick J Brashear wrote:
>On Mon, 19 Mar 2007, Garance A Drosihn wrote:
>
>>The run of my ruby script finished on my intel-based
Mac-mini.  It
>>checked almost 200,000 directories, found 2067 AFS
volumes, and all of
>>those AFS volumes worked fine.  No getcwd()
problems.
>
>Ok. Can you produce the getcwd problem with 1.5 on that
machine?

So far, I have never seen the getcwd problem on the
intel-based
Mac-mini.  It's a "Core Duo" machine, so two-CPUs
in one chip, but
the version which is 32-bit only.  I do not have a Core 2
Duo machine
to test it on.  I have tried to generate the getcwd-problem
on both
OpenAFS 1.4.2 and OpenAFS 1.5.1.  I *did* have one case
where my
Mac-mini hung up at shutdown after running the script, so my
script
might still be tickling some bug when run on the Mac-mini,
but the
getcwd() problem does not show up.

On my Dual-G5 machine, I have seen the getcwd() problem when
running
OpenAFS 1.4.1, 1.4.2, and 1.5.1.  Both machines are running
the same
version of MacOS (I've tested this with both 10.4.8 and
10.4.9, on
both machines).

As much as I didn't want to disrupt my regular work on my
desktop,
curiosity got the better of me and I tried a few more tests.
 I shutdown
the Mac-mini, and switched the network cable to my desktop. 
The problem
still shows up when the desktop is using the same network
cable and the
same network address as the Mac-mini, so this isn't a
networking issue.

This problem has been particularly irksome to me, because it
seemed
much more likely to show up when I was ready to do some
serious work,
and never seemed to show up when I was just doing a quick
check of
something in the same directories.  I kept telling myself I
was just
imagining that, but it turns out there seems to be something
to that.

It turns out that if I point the script into somewhere
/afs/rpi.edu,
then it is much less likely to have any problems.  But if I
point it
into /afs/.rpi.edu, then the getcwd problem is much more
likely to
happen.  And if I 'cd' into the /afs/.rpi.edu version of one
of the
problematic AFS volumes, then I *will* see a problem at that
volume
when I point the script at the /afs/rpi.edu version.  Now,
when I'm
just checking some files in afs, I probably always 'cd' into
the
rpi.edu version, but when I'm ready to do serious work I'll
'cd' to
the /afs/.rpi.edu version, even for a volume where I don't
need to,
just by habit.

It is also interesting that none of the problematic AFS
volumes are
replicated.  So, for all of them, the volume seen under
/afs/rpi.edu
is exactly the same volume as seen under /afs/.rpi.edu.

Even armed with all the above info, I have not been able to
trigger
the getcwd() problem on the Mac-mini, but I'll keep trying.

The script I use is a quickly thrown together hack, with
various
sections copy&pasted from other scripts.  So, it ain't
pretty, but if
anyone else wants to try it, the script is at:

ht
tp://www.rpi.edu/~drosehn/openafs/getwd_AFS_vols

That is the actual script, so it might be that your web
browser will
just download it as a file.

You have to give it a directory to start at, and it will
check that
directory and all directories below that one.  It will do an
'fs lq'
for every directory, and when it notices a new AFS volume it
will
'cd' into the directory and then call the Dir.getwd of ruby
(which
in turn calls getcwd).  The script will keep track of which
calls fail.

Example:   getwd_AFS_vols
--start-dir=/afs/rpi.edu/campus/print

although it'd probably be best to point into your local
cell!

-- 
Garance Alistair Drosehn            =   gadgilead.netel.rpi.edu
Senior Systems Programmer           or  gadfreebsd.org
Rensselaer Polytechnic Institute    or  drosihrpi.edu
_______________________________________________
port-darwin mailing list
port-darwinopenafs.org
https://lists.openafs.org/mailman/listinfo/port-darwin

At 5:32 PM -0400 3/19/07, Garance A Drosihn wrote:
>
>On my Dual-G5 machine, I have seen the getcwd() problem
when running
>OpenAFS 1.4.1, 1.4.2, and 1.5.1.  Both machines are
running the same
>version of MacOS (I've tested this with both 10.4.8 and
10.4.9, on
>both machines).

Just to clarify: In my description of my adventures, I often
state that
some of my testing was with 1.5.1, but in fact that was with
1.5.15.

-- 
Garance Alistair Drosehn            =   gadgilead.netel.rpi.edu
Senior Systems Programmer           or  gadfreebsd.org
Rensselaer Polytechnic Institute    or  drosihrpi.edu
_______________________________________________
port-darwin mailing list
port-darwinopenafs.org
https://lists.openafs.org/mailman/listinfo/port-darwin