Need help debuging semaphores

Hi!

I've been running Apache for a while and been quite happy.

Now apache "suddenly" won't start anymore and
gives the error:
[Sat Oct 14 03:18:30 2006] [emerg] (28)No space left on
device: Couldn't 
create accept lock

A little googling suggests that it should be related to
semaphores - alas 
my ipcs seems to be fine.

After a cold boot I am seeing:

# ipcs -s
IPC status from <running system> as of Sat Oct 14
03:29:33 2006

Semaphores:
T        ID     KEY        MODE       OWNER    GROUP

# ipcs -S
IPC status from <running system> as of Sat Oct 14
03:29:36 2006

seminfo:
         semmap:     30  (# of entries in semaphore map)
         semmni:     10  (# of semaphore identifiers)
         semmns:     60  (# of semaphores in system)
         semmnu:     30  (# of undo structures in system)
         semmsl:     60  (max # of semaphores per id)
         semopm:    100  (max # of operations per semop
call)
         semume:     10  (max # of undo entries per process)
         semusz:    100  (size in bytes of undo structure)
         semvmx:  32767  (semaphore maximum value)
         semaem:  16384  (adjust on exit max value)


I am really at a loss and in doubt where too look for
debuging. Any hints 
would be _highly_ appreciated.

Kind Regards,
Claus Andersen

On Sat, Oct 14, 2006 at 03:38:15AM +0200, Claus Andersen
wrote:
>I am really at a loss and in doubt where too look for
debuging. Any hints 
>would be _highly_ appreciated.

what version of apache are you using?
apache-ssl? mod_perl, mod_python?
how about logging, are you using pipes
or regular access.log?

I think you need a custom kernel for
apache-ssl with mod_python, extra semaphores
or bigger block, don't remember exactly what
I did but it was just guess work because I
couldn't find any doc on how to set them
"correctly"; everything else might be
involved.... the apache-ssl before the last
version of 2006Q2, wouldn't log to pipes, I
think I got that error too.

// George


-- 
George Georgalis, systems architect, administrator
<IXOYE><

On Fri, 13 Oct 2006, George Georgalis wrote:

> On Sat, Oct 14, 2006 at 03:38:15AM +0200, Claus
Andersen wrote:
>> I am really at a loss and in doubt where too look
for debuging. Any hints
>> would be _highly_ appreciated.
>
> what version of apache are you using?
> apache-ssl? mod_perl, mod_python?
> how about logging, are you using pipes
> or regular access.log?

apache-2.0.59
ap2-php5-5.1.5 (which broke my postgresql client 8.1)
ap2-py24-python-3.2.8
logging to access.log

> I think you need a custom kernel for
> apache-ssl with mod_python, extra semaphores
> or bigger block, don't remember exactly what

You are spot on! Thanks! I removed LoadModule python_module
and apache 
starts.

The semaphores however bugs me as ipcs shows nothing.

> I did but it was just guess work because I
> couldn't find any doc on how to set them
> "correctly";

I found a lot of linux centric references to:
kernel.sem = 512 32000 32 512
...but none which explains what those numbers expands to.

The closest thing to an explanation was from a DB2 article:
"From here, modify the kernel parameters by adding the
following entries 
to the /etc/sysctl.conf default system control configuration
file:

kernel.msgmni =512
kernel.sem =250 128000 32 1024

where:

max semaphores system wide = max number of arrays x max
semaphores per 
array."

Would that perhaps be something like:
SEGMNS = SEGMNI x ??

Anyhow I was getting ready to roll a new kernel with:
options        SEMMNI=1024       # number of semaphore
identifiers
options        SEMMNS=1024       # number of semaphores in
system

...when I got a little sidetracked - and I am glad I did!

My postgresql client recently got hosed with client 8.0
overwriting client 
8.1. I noticed that Postgresql didn't start because pg_ctl
was the wrong 
version. I did a brute force replace of pg_ctl so Postgresql
would start.

...and with postgresql running, apache does not choke on
mod_python?!?

Results:
1) Apache runs fine without "LoadModule
python_module"
2) Apache chokes with "LoadModule python_module"
and postgresql not 
running. (_No_ other python references in httpd.conf - so no
"SetHandler 
python-program")
3) Apache, mod_python is happy when postgresql is running.

It baffles me that there would be any kind of
dependecy/relationship 
between mod_python and postgresql.

I am suspecting that postgresql might be tweaking the number
of semaphores 
at run-time which causes the breakage of mod_python when it
is not 
running.

I will try to confirm this by rolling a new kernel with more
semaphores.

Kind Regards,
Claus Andersen

On Sat, 14 Oct 2006, Claus Andersen wrote:

> On Fri, 13 Oct 2006, George Georgalis wrote:
>
>> I think you need a custom kernel for
>> apache-ssl with mod_python, extra semaphores
>> or bigger block, don't remember exactly what

> Anyhow I was getting ready to roll a new kernel with:
> options        SEMMNI=1024       # number of semaphore
identifiers
> options        SEMMNS=1024       # number of semaphores
in system

This was the solution. Postgresql tweaks these on the fly
which was why 
apache played along for a time.
With the above kernel options apache+mod_python runs fine
without 
postgresql.

Thanks alot, George!

Kind Regards,
Claus Andersen