Lighttpd keeps log files open after HUP when using multiple workers

Hi,

I've got a problem where I have too much disk space used.
After
tracking the problem down, it appears that when using
"server.max-worker = 2", the master process keeps
some old log files
open even after receiving a HUP signal. The two workers only
have the
current log files open, as they should.

Is this a known bug or limitation? I searched trac quickly
and couldn't
find anything about this particular issue.

Using "lsof" I clearly see the master lighttpd
process still having
those files open, although they're in "deleted"
state. It's worth
noting also that it doesn't have any of the new files open,
only
the two workers have them open. Sending another HUP doesn't
help.
Completely restarting lighttpd recovers all free space from
those log
files.

Since the disk space used doesn't increase forever, I assume
that the
master lighttpd process will always keep these initial log
files open,
and never open newer ones. The log rotation for the workers
seems to be
working fine. So the only space "lost" is from the
initial uncompressed
log files created when lighttpd was first started.

This is with 1.4.16 on RHEL5 x86.

Matthias

-- 
Clean custom Red Hat Linux rpm packages : http://freshrpms.net/
Fedora release 7 (Moonshine) - Linux kernel 2.6.22.5-71.fc7
Load : 1.86 1.13 0.54

On Mon, September 3, 2007 11:42, Matthias Saou wrote:
> I've got a problem where I have too much disk space
used. After
> tracking the problem down, it appears that when using
"server.max-worker =
> 2", the master process keeps some old log files
> open even after receiving a HUP signal.

IIRC, a HUP does not tell lighttpd (and apache and a whole
lot of other
deamons too) too change file descriptors. From a quick look
of my
logrotate scripts, all daemons have to be restarted to open
new logfiles,
a HUP will not suffice.

One could rotate logfiles without restarting the daemon, but
this will
require more (temporary) space:

# gzip -c logfile > logfile.gz
# > logfile

however, this would lose some entries while the logfile is
being gzipped...

C.
-- 
BOFH excuse #442:

Trojan horse ran out of hay

On 2007-09-03 15:12:17 +0200, Christian Kujau wrote:
> On Mon, September 3, 2007 11:42, Matthias Saou wrote:
> > I've got a problem where I have too much disk
space used. After
> > tracking the problem down, it appears that when
using "server.max-worker =
> > 2", the master process keeps some old log
files
> > open even after receiving a HUP signal.
> 
> IIRC, a HUP does not tell lighttpd (and apache and a
whole lot of other
> deamons too) too change file descriptors. From a quick
look of my
> logrotate scripts, all daemons have to be restarted to
open new logfiles,
> a HUP will not suffice.

lighttpd handles HUP correctly without max-workers.

  darix

-- 
           openSUSE - SUSE Linux is my linux
               openSUSE is good for you
                   www.opensuse.org

On Mon, September 3, 2007 15:21, Marcus Rueckert wrote:
>> IIRC, a HUP does not tell lighttpd (and apache and
a whole lot of other
>>  deamons too) too change file descriptors. From a
quick look of my
>> logrotate scripts, all daemons have to be restarted
to open new
>> logfiles, a HUP will not suffice.
>
> lighttpd handles HUP correctly without max-workers.

I'm not sure if I understood: what do you mean by
"correctly", and what
has this to do with max-workers?

-- 
BOFH excuse #442:

Trojan horse ran out of hay

On Mon, 3 Sep 2007, Christian Kujau wrote:

> I'm not sure if I understood: what do you mean by
"correctly", and what
> has this to do with max-workers?

I think what he means is that lighttpd, when run in
single-process mode,
correctly closes and re-opens the logfiles when it proceses
a SIGHUP.
However, when used with the max-workers configuration
directive, it
spawns child processes to handle some of the work, these
child process
appear to write to the log files directly, and they do not
close and
re-open the log files when the parent gets a SIGHUP.

Interestingly enough, the release notes for version 1.4.12
say, "fixed
SIGHUP handling if max-workers is used":

     http://blog.
lighttpd.net/articles/2006/9

Is this a reversion?

As a workaround, I wonder if anybody's tried sending SIGHUP
to the
children as well.

cjs
-- 
Curt Sampson         <cjscynic.net>         +81
90 7737 2974
              http://www.starling-
software.com
The power of accurate observation is commonly called
cynicism
by those who have not got it.    --George Bernard Shaw

Curt Sampson wrote :

> On Mon, 3 Sep 2007, Christian Kujau wrote:
> 
> > I'm not sure if I understood: what do you mean by
"correctly", and what
> > has this to do with max-workers?
> 
> I think what he means is that lighttpd, when run in
single-process mode,
> correctly closes and re-opens the logfiles when it
proceses a SIGHUP.
> However, when used with the max-workers configuration
directive, it
> spawns child processes to handle some of the work,
these child process
> appear to write to the log files directly, and they do
not close and
> re-open the log files when the parent gets a SIGHUP.

Almost. What I've seen is that the children do reopen the
new log files
properly, it's the parent process that keeps the initial log
files
open forever (and never opens any new ones).

> Interestingly enough, the release notes for version
1.4.12 say, "fixed
> SIGHUP handling if max-workers is used":
> 
>      http://blog.
lighttpd.net/articles/2006/9
> 
> Is this a reversion?

Hmmm, indeed. Either that, or the fix wasn't effective.

> As a workaround, I wonder if anybody's tried sending
SIGHUP to the
> children as well.

My logrotate script sends a SIGHUP to all
"lighttpd" processes, parent
and children all at once.

Matthias

-- 
Clean custom Red Hat Linux rpm packages : http://freshrpms.net/
Fedora release 7 (Moonshine) - Linux kernel 2.6.22.5-71.fc7
Load : 0.66 0.77 0.89

Jan Kneschke wrote :

> Matthias Saou wrote:
> > Hi,
> > 
> > I've got a problem where I have too much disk
space used. After
> > tracking the problem down, it appears that when
using
> > "server.max-worker = 2", the master
process keeps some old log files
> > open even after receiving a HUP signal. The two
workers only have the
> > current log files open, as they should.
> > 
> > Is this a known bug or limitation? I searched trac
quickly and couldn't
> > find anything about this particular issue.
> 
> See
> 
> http://trac.
lighttpd.net/trac/ticket/902
> 
> It also includes a patch.

Great! Thanks a lot 

I also just noticed 1.4.17 got released... maybe it could
get also
posted to this list as it used to 

Matthias

-- 
Clean custom Red Hat Linux rpm packages : http://freshrpms.net/
Fedora release 7 (Moonshine) - Linux kernel 2.6.22.5-71.fc7
Load : 1.14 1.40 1.27

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 03/09/07 09:12 AM, Christian Kujau wrote:
> 
> One could rotate logfiles without restarting the
daemon, but this will
> require more (temporary) space:
> 
> # gzip -c logfile > logfile.gz
> # > logfile

AFAIK truncating your log file will result in a spare file
as soon as it
gets written again. The file descriptor won't get rewinded
in Lighty so
the next write will start at the same offset as if the log
wasn't truncated.

Of course if you cat the file you won't see it because it'll
be filled
with NULL's but opening it with vim sill show it.

Thomas
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org


iD8DBQFG31j66dZ+Kt5BchYRAn02AKCdMhULUjtDdTvIslKAgWkVueugUACd
GrBW
Wt5D2ju7ia/v8RIhj0LG2po=
=flOj
-----END PGP SIGNATURE-----

Hi,

The 1.4.18 release states that it should have fixed this bug
:

http://trac.
lighttpd.net/trac/ticket/902

"fixed forwarding a SIGINT and SIGHUP when using
max-workers (#902)"

But I can confirm that it's not the case. It's quite easy to
reproduce,
actually :
- Set "server.max-worker = 2" in lighttpd.conf,
restart lighty
- Run "lsof -n | grep /var/log/lighttpd" you'll
see the log files open
- Remove the log files "rm -f
/var/log/lighttpd/*"
- Run "lsof -n | grep /var/log/lighttpd" you'll
see logs as "deleted"
- Send a SIGHUP to the master lighty or to all (killall)
- Run "lsof -n | grep /var/log/lighttpd" again

Now you'll see that the two workers no longer have
"deleted", because
they've switched to new log files properly, but the master
process
still has the initial log files as "deleted".

This is on GNU/Linux RHEL5 x86_64 with lighty 1.4.18.

Matthias

-- 
Clean custom Red Hat Linux rpm packages : http://freshrpms.net/
Fedora release 7 (Moonshine) - Linux kernel 2.6.22.5-71.fc7
Load : 0.27 0.54 1.65

Thanks. Good to know. I was about to reinstall ...
If this is the case, will there be a 1.4.19 coming out
soon?

-- 
Best regards,
Jack

Monday, September 10, 2007, 8:28:17 AM, you wrote:

> Hi,

> The 1.4.18 release states that it should have fixed
this bug :

> http://trac.
lighttpd.net/trac/ticket/902

> "fixed forwarding a SIGINT and SIGHUP when using
max-workers (#902)"

> But I can confirm that it's not the case. It's quite
easy to reproduce,
> actually :
> - Set "server.max-worker = 2" in
lighttpd.conf, restart lighty
> - Run "lsof -n | grep /var/log/lighttpd"
you'll see the log files open
> - Remove the log files "rm -f
/var/log/lighttpd/*"
> - Run "lsof -n | grep /var/log/lighttpd"
you'll see logs as "deleted"
> - Send a SIGHUP to the master lighty or to all
(killall)
> - Run "lsof -n | grep /var/log/lighttpd"
again

> Now you'll see that the two workers no longer have
"deleted", because
> they've switched to new log files properly, but the
master process
> still has the initial log files as
"deleted".

> This is on GNU/Linux RHEL5 x86_64 with lighty 1.4.18.

> Matthias