FCGI automatic restart / memory leaks

I just deployed our quite large Catalyst app and I'm seeing
a constant increase
in memory usage. Here's a current snapshop from top:

PID USER   PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+ 
COMMAND
29342 www  16   0 67156  60m 3180 S    1  3.0   0:13.87
perl
29332 www  25   0 41052  34m 1204 S    0  1.7   0:00.01
perl
29333 www  25   0 68424  60m 3172 S    0  3.0   0:13.42
perl
29334 www  22   0 68944  60m 3308 S    0  3.0   0:13.58
perl
29335 www  16   0 67428  60m 3168 S    0  3.0   0:13.86
perl
29336 www  16   0 68736  60m 3192 S    0  3.0   0:13.88
perl
29337 www  16   0 67392  59m 3172 S    0  2.9   0:14.02
perl
29338 www  18   0 66404  59m 3180 S    0  2.9   0:13.49
perl
29339 www  17   0 69216  62m 3172 S    0  3.1   0:14.12
perl
29340 www  16   0 67992  60m 3188 S    0  3.0   0:13.89
perl
29341 www  16   0 68812  60m 3180 S    0  3.0   0:14.88
perl

We're using lighttpd and handle the starting of the FCGI
server manually. Is
there a way to have the FCGI children restart once in a
while without having to
restart the whole FCGI process manager (and bringing down
the application for a
few seconds)? Something like Apache's MaxRequestsPerChild
directive would be
cool.

Then it's time to go memory leak hunting :(

Thanks a lot!

--Tobias

_______________________________________________
List: Catalystlists.rawmode.org
Listinfo: ht
tp://lists.rawmode.org/mailman/listinfo/catalyst
Searchable archive: http://www.mail-
archive.com/catalystlists.rawmode.org/
Dev site: http://dev.catalyst.per
l.org/

On Thursday 16 August 2007 10:40, Tobias Kremer wrote:
> I just deployed our quite large Catalyst app and I'm
seeing a
> constant increase in memory usage. Here's a current
snapshop from
> top:

> We're using lighttpd and handle the starting of the
FCGI server
> manually. Is there a way to have the FCGI children
restart once in a
> while without having to restart the whole FCGI process
manager (and
> bringing down the application for a few seconds)?
Something like
> Apache's MaxRequestsPerChild directive would be cool.

I use daemontools similar to
http
://www.catalystframework.org/calendar/2006/4 .

My run file looks like this:

============================================================
============
#!/bin/sh

cd /srv/www/vhosts/myapp

exec 2>&1 
    setuidgid myappuser 
    softlimit -m 150000000 
        script/myapp_fastcgi.pl -e -l
/var/tmp/myapp/fastcgi.socket -n 10
============================================================
============

Sets uid and gid to "myappuser ", Spawns 10
processes and listens on the
specified UNIX socket. If a process gets too big, it gets
terminated and
replaced by a fresh one (softlimit is part of daemontools,
but you could
use ulimit instead).

> Then it's time to go memory leak hunting :(

I don't care much since softlimit restarts the FCGI
processes (under
normal conditions approx. every 12 hours).

BTW - here are the manpages for daemontools
http://smarden.sunsite.dk/pape/djb/manpag
es/daemontools-0.76-man.tar.gz


HTH
-- 
Bernhard Graf

_______________________________________________
List: Catalystlists.rawmode.org
Listinfo: ht
tp://lists.rawmode.org/mailman/listinfo/catalyst
Searchable archive: http://www.mail-
archive.com/catalystlists.rawmode.org/
Dev site: http://dev.catalyst.per
l.org/

IIRC you see this under mod_perl because of the way that
memory is
shared between processes until it is written to, when a
seperate copy
is created per process. It happens even with no leaks.

I have no idea whether FCGI has a similar mechanism, but
just
something to maybe look at.


On 8/16/07, Tobias Kremer <listfunkreich.de> wrote:
> I just deployed our quite large Catalyst app and I'm
seeing a constant increase
> in memory usage. Here's a current snapshop from top:
>
> PID USER   PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+ 
COMMAND
> 29342 www  16   0 67156  60m 3180 S    1  3.0   0:13.87
perl
> 29332 www  25   0 41052  34m 1204 S    0  1.7   0:00.01
perl
> 29333 www  25   0 68424  60m 3172 S    0  3.0   0:13.42
perl
> 29334 www  22   0 68944  60m 3308 S    0  3.0   0:13.58
perl
> 29335 www  16   0 67428  60m 3168 S    0  3.0   0:13.86
perl
> 29336 www  16   0 68736  60m 3192 S    0  3.0   0:13.88
perl
> 29337 www  16   0 67392  59m 3172 S    0  2.9   0:14.02
perl
> 29338 www  18   0 66404  59m 3180 S    0  2.9   0:13.49
perl
> 29339 www  17   0 69216  62m 3172 S    0  3.1   0:14.12
perl
> 29340 www  16   0 67992  60m 3188 S    0  3.0   0:13.89
perl
> 29341 www  16   0 68812  60m 3180 S    0  3.0   0:14.88
perl
>
> We're using lighttpd and handle the starting of the
FCGI server manually. Is
> there a way to have the FCGI children restart once in a
while without having to
> restart the whole FCGI process manager (and bringing
down the application for a
> few seconds)? Something like Apache's
MaxRequestsPerChild directive would be
> cool.
>
> Then it's time to go memory leak hunting :(
>
> Thanks a lot!
>
> --Tobias
>
> _______________________________________________
> List: Catalystlists.rawmode.org
> Listinfo: ht
tp://lists.rawmode.org/mailman/listinfo/catalyst
> Searchable archive: http://www.mail-
archive.com/catalystlists.rawmode.org/
> Dev site: http://dev.catalyst.per
l.org/
>


-- 
Daniel McBrearty
email : danielmcbrearty at gmail.com
http://www.engoi.com
http://danmcb.vox.com
http://danmcb.blogger.com
find me on linkedin and facebook
BTW : 0873928131

_______________________________________________
List: Catalystlists.rawmode.org
Listinfo: ht
tp://lists.rawmode.org/mailman/listinfo/catalyst
Searchable archive: http://www.mail-
archive.com/catalystlists.rawmode.org/
Dev site: http://dev.catalyst.per
l.org/

Quoting Bernhard Graf <catalyst2augensalat.de>:
> On Thursday 16 August 2007 10:40, Tobias Kremer wrote:
> > We're using lighttpd and handle the starting of
the FCGI server
> > manually. Is there a way to have the FCGI children
restart once in a
> > while without having to restart the whole FCGI
process manager (and
> > bringing down the application for a few seconds)?
Something like
> > Apache's MaxRequestsPerChild directive would be
cool.
>
> I use daemontools similar to
> http
://www.catalystframework.org/calendar/2006/4 .

Thanks! I'll take a look at it.

For now I'm gonna try running two independent FastCGI
process managers and
restart them on different hours via a cronjob so that one of
them will always
be available to lighttpd. Other solutions are still welcome!


--Tobias

_______________________________________________
List: Catalystlists.rawmode.org
Listinfo: ht
tp://lists.rawmode.org/mailman/listinfo/catalyst
Searchable archive: http://www.mail-
archive.com/catalystlists.rawmode.org/
Dev site: http://dev.catalyst.per
l.org/

On Thursday 16 August 2007 13:42, Tobias Kremer wrote:
> Quoting Bernhard Graf <catalyst2augensalat.de>:

> > I use daemontools similar to
> > http
://www.catalystframework.org/calendar/2006/4 .
>
> Thanks! I'll take a look at it.

You should definitely do that. Not only for this case -
daemontools (or 
similar like runit) are superior for nearly every server
service on *ix 
OSes. Check out http://smard
en.org/runit/runscripts.html and compare 
the scripts with typical System-V run scripts (not
mentioning the 
supervision concept in general).

> For now I'm gonna try running two independent FastCGI
process
> managers and restart them on different hours via a
cronjob so that
> one of them will always be available to lighttpd. Other
solutions are
> still welcome! 

Why two FCGI process managers? To compensate FCGI downtimes?
Did you 
find a way to tell lighttpd not to talk to an FCGI process
that is 
down? I only get a 500 error in that case.

-- 
Bernhard Graf

_______________________________________________
List: Catalystlists.rawmode.org
Listinfo: ht
tp://lists.rawmode.org/mailman/listinfo/catalyst
Searchable archive: http://www.mail-
archive.com/catalystlists.rawmode.org/
Dev site: http://dev.catalyst.per
l.org/

Quoting Bernhard Graf <catalyst2augensalat.de>:
> You should definitely do that. Not only for this case -
daemontools (or
> similar like runit) are superior for nearly every
server service on *ix
> OSes. Check out http://smard
en.org/runit/runscripts.html and compare
> the scripts with typical System-V run scripts (not
mentioning the
> supervision concept in general).

I suppose that the whole fcgi-pm process will be restarted
instead of individual
fcgi processes, correct? That'd mean a downtime of about 5
seconds which would
render it useless to me :(

> Why two FCGI process managers? To compensate FCGI
downtimes? Did you
> find a way to tell lighttpd not to talk to an FCGI
process that is
> down? I only get a 500 error in that case.

I just _hope_ that lighttpd will do the right thing as the
error log tells me
this when a backend server goes down:

connect failed: Connection refused on
unix:/srv/webapp.socket
backend died; we'll disable it for 5 seconds and send the
request to another
backend instead: reconnects: 0 load: 1

--Tobias

_______________________________________________
List: Catalystlists.rawmode.org
Listinfo: ht
tp://lists.rawmode.org/mailman/listinfo/catalyst
Searchable archive: http://www.mail-
archive.com/catalystlists.rawmode.org/
Dev site: http://dev.catalyst.per
l.org/

daemontools is brilliant. I use it for several services now.
it is so
easy to set up and use.

hot tip : don't forget the -e to myapp_fastcgi.pl option to
get
multilog writing to your log files.




On 8/16/07, Tobias Kremer <listfunkreich.de> wrote:
> Quoting Bernhard Graf <catalyst2augensalat.de>:
> > You should definitely do that. Not only for this
case - daemontools (or
> > similar like runit) are superior for nearly every
server service on *ix
> > OSes. Check out http://smard
en.org/runit/runscripts.html and compare
> > the scripts with typical System-V run scripts (not
mentioning the
> > supervision concept in general).
>
> I suppose that the whole fcgi-pm process will be
restarted instead of individual
> fcgi processes, correct? That'd mean a downtime of
about 5 seconds which would
> render it useless to me :(
>
> > Why two FCGI process managers? To compensate FCGI
downtimes? Did you
> > find a way to tell lighttpd not to talk to an FCGI
process that is
> > down? I only get a 500 error in that case.
>
> I just _hope_ that lighttpd will do the right thing as
the error log tells me
> this when a backend server goes down:
>
> connect failed: Connection refused on
unix:/srv/webapp.socket
> backend died; we'll disable it for 5 seconds and send
the request to another
> backend instead: reconnects: 0 load: 1
>
> --Tobias
>
> _______________________________________________
> List: Catalystlists.rawmode.org
> Listinfo: ht
tp://lists.rawmode.org/mailman/listinfo/catalyst
> Searchable archive: http://www.mail-
archive.com/catalystlists.rawmode.org/
> Dev site: http://dev.catalyst.per
l.org/
>


-- 
Daniel McBrearty
email : danielmcbrearty at gmail.com
http://www.engoi.com
http://danmcb.vox.com
http://danmcb.blogger.com
find me on linkedin and facebook
BTW : 0873928131

_______________________________________________
List: Catalystlists.rawmode.org
Listinfo: ht
tp://lists.rawmode.org/mailman/listinfo/catalyst
Searchable archive: http://www.mail-
archive.com/catalystlists.rawmode.org/
Dev site: http://dev.catalyst.per
l.org/

On Thursday 16 August 2007 14:32, Tobias Kremer wrote:
> Quoting Bernhard Graf <catalyst2augensalat.de>:
> > You should definitely do that. Not only for this
case - daemontools
> > (or similar like runit) are superior for nearly
every server
> > service on *ix OSes. Check out
> > http://smard
en.org/runit/runscripts.html and compare the scripts
> > with typical System-V run scripts (not mentioning
the supervision
> > concept in general).
>
> I suppose that the whole fcgi-pm process will be
restarted instead of
> individual fcgi processes, correct? That'd mean a
downtime of about 5
> seconds which would render it useless to me :(

No. The softlimit applies to each of the processes forked by
the master, 
so each process is terminated if it gets too fat.

> > Why two FCGI process managers? To compensate FCGI
downtimes? Did
> > you find a way to tell lighttpd not to talk to an
FCGI process that
> > is down? I only get a 500 error in that case.
>
> I just _hope_ that lighttpd will do the right thing as
the error log
> tells me this when a backend server goes down:
>
> connect failed: Connection refused on
unix:/srv/webapp.socket
> backend died; we'll disable it for 5 seconds and send
the request to
> another backend instead: reconnects: 0 load: 1

Just by setting multiple "socket" entries in the
lighttpd-conf?

-- 
Bernhard Graf

_______________________________________________
List: Catalystlists.rawmode.org
Listinfo: ht
tp://lists.rawmode.org/mailman/listinfo/catalyst
Searchable archive: http://www.mail-
archive.com/catalystlists.rawmode.org/
Dev site: http://dev.catalyst.per
l.org/

Quoting Bernhard Graf <catalyst2augensalat.de>:
> On Thursday 16 August 2007 14:32, Tobias Kremer wrote:
> > Quoting Bernhard Graf <catalyst2augensalat.de>:
> > > You should definitely do that. Not only for
this case - daemontools
> > > (or similar like runit) are superior for
nearly every server
> > > service on *ix OSes. Check out
> > > http://smard
en.org/runit/runscripts.html and compare the scripts
> > > with typical System-V run scripts (not
mentioning the supervision
> > > concept in general).
> > I suppose that the whole fcgi-pm process will be
restarted instead of
> > individual fcgi processes, correct? That'd mean a
downtime of about 5
> > seconds which would render it useless to me :(
> No. The softlimit applies to each of the processes
forked by the master,
> so each process is terminated if it gets too fat.

Cool! 

> > I just _hope_ that lighttpd will do the right
thing as the error log
> > tells me this when a backend server goes down:
> > connect failed: Connection refused on
unix:/srv/webapp.socket
> > backend died; we'll disable it for 5 seconds and
send the request to
> > another backend instead: reconnects: 0 load: 1
> Just by setting multiple "socket" entries in
the lighttpd-conf?

Yes, like so:

$HTTP["host"] =~ "bla" {
    fastcgi.server = (
        "" => (
            "MyApp" => (
                "socket" =>
"/srv/socket_1",
                "check-local" =>
"disable"
            ),
            (
                "socket" =>
"/srv/socket_2",
                "check-local" =>
"disable"
            )
        )
    )
}

--Tobias

_______________________________________________
List: Catalystlists.rawmode.org
Listinfo: ht
tp://lists.rawmode.org/mailman/listinfo/catalyst
Searchable archive: http://www.mail-
archive.com/catalystlists.rawmode.org/
Dev site: http://dev.catalyst.per
l.org/