Django/mod_python high memory usage - possible leak?

I'm running a production Django application on two
loadbalanced
webservers and a single, dedicated Postgres server handling
around
500k requests/day. I'm using memcached, and my database
server
performance has been fantastic.

Lately, I've been hitting very high percentages of free
memory used,
and occasionally spiking to very high percentages of swap
memory used.
I'm pretty familiar with Apache and Django, but not enough
to diagnose
what is happening - I'm hoping someone will be able to help
fill in
the gaps or tell me my understanding is off-base.

Symptoms:

What seems to be happening is that Apache grows to use all
available
memory, and my memused percentage will hit about 98%. From
what I can
tell, this does not necessarily mean that Apache is actively
using 98%
of memory. It is also my understanding that Apache won't
automatically
free up any memory until the process is killed. Perhaps
someone could
explain this? Is it strange that Apache never goes to 99%,
instead
always peaking at 98.x%?

My swap % has spiked badly a few times, up to 69%. I'm told
anything
roughly > 5% is bad. Could inefficiently designed views
lead to memory
problems? I've made sure on my most popular pages that I'm
only
importing what is absolutely needed, and I've even
dereferenced
variables to try and help garbage collection.

I grepped out the offending time period from my system
activity
report, and found the requests were low during the 10-20
minutes that
the swap % spiked. I'm not sure that the request load was
the
culprit.

Incidentals:

My most popular view looks for a memcached RSS feed, and if
it doesn't
exist, uses URLLIB2 to go out and get it (then cache it).
I've read
about possible problems with URLLIB2.openurl() that it might
not free
the socket after use. I've made sure to close all
connections and
dereference all variables/objects before the view returns.
Not sure if
this is relevant, but it seems worth mentioning. My
memcached hit
ratio is 99%, so I'm comfortable that it's not opening a
connection
too often.

What I've tried:

I've tweaked one web server and left the other as-is for
comparison.
(both are running identical Django codebase) I've upgraded
mod_python
from 3.1 to 3.3.1, and I've upgraded Python from 2.4 to 2.5.
Neither
seems to have made a noticeable difference. I dropped down
the max
requests per child from 4k to 2.5k, and this shorter
lifetime for the
process seems to have helped.

Questions:

It is my understanding that Apache Prefork MPM is
memory-intensive.
Will my httpd processes keep growing in size until they are
killed?
Should I ever expect them to drop in size during their
life?

>From an application design perspective, are there any
obvious culprits
for high swap percentages in a Django project?

Will increasing my memory from 1GB to 2GB help, or will
Apache just
swell to fill this new memory?

-------------------------------------------------------

System:

OS: Enterprise Red Hat 4
Memory: 1GB
Apache: 2.0.2 Prefork MPM
Python Version: 2.5
mod_python: 3.3.1
Django: .91-bug-fixes
Caching: Memcached

Relevant httpd.conf:

Keepalives off
StartServers       8
MinSpareServers    5
MaxSpareServers   15
ServerLimit      256
MaxClients       256
MaxRequestsPerChild  2500

Recent Top Output:

top - 12:36:23 up 108 days,  8:43,  1 user,  load average:
0.01, 0.03,
0.00
Tasks:  92 total,   1 running,  91 sleeping,   0 stopped,  
0 zombie
Cpu(s):  0.0% us, 50.0% sy,  0.0% ni, 50.0% id,  0.0% wa, 
0.0% hi,
0.0% si
Mem:   1034636k total,   943036k used,    91600k free,   
87320k
buffers
Swap:  1052216k total,    22560k used,  1029656k free,  
177080k
cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+
COMMAND
 5858 apache    16   0 72192  68m  404 S    0  6.8  
0:05.70
memcached
 5766 apache    15   0 70404  63m 3120 S    0  6.2  
0:09.98
httpd
  995 apache    15   0 69600  62m 3120 S    0  6.2  
0:10.16
httpd
21319 apache    15   0 69424  62m 3112 S    0  6.2  
0:06.73
httpd
25133 apache    15   0 69492  62m 3112 S    0  6.1  
0:06.81
httpd
24232 apache    15   0 68784  61m 3112 S    0  6.1  
0:06.42
httpd
25134 apache    16   0 68848  61m 3112 S    0  6.1  
0:06.20
httpd
25132 apache    15   0 68368  61m 3112 S    0  6.0  
0:06.24
httpd
29229 apache    16   0 59580  52m 3112 S    0  5.2  
0:04.92
httpd
26287 apache    15   0 59232  52m 3112 S    0  5.2   0:04.95
httpd

Sample of Bad Recent Sar -r output:

08:50:01 AM kbmemfree kbmemused  %memused kbbuffers 
kbcached
kbswpfree kbswpused  %swpused  kbswpcad
08:50:01 AM    143984    890652     86.08     10064    
28176
962864     89352      8.49     22860
09:00:01 AM     15536   1019100     98.50      3608    
25516
318740    733476     69.71     34576
09:10:01 AM    586352    448284     43.33      5384    
39268
797504    254712     24.21     67148
09:20:01 AM    334088    700548     67.71      6596    
39352
797520    254696     24.21     86132
09:30:01 AM    252024    782612     75.64      7600    
39952
797832    254384     24.18    102208
09:40:01 AM    527368    507268     49.03      2976    
24272
898520    153696     14.61     31512
09:50:01 AM    447144    587492     56.78      4184    
24628
898520    153696     14.61     33848
10:00:01 AM    398904    635732     61.44      5244    
25868
898520    153696     14.61     35188
10:10:01 AM    128408    906228     87.59      2952    
31112
584540    467676     44.45     79816
10:20:01 AM    563540    471096     45.53      4344    
33740
893044    159172     15.13     41996
10:30:01 AM    536080    498556     48.19      5332    
33968
893044    159172     15.13     42600
10:40:01 AM    432272    602364     58.22      7316    
41144
1022792     29424      2.80     11860
10:50:01 AM    576528    458108     44.28      8472    
42068
1022792     29424      2.80     11860


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Django users" group.
To post to this group, send email to django-usersgooglegroups.com
To unsubscribe from this group, send email to
django-users-unsubscribegooglegroups.com
For more options, visit this group at htt
p://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---

You stated that things were working fine and lately they
have gone
South.  Usually in this case it isn't about something
breaking all of
a sudden.  Something has probably changed.  Start with what
you have
changed.  Added any new features or code lately?  Moving out
further
have there been any updates to the server (besides your
upgrades to
the one)?  Next look at your users.  Have they started doing
something
different with the site?  Perhaps the volume of data that
you now have
is overwhelming your app.  I've seen sites that work well
in
development and the early days after release fall to their
knees
because the algorithms that worked well with low amounts of
data get
bogged down with large amounts of data.  Failing all else
start
profiling.  Coordinate those memory usage stats you have
with what is
going on in the server to help narrow down where in your
code to
look.  Hope that helps and good luck.

-Sam


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Django users" group.
To post to this group, send email to django-usersgooglegroups.com
To unsubscribe from this group, send email to
django-users-unsubscribegooglegroups.com
For more options, visit this group at htt
p://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On 3/29/07, snewman18gmail.com <snewman18gmail.com> wrote:
> My most popular view looks for a memcached RSS feed,
and if it doesn't
> exist, uses URLLIB2 to go out and get it (then cache
it). I've read
> about possible problems with URLLIB2.openurl() that it
might not free
> the socket after use. I've made sure to close all
connections and
> dereference all variables/objects before the view
returns. Not sure if
> this is relevant, but it seems worth mentioning. My
memcached hit
> ratio is 99%, so I'm comfortable that it's not opening
a connection
> too often.

I'm not sure this will fix your problem, but, as an aside,
I'd suggest
that you not perform HTTP requests (using urllib2 or any
other
library) from within a view. It's generally just not a good
thing to
do, because it makes your site's performance dependent on
the
performance of another Web site that's out of your control.
A better
solution would be to set up a regular update of the given
Web page(s)
via cron.

With that said, you could only do this sort of caching if
the set of
Web pages that you're requesting is static and known ahead
of time. If
you're operating a site like feedvalidator.org whose main
purpose is
to retrieve Web pages dynamically and do some operations on
them,
you're stuck making the requests from within the view.

Anyway, hope this helps.

Adrian

-- 
Adrian Holovaty
holovaty.com | djangoproject.com

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Django users" group.
To post to this group, send email to django-usersgooglegroups.com
To unsubscribe from this group, send email to
django-users-unsubscribegooglegroups.com
For more options, visit this group at htt
p://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---

> ... Failing all else start
> profiling.  Coordinate those memory usage stats you
have with what is
> going on in the server to help narrow down where in
your code to
> look.  Hope that helps and good luck.

Is there a way of profiling mod_python with hotshot (or
another tool)
that will give me an indication of the memory consumption?
I've only
dabbled with it.


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Django users" group.
To post to this group, send email to django-usersgooglegroups.com
To unsubscribe from this group, send email to
django-users-unsubscribegooglegroups.com
For more options, visit this group at htt
p://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Warning: almost no actual Django-related content below...

On Thu, 2007-03-29 at 18:15 +0000, snewman18gmail.com
wrote:
> I'm running a production Django application on two
loadbalanced
> webservers and a single, dedicated Postgres server
handling around
> 500k requests/day. I'm using memcached, and my database
server
> performance has been fantastic.
> 
> Lately, I've been hitting very high percentages of free
memory used,
> and occasionally spiking to very high percentages of
swap memory used.
> I'm pretty familiar with Apache and Django, but not
enough to diagnose
> what is happening - I'm hoping someone will be able to
help fill in
> the gaps or tell me my understanding is off-base.
> 
> Symptoms:
> 
> What seems to be happening is that Apache grows to use
all available
> memory, and my memused percentage will hit about 98%.
From what I can
> tell, this does not necessarily mean that Apache is
actively using 98%
> of memory. It is also my understanding that Apache
won't automatically
> free up any memory until the process is killed. 

That's fairly normal: the free() call in C will often keep
the memory
for the process's further use on a subsequent malloc() call.
So memory
usage appears to only go up and never down.

> Perhaps someone could
> explain this? Is it strange that Apache never goes to
99%, instead
> always peaking at 98.x%?

I'm not familiar enough with Apache internals to know about
this.
However, at some point, being able to effectively utilise
99.x% of
memory is difficult because of fragmentation of the memory
blocks. A lot
of malloc()/free() pairs for differing sizes could lead to
fragmentation
over time. I don't know for sure that this is what's
happening, but it's
possible and so what you're seeing is not unbelievable.

> My swap % has spiked badly a few times, up to 69%. I'm
told anything
> roughly > 5% is bad.

An actual percentage of swap usage isn't really indicative
of anything
much. The real performance problem is when swap is actually
being
swapped in and out. Because that is very slow compared to
memory
accesses (it has to do a disk access). So if your machine is
actively
swapping, the performance is going to go down. If it has
just moved some
very rarely accessed stuff into swap, that isn't necessarily
a
performance hit because that stuff could be just sitting
there
unaccessed, so it's never being waited on. On a server,
though, if swap
has non-trivial amounts of stuff in it, you have to wonder
how it got
there. On a desktop it might happen because you have
multiple apps open
and are only using a few at once. On a server that is less
likely -- all
applications are generally in active use. So keep an eye on
swap usage,
but particularly pay attention to active swapping, which is
the real
performance killer and a sign of something using a lot of
memory.

>  Could inefficiently designed views lead to memory
> problems? I've made sure on my most popular pages that
I'm only
> importing what is absolutely needed, and I've even
dereferenced
> variables to try and help garbage collection.

Unlikely. Profiling actively running Python processes for
memory usage
is a bear, though. So it's going to be very hard to tell.
You'll end up
having to poke around in places like /proc/<fd>/maps
(on a Linux
system).

A tool like pcat from The Coroner's Toolkit is very useful
here, too. Be
aware that tracking down what is really using the memory
takes time and
is often a futile task if there is no single large user.

[...]
> I've tweaked one web server and left the other as-is
for comparison.
> (both are running identical Django codebase) I've
upgraded mod_python
> from 3.1 to 3.3.1, and I've upgraded Python from 2.4 to
2.5. Neither
> seems to have made a noticeable difference. I dropped
down the max
> requests per child from 4k to 2.5k, and this shorter
lifetime for the
> process seems to have helped.

This is a lot higher than the default value. I've seen some
pretty high
volume (e.g. internet banking) sites running happily with
the default of
1000 for MaxRequestsPerChild. Do you have a reason for
cranking this up
so high, or it was just an experiment? Quite possibly
there's logic in
your settings -- you clearly seem to have done your research
-- but I
would be trying to keep this number down a bit more. Memory
management
is one of the main reasons MaxRequestsPerChild exists -- it
mitigates
any creeping memory leaks, for example -- so if you are
seeing excess
usage over time, restarting the child processes more
frequently is one
solution. Juggling this number and MinSpareServers requires
some
experimenting and knowing the usage patterns of your server
(#
simultaneous requests, burstiness behaviour, etc).

Based on the sar output you showed, it looks like the number
of servers
times the amount of memory they want is too much for your
system, so I
would definitely be looking to reduce one of those numbers.
Restarting
the processes more frequently is a way to reduce the second
number.

> Questions:
> 
> It is my understanding that Apache Prefork MPM is
memory-intensive.
> Will my httpd processes keep growing in size until they
are killed?
> Should I ever expect them to drop in size during their
life?
> 
> >From an application design perspective, are there
any obvious culprits
> for high swap percentages in a Django project?

Not that I can think of. There's very little persistent
information
between requests (the imported Python code is basically
nothing).
Memchached is obviously going to use some memory.

> 
> Will increasing my memory from 1GB to 2GB help, or will
Apache just
> swell to fill this new memory?

Realise that having all your memory used (or nearly used) is
not a bad
thing. Many server processes are designed to allow them to
grow to fill
available space (and not require more, so they can operate
with more as
well as less memory).

It just occurred to me that you haven't mentioned whether
you are
actually seeing performance problems or not. If you aren't
seeing
performance problems, then I'm not sure there is a real
problem here --
Apache is probably behaving fairly normally.

So, unfortunately, this isn't really helping diagnose the
root cause.
I'm just suggesting some mitigating measures. If you really
want to
delve deeply, break out the Coroner's Toolkit and go crazy.
It's fun,
it's educational, it's often frustrating (oops :( ), but
it's sometimes
necessary.

Regards,
Malcolm


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Django users" group.
To post to this group, send email to django-usersgooglegroups.com
To unsubscribe from this group, send email to
django-users-unsubscribegooglegroups.com
For more options, visit this group at htt
p://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---

snewman18gmail.com wrote on 03/29/07 20:15:
> I'm running a production Django application on two
loadbalanced
> webservers and a single, dedicated Postgres server
handling around
> 500k requests/day. I'm using memcached, and my database
server
> performance has been fantastic.
> 
> Lately, I've been hitting very high percentages of free
memory used,
> and occasionally spiking to very high percentages of
swap memory used.
> I'm pretty familiar with Apache and Django, but not
enough to diagnose
> what is happening - I'm hoping someone will be able to
help fill in
> the gaps or tell me my understanding is off-base.
> 
> Symptoms:
> 
> What seems to be happening is that Apache grows to use
all available
> memory, and my memused percentage will hit about 98%.
From what I can
> tell, this does not necessarily mean that Apache is
actively using 98%
> of memory. It is also my understanding that Apache
won't automatically
> free up any memory until the process is killed. Perhaps
someone could
> explain this? Is it strange that Apache never goes to
99%, instead
> always peaking at 98.x%?
> 
> My swap % has spiked badly a few times, up to 69%. I'm
told anything
> roughly > 5% is bad. Could inefficiently designed
views lead to memory
> problems? I've made sure on my most popular pages that
I'm only
> importing what is absolutely needed, and I've even
dereferenced
> variables to try and help garbage collection.
> 
> I grepped out the offending time period from my system
activity
> report, and found the requests were low during the
10-20 minutes that
> the swap % spiked. I'm not sure that the request load
was the
> culprit.
> 
> Incidentals:
> 
> My most popular view looks for a memcached RSS feed,
and if it doesn't
> exist, uses URLLIB2 to go out and get it (then cache
it). I've read
> about possible problems with URLLIB2.openurl() that it
might not free
> the socket after use. I've made sure to close all
connections and
> dereference all variables/objects before the view
returns. Not sure if
> this is relevant, but it seems worth mentioning. My
memcached hit
> ratio is 99%, so I'm comfortable that it's not opening
a connection
> too often.
> 
> What I've tried:
> 
> I've tweaked one web server and left the other as-is
for comparison.
> (both are running identical Django codebase) I've
upgraded mod_python
> from 3.1 to 3.3.1, and I've upgraded Python from 2.4 to
2.5. Neither
> seems to have made a noticeable difference. I dropped
down the max
> requests per child from 4k to 2.5k, and this shorter
lifetime for the
> process seems to have helped.
> 
> Questions:
> 
> It is my understanding that Apache Prefork MPM is
memory-intensive.
> Will my httpd processes keep growing in size until they
are killed?
> Should I ever expect them to drop in size during their
life?
> 
>>From an application design perspective, are there
any obvious culprits
> for high swap percentages in a Django project?
> 
> Will increasing my memory from 1GB to 2GB help, or will
Apache just
> swell to fill this new memory?
> 
>
-------------------------------------------------------
> 
> System:
> 
> OS: Enterprise Red Hat 4
> Memory: 1GB
> Apache: 2.0.2 Prefork MPM
> Python Version: 2.5
> mod_python: 3.3.1
> Django: .91-bug-fixes
> Caching: Memcached
> 
> Relevant httpd.conf:
> 
> Keepalives off
> StartServers       8
> MinSpareServers    5
> MaxSpareServers   15
> ServerLimit      256
> MaxClients       256
> MaxRequestsPerChild  2500
> 
> Recent Top Output:
> 
> top - 12:36:23 up 108 days,  8:43,  1 user,  load
average: 0.01, 0.03,
> 0.00
> Tasks:  92 total,   1 running,  91 sleeping,   0
stopped,   0 zombie
> Cpu(s):  0.0% us, 50.0% sy,  0.0% ni, 50.0% id,  0.0%
wa,  0.0% hi,
> 0.0% si
> Mem:   1034636k total,   943036k used,    91600k free, 
  87320k
> buffers
> Swap:  1052216k total,    22560k used,  1029656k free, 
 177080k
> cached
> 
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM   
TIME+
> COMMAND
>  5858 apache    16   0 72192  68m  404 S    0  6.8  
0:05.70
> memcached
>  5766 apache    15   0 70404  63m 3120 S    0  6.2  
0:09.98
> httpd
>   995 apache    15   0 69600  62m 3120 S    0  6.2  
0:10.16
> httpd
> 21319 apache    15   0 69424  62m 3112 S    0  6.2  
0:06.73
> httpd
> 25133 apache    15   0 69492  62m 3112 S    0  6.1  
0:06.81
> httpd
> 24232 apache    15   0 68784  61m 3112 S    0  6.1  
0:06.42
> httpd
> 25134 apache    16   0 68848  61m 3112 S    0  6.1  
0:06.20
> httpd
> 25132 apache    15   0 68368  61m 3112 S    0  6.0  
0:06.24
> httpd
> 29229 apache    16   0 59580  52m 3112 S    0  5.2  
0:04.92
> httpd
> 26287 apache    15   0 59232  52m 3112 S    0  5.2  
0:04.95 httpd
> 
> Sample of Bad Recent Sar -r output:
> 
> 08:50:01 AM kbmemfree kbmemused  %memused kbbuffers 
kbcached
> kbswpfree kbswpused  %swpused  kbswpcad
> 08:50:01 AM    143984    890652     86.08     10064    
28176
> 962864     89352      8.49     22860
> 09:00:01 AM     15536   1019100     98.50      3608    
25516
> 318740    733476     69.71     34576
> 09:10:01 AM    586352    448284     43.33      5384    
39268
> 797504    254712     24.21     67148
> 09:20:01 AM    334088    700548     67.71      6596    
39352
> 797520    254696     24.21     86132
> 09:30:01 AM    252024    782612     75.64      7600    
39952
> 797832    254384     24.18    102208
> 09:40:01 AM    527368    507268     49.03      2976    
24272
> 898520    153696     14.61     31512
> 09:50:01 AM    447144    587492     56.78      4184    
24628
> 898520    153696     14.61     33848
> 10:00:01 AM    398904    635732     61.44      5244    
25868
> 898520    153696     14.61     35188
> 10:10:01 AM    128408    906228     87.59      2952    
31112
> 584540    467676     44.45     79816
> 10:20:01 AM    563540    471096     45.53      4344    
33740
> 893044    159172     15.13     41996
> 10:30:01 AM    536080    498556     48.19      5332    
33968
> 893044    159172     15.13     42600
> 10:40:01 AM    432272    602364     58.22      7316    
41144
> 1022792     29424      2.80     11860
> 10:50:01 AM    576528    458108     44.28      8472    
42068
> 1022792     29424      2.80     11860
> 
> 

Maybe it helps serving static media (images,css,) through
e.g lighttpd 
instead of the same apache instance?

My understanding is that every apache child pulls in all
modules 
(mod_python, mod_perl, mod_php, mod_your_favorite_mod_here)
which all 
consume memory. So you end up with loads of processes using
??MB of RAM 
to serve up e.g. a 1k image.

I am by now means an expert on this. But maybe it's worth a
shot.


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Django users" group.
To post to this group, send email to django-usersgooglegroups.com
To unsubscribe from this group, send email to
django-users-unsubscribegooglegroups.com
For more options, visit this group at htt
p://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Mar 30, 6:31 pm, Steven Armstrong <s...c-area.ch> wrote:
> My understanding is that every apache child pulls in
all modules
> (mod_python, mod_perl, mod_php,
mod_your_favorite_mod_here) which all
> consume memory. So you end up with loads of processes
using ??MB of RAM
> to serve up e.g. a 1k image.
>
> I am by now means an expert on this. But maybe it's
worth a shot.

This doesn't really tell the whole story. Apache modules,
like C
extensions for Python are implemented as shared objects,
thus the code
segments of the Apache modules themselves should only incur
a memory
hit once across all process. For example, the mod_php.so for
some
versions of PHP is about 5MB. This does not mean though that
each
Apache child process has 5MB of private memory dedicated
just to it.
Instead their would generally only be a 5MB hit to the
operating
system as a whole. Note though that I am only talking about
code here
and not runtime memory usage.

For mod_python.so though this unfortunately doesn't
necessarily apply.
In the best case the mod_python.so file would be about 400KB
in size
and it would be shared as described above. If your Python
installation
was not built so as to generate a shared library though,
when
mod_python.so is created the contents of the static Python
library are
incorporated into the mod_python.so file itself, rather than
being
linked at run time. As a result, the mod_python.so file can
balloon
out to be 1.5-2.0MB in size, or much much more on certain
operating
systems if debugger support was enabled when Python was
compiled.
Worse is that because the object files from the Python
static library
are not position independent, when the mod_python.so file is
loaded
into memory, on some systems the operating system is forced
to perform
address relocations on the code segments from the Python
library
objects where they reside in mod_python.so. Because the
address
relocations require modifying the memory it is no longer
shared and as
a result it will actually end up consuming 1.5-2.0MB of
private memory
in each Apache child process.

Thus, one way of reducing memory usage with Apache is to
ensure that
your Python is built with a shared library and not a static
library
and that mod_python.so is actually using it. On
Linux/Solaris systems,
running 'ldd' on mod_python.so will tell you. If you don't
see a
dependency on libpythonX.Y.so then it isn't using Python as
a shared
library and you will get this extra memory hit on every
Apache child
process.

This still isn't the end of the story. Because mod_python is
a mix of
C code and Python code and it uses Python code as part of
its dispatch
mechanism, when it is first loaded it loads a whole bunch of
Python
modules before any request is even handled. It is because of
these
Python modules being loaded that the startup memory usage
of
mod_python appears to be so big. For a lot of these modules
a web
application will generally load them in eventually anyway,
so in the
bigger scheme of things it makes no difference, but for some
modules a
web application may not use them so the memory usage is an
unwelcome
hit.

Worse is that recent investigations into mod_python show
that some of
those Python modules are being loaded to get access to just
a single
function. Other modules are being loaded all the time even
though the
feature it is required for is optional and not often used.
When
changes were made to mod_python to eliminate as many modules
as one
could it actually dropped 1MB from the startup memory
usage.
Realistically a web application might take back half of that
later
when it wants the modules that were eliminated, but the
modules for
the optional feature wouldn't. So, mod_python itself could
see some
improvements in that area to potentially reduce memory usage
as well.

All in all though, this is still not going to help in the
grand scheme
of things because Python web frameworks are just getting too
fat.
Django is actually at the better end of the scale here as at
minimum
it adds only about an extra 5MB of modules to size of each
Apache
child process. TurboGears appears to be the worst, adding
about 14MB,
with Pylons somewhere in the middle.

So, although mod_python does have some things that can be
cleaned up,
in the main it is the Python web frameworks themselves which
add the
most to process size and that is before any code is run and
data
caching becomes a factor.

Graham


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Django users" group.
To post to this group, send email to django-usersgooglegroups.com
To unsubscribe from this group, send email to
django-users-unsubscribegooglegroups.com
For more options, visit this group at htt
p://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Graham Dumpleton wrote on 03/30/07 12:27:
> On Mar 30, 6:31 pm, Steven Armstrong <s...c-area.ch> wrote:
>> My understanding is that every apache child pulls
in all modules
>> (mod_python, mod_perl, mod_php,
mod_your_favorite_mod_here) which all
>> consume memory. So you end up with loads of
processes using ??MB of RAM
>> to serve up e.g. a 1k image.
>>
>> I am by now means an expert on this. But maybe it's
worth a shot.
> 
> This doesn't really tell the whole story. Apache
modules, like C
> extensions for Python are implemented as shared
objects, thus the code
> segments of the Apache modules themselves should only
incur a memory
> hit once across all process. For example, the
mod_php.so for some
> versions of PHP is about 5MB. This does not mean though
that each
> Apache child process has 5MB of private memory
dedicated just to it.
> Instead their would generally only be a 5MB hit to the
operating
> system as a whole. Note though that I am only talking
about code here
> and not runtime memory usage.
> 
> For mod_python.so though this unfortunately doesn't
necessarily apply.
> In the best case the mod_python.so file would be about
400KB in size
> and it would be shared as described above. If your
Python installation
> was not built so as to generate a shared library
though, when
> mod_python.so is created the contents of the static
Python library are
> incorporated into the mod_python.so file itself, rather
than being
> linked at run time. As a result, the mod_python.so file
can balloon
> out to be 1.5-2.0MB in size, or much much more on
certain operating
> systems if debugger support was enabled when Python was
compiled.
> Worse is that because the object files from the Python
static library
> are not position independent, when the mod_python.so
file is loaded
> into memory, on some systems the operating system is
forced to perform
> address relocations on the code segments from the
Python library
> objects where they reside in mod_python.so. Because the
address
> relocations require modifying the memory it is no
longer shared and as
> a result it will actually end up consuming 1.5-2.0MB of
private memory
> in each Apache child process.
> 
> Thus, one way of reducing memory usage with Apache is
to ensure that
> your Python is built with a shared library and not a
static library
> and that mod_python.so is actually using it. On
Linux/Solaris systems,
> running 'ldd' on mod_python.so will tell you. If you
don't see a
> dependency on libpythonX.Y.so then it isn't using
Python as a shared
> library and you will get this extra memory hit on every
Apache child
> process.
> 
> This still isn't the end of the story. Because
mod_python is a mix of
> C code and Python code and it uses Python code as part
of its dispatch
> mechanism, when it is first loaded it loads a whole
bunch of Python
> modules before any request is even handled. It is
because of these
> Python modules being loaded that the startup memory
usage of
> mod_python appears to be so big. For a lot of these
modules a web
> application will generally load them in eventually
anyway, so in the
> bigger scheme of things it makes no difference, but for
some modules a
> web application may not use them so the memory usage is
an unwelcome
> hit.
> 
> Worse is that recent investigations into mod_python
show that some of
> those Python modules are being loaded to get access to
just a single
> function. Other modules are being loaded all the time
even though the
> feature it is required for is optional and not often
used. When
> changes were made to mod_python to eliminate as many
modules as one
> could it actually dropped 1MB from the startup memory
usage.
> Realistically a web application might take back half of
that later
> when it wants the modules that were eliminated, but the
modules for
> the optional feature wouldn't. So, mod_python itself
could see some
> improvements in that area to potentially reduce memory
usage as well.
> 
> All in all though, this is still not going to help in
the grand scheme
> of things because Python web frameworks are just
getting too fat.
> Django is actually at the better end of the scale here
as at minimum
> it adds only about an extra 5MB of modules to size of
each Apache
> child process. TurboGears appears to be the worst,
adding about 14MB,
> with Pylons somewhere in the middle.
> 
> So, although mod_python does have some things that can
be cleaned up,
> in the main it is the Python web frameworks themselves
which add the
> most to process size and that is before any code is run
and data
> caching becomes a factor.
> 
> Graham
> 

Wow! Many thanks for the detailed explanation.

Still, after reading your post, it seems one could save
quite some 
resources by not serving static media trough the same apache
instance 
which runs django and friends.

I mean, usually you get a request, django does it's thing,
sends back 
the page, which in turn requests another 10+ resources which
don't 
require django or other server side processing.

If those 10+ requests are handled by a lightweight server
that could 
make quite a difference for a busy site.

The section 'Scaling' at [1] has some other interesting info
about all this.

[1] http://w
ww.djangobook.com/en/beta/chapter21/


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Django users" group.
To post to this group, send email to django-usersgooglegroups.com
To unsubscribe from this group, send email to
django-users-unsubscribegooglegroups.com
For more options, visit this group at htt
p://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Hello,

I've notice the same memory consuption on my servers.
I'm running django appz on 3 LVS loadbalanced mod_python
apache http
servers.
If, for instance, I do a maintnance on one of the server
Swap is
increasing a lot on the two remaining servers, even if those
two
servers should handle the traffic.

This week I decided to give a try to a new configuration.
I'm now
running django on lighttpd / fcgi and I have no swap at
all.
So I'm aware the mod_python is the prefered way to use
django ... buy
I'm quite happy with that new server configuration 

 More over, despite django caching is really good, it still
need to
talk to the webserver, a way to avoid this is to use Squid
on top of
that.

Regards,

xav


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Django users" group.
To post to this group, send email to django-usersgooglegroups.com
To unsubscribe from this group, send email to
django-users-unsubscribegooglegroups.com
For more options, visit this group at htt
p://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On 3/30/07, xgdlm <grangiergmail.com> wrote:
...
> This week I decided to give a try to a new
configuration. I'm now
> running django on lighttpd / fcgi and I have no swap at
all.
> So I'm aware the mod_python is the prefered way to use
django ... buy
> I'm quite happy with that new server configuration 

Can you give some more details about any changes you made on
both
apache and lightty that might account for this?

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Django users" group.
To post to this group, send email to django-usersgooglegroups.com
To unsubscribe from this group, send email to
django-users-unsubscribegooglegroups.com
For more options, visit this group at htt
p://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---