Scaling the server

Hi guys,

A few weeks ago I had problems with my dedicated server, it
was
stucking twice or more times a week. After some digging, I
solve the
problem updating the mysqldb lib version. The server goes ok
almost
all the time, but I still have server stuck when I had too
much hits.

My server had 1.8mi to 2.2mi of hits a day, I found that I
must scale
my server. Before start the machine acquisition and money
expending, I
wanted to get some experience from you. I donīt want to kill
a fly
with a cannon ball ;)

I had django running over a mod_python/apache instance that
are
serving my media files too. One of my bottlenecks was the
home page,
yesterday I start to generate it through a script called by
cron. I
follow this approach even using locmen cache backend. Is it
a good
idea to use lighttpd on the same server to serve static
files or
should I need to setup a different server to database and
maybe
another to media files?

Thanks for any help,
Best regards.

--
Michel Thadeu Sabchuk


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Django users" group.
To post to this group, send email to django-usersgooglegroups.com
To unsubscribe from this group, send email to
django-users-unsubscribegooglegroups.com
For more options, visit this group at htt
p://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Oct 24, 5:13 am, Michel Thadeu Sabchuk <miche...gmail.com> wrote:
> Hi guys,
>
> A few weeks ago I had problems with my dedicated
server, it was
> stucking twice or more times a week. After some
digging, I solve the
> problem updating the mysqldb lib version. The server
goes ok almost
> all the time, but I still have server stuck when I had
too much hits.
>
> My server had 1.8mi to 2.2mi of hits a day, I found
that I must scale
> my server. Before start the machine acquisition and
money expending, I
> wanted to get some experience from you. I donīt want to
kill a fly
> with a cannon ball ;)
>
> I had django running over a mod_python/apache instance
that are
> serving my media files too. One of my bottlenecks was
the home page,
> yesterday I start to generate it through a script
called by cron. I
> follow this approach even using locmen cache backend.
Is it a good
> idea to use lighttpd on the same server to serve static
files or
> should I need to setup a different server to database
and maybe
> another to media files?

What percentage of requests are actually yielding static
pages?

People often too quickly run to lighttpd thinking it will
somehow
magically solve all their problems, but if the bulk of the
requests
are actually dynamic and served by Django, and it is Django
and the
database which are the bottleneck, it will make very little
if any
difference.

Now to try and get to what the real issue may be, expand on
what you
mean by 'server stuck when I had too much hits'. What
exactly is
getting stuck, the connections through to the database, or
Apache
itself running out of processes and/or worker threads to
keep up with
the volume of requests? Or is the box starting to run out of
physical
memory and starting to swap excessively.

Where the problem lies will guide what you may need to do.

Graham


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Django users" group.
To post to this group, send email to django-usersgooglegroups.com
To unsubscribe from this group, send email to
django-users-unsubscribegooglegroups.com
For more options, visit this group at htt
p://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Hi Graham!


> What percentage of requests are actually yielding
static pages?
> People often too quickly run to lighttpd thinking it
will somehow
> magically solve all their problems, but if the bulk of
the requests
> are actually dynamic and served by Django, and it is
Django and the
> database which are the bottleneck, it will make very
little if any
> difference.

Ok,

> Now to try and get to what the real issue may be,
expand on what you
> mean by 'server stuck when I had too much hits'.

I couldnīt figure out exactly my problem. The key pages of
my site is
cached through locmen, the index page is cron generated. I
have too
much media files, my site is a games portal and I have
videos, images
and stuff served on the same server, a Dual Intel(R)
Xeon(TM) CPU
3.20GHz with 4GB of RAM space. I have the following
installation:

* Apache2.2
* Python-2.4.2
* Django-0.96pre served through mod_python
* MySQLdb-1.2.2
* MySQL-5.0.18

At some key pages I use djangoīs locmen cache. My home page
is
somewhat slow because it has too many interchains of news
sections
(too much filtering), this way, I generate the home page
every 30
minutes and serve it as a static page. I start this approach
this
week, in the past time I let the home page to be cached by
locmen too.

My RAM usage frequently reaches 3GB or more, I canīt tell it
uses the
swap
space frequently. Last days, when the server is slow, a
`top` tells me
that
processing is at high levels (60% to 90%).

At this exact moment, when I have a big number of users
accessing my
page the server seems stable.

top - 17:46:48 up  4:27,  2 users,  load average: 0.22,
0.52, 0.66
Tasks: 117 total,   1 running, 116 sleeping,   0 stopped,  
0 zombie
Cpu(s):  7.1% us,  0.7% sy,  0.0% ni, 91.8% id,  0.1% wa, 
0.1% hi,
0.2% si
Mem:   4045568k total,  2250328k used,  1795240k free,   
55536k
buffers
Swap:  2104472k total,        0k used,  2104472k free, 
1374412k
cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+ 
COMMAND
 2632 wwwrun    15   0  157m  28m 3996 S   12  0.7   0:00.70
httpd2-
prefork
 2634 wwwrun    15   0  157m  29m 4044 S   10  0.7   0:00.83
httpd2-
prefork
 2553 wwwrun    15   0  160m  31m 4080 S    3  0.8   0:01.96
httpd2-
prefork
 2434 wwwrun    15   0  171m  42m 4076 S    3  1.1   0:06.69
httpd2-
prefork
 2907 mysql     15   0  130m  28m 5116 S    2  0.7  14:31.15
mysqld
 2313 wwwrun    15   0  160m  31m 4088 S    0  0.8   0:03.40
httpd2-
prefork
 2449 wwwrun    15   0  161m  33m 5324 S    0  0.9   0:03.01
httpd2-
prefork
 ...

A friend told me that I can make python library shared. I
read the
Jacobīs post [1] about performance tips and the Chase Davis
[2]
recomendations. I get another box to make server scaling, it
is a
Pentium 4 CPU 3.4GHz with 2GB of RAM space. What is the
best
alternative to this server? Should I use it as a database
server or
should I use it as a media files server, or maybe I should
use it for
apache and the another server for database...

I had problems like the about 2 months ago and I updated the
python
mysql library (my last one had a bug with garbage
collection), and the
server goes ok for these 2 months, it start to show me
problems again
when I had an increment of users. I could not tell how many
was this
increment, but on monday I launch a new online game (it is
served on a
partner) and I could measure the incrementation of users on
site by
the forum and other interactive sections. The server stopped
to run
again today just after I change a header file used by this
webgame
(the webgame is hosted by another server, but the header is
on my
server), but I donīt know if it is a pertinent information
:P

I am very thankfull for all your help,
Best regards.

[1] http://www.jacobian.org/writing/2005/dec/12/
django-performance-tips/
[2] http://www.car-chase.net/2007/apr/30/djan
go-deployment-lessons-learned/

--
Michel Thadeu Sabchuk
MisterApe


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Django users" group.
To post to this group, send email to django-usersgooglegroups.com
To unsubscribe from this group, send email to
django-users-unsubscribegooglegroups.com
For more options, visit this group at htt
p://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---