a very slow view, > 30secs to load

Hellow,

Any advice you can offer on optimizing this view would be
greatly
appreciated.

http://dpaste.com/7543/

I was planning on trying to reduce the amount of data in
pForms, the
list that is passed to the template.  Since i'm saving 19
presentations into this list along with the forms, i'm
assuming this
is the bottleneck.  Any ideas how to work around this?

I was going to start by only saving the data that I need,
rather than
the entire object.

Also, since most of this data does not change very often, I
should
probably cache it.  Do you have a recommendation on how to
cache the
objects in this view?

Thanks,

Milan


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Django users" group.
To post to this group, send email to django-usersgooglegroups.com
To unsubscribe from this group, send email to
django-users-unsubscribegooglegroups.com
For more options, visit this group at htt
p://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Wed, 2007-03-28 at 19:59 +0000, Milan Andric wrote:
> Hellow,
> 
> Any advice you can offer on optimizing this view would
be greatly
> appreciated.
> 
> http://dpaste.com/7543/
> 
> I was planning on trying to reduce the amount of data
in pForms, the
> list that is passed to the template.  Since i'm saving
19
> presentations into this list along with the forms, i'm
assuming this
> is the bottleneck.  Any ideas how to work around this?

Don't assume... test! You can put calls to time.time() in
various places
in the view and then print out the differences between them
to a file or
to stderr (or even stdout if you are using the development
server). That
will show you which portions are taking the longest. With a
few
iterations, moving the time.time() calls around, you should
be able to
work out the exact lines that are taking the most time.

Then come back and we can talk.

You might discover, for example, that it's not the view that
is taking
all the time, but the template rendering for some reason
(such as making
a lot of database calls by constructing querysets whilst
rendering the
template).

Regards,
Malcolm


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Django users" group.
To post to this group, send email to django-usersgooglegroups.com
To unsubscribe from this group, send email to
django-users-unsubscribegooglegroups.com
For more options, visit this group at htt
p://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Mar 28, 6:26 pm, Malcolm Tredinnick <malc...pointy-stick.com>
wrote:
> On Wed, 2007-03-28 at 19:59 +0000, Milan Andric wrote:
> > Hellow,
>
> > Any advice you can offer on optimizing this view
would be greatly
> > appreciated.
>
> >http://dpaste.com/7543/
>
> > I was planning on trying to reduce the amount of
data in pForms, the
> > list that is passed to the template.  Since i'm
saving 19
> > presentations into this list along with the forms,
i'm assuming this
> > is the bottleneck.  Any ideas how to work around
this?
>
> Don't assume... test! You can put calls to time.time()
in various places
> in the view and then print out the differences between
them to a file or
> to stderr (or even stdout if you are using the
development server). That
> will show you which portions are taking the longest.
With a few
> iterations, moving the time.time() calls around, you
should be able to
> work out the exact lines that are taking the most
time.
>
> Then come back and we can talk.
>
> You might discover, for example, that it's not the view
that is taking
> all the time, but the template rendering for some
reason (such as making
> a lot of database calls by constructing querysets
whilst rendering the
> template).
>

Hi Malcolm,

Thanks for your response.  I'm failing at getting any debug
information out of the production (apache2/mod_python)
environment.  I
tried using print and  sys.stderr.write but the logs show no
signs of
it.  So there might be something broken there.  Will get to
that
later.

But in the dev environment I can see pretty well that the
view is
functioning very quickly.  So your hunch about the problem
lying in
the templates is probably right.

So I posted the template: http://dpaste.com/7603/
.
I'm not sure how to optimize the querysets.  Don't make the
queries in
the template?   What are some ways to optimize this
template.  In the
meantime I'll try to scan the ML archives and looking at
caching.

Let me know what you think.

The problem with caching is that this view is unique for
each user.
That's how it is now at least, but maybe I need to take a
new
approach.  My current one takes little thought. 
Pre-optimization is
the root of all evil. ;)

Milan


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Django users" group.
To post to this group, send email to django-usersgooglegroups.com
To unsubscribe from this group, send email to
django-users-unsubscribegooglegroups.com
For more options, visit this group at htt
p://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Mar 30, 5:26 am, "Milan Andric" <mand...gmail.com> wrote:
> On Mar 28, 6:26 pm, Malcolm Tredinnick <malc...pointy-stick.com>
> wrote:
>
>
>
> > On Wed, 2007-03-28 at 19:59 +0000, Milan Andric
wrote:
> > > Hellow,
>
> > > Any advice you can offer on optimizing this
view would be greatly
> > > appreciated.
>
> > >http://dpaste.com/7543/
>
> > > I was planning on trying to reduce the amount
of data in pForms, the
> > > list that is passed to the template.  Since
i'm saving 19
> > > presentations into this list along with the
forms, i'm assuming this
> > > is the bottleneck.  Any ideas how to work
around this?
>
> > Don't assume... test! You can put calls to
time.time() in various places
> > in the view and then print out the differences
between them to a file or
> > to stderr (or even stdout if you are using the
development server). That
> > will show you which portions are taking the
longest. With a few
> > iterations, moving the time.time() calls around,
you should be able to
> > work out the exact lines that are taking the most
time.
>
> > Then come back and we can talk.
>
> > You might discover, for example, that it's not the
view that is taking
> > all the time, but the template rendering for some
reason (such as making
> > a lot of database calls by constructing querysets
whilst rendering the
> > template).
>
> Hi Malcolm,
>
> Thanks for your response.  I'm failing at getting any
debug
> information out of the production (apache2/mod_python)
environment.  I
> tried using print and  sys.stderr.write but the logs
show no signs of
> it.  So there might be something broken there.  Will
get to that
> later.

Under mod_python, you must explicitly call flush() on
sys.stderr to
have anything appear in Apache logs promptly. Ie.,

  print >> sys.stderr, "Hi there!"
  sys.stderr.flush()

If you know you are running under mod_python, you may be
better off
using:

  from mod_python import apache
  apache.log_error("Hi there!")

Graham


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Django users" group.
To post to this group, send email to django-usersgooglegroups.com
To unsubscribe from this group, send email to
django-users-unsubscribegooglegroups.com
For more options, visit this group at htt
p://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Great pointers, thanks!  I will look at that.  In the
meantime I
thought i would post my debug info.   I used this snippet in
my
template to see what queries are happening.

http://www
.djangosnippets.org/snippets/93/

Looks like I'm making quite a few queries there, 71, but
they still
don't seem to add up to 30+ seconds.

http://dpaste.com/7617/

Hrm, any thoughts so far?

--
Milan


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Django users" group.
To post to this group, send email to django-usersgooglegroups.com
To unsubscribe from this group, send email to
django-users-unsubscribegooglegroups.com
For more options, visit this group at htt
p://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Mar 29, 2:26 pm, "Milan Andric" <mand...gmail.com> wrote:
> But in the dev environment I can see pretty well that
the view is
> functioning very quickly.  So your hunch about the
problem lying in
> the templates is probably right.

If you aren't seeing the slowdown on your dev server, then
you
shouldn't be seeing a slowdown on your production server
either.  Is
your dev database loaded with the same amount of data as
production?
When you say that the view is functioning very quickly, do
you mean
the time from request to page render?  If so, the template
is already
getting run and is probably not your problem.

> Let me know what you think.

In your view, I think you can take this code:

f = newforms.form_for_model(PresentationEval)
f.base_fields['presentation'].widget =
widgets.HiddenInput()
f.base_fields['application'].widget = widgets.HiddenInput()
f.base_fields['topic_rating'].widget =
widgets.Select( choices=PresentationEval.RATINGS)
f.base_fields['substance_rating'].widget =
widgets.Select( choices=PresentationEval.RATINGS)
f.base_fields['expertise_rating'].widget =
widgets.Select( choices=PresentationEval.RATINGS)
f.base_fields['relevance_rating'].widget =
widgets.Select( choices=PresentationEval.RATINGS)
f.base_fields['overall_rating'].widget =
widgets.Select( choices=PresentationEval.RATINGS)

out of the loop, as it is just constructing the form class
that will
be used to instantiate your pForms instances.

Gary


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Django users" group.
To post to this group, send email to django-usersgooglegroups.com
To unsubscribe from this group, send email to
django-users-unsubscribegooglegroups.com
For more options, visit this group at htt
p://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Have you obtained the raw queries that are being run?  If
so, how long
do they take to run against your db when you execute them
yourself?
This could be a database performance problem as opposed to a
Django
problem.

On Mar 29, 11:31 pm, "Gary Wilson"
<gary.wil...gmail.com> wrote:
> On Mar 29, 2:26 pm, "Milan Andric"
<mand...gmail.com> wrote:
>
> > But in the dev environment I can see pretty well
that the view is
> > functioning very quickly.  So your hunch about the
problem lying in
> > the templates is probably right.
>
> If you aren't seeing the slowdown on your dev server,
then you
> shouldn't be seeing a slowdown on your production
server either.  Is
> your dev database loaded with the same amount of data
as production?
> When you say that the view is functioning very quickly,
do you mean
> the time from request to page render?  If so, the
template is already
> getting run and is probably not your problem.
>
> > Let me know what you think.
>
> In your view, I think you can take this code:
>
> f = newforms.form_for_model(PresentationEval)
> f.base_fields['presentation'].widget =
widgets.HiddenInput()
> f.base_fields['application'].widget =
widgets.HiddenInput()
> f.base_fields['topic_rating'].widget =
> widgets.Select( choices=PresentationEval.RATINGS)
> f.base_fields['substance_rating'].widget =
> widgets.Select( choices=PresentationEval.RATINGS)
> f.base_fields['expertise_rating'].widget =
> widgets.Select( choices=PresentationEval.RATINGS)
> f.base_fields['relevance_rating'].widget =
> widgets.Select( choices=PresentationEval.RATINGS)
> f.base_fields['overall_rating'].widget =
> widgets.Select( choices=PresentationEval.RATINGS)
>
> out of the loop, as it is just constructing the form
class that will
> be used to instantiate your pForms instances.
>
> Gary


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Django users" group.
To post to this group, send email to django-usersgooglegroups.com
To unsubscribe from this group, send email to
django-users-unsubscribegooglegroups.com
For more options, visit this group at htt
p://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Mar 29, 8:31 pm, "Gary Wilson"
<gary.wil...gmail.com> wrote:
> On Mar 29, 2:26 pm, "Milan Andric"
<mand...gmail.com> wrote:
>
> > But in the dev environment I can see pretty well
that the view is
> > functioning very quickly.  So your hunch about the
problem lying in
> > the templates is probably right.
>
> If you aren't seeing the slowdown on your dev server,
then you
> shouldn't be seeing a slowdown on your production
server either.  Is
> your dev database loaded with the same amount of data
as production?

Yes

> When you say that the view is functioning very quickly,
do you mean
> the time from request to page render?  If so, the
template is already
> getting run and is probably not your problem.

My conclusion that the view was functioning quickly was from
a print
time.time() test I did.  According to that it took .2 secs
from start
to the point right before the view called
render_to_response().
Again, this was on my dev site which doesn't have as much
data.

>
> > Let me know what you think.
>
> In your view, I think you can take this code:
>
> f = newforms.form_for_model(PresentationEval)
> f.base_fields['presentation'].widget =
widgets.HiddenInput()
> f.base_fields['application'].widget =
widgets.HiddenInput()
> f.base_fields['topic_rating'].widget =
> widgets.Select( choices=PresentationEval.RATINGS)
> f.base_fields['substance_rating'].widget =
> widgets.Select( choices=PresentationEval.RATINGS)
> f.base_fields['expertise_rating'].widget =
> widgets.Select( choices=PresentationEval.RATINGS)
> f.base_fields['relevance_rating'].widget =
> widgets.Select( choices=PresentationEval.RATINGS)
> f.base_fields['overall_rating'].widget =
> widgets.Select( choices=PresentationEval.RATINGS)
>
> out of the loop, as it is just constructing the form
class that will
> be used to instantiate your pForms instances.

I took that out of the loop and it made a big difference. 
Makes total
sense too, I was doing a lot of extra work there but didn't
realize
it.  The time difference was dramatic.  I timed it using
basic
sys.stderr.write() and .flush() with apache2/mod_python.

creating classes inside the loop, calling
newforms.form_for_model() 19
times:
for p in ... 1175239555.13
done with for p in ... 1175239601.91
47 Seconds?!

only creating the class once:
for p in ... 1175239882.93
done with for p in ... 1175239886.7
3 Seconds!

So I think that was it.

This is on django-svn 4455, a bit outdated. I tried updating
a few
days ago but some non-backwards compat code bit me and had
to back off
for a bit.

Thanks Gary and Everyone!


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Django users" group.
To post to this group, send email to django-usersgooglegroups.com
To unsubscribe from this group, send email to
django-users-unsubscribegooglegroups.com
For more options, visit this group at htt
p://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Fri, 2007-03-30 at 07:49 +0000, Milan Andric wrote:
> 
> 
> On Mar 29, 8:31 pm, "Gary Wilson"
<gary.wil...gmail.com> wrote:
[...]
> > When you say that the view is functioning very
quickly, do you mean
> > the time from request to page render?  If so, the
template is already
> > getting run and is probably not your problem.
> 
> My conclusion that the view was functioning quickly was
from a print
> time.time() test I did.  According to that it took .2
secs from start
> to the point right before the view called
render_to_response().
> Again, this was on my dev site which doesn't have as
much data.

[... useful suggestion from Gary snipped...]

> creating classes inside the loop, calling
newforms.form_for_model() 19
> times:
> for p in ... 1175239555.13
> done with for p in ... 1175239601.91
> 47 Seconds?!
> 
> only creating the class once:
> for p in ... 1175239882.93
> done with for p in ... 1175239886.7
> 3 Seconds!
> 
> So I think that was it.

I'm confused. First you said it took 0.2 seconds, then you
say it took
47  seconds and then 3 seconds. Both of those numbers are
bigger than
0.2. I know this because I took advanced maths courses at
university.

Even so, around three seconds for 19 iterations of that loop
without the
form seems a bit slow. That's only just over five per second
(since it's
really 3.8 seconds). We may need to look at newforms
performance when
things have settled down a bit. Not worth being too
premature in the
optimisation, but this is an interesting data point.

Malcolm




--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Django users" group.
To post to this group, send email to django-usersgooglegroups.com
To unsubscribe from this group, send email to
django-users-unsubscribegooglegroups.com
For more options, visit this group at htt
p://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---