Long time process and FCGI

Hello, I have a process that sends an e-mail to more then
5000 addresses 
(grouping it in 50 each time) and after a while it gives me
the error:

-------
Internal Server Error

The server encountered an internal error or misconfiguration
and was 
unable to complete your request.

Please contact the server administrator, webmasterutopia-design.net and 
inform them of the time the error occurred, and anything you
might have 
done that may have caused the error.

More information about this error may be available in the
server error log.

Additionally, a 404 Not Found error was encountered while
trying to use 
an ErrorDocument to handle the request.

-----------
Apache/1.3.37 Server at www.utopia-design.net Port 80
--------------------------------------


So, trying to realize what's wrong, I found this on the log
of my server:

[Thu Mar 29 14:26:17 2007] [warn] FastCGI: (dynamic) server

"/home/curalouc/public_html/enercons/django.fcgi"
(uid 32016, gid 32018) 
restarted (pid 21406)

[Thu Mar 29 14:26:17 2007] [warn] FastCGI: scheduled the
restart of the last (dynamic) server
"/home/curalouc/public_html/enercons/django.fcgi"
process: reached dynamicMaxClassProcs (10)
[Thu Mar 29 14:26:12 2007] [warn] FastCGI: (dynamic) server
"/home/curalouc/public_html/enercons/django.fcgi"
(pid 20125) terminated by calling exit with status '0'
[Thu Mar 29 14:26:12 2007] [warn] FastCGI: (dynamic) server
"/home/curalouc/public_html/enercons/django.fcgi"
(pid 20125) termination signaled
[Thu Mar 29 14:22:59 2007] [error] [client 200.146.81.46]
File does not exist: /home/curalouc/public_html/500.shtml



And now, I just get stucked and I have no idea what I can do
to send all 
those mails in a view, and I am not sure if this is the
problem.
When I tried the process with less then 10 mails, it was
okay.

Any help will be appreciate,

regards


4nderson


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Django users" group.
To post to this group, send email to django-usersgooglegroups.com
To unsubscribe from this group, send email to
django-users-unsubscribegooglegroups.com
For more options, visit this group at htt
p://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Hello,

By the looks of it, mod_fcgi has a limit as to how much time
it could
wait for response from a script. Since you're doing a heavy
operation
that takes a while, i guess it times out.

The point is  - if your executed action takes so much time,
you're
clearly having a wrong conception on when and how you should
be
executing it. It could be better off as an external process
(that
still uses django in its code) that is triggered by certain
actions in
your web-application, or scheduled by cron. In any case - it
is not
the job of your django view to handle bulk transfers, it was
not meant
to be doing that, even if the mod_fcgi didn't have the
timeout.


On 30/03/07, Anderson Santos Silva <curaloucuragmail.com> wrote:
>
> Hello, I have a process that sends an e-mail to more
then 5000 addresses
> (grouping it in 50 each time) and after a while it
gives me the error:
>
> -------
> Internal Server Error
>
> The server encountered an internal error or
misconfiguration and was
> unable to complete your request.
>
> Please contact the server administrator, webmasterutopia-design.net and
> inform them of the time the error occurred, and
anything you might have
> done that may have caused the error.
>
> More information about this error may be available in
the server error log.
>
> Additionally, a 404 Not Found error was encountered
while trying to use
> an ErrorDocument to handle the request.
>

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Django users" group.
To post to this group, send email to django-usersgooglegroups.com
To unsubscribe from this group, send email to
django-users-unsubscribegooglegroups.com
For more options, visit this group at htt
p://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---

    That's exactly what I thought when I dealed with that,
but I can't 
imagine a separated process running in the background and
return a 
response to the user that his "action" was
activated if every method 
waits to get back instead of a kind of thread operation.
(Yes, I came 
from windows programming  to Web, so my knowledge is
kind of 
limited with Python)

     Any suggestion?




Atilla wrote:
> Hello,
>
> By the looks of it, mod_fcgi has a limit as to how much
time it could
> wait for response from a script. Since you're doing a
heavy operation
> that takes a while, i guess it times out.
>
> The point is  - if your executed action takes so much
time, you're
> clearly having a wrong conception on when and how you
should be
> executing it. It could be better off as an external
process (that
> still uses django in its code) that is triggered by
certain actions in
> your web-application, or scheduled by cron. In any case
- it is not
> the job of your django view to handle bulk transfers,
it was not meant
> to be doing that, even if the mod_fcgi didn't have the
timeout.
>
>
> On 30/03/07, Anderson Santos Silva <curaloucuragmail.com> wrote:
>   
>> Hello, I have a process that sends an e-mail to
more then 5000 addresses
>> (grouping it in 50 each time) and after a while it
gives me the error:
>>
>> -------
>> Internal Server Error
>>
>> The server encountered an internal error or
misconfiguration and was
>> unable to complete your request.
>>
>> Please contact the server administrator,
webmasterutopia-design.net and
>> inform them of the time the error occurred, and
anything you might have
>> done that may have caused the error.
>>
>> More information about this error may be available
in the server error log.
>>
>> Additionally, a 404 Not Found error was encountered
while trying to use
>> an ErrorDocument to handle the request.
>>
>>     
>
> >
>
>   


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Django users" group.
To post to this group, send email to django-usersgooglegroups.com
To unsubscribe from this group, send email to
django-users-unsubscribegooglegroups.com
For more options, visit this group at htt
p://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Well, one thing that you could do is to execute an external
script
from your view. Then you'd immediatelly return from the
view, and the
script could mail them in the background. That is, of
course, if you
don't need any feedback to be displayed.

If you do need to do something like that, you'll need to
make your
view provide some output every once in a while, so your
webserver does
not time it out. However - that would cause your view to
return data
to the browser quite slow. If you want it more interactive,
you could
make the view generate output to a page with something
AJAX-y, instead
of a normal web page.
http://my.opera.com/WebApplications/blog/show.dml/438711
 - this idea
is quite nice, although I guess you don't need anything so
flashy in
your case.


On 30/03/07, Anderson Santos Silva <curaloucuragmail.com> wrote:
>
>     That's exactly what I thought when I dealed with
that, but I can't
> imagine a separated process running in the background
and return a
> response to the user that his "action" was
activated if every method
> waits to get back instead of a kind of thread
operation. (Yes, I came
> from windows programming  to Web, so my
knowledge is kind of
> limited with Python)
>
>      Any suggestion?
>
>
>
>
> Atilla wrote:
> > Hello,
> >
> > By the looks of it, mod_fcgi has a limit as to how
much time it could
> > wait for response from a script. Since you're
doing a heavy operation
> > that takes a while, i guess it times out.
> >
> > The point is  - if your executed action takes so
much time, you're
> > clearly having a wrong conception on when and how
you should be
> > executing it. It could be better off as an
external process (that
> > still uses django in its code) that is triggered
by certain actions in
> > your web-application, or scheduled by cron. In any
case - it is not
> > the job of your django view to handle bulk
transfers, it was not meant
> > to be doing that, even if the mod_fcgi didn't have
the timeout.
> >
> >
> > On 30/03/07, Anderson Santos Silva
<curaloucuragmail.com> wrote:
> >
> >> Hello, I have a process that sends an e-mail
to more then 5000 addresses
> >> (grouping it in 50 each time) and after a
while it gives me the error:
> >>
> >> -------
> >> Internal Server Error
> >>
> >> The server encountered an internal error or
misconfiguration and was
> >> unable to complete your request.
> >>
> >> Please contact the server administrator,
webmasterutopia-design.net and
> >> inform them of the time the error occurred,
and anything you might have
> >> done that may have caused the error.
> >>
> >> More information about this error may be
available in the server error log.
> >>
> >> Additionally, a 404 Not Found error was
encountered while trying to use
> >> an ErrorDocument to handle the request.
> >>
> >>
> >
> > >
> >
> >
>

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Django users" group.
To post to this group, send email to django-usersgooglegroups.com
To unsubscribe from this group, send email to
django-users-unsubscribegooglegroups.com
For more options, visit this group at htt
p://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---

  In my case, it's quite urgent (it must work today), so do
you have
any example to execute this external script? Is it simple
python
(*.py) files?
I was reading the link you sent, it's really interesting and
it could
be useful in a visual progress process, I will try it with
another
project.

   Thanks in advance,

4nderson



On 3/30/07, Atilla <theatillagmail.com> wrote:
>
> Well, one thing that you could do is to execute an
external script
> from your view. Then you'd immediatelly return from the
view, and the
> script could mail them in the background. That is, of
course, if you
> don't need any feedback to be displayed.
>
> If you do need to do something like that, you'll need
to make your
> view provide some output every once in a while, so your
webserver does
> not time it out. However - that would cause your view
to return data
> to the browser quite slow. If you want it more
interactive, you could
> make the view generate output to a page with something
AJAX-y, instead
> of a normal web page.
> http://my.opera.com/WebApplications/blog/show.dml/438711
 - this idea
> is quite nice, although I guess you don't need anything
so flashy in
> your case.
>
>
> On 30/03/07, Anderson Santos Silva <curaloucuragmail.com> wrote:
> >
> >     That's exactly what I thought when I dealed
with that, but I can't
> > imagine a separated process running in the
background and return a
> > response to the user that his "action"
was activated if every method
> > waits to get back instead of a kind of thread
operation. (Yes, I came
> > from windows programming  to Web, so my
knowledge is kind of
> > limited with Python)
> >
> >      Any suggestion?
> >
> >
> >
> >
> > Atilla wrote:
> > > Hello,
> > >
> > > By the looks of it, mod_fcgi has a limit as
to how much time it could
> > > wait for response from a script. Since you're
doing a heavy operation
> > > that takes a while, i guess it times out.
> > >
> > > The point is  - if your executed action takes
so much time, you're
> > > clearly having a wrong conception on when and
how you should be
> > > executing it. It could be better off as an
external process (that
> > > still uses django in its code) that is
triggered by certain actions in
> > > your web-application, or scheduled by cron.
In any case - it is not
> > > the job of your django view to handle bulk
transfers, it was not meant
> > > to be doing that, even if the mod_fcgi didn't
have the timeout.
> > >
> > >
> > > On 30/03/07, Anderson Santos Silva
<curaloucuragmail.com> wrote:
> > >
> > >> Hello, I have a process that sends an
e-mail to more then 5000 addresses
> > >> (grouping it in 50 each time) and after a
while it gives me the error:
> > >>
> > >> -------
> > >> Internal Server Error
> > >>
> > >> The server encountered an internal error
or misconfiguration and was
> > >> unable to complete your request.
> > >>
> > >> Please contact the server administrator,
webmasterutopia-design.net and
> > >> inform them of the time the error
occurred, and anything you might have
> > >> done that may have caused the error.
> > >>
> > >> More information about this error may be
available in the server error log.
> > >>
> > >> Additionally, a 404 Not Found error was
encountered while trying to use
> > >> an ErrorDocument to handle the request.
> > >>
> > >>
> > >
> > > >
> > >
> > >
> >
>
> >
>

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Django users" group.
To post to this group, send email to django-usersgooglegroups.com
To unsubscribe from this group, send email to
django-users-unsubscribegooglegroups.com
For more options, visit this group at htt
p://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Hello,

Sending email within a view seems to me quite a strange
process 
You should send your emails within a shell python (or
whatever)
process.

What I would do ...
I would within my view, write some kind of data to a
database or a
text file or whatever.
Then, have a cronjob that check that data (should I send
emails or
not ?)
Then if the answer is 'True' fire a script that sends
emails.

In this case your long process won't involve any
fcgi/webserver
process.

Of course you can use Django in shell python script. You
juste have to
wrtite some script with the flowing caneva :
------------------------------------------------------------
-----------------------
#!/usr/bin/env python

import sys
sys.path.append('/path/to/django/source')
sys.path.append('/path/to/projects')

import os

# assuming /path/to/your/projects/myproject/settings.py
os.environ['DJANGO_SETTINGS_MODULE'] = 'project.settings'

from project.models import whatever

django code
------------------------------------------------------------
-----------------------


Regards,
xav


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Django users" group.
To post to this group, send email to django-usersgooglegroups.com
To unsubscribe from this group, send email to
django-users-unsubscribegooglegroups.com
For more options, visit this group at htt
p://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---

I was looking for how to execute a script and I was just
thinking how
to use django code and here is your e-mail! Thank you!

I am just not sure about how to call the shell script.

should I use the os.system(command) or the os.popen ?  That
seems that
popen just return when I close the script and this is
exactly what I
want to avoid.

and considering that a file called sendmail.py with all the
script to
read from a database and send the e-mails will be in the
same path as
the view, how can I call it?

def startsending(request):
     os.system("sendmail.py")
     return HttpResponse('Your e-mails are being
send")

?

and I got an example that does a python script, on the end 
of the
script it does:


----------------------------------------------
[script code]

if __name__ == "__main__":
    while something:
            execute(command)

[end of the script]
----------------------

if I call it from a view using os.system, it still will be
the main so
I have to use this code?






Thanks for your help  =)




On 3/30/07, xgdlm <grangiergmail.com> wrote:
>
> Hello,
>
> Sending email within a view seems to me quite a strange
process 
> You should send your emails within a shell python (or
whatever)
> process.
>
> What I would do ...
> I would within my view, write some kind of data to a
database or a
> text file or whatever.
> Then, have a cronjob that check that data (should I
send emails or
> not ?)
> Then if the answer is 'True' fire a script that sends
emails.
>
> In this case your long process won't involve any
fcgi/webserver
> process.
>
> Of course you can use Django in shell python script.
You juste have to
> wrtite some script with the flowing caneva :
>
------------------------------------------------------------
-----------------------
> #!/usr/bin/env python
>
> import sys
> sys.path.append('/path/to/django/source')
> sys.path.append('/path/to/projects')
>
> import os
>
> # assuming
/path/to/your/projects/myproject/settings.py
> os.environ['DJANGO_SETTINGS_MODULE'] =
'project.settings'
>
> from project.models import whatever
>
> django code
>
------------------------------------------------------------
-----------------------
>
>
> Regards,
> xav
>
>
> >
>

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Django users" group.
To post to this group, send email to django-usersgooglegroups.com
To unsubscribe from this group, send email to
django-users-unsubscribegooglegroups.com
For more options, visit this group at htt
p://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---

> I am just not sure about how to call the shell script.

I would call the shell script withing a cronjob command :
python /path/to/sendmail.py

this script would do someting roughly like :
-----
if there_is_a_mailing_to_send :
 for mail in mailtosend :
 do
   send mail
-----

> read from a database and send the e-mails will be in
the same path as
> the view, how can I call it?

in the view I won't call any system command. I would just
write some
where (in a database or in a file) if there is a mailing to
send.

def startsending(request):
      ''' do someting here so cronjob would know there is a
mailign to
send '''
      there_is_a_mailing_to_send = True
      return HttpResponse('Your e-mails are being send
shortly")

xav


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Django users" group.
To post to this group, send email to django-usersgooglegroups.com
To unsubscribe from this group, send email to
django-users-unsubscribegooglegroups.com
For more options, visit this group at htt
p://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Thank you, again. Now I think I have enough information to
find my way  =D



On 3/30/07, xgdlm <grangiergmail.com> wrote:
>
> > I am just not sure about how to call the shell
script.
>
> I would call the shell script withing a cronjob command
:
> python /path/to/sendmail.py
>
> this script would do someting roughly like :
> -----
> if there_is_a_mailing_to_send :
>  for mail in mailtosend :
>  do
>    send mail
> -----
>
> > read from a database and send the e-mails will be
in the same path as
> > the view, how can I call it?
>
> in the view I won't call any system command. I would
just write some
> where (in a database or in a file) if there is a
mailing to send.
>
> def startsending(request):
>       ''' do someting here so cronjob would know there
is a mailign to
> send '''
>       there_is_a_mailing_to_send = True
>       return HttpResponse('Your e-mails are being send
shortly")
>
> xav
>
>
> >
>

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Django users" group.
To post to this group, send email to django-usersgooglegroups.com
To unsubscribe from this group, send email to
django-users-unsubscribegooglegroups.com
For more options, visit this group at htt
p://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---