Using LimitRequestBody to Limit File Uploads

Hello,

I have recently begun testing LimitRequestBody to limit file
uploads.
It works but there are some issues. Whenever the uploaded
file hits
the limit, I get "Connection reset by peer".
Ideally I would like to
be able to redirect the user to an error page but it seems
that Django
tries to run but it hits an exception:

Traceback (most recent call last):

 File
"/opt/ActivePython/lib/python2.4/site-packages/django/c
ore/handlers/base.py",
line 77, in get_response
   response = callback(request, *callback_args,
**callback_kwargs)

 File
"/opt/ActivePython/lib/python2.4/site-packages/django/d
b/transaction.py",
line 194, in _commit_on_success
   res = func(*args, **kw)

 File
"/home/preownedcar/djangoapps/preownedcar/CoreApp/views
.py",
line 1108, in offer_post
   new_data = request.POST.copy()

 File
"/opt/ActivePython/lib/python2.4/site-packages/django/c
ore/handlers/modpython.py",
line 69, in _get_post
   self._load_post_and_files()

 File
"/opt/ActivePython/lib/python2.4/site-packages/django/c
ore/handlers/modpython.py",
line 50, in _load_post_and_files
   self._post, self._files =
http.parse_file_upload(self._req.headers_in,
self.raw_post_data)

 File
"/opt/ActivePython/lib/python2.4/site-packages/django/c
ore/handlers/modpython.py",
line 119, in _get_raw_post_data
   self._raw_post_data = self._req.read()

SystemError: Objects/stringobject.c:3518: bad argument to
internal function

I get that traceback emailed to me every time an uploaded
file hits
the limit set by LimitRequestBody.

I am using Django 0.96 and this is on the production
server.

Someone posted something similar on this list last December
but the
last question by the OP was never really addressed:

http://groups.
google.com/group/django-users/browse_thread/thread/162a66bf9
2cc5c37/b33c637c502b669e

Has any progress been made with regards to handling 413 from
within Django?
-- 
_nimrod_a_abing_

http://abing.gotdns.com/

http://www.preownedcar.co
m/

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Django users" group.
To post to this group, send email to django-usersgooglegroups.com
To unsubscribe from this group, send email to
django-users-unsubscribegooglegroups.com
For more options, visit this group at htt
p://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Jul 29, 11:20 am, "Nimrod A. Abing"
<nimrod.ab...gmail.com> wrote:
> Hello,
>
> I have recently begun testing LimitRequestBody to limit
file uploads.
> It works but there are some issues. Whenever the
uploaded file hits
> the limit, I get "Connection reset by peer".
Ideally I would like to
> be able to redirect the user to an error page but it
seems that Django
> tries to run but it hits an exception:
>
> Traceback (most recent call last):
>
>  File
"/opt/ActivePython/lib/python2.4/site-packages/django/c
ore/handlers/base.py",
> line 77, in get_response
>    response = callback(request, *callback_args,
**callback_kwargs)
>
>  File
"/opt/ActivePython/lib/python2.4/site-packages/django/d
b/transaction.py",
> line 194, in _commit_on_success
>    res = func(*args, **kw)
>
>  File
"/home/preownedcar/djangoapps/preownedcar/CoreApp/views
.py",
> line 1108, in offer_post
>    new_data = request.POST.copy()
>
>  File
"/opt/ActivePython/lib/python2.4/site-packages/django/c
ore/handlers/modpython.py",
> line 69, in _get_post
>    self._load_post_and_files()
>
>  File
"/opt/ActivePython/lib/python2.4/site-packages/django/c
ore/handlers/modpython.py",
> line 50, in _load_post_and_files
>    self._post, self._files =
> http.parse_file_upload(self._req.headers_in,
self.raw_post_data)
>
>  File
"/opt/ActivePython/lib/python2.4/site-packages/django/c
ore/handlers/modpython.py",
> line 119, in _get_raw_post_data
>    self._raw_post_data = self._req.read()
>
> SystemError: Objects/stringobject.c:3518: bad argument
to internal function

This error message possibly relates to this bug in
mod_python.

  h
ttps://issues.apache.org/jira/browse/MODPYTHON-234

We hadn't been able to come up with a way of reproducing the
problem
on demand so hadn't yet been able to investigate. Where
people had
seen this exception it was very rare and very random.

I'll link this discussion to that issue as it may help
finding
underlying problem in mod_python.

> I get that traceback emailed to me every time an
uploaded file hits
> the limit set by LimitRequestBody.
>
> I am using Django 0.96 and this is on the production
server.
>
> Someone posted something similar on this list last
December but the
> last question by the OP was never really addressed:
>
> http://groups.google.com/group/django-us
ers/browse_thread/thread/162a...
>
> Has any progress been made with regards to handling 413
from within Django?

FWIW, how mod_python handle the Apache LimitRequestBody
directive is
wrong. I have it one of my TODO lists to log a bug against
mod_python
for it.

The problem is that mod_python doesn't check whether
LimitRequestBody
would be triggered before actually calling the mod_python
handler.
Thus, that the post data exceeds the limit is only found
when
req.read() of mod_python request object is called and a
Python
exception is generated because of an unknown read error.
Well at least
it normally is an unknown read error. In your case you seem
to be
triggered this other Python bug and getting that error
instead.

Now, because a read error of some sort is generated,
mod_python
handlers would normally see it as an unexpected exception
and it would
propagate back and result in a 500 Internal Server Error
page rather
than a 413 error back to the client. Whether Django does
anything
different or still sends back a 500 error page I don't
know.

In comparison, in mod_wsgi where I uncovered this problem
with
LimitRequestBody, the code checks to see if LimitRequestBody
would be
triggered before calling the WSGI application. If triggered
the code
immediately returns without calling the application,
returning the 413
error. If an ErrorDocument for 413 is defined that would
then be
invoked, else the standard Apache 413 error page would be
returned.

In summary, mod_python LimitRequestBody handling is broken.
Even when
fixed, it would result in it being detected before Django is
even
called and thus only custom error page can be one setup
using
ErrorDocument directive of Apache. This ErrorDocument
directive would
reference a Django URL if desired.

Only other choice when using mod_python is not to use
Apache
LimitRequestBody directive and for your actual Django URL
handler to
check Content-Length of request itself and generate an
appropriate 413
error page. You wouldn't be able to apply it in one go for
whole
Django application.

If not bound to mod_python because of need to use it for
custom
authentication/authorisation handlers connected to Django,
you might
also try using mod_wsgi instead, where the LimitRequestBody
directive
isn't a problem.

Hope this helps to explain the issue.

Graham


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Django users" group.
To post to this group, send email to django-usersgooglegroups.com
To unsubscribe from this group, send email to
django-users-unsubscribegooglegroups.com
For more options, visit this group at htt
p://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Jul 29, 8:00 pm, Graham Dumpleton
<Graham.Dumple...gmail.com>
wrote:
> On Jul 29, 11:20 am, "Nimrod A. Abing"
<nimrod.ab...gmail.com> wrote:
>
>
>
> > Hello,
>
> > I have recently begun testing LimitRequestBody to
limit file uploads.
> > It works but there are some issues. Whenever the
uploaded file hits
> > the limit, I get "Connection reset by
peer". Ideally I would like to
> > be able to redirect the user to an error page but
it seems that Django
> > tries to run but it hits an exception:
>
> > Traceback (most recent call last):
>
> >  File
"/opt/ActivePython/lib/python2.4/site-packages/django/c
ore/handlers/base.py",
> > line 77, in get_response
> >    response = callback(request, *callback_args,
**callback_kwargs)
>
> >  File
"/opt/ActivePython/lib/python2.4/site-packages/django/d
b/transaction.py",
> > line 194, in _commit_on_success
> >    res = func(*args, **kw)
>
> >  File
"/home/preownedcar/djangoapps/preownedcar/CoreApp/views
.py",
> > line 1108, in offer_post
> >    new_data = request.POST.copy()
>
> >  File
"/opt/ActivePython/lib/python2.4/site-packages/django/c
ore/handlers/modpython.py",
> > line 69, in _get_post
> >    self._load_post_and_files()
>
> >  File
"/opt/ActivePython/lib/python2.4/site-packages/django/c
ore/handlers/modpython.py",
> > line 50, in _load_post_and_files
> >    self._post, self._files =
> > http.parse_file_upload(self._req.headers_in,
self.raw_post_data)
>
> >  File
"/opt/ActivePython/lib/python2.4/site-packages/django/c
ore/handlers/modpython.py",
> > line 119, in _get_raw_post_data
> >    self._raw_post_data = self._req.read()
>
> > SystemError: Objects/stringobject.c:3518: bad
argument to internal function
>
> This error message possibly relates to this bug in
mod_python.
>
>  h
ttps://issues.apache.org/jira/browse/MODPYTHON-234
>
> We hadn't been able to come up with a way of
reproducing the problem
> on demand so hadn't yet been able to investigate. Where
people had
> seen this exception it was very rare and very random.
>
> I'll link this discussion to that issue as it may help
finding
> underlying problem in mod_python.
>
> > I get that traceback emailed to me every time an
uploaded file hits
> > the limit set by LimitRequestBody.
>
> > I am using Django 0.96 and this is on the
production server.
>
> > Someone posted something similar on this list last
December but the
> > last question by the OP was never really
addressed:
>
> >http://groups.google.com/group/django-us
ers/browse_thread/thread/162a...
>
> > Has any progress been made with regards to
handling 413 from within Django?
>
> FWIW, how mod_python handle the Apache LimitRequestBody
directive is
> wrong. I have it one of my TODO lists to log a bug
against mod_python
> for it.
>
> The problem is that mod_python doesn't check whether
LimitRequestBody
> would be triggered before actually calling the
mod_python handler.
> Thus, that the post data exceeds the limit is only
found when
> req.read() of mod_python request object is called and a
Python
> exception is generated because of an unknown read
error. Well at least
> it normally is an unknown read error. In your case you
seem to be
> triggered this other Python bug and getting that error
instead.
>
> Now, because a read error of some sort is generated,
mod_python
> handlers would normally see it as an unexpected
exception and it would
> propagate back and result in a 500 Internal Server
Error page rather
> than a 413 error back to the client. Whether Django
does anything
> different or still sends back a 500 error page I don't
know.
>
> In comparison, in mod_wsgi where I uncovered this
problem with
> LimitRequestBody, the code checks to see if
LimitRequestBody would be
> triggered before calling the WSGI application. If
triggered the code
> immediately returns without calling the application,
returning the 413
> error. If an ErrorDocument for 413 is defined that
would then be
> invoked, else the standard Apache 413 error page would
be returned.
>
> In summary, mod_python LimitRequestBody handling is
broken. Even when
> fixed, it would result in it being detected before
Django is even
> called and thus only custom error page can be one setup
using
> ErrorDocument directive of Apache. This ErrorDocument
directive would
> reference a Django URL if desired.
>
> Only other choice when using mod_python is not to use
Apache
> LimitRequestBody directive and for your actual Django
URL handler to
> check Content-Length of request itself and generate an
appropriate 413
> error page. You wouldn't be able to apply it in one go
for whole
> Django application.
>
> If not bound to mod_python because of need to use it
for custom
> authentication/authorisation handlers connected to
Django, you might
> also try using mod_wsgi instead, where the
LimitRequestBody directive
> isn't a problem.
>
> Hope this helps to explain the issue.

And here is the mod_python issue I have created for the
LimitRequestBody issue.

  h
ttps://issues.apache.org/jira/browse/MODPYTHON-240

I can now mark doing this off my TODO list. 

Graham


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Django users" group.
To post to this group, send email to django-usersgooglegroups.com
To unsubscribe from this group, send email to
django-users-unsubscribegooglegroups.com
For more options, visit this group at htt
p://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On 7/29/07, Graham Dumpleton <Graham.Dumpletongmail.com> wrote:
> We hadn't been able to come up with a way of
reproducing the problem
> on demand so hadn't yet been able to investigate. Where
people had
> seen this exception it was very rare and very random.

I think the following conditions trigger the exception:

1. Set LimitRequestBody to an arbitrary limit, in our case
15728640
2. Upload a file larger than LimitRequestBody using POST to
a Django
app. In the test case that caused the error, the file size
is 24952238
(based on our logs).

The case above is for only one file of that specific size
for that
specific limit.

If I find the time I will set up a few test cases for this
error and
see if I can trigger it on demand.

> I'll link this discussion to that issue as it may help
finding
> underlying problem in mod_python.
>
> > I get that traceback emailed to me every time an
uploaded file hits
> > the limit set by LimitRequestBody.
> >
> > I am using Django 0.96 and this is on the
production server.
> >
> > Someone posted something similar on this list last
December but the
> > last question by the OP was never really
addressed:
> >
> > http://groups.google.com/group/django-us
ers/browse_thread/thread/162a...
> >
> > Has any progress been made with regards to
handling 413 from within Django?
>
> FWIW, how mod_python handle the Apache LimitRequestBody
directive is
> wrong. I have it one of my TODO lists to log a bug
against mod_python
> for it.
>
> The problem is that mod_python doesn't check whether
LimitRequestBody
> would be triggered before actually calling the
mod_python handler.
> Thus, that the post data exceeds the limit is only
found when
> req.read() of mod_python request object is called and a
Python
> exception is generated because of an unknown read
error. Well at least
> it normally is an unknown read error. In your case you
seem to be
> triggered this other Python bug and getting that error
instead.
>
> Now, because a read error of some sort is generated,
mod_python
> handlers would normally see it as an unexpected
exception and it would
> propagate back and result in a 500 Internal Server
Error page rather
> than a 413 error back to the client. Whether Django
does anything
> different or still sends back a 500 error page I don't
know.

The user reported that he just gets "Connection reset
by peer."

> In comparison, in mod_wsgi where I uncovered this
problem with
> LimitRequestBody, the code checks to see if
LimitRequestBody would be
> triggered before calling the WSGI application. If
triggered the code
> immediately returns without calling the application,
returning the 413
> error. If an ErrorDocument for 413 is defined that
would then be
> invoked, else the standard Apache 413 error page would
be returned.
>
> In summary, mod_python LimitRequestBody handling is
broken. Even when
> fixed, it would result in it being detected before
Django is even
> called and thus only custom error page can be one setup
using
> ErrorDocument directive of Apache. This ErrorDocument
directive would
> reference a Django URL if desired.

I guess that would be an acceptable fall-back behavior.

> Only other choice when using mod_python is not to use
Apache
> LimitRequestBody directive and for your actual Django
URL handler to
> check Content-Length of request itself and generate an
appropriate 413
> error page. You wouldn't be able to apply it in one go
for whole
> Django application.

One reason we turned to using LimitRequestBody is to prevent
DoS
attacks on our server. IIRC, Django keeps POST uploaded
files in
memory. I don't know if this is still the case in SVN trunk,
so please
correct me if I am dead wrong 

Deferring the actual size check to a Django view would be a
bit late
(and useless for our intended purpose) because by that time,
Django
and mod_python would have handled the request already.

> If not bound to mod_python because of need to use it
for custom
> authentication/authorisation handlers connected to
Django, you might
> also try using mod_wsgi instead, where the
LimitRequestBody directive
> isn't a problem.

I would love to try that on my development server but I
can't seem to
find documentation on how to do it:

http://code.djangoproject.com/wiki/ServerArrangements

Can someone point me in the right direction?

> Hope this helps to explain the issue.

Very informative explanation. Thank you 
-- 
_nimrod_a_abing_

http://abing.gotdns.com/

http://www.preownedcar.co
m/

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Django users" group.
To post to this group, send email to django-usersgooglegroups.com
To unsubscribe from this group, send email to
django-users-unsubscribegooglegroups.com
For more options, visit this group at htt
p://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Jul 29, 8:35 pm, "Nimrod A. Abing"
<nimrod.ab...gmail.com> wrote:
> On 7/29/07, Graham Dumpleton <Graham.Dumple...gmail.com> wrote:
> > If not bound to mod_python because of need to use
it for custom
> > authentication/authorisation handlers connected to
Django, you might
> > also try using mod_wsgi instead, where the
LimitRequestBody directive
> > isn't a problem.
>
> I would love to try that on my development server but I
can't seem to
> find documentation on how to do it:
>
> http://code.djangoproject.com/wiki/ServerArrangements
>
> Can someone point me in the right direction?

Available from:

  http://www.modwsgi.org

Django specific integration guide at:

  http://code.google.com/p/modwsgi/wiki/IntegrationWith
Django

Graham


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Django users" group.
To post to this group, send email to django-usersgooglegroups.com
To unsubscribe from this group, send email to
django-users-unsubscribegooglegroups.com
For more options, visit this group at htt
p://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On 7/29/07, Graham Dumpleton <Graham.Dumpletongmail.com> wrote:
> > I would love to try that on my development server
but I can't seem to
> > find documentation on how to do it:
> >
> > http://code.djangoproject.com/wiki/ServerArrangements
> >
> > Can someone point me in the right direction?
>
> Available from:
>
>   http://www.modwsgi.org
>
> Django specific integration guide at:
>
>   http://code.google.com/p/modwsgi/wiki/IntegrationWith
Django

I'm looking at the Django integration guide but I am
confused as to
what I do with the following:

> Using this function, a script file for a Django
application which is compatible with mod_wsgi would be
constructed as follows:
>
> import os, sys
> sys.path.append('/usr/local/django')
> os.environ['DJANGO_SETTINGS_MODULE'] =
'mysite.settings'
>
> import django.core.handlers.wsgi
>
> application = django.core.handlers.wsgi.WSGIHandler()

Where do I put that? views.py?

Thanks in advance.
-- 
_nimrod_a_abing_

http://abing.gotdns.com/

http://www.preownedcar.co
m/

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Django users" group.
To post to this group, send email to django-usersgooglegroups.com
To unsubscribe from this group, send email to
django-users-unsubscribegooglegroups.com
For more options, visit this group at htt
p://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Jul 30, 12:26 am, "Nimrod A. Abing"
<nimrod.ab...gmail.com> wrote:
> On 7/29/07, Graham Dumpleton <Graham.Dumple...gmail.com> wrote:
>
> > > I would love to try that on my development
server but I can't seem to
> > > find documentation on how to do it:
>
> > >http://code.djangoproject.com/wiki/ServerArrangements
>
> > > Can someone point me in the right direction?
>
> > Available from:
>
> >  http://www.modwsgi.org
>
> > Django specific integration guide at:
>
> >  http://code.google.com/p/modwsgi/wiki/IntegrationWith
Django
>
> I'm looking at the Django integration guide but I am
confused as to
> what I do with the following:
>
> > Using this function, a script file for a Django
application which is compatible with mod_wsgi would be
constructed as follows:
>
> > import os, sys
> > sys.path.append('/usr/local/django')
> > os.environ['DJANGO_SETTINGS_MODULE'] =
'mysite.settings'
>
> > import django.core.handlers.wsgi
>
> > application =
django.core.handlers.wsgi.WSGIHandler()
>
> Where do I put that? views.py?

The documentation says:

"""Similarly, an 'apache' subdirectory should
have been created within
the package and the script file stored there under the name
'django.wsgi'."""

Rather than trying to launch straight into setting up
Django, you
might want to get a simple WSGI hello world application
going first.
For more basic instructions read:

  http://code.google.com/p/modwsgi/wiki/Configuration
Guidelines

elsewhere on the same site.

Graham


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Django users" group.
To post to this group, send email to django-usersgooglegroups.com
To unsubscribe from this group, send email to
django-users-unsubscribegooglegroups.com
For more options, visit this group at htt
p://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On 7/30/07, Graham Dumpleton <Graham.Dumpletongmail.com> wrote:
> The documentation says:
>
> """Similarly, an 'apache' subdirectory
should have been created within
> the package and the script file stored there under the
name
> 'django.wsgi'."""
>
> Rather than trying to launch straight into setting up
Django, you
> might want to get a simple WSGI hello world application
going first.
> For more basic instructions read:
>
>   http://code.google.com/p/modwsgi/wiki/Configuration
Guidelines
>
> elsewhere on the same site.

Ok, thanks again. After going through postings on modswgi
Google Group
as well as your blog, I'm sold 

I'm looking at the docs and preparing for eventual migration
to mod_wsgi now...
-- 
_nimrod_a_abing_

http://abing.gotdns.com/

http://www.preownedcar.co
m/

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "Django users" group.
To post to this group, send email to django-usersgooglegroups.com
To unsubscribe from this group, send email to
django-users-unsubscribegooglegroups.com
For more options, visit this group at htt
p://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---