Python pickle and web security.

Hello,

I posted this on my blog the other day about people using
pickle for
sessions, but got no response.  Do you guys think using
pickles for
sessions is an ok thing to do?




...........

Some python web frame works are using pickle to store
session data.
Pickle is a well known poor choice for secure systems.
However it
seems to be more widely known by those writing network
applications,
than those making web frameworks.

Is your web framework using pickle for sessions despite the
warnings
in the python documentation about it being insecure?

By using sessions with pickle people who can write to the
database
servers session table can execute code on the app server. Or
people
who can get data into the session file/memcache data store
can execute
data.

This might be an issue if the database server is run by
separate
people than the app server. Or if the session table is
compromised by
an sql injection attack elsewhere.

There are some more secure ways of storing pickled data.

Pickle is deemed to be untrustworthy for data. In that it is
not
certain that code can not be snuck into the data that will
be executed
by pickle. So if some data from user input is put into the
pickle,
then it is possible that code could be run.

There are some people who know more about how to exploit
pickle,
however the warning in the python documentation is this:

""Warning:
The pickle module is not intended to be secure against
erroneous or
maliciously constructed data. Never unpickle data received
from an
untrusted or unauthenticated source."""


Cerealizer might be an alternative option...
http://home.gna.org/oomadness/en/cerealizer/index.html

Or maybe these other two.
http://aspn.activestate.com/ASPN/Cookbook/Python/R
ecipe/415503
http://barnesc.blogspot.com/2006/01/ren
code-reduced-length-encodings.html
_______________________________________________
Web-SIG mailing list
Web-SIGpython.org
Web SIG: http://www.python.
org/sigs/web-sig
Unsubscribe: http://mail.python.org/mailman/options/web-sig/bo
nd%40yahoo.com

On Sep 15, 2006, at 4:29 AM, René Dudfield wrote:

> Hello,
>
> I posted this on my blog the other day about people
using pickle for
> sessions, but got no response.  Do you guys think using
pickles for
> sessions is an ok thing to do?

You don't want to accept pickles from an untrusted source,
which  
typically means you don't want to accept pickles over the
network.   
Even then, there are ways to use pickles securely. For
example, you  
can, if you know what you're doing, arrange to prevent
pickle from  
calling global objects or control specifically what global
objects  
are callable.

There is nothing wrong with using pickles to store data
internally.   
As long as the pickles are generated by the application,
there is no  
risk to the application reading them again, assuming that
they are  
stored where they can't be tampered with.

Saying pickle is inherently insecure is like saying Python
is  
inherently insecure.  You don't want to execute Python from
an  
untrusted source.  If someone can tamper with your Python
code, then  
you have a serious security problem as well.

Jim

--
Jim Fulton			mailto:jimzope.com		Python Powered!
CTO 				(540) 361-1714			http://www.python.org
Zope Corporation	http://www.zope.com		http://www.zope.org



_______________________________________________
Web-SIG mailing list
Web-SIGpython.org
Web SIG: http://www.python.
org/sigs/web-sig
Unsubscribe: http://mail.python.org/mailman/options/web-sig/bo
nd%40yahoo.com

On Sep 15, 2006, at 4:29 AM, René Dudfield wrote:

> Hello,
>
> I posted this on my blog the other day about people
using pickle for
> sessions, but got no response.  Do you guys think using
pickles for
> sessions is an ok thing to do?

You don't want to accept pickles from an untrusted source,
which  
typically means you don't want to accept pickles over the
network.   
Even then, there are ways to use pickles securely. For
example, you  
can, if you know what you're doing, arrange to prevent
pickle from  
calling global objects or control specifically what global
objects  
are callable.

There is nothing wrong with using pickles to store data
internally.   
As long as the pickles are generated by the application,
there is no  
risk to the application reading them again, assuming that
they are  
stored where they can't be tampered with.

Saying pickle is inherently insecure is like saying Python
is  
inherently insecure.  You don't want to execute Python from
an  
untrusted source.  If someone can tamper with your Python
code, then  
you have a serious security problem as well.

Jim

--
Jim Fulton			mailto:jimzope.com		Python Powered!
CTO 				(540) 361-1714			http://www.python.org
Zope Corporation	http://www.zope.com		http://www.zope.org



_______________________________________________
Web-SIG mailing list
Web-SIGpython.org
Web SIG: http://www.python.
org/sigs/web-sig
Unsubscribe: http://mail.python.org/mailman/options/web-sig/bo
nd%40yahoo.com

On Fri, 2006-09-15 at 18:29 +1000, René Dudfield wrote:
> Hello,
> 
> I posted this on my blog the other day about people
using pickle for
> sessions, but got no response.  Do you guys think using
pickles for
> sessions is an ok thing to do?

Either encrypt the pickle or have a seeded (md5) signature
so that you
can verify that the pickle has not been tampered.  I use
pickles
routinely, but with an md5 signature that combines a seed
and the
pickle.

Someone cannot generate a valid signature without also
knowing the seed.
I am paranoid enough so that I only pickle dictionaries and
then only
extract and verify my list of expected keys after
unpickling.  I can't
prove that's secure, but I am not losing sleep over it.  

Presumably someone who knew the seed could generate a valid
signature
*and* inject code into the pickle that got executed by the
unpickle
operation.

> 
> 
> 
> 
> ...........
> 
> Some python web frame works are using pickle to store
session data.
> Pickle is a well known poor choice for secure systems.
However it
> seems to be more widely known by those writing network
applications,
> than those making web frameworks.
> 
> Is your web framework using pickle for sessions despite
the warnings
> in the python documentation about it being insecure?
> 
> By using sessions with pickle people who can write to
the database
> servers session table can execute code on the app
server. Or people
> who can get data into the session file/memcache data
store can execute
> data.
> 
> This might be an issue if the database server is run by
separate
> people than the app server. Or if the session table is
compromised by
> an sql injection attack elsewhere.
> 
> There are some more secure ways of storing pickled
data.
> 
> Pickle is deemed to be untrustworthy for data. In that
it is not
> certain that code can not be snuck into the data that
will be executed
> by pickle. So if some data from user input is put into
the pickle,
> then it is possible that code could be run.
> 
> There are some people who know more about how to
exploit pickle,
> however the warning in the python documentation is
this:
> 
> ""Warning:
> The pickle module is not intended to be secure against
erroneous or
> maliciously constructed data. Never unpickle data
received from an
> untrusted or unauthenticated
source."""
> 
> 
> Cerealizer might be an alternative option...
> http://home.gna.org/oomadness/en/cerealizer/index.html
> 
> Or maybe these other two.
> http://aspn.activestate.com/ASPN/Cookbook/Python/R
ecipe/415503
> http://barnesc.blogspot.com/2006/01/ren
code-reduced-length-encodings.html
> _______________________________________________
> Web-SIG mailing list
> Web-SIGpython.org
> Web SIG: http://www.python.
org/sigs/web-sig
> Unsubscribe: http://mail.python.org/mailman/options/web-sig/
python%40venix.com
-- 
Lloyd Kvam
Venix Corp

_______________________________________________
Web-SIG mailing list
Web-SIGpython.org
Web SIG: http://www.python.
org/sigs/web-sig
Unsubscribe: http://mail.python.org/mailman/options/web-sig/bo
nd%40yahoo.com

On Fri, 2006-09-15 at 18:29 +1000, René Dudfield wrote:
> Hello,
> 
> I posted this on my blog the other day about people
using pickle for
> sessions, but got no response.  Do you guys think using
pickles for
> sessions is an ok thing to do?

Either encrypt the pickle or have a seeded (md5) signature
so that you
can verify that the pickle has not been tampered.  I use
pickles
routinely, but with an md5 signature that combines a seed
and the
pickle.

Someone cannot generate a valid signature without also
knowing the seed.
I am paranoid enough so that I only pickle dictionaries and
then only
extract and verify my list of expected keys after
unpickling.  I can't
prove that's secure, but I am not losing sleep over it.  

Presumably someone who knew the seed could generate a valid
signature
*and* inject code into the pickle that got executed by the
unpickle
operation.

> 
> 
> 
> 
> ...........
> 
> Some python web frame works are using pickle to store
session data.
> Pickle is a well known poor choice for secure systems.
However it
> seems to be more widely known by those writing network
applications,
> than those making web frameworks.
> 
> Is your web framework using pickle for sessions despite
the warnings
> in the python documentation about it being insecure?
> 
> By using sessions with pickle people who can write to
the database
> servers session table can execute code on the app
server. Or people
> who can get data into the session file/memcache data
store can execute
> data.
> 
> This might be an issue if the database server is run by
separate
> people than the app server. Or if the session table is
compromised by
> an sql injection attack elsewhere.
> 
> There are some more secure ways of storing pickled
data.
> 
> Pickle is deemed to be untrustworthy for data. In that
it is not
> certain that code can not be snuck into the data that
will be executed
> by pickle. So if some data from user input is put into
the pickle,
> then it is possible that code could be run.
> 
> There are some people who know more about how to
exploit pickle,
> however the warning in the python documentation is
this:
> 
> ""Warning:
> The pickle module is not intended to be secure against
erroneous or
> maliciously constructed data. Never unpickle data
received from an
> untrusted or unauthenticated
source."""
> 
> 
> Cerealizer might be an alternative option...
> http://home.gna.org/oomadness/en/cerealizer/index.html
> 
> Or maybe these other two.
> http://aspn.activestate.com/ASPN/Cookbook/Python/R
ecipe/415503
> http://barnesc.blogspot.com/2006/01/ren
code-reduced-length-encodings.html
> _______________________________________________
> Web-SIG mailing list
> Web-SIGpython.org
> Web SIG: http://www.python.
org/sigs/web-sig
> Unsubscribe: http://mail.python.org/mailman/options/web-sig/
python%40venix.com
-- 
Lloyd Kvam
Venix Corp

_______________________________________________
Web-SIG mailing list
Web-SIGpython.org
Web SIG: http://www.python.
org/sigs/web-sig
Unsubscribe: http://mail.python.org/mailman/options/web-sig/bo
nd%40yahoo.com

Hi,

I think my main point was about using pickle for sessions,
not just
using pickle by itself.

Unlike loading other data, code gets run when you load a
pickle.  It
is indeed like running python code.  So if you do not trust
where you
store your pickles to run python code, then that is a
problem.

If the unpickle or pickle code is not bug free, then you can
not trust
that unpickling a pickle will not allow data to be made
which can
trick the unpickle escaping code.

With the history of bugs with the unpickle code, I don't
think relying
on it is a good idea.

For a list of pickle bugs you can search the python bug
tracker.
There are over 70 bugs listed including the open, closed,
and deleted
bugs.  With 13 open bugs listed.

One of the bugs was closed because: 'Closing due to lack of
response.
cPickle is such a complex module, without a test case the
leak cannot
be found.'

I think that line says best about how much you should trust
the C
module pickle code that is 5753 lines long, and has not been
audited.

Will pickle *always* escape data you pass it correctly when
it encodes
it into a pickle?  Will unpickle *always* unescape parts of
the pickle
correctly?  If not then those pickles can run code.

The risk of using pickle does not seem to be worth the
convenience
that it gives.  With alternatives to pickle which do not
execute code
being available why not use them?

By using pickle for session data you allow people the
oportunity to
put data into the pickle.  For example say you store a given
GET
variable in the session.

Combining that you allow people with pickle-sessions to put
data into
the pickle, and the risk that pickle might not encode/decode
it
correctly is the problem I see.

However if allowing untrusted data to be placed into a
pickle is ok,
then this is not a problem.  That only leaves the problem of
allowing
the data store of your sessions to be able to execute code
where you
load sessions.

This means you allow execution of code from your data store
to your
session loading code.  Which means if you use a separate
database
machine(quite common), or if you use a separate memcache
server(not
unheard of) you allow these machines to execute code on the
session
using machine.

There's a reason why people use separate user accounts, and
separate
machines for doing different tasks.  That reason is to limit
what each
user or machine can do.  By using pickles for sessions those
benefits
are removed in some cases.

Cheers,

On 9/15/06, Jim Fulton <jimzope.com> wrote:
>
> On Sep 15, 2006, at 4:29 AM, René Dudfield wrote:
>
> > Hello,
> >
> > I posted this on my blog the other day about
people using pickle for
> > sessions, but got no response.  Do you guys think
using pickles for
> > sessions is an ok thing to do?
>
> You don't want to accept pickles from an untrusted
source, which
> typically means you don't want to accept pickles over
the network.
> Even then, there are ways to use pickles securely. For
example, you
> can, if you know what you're doing, arrange to prevent
pickle from
> calling global objects or control specifically what
global objects
> are callable.
>
> There is nothing wrong with using pickles to store data
internally.
> As long as the pickles are generated by the
application, there is no
> risk to the application reading them again, assuming
that they are
> stored where they can't be tampered with.
>
> Saying pickle is inherently insecure is like saying
Python is
> inherently insecure.  You don't want to execute Python
from an
> untrusted source.  If someone can tamper with your
Python code, then
> you have a serious security problem as well.
>
> Jim
>
_______________________________________________
Web-SIG mailing list
Web-SIGpython.org
Web SIG: http://www.python.
org/sigs/web-sig
Unsubscribe: http://mail.python.org/mailman/options/web-sig/bo
nd%40yahoo.com

That seems like a good way to stop the untrusted session
store from
being able to inject sessions in there.  That could at least
solve the
problem of using pickles from untrusted session stores.

Are you just using the basic python types?  eg dict, string,
list,
numbers etc?  If so, perhaps using another serialiser will
remove some
more risk if you cared.


On 9/15/06, Python <pythonvenix.com> wrote:
> On Fri, 2006-09-15 at 18:29 +1000, René Dudfield wrote:
> > Hello,
> >
> > I posted this on my blog the other day about
people using pickle for
> > sessions, but got no response.  Do you guys think
using pickles for
> > sessions is an ok thing to do?
>
> Either encrypt the pickle or have a seeded (md5)
signature so that you
> can verify that the pickle has not been tampered.  I
use pickles
> routinely, but with an md5 signature that combines a
seed and the
> pickle.
>
> Someone cannot generate a valid signature without also
knowing the seed.
> I am paranoid enough so that I only pickle dictionaries
and then only
> extract and verify my list of expected keys after
unpickling.  I can't
> prove that's secure, but I am not losing sleep over
it.
>
> Presumably someone who knew the seed could generate a
valid signature
> *and* inject code into the pickle that got executed by
the unpickle
> operation.
>
> >
> >
> >
> >
> > ...........
> >
> > Some python web frame works are using pickle to
store session data.
> > Pickle is a well known poor choice for secure
systems. However it
> > seems to be more widely known by those writing
network applications,
> > than those making web frameworks.
> >
> > Is your web framework using pickle for sessions
despite the warnings
> > in the python documentation about it being
insecure?
> >
> > By using sessions with pickle people who can write
to the database
> > servers session table can execute code on the app
server. Or people
> > who can get data into the session file/memcache
data store can execute
> > data.
> >
> > This might be an issue if the database server is
run by separate
> > people than the app server. Or if the session
table is compromised by
> > an sql injection attack elsewhere.
> >
> > There are some more secure ways of storing pickled
data.
> >
> > Pickle is deemed to be untrustworthy for data. In
that it is not
> > certain that code can not be snuck into the data
that will be executed
> > by pickle. So if some data from user input is put
into the pickle,
> > then it is possible that code could be run.
> >
> > There are some people who know more about how to
exploit pickle,
> > however the warning in the python documentation is
this:
> >
> > ""Warning:
> > The pickle module is not intended to be secure
against erroneous or
> > maliciously constructed data. Never unpickle data
received from an
> > untrusted or unauthenticated
source."""
> >
> >
> > Cerealizer might be an alternative option...
> > http://home.gna.org/oomadness/en/cerealizer/index.html
> >
> > Or maybe these other two.
> > http://aspn.activestate.com/ASPN/Cookbook/Python/R
ecipe/415503
> > http://barnesc.blogspot.com/2006/01/ren
code-reduced-length-encodings.html
> > _______________________________________________
> > Web-SIG mailing list
> > Web-SIGpython.org
> > Web SIG: http://www.python.
org/sigs/web-sig
> > Unsubscribe: http://mail.python.org/mailman/options/web-sig/
python%40venix.com
> --
> Lloyd Kvam
> Venix Corp
>
>
_______________________________________________
Web-SIG mailing list
Web-SIGpython.org
Web SIG: http://www.python.
org/sigs/web-sig
Unsubscribe: http://mail.python.org/mailman/options/web-sig/bo
nd%40yahoo.com

On Sat, 2006-09-16 at 12:23 +1000, René Dudfield wrote:
> That seems like a good way to stop the untrusted
session store from
> being able to inject sessions in there.  That could at
least solve the
> problem of using pickles from untrusted session stores.
> 
> Are you just using the basic python types?  eg dict,
string, list,
> numbers etc?  If so, perhaps using another serialiser
will remove some
> more risk if you cared.

Besides the basic types, date/time objects are often
included.

My use of md5 signatures was focused primarily on preventing
unwanted
data manipulation.  I would agree that outside data should
be acquired
in formats that are simpler than pickles.  I am pickling
data that has
been checked and accepted.

> 
> 
> On 9/15/06, Python <pythonvenix.com> wrote:
> > On Fri, 2006-09-15 at 18:29 +1000, René Dudfield
wrote:
> > > Hello,
> > >
> > > I posted this on my blog the other day about
people using pickle for
> > > sessions, but got no response.  Do you guys
think using pickles for
> > > sessions is an ok thing to do?
> >
> > Either encrypt the pickle or have a seeded (md5)
signature so that you
> > can verify that the pickle has not been tampered. 
I use pickles
> > routinely, but with an md5 signature that combines
a seed and the
> > pickle.
> >
> > Someone cannot generate a valid signature without
also knowing the seed.
> > I am paranoid enough so that I only pickle
dictionaries and then only
> > extract and verify my list of expected keys after
unpickling.  I can't
> > prove that's secure, but I am not losing sleep
over it.
> >
> > Presumably someone who knew the seed could
generate a valid signature
> > *and* inject code into the pickle that got
executed by the unpickle
> > operation.
> >
> > >
> > >
> > >
> > >
> > > ...........
> > >
> > > Some python web frame works are using pickle
to store session data.
> > > Pickle is a well known poor choice for secure
systems. However it
> > > seems to be more widely known by those
writing network applications,
> > > than those making web frameworks.
> > >
> > > Is your web framework using pickle for
sessions despite the warnings
> > > in the python documentation about it being
insecure?
> > >
> > > By using sessions with pickle people who can
write to the database
> > > servers session table can execute code on the
app server. Or people
> > > who can get data into the session
file/memcache data store can execute
> > > data.
> > >
> > > This might be an issue if the database server
is run by separate
> > > people than the app server. Or if the session
table is compromised by
> > > an sql injection attack elsewhere.
> > >
> > > There are some more secure ways of storing
pickled data.
> > >
> > > Pickle is deemed to be untrustworthy for
data. In that it is not
> > > certain that code can not be snuck into the
data that will be executed
> > > by pickle. So if some data from user input is
put into the pickle,
> > > then it is possible that code could be run.
> > >
> > > There are some people who know more about how
to exploit pickle,
> > > however the warning in the python
documentation is this:
> > >
> > > ""Warning:
> > > The pickle module is not intended to be
secure against erroneous or
> > > maliciously constructed data. Never unpickle
data received from an
> > > untrusted or unauthenticated
source."""
> > >
> > >
> > > Cerealizer might be an alternative option...
> > > http://home.gna.org/oomadness/en/cerealizer/index.html
> > >
> > > Or maybe these other two.
> > > http://aspn.activestate.com/ASPN/Cookbook/Python/R
ecipe/415503
> > > http://barnesc.blogspot.com/2006/01/ren
code-reduced-length-encodings.html
> > >
_______________________________________________
> > > Web-SIG mailing list
> > > Web-SIGpython.org
> > > Web SIG: http://www.python.
org/sigs/web-sig
> > > Unsubscribe: http://mail.python.org/mailman/options/web-sig/
python%40venix.com
> > --
> > Lloyd Kvam
> > Venix Corp
> >
> >
> _______________________________________________
> Web-SIG mailing list
> Web-SIGpython.org
> Web SIG: http://www.python.
org/sigs/web-sig
> Unsubscribe: http://mail.python.org/mailman/options/web-sig/
python%40venix.com
-- 
Lloyd Kvam
Venix Corp

_______________________________________________
Web-SIG mailing list
Web-SIGpython.org
Web SIG: http://www.python.
org/sigs/web-sig
Unsubscribe: http://mail.python.org/mailman/options/web-sig/bo
nd%40yahoo.com

On Sep 15, 2006, at 7:23 PM, René Dudfield wrote:

> That seems like a good way to stop the untrusted
session store from
> being able to inject sessions in there.  That could at
least solve the
> problem of using pickles from untrusted session stores.
>
> Are you just using the basic python types?  eg dict,
string, list,
> numbers etc?  If so, perhaps using another serialiser
will remove some
> more risk if you cared.

Why do you assume the session store is untrusted? If someone
can hack 
into my database, they can typically hack into my web
application so 
its pretty weird to consider the backend session store to be

"untrusted". I think this is why using pickle
for sessions is pretty 
harmless as you're the one writing to them, not the user.

While I can imagine a few situations where an untrusted
session store 
might come into play, I'd generally imagine that the vast
majority of 
the time one does trust their session storage as much as
they trust 
that their application can't have its source code modified.

Cheers,
Ben

_______________________________________________
Web-SIG mailing list
Web-SIGpython.org
Web SIG: http://www.python.
org/sigs/web-sig
Unsubscribe: http://mail.python.org/mailman/options/web-sig/bo
nd%40yahoo.com

On Mon, 2006-09-18 at 10:27 -0700, Ben Bangert wrote:
> Why do you assume the session store is untrusted? If
someone can hack 
> into my database, they can typically hack into my web
application so 
> its pretty weird to consider the backend session store
to be 
> "untrusted".

You are assuming that the pickle is stored in a secure
database.  If the
pickle is in a cookie or some other client side storage,
then it is
definitely not to be trusted.

-- 
Lloyd Kvam
Venix Corp

_______________________________________________
Web-SIG mailing list
Web-SIGpython.org
Web SIG: http://www.python.
org/sigs/web-sig
Unsubscribe: http://mail.python.org/mailman/options/web-sig/bo
nd%40yahoo.com