inst_persistent_id

Recently, I reviewed the documentation source for pickle and
came  
across the following comment:

    BAW: Both pickle and cPickle support something called  
inst_persistent_id()
    which appears to give unknown types a second shot at
producing a  
persistent
    id.  Since Jim Fulton can't remember why it was added or
what it's  
for, I'm
    leaving it undocumented.

I couldn't remember this and decided to dig into it and
realized that  
this was a very useful experimental feature I added way back
when  
cPickle was in its infancy.  This is a fairly useful
optimization that  
I never got around to evaluating.  (Yeah, I know.) It is a
hook, like  
the persistent_id hook that is called with objects to
determine if  
they should be pickled by persistent reference.  Unlike
persistent_id,  
it isn't called for built-in types (really types for which
pickle has  
specific handlers), like strings, numbers, lists, tuples and
so on. It  
is only called for "instances" (types for which
pickle doesn't have  
specific handlers).  This vastly reduces the number of calls
to the  
hook.  Some tests with ZODB indicated significant
improvements in  
pickling speed when a hook is used.

If there are no objections, I'll update the Python 2
documentation to  
describe this and add a test.  The comment above suggests
that this  
hook is in pickle and cPickle.  It is in cPickle, but was
removed from  
pickle.  I propose to add it back to pickle -- mainly for
consistency  
with cPickle.  I'll need to double check how this interacts
with the  
type dispatching in pickle protocol 2.

Any objections?

Jim

--
Jim Fulton
Zope Corporation


_______________________________________________
Python-Dev mailing list
Python-Devpython.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com

On Jan 12, 2008 12:25 PM, Jim Fulton <jimzope.com> wrote:
> If there are no objections, I'll update the Python 2
documentation to
> describe this and add a test.  The comment above
suggests that this
> hook is in pickle and cPickle.  It is in cPickle, but
was removed from
> pickle.  I propose to add it back to pickle -- mainly
for consistency
> with cPickle.  I'll need to double check how this
interacts with the
> type dispatching in pickle protocol 2.
>
> Any objections?
>

Well, in Python 3K, inst_persistent_id() won't be usable,
since
PyInstance_Type was removed. Adding (actually supporting)
this feature
in Python 2.x will make it slightly harder to port code. So,
I think
it would probably best to leave it as it is right now --
i.e.,
undocumented and unsupported.

By the way, you might be interested to look at my work on
pickle [1]
for Python 3K. As part of last year Summer of Code, I
removed the
differences between the Python and C implementation of
pickle, and
thus allowing the C version to be transparently used,
instead of the
Python one, when it is available. Currently, I am polishing
the code
for inclusion into the main branch. I also started to work
on the next
version of the pickle protocol, that will make it more
suitable for
Python 3K. If you are interested to help out, just send me
an email
and I will explain you what needs to be done.


-- Alexandre

.. [1] http://peadrop.com/alex-py3k/?file/91639e
5487dc/Modules/_picklemodule.c
_______________________________________________
Python-Dev mailing list
Python-Devpython.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com

Hi,

On Sat, Jan 12, 2008 at 07:33:38PM -0500, Alexandre
Vassalotti wrote:
> Well, in Python 3K, inst_persistent_id() won't be
usable, since
> PyInstance_Type was removed.

Looking at the code, inst_persistent_id() is just a
confusing name.  It
has got nothing to do with PyInstance_Type; it's called for
any object
type that cPickle.c doesn't know how to handle.  In fact, it
seems that
cPickle.c never calls inst_persistent_id() for objects of
type
PyInstance_Type...


A bientot,

Armin.
_______________________________________________
Python-Dev mailing list
Python-Devpython.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com

Oh, you are right. I thought that save_inst() used
inst_persistent_id,
but that isn't the case. Now, I have checked more thoroughly
and found
the relevant piece of code:

	if (!pers_save && self->inst_pers_func) {
		if ((tmp = save_pers(self, args, self->inst_pers_func))
!= 0) {
			res = tmp;
			goto finally;
		}
	}

which is indeed called only when the object is not
"supported" by pickle.

I guess my original argument doesn't hold anymore, thus I
don't have
anything against supporting this feature officially.

Thanks for correcting me!
-- Alexandre


On Jan 14, 2008 12:59 PM, Armin Rigo <arigotunes.org> wrote:
> Hi,
>
> On Sat, Jan 12, 2008 at 07:33:38PM -0500, Alexandre
Vassalotti wrote:
> > Well, in Python 3K, inst_persistent_id() won't be
usable, since
> > PyInstance_Type was removed.
>
> Looking at the code, inst_persistent_id() is just a
confusing name.  It
> has got nothing to do with PyInstance_Type; it's called
for any object
> type that cPickle.c doesn't know how to handle.  In
fact, it seems that
> cPickle.c never calls inst_persistent_id() for objects
of type
> PyInstance_Type...
>
>
> A bientot,
>
> Armin.
>
_______________________________________________
Python-Dev mailing list
Python-Devpython.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com

I took Python-3000 out of the cc list as I originally just
wanted to  
make them aware of this issue.

On Jan 14, 2008, at 12:59 PM, Armin Rigo wrote:

> Hi,
>
> On Sat, Jan 12, 2008 at 07:33:38PM -0500, Alexandre
Vassalotti wrote:
>> Well, in Python 3K, inst_persistent_id() won't be
usable, since
>> PyInstance_Type was removed.
>
> Looking at the code, inst_persistent_id() is just a
confusing name.   
> It
> has got nothing to do with PyInstance_Type; it's called
for any object
> type that cPickle.c doesn't know how to handle.

It has to do with instances in a more general sense.

>  In fact, it seems that
> cPickle.c never calls inst_persistent_id() for objects
of type
> PyInstance_Type...


Hm, good point.

At the time, all I cared about were ExtensionClass
instances, which  
were akin to modern-day new-style instances.

It doesn't make sense to not call this function for classic
instances.

This optimization is fairly useful.  I propose to update the
logic to  
call this function for classic instances too.  I'm also open
to  
renaming it if that would make people happier. 

Thoughts?

Jim

--
Jim Fulton
Zope Corporation


_______________________________________________
Python-Dev mailing list
Python-Devpython.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com