Cloning threading.py using proccesses

Fredrik Lundh <fredrikpythonware.com> wrote:
> Josiah Carlson wrote:
> 
> > It would basically be something along the lines of
cPickle, but would
> > only support the basic types of: int, long, float,
str, unicode, tuple,
> > list, dictionary.
> 
> if you're aware of a way to do that faster than the
current marshal 
> implementation, maybe you could work on speeding up
marshal instead?

The current implementation uses a fixed resize semantic
(1024 bytes at a
time) that makes large marshal operations slow.  If we were
to switch to
a list resize-like or cStringIO semantic (overallocate by
~size>>3,
or at least double, respectively), it would likely increase
the speed
for large resize operations. (see the w_more definition) 
This should
make it significantly faster than cPickle in basically all
cases.

w_object uses a giant if/else if block to handle all of the
possible
cases, both for identity checks against None, False, True,
etc., as well
as with the various Py*_Check().  This is necessary due to
marshal
supporting subclasses (the Py*_Check() calls) and the
dynamic layout of
memory during Python startup.  The identity checks may be
able to be
replaced with a small array-like thing if we were to
statically allocate
them from a single array to guarantee that their addresses
are a fixed
distance apart...

char base_objects[320];
PyObject* IDENTITY[8];
int cases[8];

/* 
64 bytes per object is overkill, and we may want to allocate
enough room
for 15 objects, to make sure that IDENTITY[0] = NULL;
*/
p = 0
for obj_init in objs_to_init:
    init_object(base_objects+p, obj_init)
    x = ((base_objects+p)>>6)&7
    IDENTITY[x] = (PyObject*)(base_objects+p)
    cases[x] = p//64
    p += 64

Then we could use the following in w_object...

x = (v>>6)&7
if v == IDENTITY[x] {
    switch (cases[x]) {
    case 0: /* should be null */
    ...
    case 1: /* based on the order of objs_to_init */
    }
}


The Py*_Check() stuff isn't so amenable to potential
speedup, but in a
custom no-subclasses only base types version, we ccould use
a variant of
the above mechanism to look directly at types, then use a
second
switch/case statement, which should be significantly faster
than the
if/else if tests that it currently uses.  An identity check,
then a fast
type check, otherwise fail.

 - Josiah

_______________________________________________
Python-Dev mailing list
Python-Devpython.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com