Request for inclusion in 2.5.2 (5-for-1)

Issue http://bugs.pytho
n.org/issue1663329 details an annoyance in the  
subprocess module that has affected several users, including
me.   
Essentially, closing hundreds of thousands of file
descriptors by  
round-tripping through the python exception machinery is
very slow,  
taking hundreds of milliseconds and at times many seconds. 
The  
proposed fix is to write this loop in c.  The c function is
but a  
handful of lines long.  I purposefully kept the
implementation  
trivial so that it will work on all unix variants (there is
another  
issue that contains a super-duper optimization for AIX, and
other  
possibilities exist for Solaris, but the simple fix yields a
ten-fold  
speedup everywhere but windows, so I didn't think that it
was worth  
the complexity).

Though technically relating only to performance, I consider
this a  
bug-fix candidate as mysterious multi-second delays when
launching a  
subprocess end up making the functionality of close_fds
unusable on  
some platform configurations (namely, those with high MAX_FD
set).

It would be great to see this is 2.5.2.  Understanding that
issue  
evaluation takes significant effort, I've done some
evaluation/triage  
on other open tickets:
----------------------------------------
See issues for detailed comments.

http://b
ugs.python.org/issue1516327:  No clear problem, no
patch.  
Recommend closing.

http://b
ugs.python.org/issue1773632: tested patch.  Recommend  
accepting unless there are things I don't know about this
mysterious  
_xmlrpclib extension (which there doubtlessly are)

python.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com

Mike Klaas wrote:
> gmail.com   |   Brisbane,
Australia
------------------------------------------------------------
---
             http://www.boredoma
ndlaziness.org
_______________________________________________
Python-Dev mailing list
Python-Devpython.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com

On 10/31/07, Mike Klaas <mike.klaasgmail.com> wrote:
> Issue http://bugs.pytho
n.org/issue1663329 details an annoyance in the
> subprocess module that has affected several users,
including me.
> Essentially, closing hundreds of thousands of file
descriptors by
> round-tripping through the python exception machinery
is very slow,
> taking hundreds of milliseconds and at times many
seconds.  The
> proposed fix is to write this loop in c.  The c
function is but a
> handful of lines long.  I purposefully kept the
implementation
> trivial so that it will work on all unix variants
(there is another
> issue that contains a super-duper optimization for AIX,
and other
> possibilities exist for Solaris, but the simple fix
yields a ten-fold
> speedup everywhere but windows, so I didn't think that
it was worth
> the complexity).
>
> Though technically relating only to performance, I
consider this a
> bug-fix candidate as mysterious multi-second delays
when launching a
> subprocess end up making the functionality of close_fds
unusable on
> some platform configurations (namely, those with high
MAX_FD set).
>
> It would be great to see this is 2.5.2.  Understanding
that issue
> evaluation takes significant effort, I've done some
evaluation/triage
> on other open tickets:

Thanks for doing these!

Since people are already jumping on those bugs but nobody
has voiced
an opinion on your own patch, let me say that I think it's a
good
patch, and I want it in 2.6, but I'm reluctant to add it to
2.5.2 as
it goes well beyond a bugfix (adding a new C API and all
that).

-- 
--Guido van Rossum (home page: http://www.python.org/~
guido/)
_______________________________________________
Python-Dev mailing list
Python-Devpython.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com

On 2-Nov-07, at 6:57 AM, Guido van Rossum wrote:

>
> Since people are already jumping on those bugs but
nobody has voiced
> an opinion on your own patch, let me say that I think
it's a good
> patch, and I want it in 2.6, but I'm reluctant to add
it to 2.5.2 as
> it goes well beyond a bugfix (adding a new C API and
all that).

Thanks for looking at it!

Is there a better way of exposing some c-helper code for a
stdlib  
module written in python?  It seems that the canonical
pattern is to  
write a separate extension module called _<modulename>
and import the  
functionality from there, but that seemed like a
significantly more  
invasive patch.

Might it help to tack on the helper function in posix only,
deleting  
it from the os namespace?

Thanks again,
-Mike

_______________________________________________
Python-Dev mailing list
Python-Devpython.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com

On 11/2/07, Mike Klaas <mike.klaasgmail.com> wrote:
> On 2-Nov-07, at 6:57 AM, Guido van Rossum wrote:
>
> > Since people are already jumping on those bugs but
nobody has voiced
> > an opinion on your own patch, let me say that I
think it's a good
> > patch, and I want it in 2.6, but I'm reluctant to
add it to 2.5.2 as
> > it goes well beyond a bugfix (adding a new C API
and all that).
>
> Thanks for looking at it!
>
> Is there a better way of exposing some c-helper code
for a stdlib
> module written in python?  It seems that the canonical
pattern is to
> write a separate extension module called
_<modulename> and import the
> functionality from there, but that seemed like a
significantly more
> invasive patch.

No, what you did was the right thing. It just doesn't feel
like a bugfix to me.

> Might it help to tack on the helper function in posix
only, deleting
> it from the os namespace?

No. Why are yo so insistent on having this in 2.5.2? You
can't force
folks who use your code to upgrade (e.g. OSX Leopard was
just shipped
with 2.5.1).

-- 
--Guido van Rossum (home page: http://www.python.org/~
guido/)
_______________________________________________
Python-Dev mailing list
Python-Devpython.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com

Mike, thanks for reviewing the bugs and for creating a
patch.  Good work!

On Nov 2, 2007 2:23 AM, Nick Coghlan <ncoghlangmail.com> wrote:
> Mike Klaas wrote:
> > python.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com

Neal Norwitz wrote:
> On Nov 2, 2007 2:23 AM, Nick Coghlan <ncoghlangmail.com> wrote:
>> Neal - where does the 2.5 branch stand at the
moment? This would be a
>> simple fix to slip into 2.5.2 if there's still
time.
> 
> I'm not sure if this was before my last status mail. 
We are hoping to
> get a release candidate out in a week or two.

It was more than a week but less than two weeks after your
last status 
message, but you hadn't said anything yet about freezing the
branch to 
cut a release candidate - hence my question 

> I agree with Guido that this isn't a bugfix and
therefore shouldn't be
> backported to 2.5.

I was actually referring to the contextmanager bug with that
comment - I 
was just a little vague with my pronouns. All sorted now
though.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlangmail.com   |   Brisbane,
Australia
------------------------------------------------------------
---
             http://www.boredoma
ndlaziness.org
_______________________________________________
Python-Dev mailing list
Python-Devpython.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com