Deadlock by a second import in a thread

Hi!

I was looking to this bug:   http://bugs.python.o
rg/issue1255

It basically creates a deadlock in Python by doing the
following:

- aa.py imports bb.py
- bb.py imports time and generates a thread
- the thread uses time.strptime

The deadlock is because the strptime function imports
another module,
line 517 of timemodule.c:

      PyObject *strptime_module =
PyImport_ImportModule("_strptime");

This situation is well known, found a lot of references to
this
import-thread-import problem in discussions and previous
bugs (i.e.:
http://bugs.python
.org/issue683658).

What I did *not* find, and why I'm asking here, is how to
solve it.

Exists a known solution to this?

Thank you!

-- 
.    Facundo

Blog: http://www.tanique
til.com.ar/plog/
PyAr: http://www.python.org/ar/
_______________________________________________
Python-Dev mailing list
Python-Devpython.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com

On 10/19/07, Facundo Batista <facundobatistagmail.com> wrote:
> Hi!
>
> I was looking to this bug:   http://bugs.python.o
rg/issue1255
>
> It basically creates a deadlock in Python by doing the
following:
>
> - aa.py imports bb.py
> - bb.py imports time and generates a thread
> - the thread uses time.strptime
>
> The deadlock is because the strptime function imports
another module,
> line 517 of timemodule.c:
>
>       PyObject *strptime_module =
PyImport_ImportModule("_strptime");
>
> This situation is well known, found a lot of references
to this
> import-thread-import problem in discussions and
previous bugs (i.e.:
> http://bugs.python
.org/issue683658).
>
> What I did *not* find, and why I'm asking here, is how
to solve it.
>
> Exists a known solution to this?

When python encounters a recursive import within a single
thread it
allows you to get access to partially-imported modules,
making the
assumption that you won't do any significant work until the
entire
import process completes.

Only one thread is allowed to do an import at a time though,
as
they'll do significant work with it immediately, so being
partially-imported would be a race condition.

Writing a python file as a script means you do significant
work in the
body, rather than in a function called after importing
completes.
Importing this python file then violates the assumption for
single-threaded recursive imports, and creating threads then
violates
their safety assumptions.

The solution then is, if your python file will ever be
imported, you
must write a main function and do all the work there
instead.  Do not
write it in the style of a script (with significant work in
the global
scope.)

-- 
Adam Olsen, aka Rhamphoryncus
_______________________________________________
Python-Dev mailing list
Python-Devpython.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com

2007/10/19, Adam Olsen <rhamphgmail.com>:

> The solution then is, if your python file will ever be
imported, you
> must write a main function and do all the work there
instead.  Do not
> write it in the style of a script (with significant
work in the global
> scope.)

I had this a as a good coding style, not so mandatory.

I agree with you that the OP shouldn't be doing that, but
note that
the main problem arises here because it's completely
unpredictable the
import in strptime for an external user.

Do you recommend to close the bug as "won't fix"
saying something like...

    The deadlock happens because strptime has an import
inside it, and
    recursive imports are not allowed in different threads.

    As a general rule and good coding style, don't run your
code when the
    module is imported, but put it in a function like
"main" in the second file,
    import it and call it from the first one. This will
solve your problem.

    Note that this happens to you with strptime, but could
happen with a lot
    of functions that do this internal import of something
else. So,
you'll never
    be sure.

What do you think?

Thank you!

--
.    Facundo

Blog: http://www.tanique
til.com.ar/plog/
PyAr: http://www.python.org/ar/
_______________________________________________
Python-Dev mailing list
Python-Devpython.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com

Facundo Batista wrote:
> What I did *not* find, and why I'm asking here, is how
to solve it.
> 
> Exists a known solution to this?

I know a possible solution. You could write a patch that
moves the
imports in C code to the module init function and stores the
modules in
a global static variable.

Christian

_______________________________________________
Python-Dev mailing list
Python-Devpython.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com

On 10/19/07, Facundo Batista <facundobatistagmail.com> wrote:
> 2007/10/19, Adam Olsen <rhamphgmail.com>:
>
> > The solution then is, if your python file will
ever be imported, you
> > must write a main function and do all the work
there instead.  Do not
> > write it in the style of a script (with
significant work in the global
> > scope.)
>
> I had this a as a good coding style, not so mandatory.
>
> I agree with you that the OP shouldn't be doing that,
but note that
> the main problem arises here because it's completely
unpredictable the
> import in strptime for an external user.
>
> Do you recommend to close the bug as "won't
fix" saying something like...
>
>     The deadlock happens because strptime has an import
inside it, and
>     recursive imports are not allowed in different
threads.
>
>     As a general rule and good coding style, don't run
your code when the
>     module is imported, but put it in a function like
"main" in the second file,
>     import it and call it from the first one. This will
solve your problem.
>
>     Note that this happens to you with strptime, but
could happen with a lot
>     of functions that do this internal import of
something else. So,
> you'll never
>     be sure.
>
> What do you think?

Whether this is a minor problem due to poor style or a major
problem
due to a language defect is a matter of perspective.  I'm
working on
redesigning Python's threading support, expecting it to be
used a
great deal more, which'd push it into the major problem
category.

For now I'd leave it open.

-- 
Adam Olsen, aka Rhamphoryncus
_______________________________________________
Python-Dev mailing list
Python-Devpython.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com

Facundo Batista wrote:
> Hi!
> 
> I was looking to this bug:   http://bugs.python.o
rg/issue1255
> 
> It basically creates a deadlock in Python by doing the
following:
> 
> - aa.py imports bb.py
> - bb.py imports time and generates a thread

bb.py is broken - importing a module should never spawn a
new thread as 
a side effect (precisely because it will deadlock if the
spawned thread 
tries to do an import, which can happen in a myriad of
ways).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlangmail.com   |   Brisbane,
Australia
------------------------------------------------------------
---
             http://www.boredoma
ndlaziness.org
_______________________________________________
Python-Dev mailing list
Python-Devpython.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com

2007/10/19, Christian Heimes <listscheimes.de>:

> I know a possible solution. You could write a patch
that moves the
> imports in C code to the module init function and
stores the modules in
> a global static variable.

I thought about this. It even will have the good side efect
of not
shooting the whole "import" machinery to see that
you already imported
it, every time you do an strptime.

One question: The program makes this:

    PyObject *strptime_module =
PyImport_ImportModule("_strptime");
    ...
    Py_DECREF(strptime_module);

If I'd import it in the beggining of the file with the
following...

    static PyObject *strptime_module =
PyImport_ImportModule("_strptime");

... I'd never decref it, right?

Thank you!

-- 
.    Facundo

Blog: http://www.tanique
til.com.ar/plog/
PyAr: http://www.python.org/ar/
_______________________________________________
Python-Dev mailing list
Python-Devpython.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com

2007/10/20, Nick Coghlan <ncoghlangmail.com>:

> bb.py is broken - importing a module should never spawn
a new thread as
> a side effect (precisely because it will deadlock if
the spawned thread
> tries to do an import, which can happen in a myriad of
ways).

Agreed.

But if we move the import of the _strptime module to do it
once when
"time" is imported, we enhance the "least
surprise" situation (and has
a marginal performance benefit of not trying to import it
again every
time.strptime() call).

And even solves this bug, which does not hurt enaybody, ;)

Regards,

-- 
.    Facundo

Blog: http://www.tanique
til.com.ar/plog/
PyAr: http://www.python.org/ar/
_______________________________________________
Python-Dev mailing list
Python-Devpython.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com

On 10/20/07, Facundo Batista <facundobatistagmail.com> wrote:
> 2007/10/19, Christian Heimes <listscheimes.de>:
>
> > I know a possible solution. You could write a
patch that moves the
> > imports in C code to the module init function and
stores the modules in
> > a global static variable.
>
> I thought about this. It even will have the good side
efect of not
> shooting the whole "import" machinery to see
that you already imported
> it, every time you do an strptime.
>
> One question: The program makes this:
>
>     PyObject *strptime_module =
PyImport_ImportModule("_strptime");
>     ...
>     Py_DECREF(strptime_module);
>
> If I'd import it in the beggining of the file with the
following...
>
>     static PyObject *strptime_module =
PyImport_ImportModule("_strptime");
>
> ... I'd never decref it, right?

Right.  Otherwise, if the module is removed from sys.modules
then it
will have a refcount of 0 and be collected, leaving your
static
variable holding junk memory.

One issue with this approach, though, is that the import is
a one-time
thing, and so replacing what is in sys.modules['_strptime']
will not
take affect in the module ever thanks to the caching of the
module's
dict.

Granted this is a small issue as normal Python code does not
pick up
changes in sys.modules, but it is something to be aware of.

-Brett
_______________________________________________
Python-Dev mailing list
Python-Devpython.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com

2007/10/19, Adam Olsen <rhamphgmail.com>:

> Whether this is a minor problem due to poor style or a
major problem
> due to a language defect is a matter of perspective. 
I'm working on
> redesigning Python's threading support, expecting it to
be used a
> great deal more, which'd push it into the major problem
category.
>
> For now I'd leave it open.

It's a matter of perspective, yes. But I'll close this bug,
because
he's hitting the problem through a weird way, doing
something that he
shouldn't.

The real problem here, if any, is that you can not make a
second
import in another thread. Feel free to open a bug for this,
but
addressing this specifically.

I'd prefer a PEP, though, ;)

Regards,

-- 
.    Facundo

Blog: http://www.tanique
til.com.ar/plog/
PyAr: http://www.python.org/ar/
_______________________________________________
Python-Dev mailing list
Python-Devpython.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com