v2 Python toolset doesn't faithfully reproduce v1

On Thursday 13 July 2006 20:05, David Abrahams wrote:
> Vladimir Prus <ghostcs.msu.su> writes:
> > On Thursday 13 July 2006 18:09, David Abrahams
wrote:
> >> Vladimir Prus <ghostcs.msu.su> writes:
> >> > We've being though this before, and you
basically agreed that this
> >> > comment is not accurate. Please see the
attached message. This topic
> >> > then died.
> >>
> >> Do we need it to un-die, or do you think all
the problems are
> >> resolved?
> >
> > I think we need it to un-die. The two current
issues are:
> >
> > 1. I've modified Boost.Python tests Jamfile.v2 to
have <threading>multi.
> > I think that fixes the errors, but is not a clean
solution. I'd prefer
> > direct linking to pthread.
>
> Why?  For example, do you believe the other options
that
> <threading>multi might add are inappropriate, and
if so, why do you
> think that?

The apparent problem we see that that Python wants symbols
from the pthread 
library. The apparent solution is to add that library.

The <threading>multi, applied to my main executable,
means I pay for thread 
safety, even if my executable is single-threaded. 

> > 2. I noticed that in V2 Boost.Python links to
Python library.
>
> Too vague to understand.

On Unix, the Boost.Python's so library is created, there's
-lpython2.3 option
on linkers' command line. This in in V2, and in V1 this is
not done.

> * Windows, Unix, both...?
>
> * You mean the boost_python.so/dll links to
libpython.a/pythonXX.lib?

I think on my system libpython is actually .so.

> > This sounds potentially problematic, I'm not sure
if Python
> > interpreter will completely hide all symbols from
Python shared lib.
>
> Sorry, you lost me completely here.  Whose symbols? 
How would a
> Python interpreter hide symbols?

Python interpreter has almost the same set of functions as
Python dynamic 
library (the Py_* functions). If Python interpreter loads an
extension which 
links to Boost.Python which is linked to Python library, you
get two copies 
of each function. And I have no idea if this is going to
work, or create some 
problems.

- Volodya


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build

On 7/13/06, Vladimir Prus <ghostcs.msu.su> wrote:

> Python interpreter has almost the same set of functions
as Python dynamic
> library (the Py_* functions). If Python interpreter
loads an extension which
> links to Boost.Python which is linked to Python
library, you get two copies
> of each function. And I have no idea if this is going
to work, or create some
> problems.

Generally speaking this should be fine; the dynamic linker
will detect
that libpython2.x.so is already in the ELF image at runtime
(the
Python interpreter is dynamically linked with the Python
library) and
just re-use that shared object.

This works almost the same way with DLLs on Windows, except
they have
a funky DllMain function as well (shouldn't affect this
scenario.)

The only way I would see this breaking is if the internals
of
boost.python somehow relied on static or initialization
ordering of
the Python library itself, but AFAIK that's not the case.

HTH,
Andreas

-- 
My other car is a cons.
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build

Vladimir Prus <ghostcs.msu.su> writes:

> On Thursday 13 July 2006 20:05, David Abrahams wrote:
>> Vladimir Prus <ghostcs.msu.su> writes:
>> > On Thursday 13 July 2006 18:09, David Abrahams
wrote:
>> >> Vladimir Prus <ghostcs.msu.su> writes:
>> >> > We've being though this before, and
you basically agreed that this
>> >> > comment is not accurate. Please see
the attached message. This topic
>> >> > then died.
>> >>
>> >> Do we need it to un-die, or do you think
all the problems are
>> >> resolved?
>> >
>> > I think we need it to un-die. The two current
issues are:
>> >
>> > 1. I've modified Boost.Python tests
Jamfile.v2 to have <threading>multi.
>> > I think that fixes the errors, but is not a
clean solution. I'd prefer
>> > direct linking to pthread.
>>
>> Why?  For example, do you believe the other options
that
>> <threading>multi might add are inappropriate,
and if so, why do you
>> think that?
>
> The apparent problem we see that that Python wants
symbols from the pthread 
> library. The apparent solution is to add that library.

I would say the right solution is to declare 

  lib pthread;
  lib python : : : <source>pthread ;

or however it is you get pthread into the usage requirements
for libpython.

> The <threading>multi, applied to my main
executable, means I pay for
> thread safety, even if my executable is
single-threaded.

I agree as long as you are convinced that there's no link
incompatibility.

>
>> > 2. I noticed that in V2 Boost.Python links to
Python library.
>>
>> Too vague to understand.
>
> On Unix, the Boost.Python's so library is created,
there's
> -lpython2.3 option on linkers' command line. This in
in V2, and in
> V1 this is not done.

Okay.  Do you know why it's done in v2, and who did it?


>> > This sounds potentially problematic, I'm not
sure if Python
>> > interpreter will completely hide all symbols
from Python shared lib.
>>
>> Sorry, you lost me completely here.  Whose symbols?
 How would a
>> Python interpreter hide symbols?
>
> Python interpreter has almost the same set of functions
as Python dynamic 
> library (the Py_* functions). If Python interpreter
loads an extension which 
> links to Boost.Python which is linked to Python
library, you get two copies 
> of each function. And I have no idea if this is going
to work, or create some 
> problems.

Yes, it's problematic.  On Unix, dynamically loaded python
extensions
get the python symbols from the executable.  If we're
extending
Python, then that's the python executable itself.  If
we're embedding
Python, it's the embedding application, which is supposed
to be linked
to libpython.a



-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build

On Thursday 13 July 2006 21:01, Andreas Fredriksson wrote:
> On 7/13/06, Vladimir Prus <ghostcs.msu.su> wrote:
> > Python interpreter has almost the same set of
functions as Python dynamic
> > library (the Py_* functions). If Python
interpreter loads an extension
> > which links to Boost.Python which is linked to
Python library, you get
> > two copies of each function. And I have no idea if
this is going to work,
> > or create some problems.
>
> Generally speaking this should be fine; the dynamic
linker will detect
> that libpython2.x.so is already in the ELF image at
runtime (the
> Python interpreter is dynamically linked with the
Python library) and
> just re-use that shared object.

On my system, Python binary is *not* linked to
libpython2.x.so:

        $ ldd `which python`
        linux-gate.so.1 =>  (0xffffe000)
        libpthread.so.0 =>
/lib/tls/i686/cmov/libpthread.so.0 (0xb7f8c000)
        libdl.so.2 => /lib/tls/i686/cmov/libdl.so.2
(0xb7f88000)
        libutil.so.1 => /lib/tls/i686/cmov/libutil.so.1
(0xb7f84000)
        libm.so.6 => /lib/tls/i686/cmov/libm.so.6
(0xb7f5e000)
        libc.so.6 => /lib/tls/i686/cmov/libc.so.6
(0xb7e28000)
        /lib/ld-linux.so.2 (0xb7fba000)

I believe that's intentional, because position independent
code (needed when 
building .so), is supposed to be slightly slower.

- Volodya

-- 
Vladimir Prus
http://vladimir_pru
s.blogspot.com
Boost.Build V2: http://boost.org/boost-
build2
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build

On Thursday 13 July 2006 22:30, David Abrahams wrote:
> Vladimir Prus <ghostcs.msu.su> writes:
> > On Thursday 13 July 2006 20:05, David Abrahams
wrote:
> >> Vladimir Prus <ghostcs.msu.su> writes:
> >> > On Thursday 13 July 2006 18:09, David
Abrahams wrote:
> >> >> Vladimir Prus <ghostcs.msu.su> writes:
> >> >> > We've being though this before,
and you basically agreed that this
> >> >> > comment is not accurate. Please
see the attached message. This
> >> >> > topic then died.
> >> >>
> >> >> Do we need it to un-die, or do you
think all the problems are
> >> >> resolved?
> >> >
> >> > I think we need it to un-die. The two
current issues are:
> >> >
> >> > 1. I've modified Boost.Python tests
Jamfile.v2 to have
> >> > <threading>multi. I think that
fixes the errors, but is not a clean
> >> > solution. I'd prefer direct linking to
pthread.
> >>
> >> Why?  For example, do you believe the other
options that
> >> <threading>multi might add are
inappropriate, and if so, why do you
> >> think that?
> >
> > The apparent problem we see that that Python wants
symbols from the
> > pthread library. The apparent solution is to add
that library.
>
> I would say the right solution is to declare
>
>   lib pthread;
>   lib python : : : <source>pthread ;
>
> or however it is you get pthread into the usage
requirements for libpython.

Yes, that's precisely what I wanted.

> > The <threading>multi, applied to my main
executable, means I pay for
> > thread safety, even if my executable is
single-threaded.
>
> I agree as long as you are convinced that there's no
link
> incompatibility.

I'm 99% sure.

> >> > 2. I noticed that in V2 Boost.Python
links to Python library.
> >>
> >> Too vague to understand.
> >
> > On Unix, the Boost.Python's so library is
created, there's
> > -lpython2.3 option on linkers' command line. This
in in V2, and in
> > V1 this is not done.
>
> Okay.  Do you know why it's done in v2, and who did
it?

I did, the log message for revision 1.8 of build/Jamfile.v2
is unfortunately 
not very descriptive.

> >> > This sounds potentially problematic, I'm
not sure if Python
> >> > interpreter will completely hide all
symbols from Python shared lib.
> >>
> >> Sorry, you lost me completely here.  Whose
symbols?  How would a
> >> Python interpreter hide symbols?
> >
> > Python interpreter has almost the same set of
functions as Python dynamic
> > library (the Py_* functions). If Python
interpreter loads an extension
> > which links to Boost.Python which is linked to
Python library, you get
> > two copies of each function. And I have no idea if
this is going to work,
> > or create some problems.
>
> Yes, it's problematic.  On Unix, dynamically loaded
python extensions
> get the python symbols from the executable.  If we're
extending
> Python, then that's the python executable itself.  If
we're embedding
> Python, it's the embedding application, which is
supposed to be linked
> to libpython.a

Yep, we'll end up with two copies of symbols, but is this
going to cause real 
problems?

Anyway, I'll try to change this. That raises, again, this
question: Do we:

   - Document that for embedding, application should be
explicitly linked to
     Python
   - Add new target "boost_python_for_embedded",
that will alias to
     Boost.Python, Python, and any auxilliary libraries.

I'm fine with just adding /boost/python to all embedding
applications, as far 
as tests are concerned. You know better, though.

- Volodya



-- 
Vladimir Prus
http://vladimir_pru
s.blogspot.com
Boost.Build V2: http://boost.org/boost-
build2
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build

On Saturday 22 July 2006 10:32, Vladimir Prus wrote:

> > Yes, it's problematic.  On Unix, dynamically
loaded python extensions
> > get the python symbols from the executable.  If
we're extending
> > Python, then that's the python executable itself.
 If we're embedding
> > Python, it's the embedding application, which is
supposed to be linked
> > to libpython.a
>
> Yep, we'll end up with two copies of symbols, but is
this going to cause
> real problems?
>
> Anyway, I'll try to change this. That raises, again,
this question: Do we:
>
>    - Document that for embedding, application should be
explicitly linked
> to Python
>    - Add new target
"boost_python_for_embedded", that will alias to
>      Boost.Python, Python, and any auxilliary
libraries.
>
> I'm fine with just adding /boost/python to all
embedding applications, as
> far as tests are concerned. You know better, though.

I've just checked in the attached. Let me know if something
looks wrong.

- Volodya

Don't link Boost.Python to python library, and don't
require
<threading>multi for embedding applications.

* libs/python/build/Jamfile.v2: (boost_python): Don't link
  to /python//python. Use /python//python_for_extensions.

* libs/python/test/Jamfile.v2: Remove <threading>multi
project
  requirements. 
  (py-run): Link to /python//python.
  (exec): Likewise.

* tools/build/v2/tools/python.jam: (pthread): Declare.
  (init-unix): Add 'pthread' to extra-libs.  
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build

Vladimir Prus <ghostcs.msu.su> writes:

> Yep, we'll end up with two copies of symbols, but is
this going to cause real 
> problems?

Yes.

> Anyway, I'll try to change this. That raises, again,
this question: Do we:
>
>    - Document that for embedding, application should be
explicitly linked to
>      Python

Not bad, but not necessary.

>    - Add new target
"boost_python_for_embedded", that will alias to
>      Boost.Python, Python, and any auxilliary
libraries.

Also not bad.

Is it impossible to make conditional usage requirements
where the
condition is "what type of target are we
building?"

> I'm fine with just adding /boost/python to all
embedding applications, as far 
> as tests are concerned. 

What is /boost/python (a path?) and how does it get added?

> You know better, though.

Not sure I do.

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build

Vladimir Prus <ghostcs.msu.su> writes:

> I've just checked in the attached. Let me know if
something looks wrong.

a. I don't see python_for_extensions defined anywhere

b. At the very least, the comments are wrong for the Windows
platform,
   where extending applications _do_ link directly to the
   libpython.lib import library.


-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build

On Saturday 22 July 2006 16:05, David Abrahams wrote:
> Vladimir Prus <ghostcs.msu.su> writes:
> > I've just checked in the attached. Let me know if
something looks wrong.
>
> a. I don't see python_for_extensions defined anywhere

It's defined in python.jam, and was defined for quite some
time, just not 
used.

> b. At the very least, the comments are wrong for the
Windows platform,
>    where extending applications _do_ link directly to
the
>    libpython.lib import library.

And which comment is wrong? This one:

+	# We don't link to Python library itself. If
+	# Boost.Python is used for extension, all Python
+	# symbols are available in Python interpreter.
+	# If Boost.Python is used for extending, client
+	# is required to link to /python//python itself.
+	<use>/python//python_for_extensions

? How about this rewording:

	# On Linux, we don't link to Python library itself. If
	# Boost.Python is used for extension, all Python
	# symbols are available in Python interpreter.
	# If Boost.Python is used for extending, client
	# is required to link to /python//python itself.
	# On Windows, all code using Python has to link
	# to python import library. The 'python_for_extension'
	# is the target that's setup to provide either just
	# include paths, or import library.       
	<library>/python//python_for_extensions

?

- Volodya

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build

On Saturday 22 July 2006 15:57, David Abrahams wrote:
> Vladimir Prus <ghostcs.msu.su> writes:
> > Yep, we'll end up with two copies of symbols, but
is this going to cause
> > real problems?
>
> Yes.

Are they already summarised somewhere? I've already changed
not to do so, but 
it would be nicer to have rationale explained somewhere.

>
> > Anyway, I'll try to change this. That raises,
again, this question: Do
> > we:
> >
> >    - Document that for embedding, application
should be explicitly linked
> > to Python
>
> Not bad, but not necessary.
>
> >    - Add new target
"boost_python_for_embedded", that will alias to
> >      Boost.Python, Python, and any auxilliary
libraries.
>
> Also not bad.
>
> Is it impossible to make conditional usage requirements
where the
> condition is "what type of target are we
building?"

I'm not sure, I never tried to do this before. I think
allowing this is a good 
idea, so I've added:

   h
ttps://zigzag.cs.msu.su:7813/boost.build/ticket/98

but we're not there yet.

> > I'm fine with just adding /boost/python to all
embedding applications, as
> > far as tests are concerned.
>
> What is /boost/python (a path?) and how does it get
added?

I meant "/boost//python" -- the id of
Boost.Python library.

- Volodya
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build