python, lipo and the future?

Ronald Oussoren schrieb:
> One problem is that python's configure script detects
the sizes of
> various types and those values will be different on
32-bit and 64-bit
> flavours.

FWIW, the PC build "solves" this problem by
providing a hand-crafted
pyconfig.h file, instead of using an autoconf-generated one.
That could work for OSX as well, although it is tedious to
keep
the hand-crafted file up-to-date.

For the PC, this isn't really a problem, since Windows
doesn't suddenly
grow new features, at least not those that configure checks
for. So
forking pyconfig.h once and then customizing it for
universal binaries
might work.

Another approach would be to override architecture-specific
defines.
For example, a block

#ifdef __APPLE__
#include "pyosx.h"
#endif

could be added to the end of pyconfig.h, and then pyosx.h
would have

#undef SIZEOF_LONG

#if defined(__i386__) || defined(__ppc__)
#define SIZEOF_LONG 4
#elif defined(__amd64__) || defined(__ppc64__)
#define SIZEOF_LONG 8
#else
#error unsupported architecture
#endif

Out of curiosity: how do the current universal binaries deal
with this
issue?

Regards,
Martin
_______________________________________________
Python-Dev mailing list
Python-Devpython.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com

"Martin v. Löwis" <martinv.loewis.de> wrote:
> Out of curiosity: how do the current universal binaries
deal with this
> issue?

If I remember correctly, usually you do two completely
independant
compile runs (optionally on the same machine with different
configure or
macro definitions, then use a packager provided by Apple to
merge the
results for each binary/so to be distributed. Each
additional platform
would just be a new compile run.


 - Josiah

_______________________________________________
Python-Dev mailing list
Python-Devpython.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com

On Sep 17, 2006, at 6:53 PM, Martin v. Löwis wrote:

> Ronald Oussoren schrieb:
>> One problem is that python's configure script
detects the sizes of
>> various types and those values will be different on
32-bit and 64-bit
>> flavours.
>
> FWIW, the PC build "solves" this problem by
providing a hand-crafted
> pyconfig.h file, instead of using an autoconf-generated
one.
> That could work for OSX as well, although it is tedious
to keep
> the hand-crafted file up-to-date.
>
> For the PC, this isn't really a problem, since Windows
doesn't  
> suddenly
> grow new features, at least not those that configure
checks for. So
> forking pyconfig.h once and then customizing it for
universal binaries
> might work.
>
> Another approach would be to override
architecture-specific defines.
> For example, a block
>
> #ifdef __APPLE__
> #include "pyosx.h"
> #endif

Thats what I had started on before Bob came up with the
endianness  
check that is in pyconfig.h.in at the moment. I'd to do
this instead  
of manually maintaining a fork of pyconfig.h, my guess it is
a lot  
less likely that pyconfig.h will grow new size related
macros than  
new feature related ones.

One possible issue here is that distutils has an API for
fetching  
definitions from pyconfig.h, code that uses this to detect  
architecture features could cause problems.

>
> could be added to the end of pyconfig.h, and then
pyosx.h would have
>
> #undef SIZEOF_LONG
>
> #if defined(__i386__) || defined(__ppc__)
> #define SIZEOF_LONG 4
> #elif defined(__amd64__) || defined(__ppc64__)
> #define SIZEOF_LONG 8
> #else
> #error unsupported architecture
> #endif
>
> Out of curiosity: how do the current universal binaries
deal with this
> issue?

The sizes of basic types are the same on PPC32 and x86 which
helps a  
lot. The byteorder is different, but we can use GCC feature
checks  
there. The relevant bit of pyconfig.h.in:

#ifdef __BIG_ENDIAN__
#define WORDS_BIGENDIAN 1
#else
#ifndef __LITTLE_ENDIAN__
#undef WORDS_BIGENDIAN
#endif
#endif

Users of pyconfig.h will see the correct definition of  
WORDS_BIGENDIAN regardless of the architecture that was used
to  
create the file.

One of the announced features of osx 10.5 is 64-bit support 

throughout the system and I definitely want to see if we can
get 4- 
way universal support on such systems. As I don't have a
system that  
is capable of running 64-bit code  I'm not going to worry
too much  
about this right now 

Ronald
_______________________________________________
Python-Dev mailing list
Python-Devpython.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com

Josiah Carlson schrieb:
> "Martin v. Löwis" <martinv.loewis.de> wrote:
>> Out of curiosity: how do the current universal
binaries deal with this
>> issue?
> 
> If I remember correctly, usually you do two completely
independant
> compile runs (optionally on the same machine with
different configure or
> macro definitions, then use a packager provided by
Apple to merge the
> results for each binary/so to be distributed. Each
additional platform
> would just be a new compile run.

It's true that the compiler is invoked twice, however, I
very much doubt
that configure is run twice. Doing so would cause the
Makefile being
regenerated, and the build starting from scratch. It would
find the
object files from the previous run, and either all overwrite
them, or
leave them in place.

The gcc driver on OSX allows to invoke cc1/as two times, and
then
combines the resulting object files into a single one (not
sure whether
or not by invoking lipo).

Regards,
Martin
_______________________________________________
Python-Dev mailing list
Python-Devpython.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com

Ronald Oussoren schrieb:
> The sizes of basic types are the same on PPC32 and x86
which helps a
> lot.

Ah, right. This was the missing piece of the puzzle.

 The byteorder is different, but we can use GCC feature
checks
> there. The relevant bit of pyconfig.h.in:
> 
> #ifdef __BIG_ENDIAN__
> #define WORDS_BIGENDIAN 1
> #else
> #ifndef __LITTLE_ENDIAN__
> #undef WORDS_BIGENDIAN
> #endif
> #endif

Yes, I remember this change very well.

> One of the announced features of osx 10.5 is 64-bit
support throughout
> the system and I definitely want to see if we can get
4-way universal
> support on such systems. As I don't have a system that
is capable of
> running 64-bit code  I'm not going to worry too much
about this right
> now 

Isn't this a size issue, also? There might be very few
users of a 64-bit
binary (fewer even on PPC64 than on AMD64).

In addition: how does the system chose whether to create a
32-bit
or a 64-bit process if the python binary is fat?

Regards,
Martin

P.S.: for distutils, I think adding special cases would
retrieving
pyconfig.h items would be necessary. In addition, I think
Python should
expose some of these in the image, e.g. as
sys.platform_config.SIZEOF_INT.

_______________________________________________
Python-Dev mailing list
Python-Devpython.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com

On Sep 17, 2006, at 8:56 PM, Martin v. Löwis wrote:

>
>> One of the announced features of osx 10.5 is 64-bit
support  
>> throughout
>> the system and I definitely want to see if we can
get 4-way universal
>> support on such systems. As I don't have a system
that is capable of
>> running 64-bit code  I'm not going to worry too
much about this right
>> now 
>
> Isn't this a size issue, also? There might be very few
users of a  
> 64-bit
> binary (fewer even on PPC64 than on AMD64).

On Tiger it's primairily a useability issue: 64-bit
binaries can't  
use most of the system API's because only the unix API
(libSystem) is  
64-bit at the moment.

The size of the python installer would grow significantly
for a 4-way  
universal distribution, it would be almost twice as large as
the  
current distribution ("almost" because only
binaries would grow in  
site, python source files and data files wouldn't grow in
size).

>
> In addition: how does the system chose whether to
create a 32-bit
> or a 64-bit process if the python binary is fat?

It should take the best fit, on 32-bit processors it picks
the 32-bit  
version and on 64-bit processors it picks the 64-bit one. 
This  
probably means that we'll have to ship multiple versions of
the  
python executable, otherwise Tiger (10.4) users would end up
with an  
interpreter that cannot use OSX-specific API's.

Ronald
_______________________________________________
Python-Dev mailing list
Python-Devpython.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com

Just wondering: is it a good idea in the first place to
create a  
universal 32/64 bit Python on MacOSX?

On MacOS you don't pay a penalty or anything for running in
32-bit  
mode on any current hardware, so the choice of whether to
use 32 or  
64 bits really depends on the application. A single Python  
interpreter that can run in both 32 and 64 bit mode would
possibly  
make this more difficult rather than easier. I think I'd
prefer a  
situation where we have python32 and python64 (with both
being ppc/ 
intel fat) and python being a symlink to either, at the
end-users'  
discretion.

For extension modules it's different, though: there it
would be nice  
to be able to have a single module that could load into any
Python  
(32/64 bit, Intel/PPC) on any applicable MacOSX version.
--
Jack Jansen, <Jack.Jansencwi.nl>, http://www.cwi.nl/~jack
If I can't dance I don't want to be part of your
revolution -- Emma  
Goldman


_______________________________________________
Python-Dev mailing list
Python-Devpython.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com

On Sep 17, 2006, at 9:29 PM, Jack Jansen wrote:

> Just wondering: is it a good idea in the first place to
create a
> universal 32/64 bit Python on MacOSX?
>
> On MacOS you don't pay a penalty or anything for
running in 32-bit
> mode on any current hardware, so the choice of whether
to use 32 or
> 64 bits really depends on the application. A single
Python
> interpreter that can run in both 32 and 64 bit mode
would possibly
> make this more difficult rather than easier. I think
I'd prefer a
> situation where we have python32 and python64 (with
both being ppc/
> intel fat) and python being a symlink to either, at the
end-users'
> discretion.
>
> For extension modules it's different, though: there it
would be nice
> to be able to have a single module that could load into
any Python
> (32/64 bit, Intel/PPC) on any applicable MacOSX
version.

A 4-way universal python framework could be useful, but I
agree that  
the python executable shouldn't be 64-bit.  I'm not too
happy about a  
symlink that selects which version you get to use, wouldn't
 
'python' (32-bit) and 'python-64' (64-bit) be just as
good. That way  
the user doesn't have to set up anything and it helps to
reinforce  
the message that 64-bit isn't necessarily better than
32-bit.

Having a 4-way universal framework would IMO be preferable
over two  
seperate python installs, that would just increase the
confusion.  
There are too many python distributions for the mac anyway.
A major  
stumbling-block for a 4-way universal installation is the  
availability of binary packages for (popular) 3th party
packages,  
this is not really relevant for python-dev but I'd prefer
not having  
64-bit support in the default installer over a 64-bit
capable  
installation where it is very  hard to get popular packages
to work.

BTW. several sites on the interweb claim that x86-64 runs
faster than  
plain x86 due to a larger register set. All my machines are
32-bit so  
I can't check if this is relevant for Python (let alone
Python on OSX).

Ronald
> --
> Jack Jansen, <Jack.Jansencwi.nl>, http://www.cwi.nl/~jack
> If I can't dance I don't want to be part of your
revolution -- Emma
> Goldman
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Devpython.org
> ht
tp://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: ht
tp://mail.python.org/mailman/options/python-dev/ 
> ronaldoussoren%40mac.com

_______________________________________________
Python-Dev mailing list
Python-Devpython.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com

Ronald Oussoren schrieb:
> BTW. several sites on the interweb claim that x86-64
runs faster than
> plain x86 due to a larger register set. All my machines
are 32-bit so I
> can't check if this is relevant for Python (let alone
Python on OSX).

That is plausible. OTOH, the AMD64 binaries will often
require twice
as much main memory, as all pointers double their size, and
the Python
implementation (or most OO languages, for that matter) is
full of
pointers. So it will be more efficient only until it starts
swapping.
(there is also a negative effect of larger pointers on the
processor
 cache; the impact of this effect is hard to estimate).

Regards,
Martin


_______________________________________________
Python-Dev mailing list
Python-Devpython.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com

Jack Jansen schrieb:
> Just wondering: is it a good idea in the first place to
create a  
> universal 32/64 bit Python on MacOSX?

I wonder about the same thing.

> For extension modules it's different, though: there it
would be nice  
> to be able to have a single module that could load into
any Python  
> (32/64 bit, Intel/PPC) on any applicable MacOSX
version.

That seems to suggest that the standard distribution should
indeed
provide a four-times fat binary, at least for libpython:
AFAIU,
to build extension modules that way, all target
architectures must
be supported in all necessary libraries on the build machine
(somebody
will surely correct me if that's wrong).

Regards,
Martin
_______________________________________________
Python-Dev mailing list
Python-Devpython.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com