py2app corrupting libraries -- patch for bug in py2app 2.5

This is a patch and explanation (written by one of my
co-workers)
related to a question I posted to this list on 5/10. 


We have a Python application with a number of C++ extension
modules,
some of
which dynamically load other C++ modules, all of which
reference
additional
C++ frameworks we've built. The module dependency graph
gets complex.

We've recently begun building those C++ pieces as
Universal, and have
upgraded
to py2app 2.5 to construct a Universal app bundle. We're
passing:

setup(
    app=['blobs.py'],
    data_files=[...],
    options=dict(
        py2app=dict(
            archs="ppc,i386",
            ...)))

However, we're getting an error of the form:

Thinning /Users/nat/Trainingr/blobs/Python/dist/Cosmic
Blobs.app/Contents/Frameworks/Events.framework/Versions/A/Ev
ents to ppc,
i386
/usr/bin/lipo: -extract ppc specified but fat file:
/Users/nat/Trainingr/blobs/Python/dist/Cosmic
Blobs.app/Contents/Frameworks/Events.framework/Versions/A/Ev
ents does
not contain that architecture

Examining previous output shows this some 25 lines above:

Thinning /Users/nat/Trainingr/blobs/Python/dist/Cosmic
Blobs.app/Contents/MacOS/../Frameworks/Events.framework/Vers
ions/A/Event
s to i386

That shouldn't be happening. We inserted a test in
util.thin_to_archs()
to
detect the case of a call that fails to specify both 'ppc'
and 'i386',
and
tracked it back up into MachOStandalone.run(). That function
constructs
an
archs dict that maps filenames to lists of architecture
keywords. Then
it
looks up the architecture list for each file of interest.

Among the entries in that dict are:

 '/Users/nat/Trainingr/blobs/Python/dist/Cosmic
Blobs.app/Contents/Frameworks/Events.framework/Versions/A/Ev
ents':
  ['ppc', 'i386'],
 ...
 '/Users/nat/Trainingr/blobs/Python/dist/Cosmic
Blobs.app/Contents/MacOS/../Frameworks/Events.framework/Vers
ions/A/Event
s': ['i386'],

with other similar duplications.

I'm guessing that it should work better to call
os.path.realpath() on
the
filenames, both when constructing and when accessing that
archs dict.
For
instance:

***
/Library/Frameworks/Python.framework/Versions/2.4/lib/python
2.4/site-pac
kages/py2app/macholib/MachOStandalone.py~	Thu May 18
14:26:59 2006
---
/Library/Frameworks/Python.framework/Versions/2.4/lib/python
2.4/site-pac
kages/py2app/macholib/MachOStandalone.py	Mon May 22 15:14:43
2006
***************
*** 117,130 ****
              mm = self.mms[arch]
              for node in mm.flatten(has_filename_filter):
                  machfiles[node.filename] = node
!                 archs.setdefault(node.filename, [])
!                 archs[node.filename].append(arch)
                  dest = os.path.join(contents,
node.filename[len(skipcontents):])
                  changemap[node.filename] = dest
  
          for node in machfiles.itervalues():
              filename = node.filename
!             nodearchs = archs[node.filename]
              changed = False
              for arch in nodearchs:
                  mm = self.mms[arch]
--- 117,129 ----
              mm = self.mms[arch]
              for node in mm.flatten(has_filename_filter):
                  machfiles[node.filename] = node
!                
archs.setdefault(os.path.realpath(node.filename),
[]).append(arch)
                  dest = os.path.join(contents,
node.filename[len(skipcontents):])
                  changemap[node.filename] = dest
  
          for node in machfiles.itervalues():
              filename = node.filename
!             nodearchs =
archs[os.path.realpath(node.filename)]
              changed = False
              for arch in nodearchs:
                  mm = self.mms[arch]

Unfortunately this starts producing:

Thinning /Users/nat/Trainingr/blobs/Python/dist/Cosmic
Blobs.app/Contents/Frameworks/MathUtils.framework/Versions/A
/MathUtils
to ppc, i386, i386
Thinning /Users/nat/Trainingr/blobs/Python/dist/Cosmic
Blobs.app/Contents/MacOS/../MacOS/../Frameworks/Python.frame
work/Version
s/2.4/Python to ppc, ppc, ppc, i386, i386, i386
/usr/bin/lipo: -extract ppc specified multiple times
ValueError: Error 1 returned by: '/usr/bin/lipo'
'/Users/nat/Trainingr/blobs/Python/dist/Cosmic
Blobs.app/Contents/MacOS/../MacOS/../Frameworks/Python.frame
work/Version
s/2.4/Python' '-output' '/tmp/tmpoYncVt' '-extract'
'ppc' '-extract'
'ppc' '-extract' 'ppc' '-extract' 'i386'
'-extract' 'i386' '-extract'
'i386'

So maybe it would work better if, instead of a list of
architectures, we
were to use a set instead:

***
/Library/Frameworks/Python.framework/Versions/2.4/lib/python
2.4/site-pac
kages/py2app/macholib/MachOStandalone.py~	Thu May 18
14:26:59 2006
---
/Library/Frameworks/Python.framework/Versions/2.4/lib/python
2.4/site-pac
kages/py2app/macholib/MachOStandalone.py	Mon May 22 15:25:19
2006
***************
*** 117,130 ****
              mm = self.mms[arch]
              for node in mm.flatten(has_filename_filter):
                  machfiles[node.filename] = node
!                 archs.setdefault(node.filename, [])
!                 archs[node.filename].append(arch)
                  dest = os.path.join(contents,
node.filename[len(skipcontents):])
                  changemap[node.filename] = dest
  
          for node in machfiles.itervalues():
              filename = node.filename
!             nodearchs = archs[node.filename]
              changed = False
              for arch in nodearchs:
                  mm = self.mms[arch]
--- 117,129 ----
              mm = self.mms[arch]
              for node in mm.flatten(has_filename_filter):
                  machfiles[node.filename] = node
!                
archs.setdefault(os.path.realpath(node.filename),
set()).add(arch)
                  dest = os.path.join(contents,
node.filename[len(skipcontents):])
                  changemap[node.filename] = dest
  
          for node in machfiles.itervalues():
              filename = node.filename
!             nodearchs =
list(archs[os.path.realpath(node.filename)])
              changed = False
              for arch in nodearchs:
                  mm = self.mms[arch]

With those changes, we can now get past this problem and
produce a
working app bundle. Please consider incorporating some such
fix into the
distributed
version of py2app. Thank you.



_______________________________________________
Pythonmac-SIG maillist  -  Pythonmac-SIGpython.org
http://mail.python.org/mailman/listinfo/pythonmac-sig

On 23-mei-2006, at 14:56, Kent Quirk wrote:

> This is a patch and explanation (written by one of my
co-workers)
> related to a question I posted to this list on 5/10.

It might be just me, but I don't really understand why
py2app tries  
to thin architectures in the first place. The only file it
may have  
to thin is its own executable stub, because the OS uses that
to  
decide if an application can run natively or not (that's
not entirely  
true, but good enough).

I want to do a new release of PyObjC soon, but want to check
the  
universal binary support in py2app first. That code seems to
have  
some rough edges at the moment.

Ronald
_______________________________________________
Pythonmac-SIG maillist  -  Pythonmac-SIGpython.org
http://mail.python.org/mailman/listinfo/pythonmac-sig

On May 23, 2006, at 2:09 PM, Ronald Oussoren wrote:

>
> On 23-mei-2006, at 14:56, Kent Quirk wrote:
>
>> This is a patch and explanation (written by one of
my co-workers)
>> related to a question I posted to this list on
5/10.
>
> It might be just me, but I don't really understand why
py2app tries
> to thin architectures in the first place. The only file
it may have
> to thin is its own executable stub, because the OS uses
that to
> decide if an application can run natively or not
(that's not entirely
> true, but good enough).

It doesn't even need to do that, because you can specify
which  
architecture to use by way of an Info.plist key.

I didn't write any of the universal support in that branch,
so I'm  
not sure why it's doing what it's doing. I'd like to go
through it  
and audit or rewrite, but I won't have time until at least
after the  
need for speed sprint.

BTW: have you looked at getting ctypes to build on Mac OS X
i386? I  
was going to take a look at ctypes performance, but I can't
build it  
on my MacBook Pro.

-bob


_______________________________________________
Pythonmac-SIG maillist  -  Pythonmac-SIGpython.org
http://mail.python.org/mailman/listinfo/pythonmac-sig

On 23-mei-2006, at 18:08, Bob Ippolito wrote:

>
> On May 23, 2006, at 2:09 PM, Ronald Oussoren wrote:
>
>>
>> On 23-mei-2006, at 14:56, Kent Quirk wrote:
>>
>>> This is a patch and explanation (written by one
of my co-workers)
>>> related to a question I posted to this list on
5/10.
>>
>> It might be just me, but I don't really understand
why py2app tries
>> to thin architectures in the first place. The only
file it may have
>> to thin is its own executable stub, because the OS
uses that to
>> decide if an application can run natively or not
(that's not entirely
>> true, but good enough).
>
> It doesn't even need to do that, because you can
specify which  
> architecture to use by way of an Info.plist key.
>
> I didn't write any of the universal support in that
branch, so I'm  
> not sure why it's doing what it's doing. I'd like to
go through it  
> and audit or rewrite, but I won't have time until at
least after  
> the need for speed sprint.

I'll probably have a look this thursday.

>
> BTW: have you looked at getting ctypes to build on Mac
OS X i386? I  
> was going to take a look at ctypes performance, but I
can't build  
> it on my MacBook Pro.

I haven't looked at this yet, I'm trying to keep python2.5
working  
first. It should be possible to use a copy of the libffi in
pyobjc to  
get ctypes going, or at least the object files that pyobjc
generates.

Ronald

_______________________________________________
Pythonmac-SIG maillist  -  Pythonmac-SIGpython.org
http://mail.python.org/mailman/listinfo/pythonmac-sig