Add a -z interpreter flag to execute a zip file

(resending this now that I'm subscribed, not sure it made it
through
the moderation the first time)

I'd like to request comments on this patch I submitted:

https://s
ourceforge.net/tracker/index.php?func=detail&aid=1739468
&group_id=5470&atid=305470

There are many details given in the comments on that page.

This can be used to deploy Python programs in a very
lightweight and
cross-platform way.  You could imagine a cgi script or a
light web app
server being deployed like this.  I have personally deployed
Python
programs using zip files and this would get rid of the need
for
boilerplate needed for each platform.

The good thing about this is that it's extremely simple --
basically
20 lines of C code to add a -z flag that calls a 3-line
Python
function in the runpy module.

I don't believe it overlaps with anything that already
exists.  py2exe
and py2app are platform specific and bundle the Python
interpreter.
This will be a cross platform binary that doesn't bundle the
Python
interpreter.  It doesn't require eggs but I think it would
work fine
with eggs, and could help fix a little bug as I mentioned on
the patch
page.

Nick Coghlan has reviewed the patch and seems to think it's
a good
idea.  Thomas Wouters also said he likes it, and I ran it by
Guido
earlier and he seemed to think the idea is good, although I
don't
think he has seen the implementation.

thanks,
Andy
_______________________________________________
Python-Dev mailing list
Python-Devpython.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com

> Nick Coghlan has reviewed the patch and seems to think
it's a good
> idea.  Thomas Wouters also said he likes it, and I ran
it by Guido
> earlier and he seemed to think the idea is good,
although I don't
> think he has seen the implementation.

See my comment: I must be missing the point of the patch,
since
I can do the same thing (make a single executable zip file
on
Linux) through a /bin/sh header just fine.

Regards,
Martin
_______________________________________________
Python-Dev mailing list
Python-Devpython.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com

On 7/11/07, "Martin v. Löwis" <martinv.loewis.de> wrote:
> > Nick Coghlan has reviewed the patch and seems to
think it's a good
> > idea.  Thomas Wouters also said he likes it, and I
ran it by Guido
> > earlier and he seemed to think the idea is good,
although I don't
> > think he has seen the implementation.
>
> See my comment: I must be missing the point of the
patch, since
> I can do the same thing (make a single executable zip
file on
> Linux) through a /bin/sh header just fine.

Right, but it's supposed to be cross platform, as mentioned
in the
patch.  This will work on Windows.  The main problem I see
is that a
shell script in front of a zip file seems like a relatively
common
idiom that people use and have different variants on, each
of which
have their own idiosyncrasies.  So it would nice to
consolidate them
and make it standard and robust.

For example, it looks like eggs have an executable format
that is
similar to this.  And see the bug I mentioned where those
executable
eggs can't be invoked through a symlink (which to me is a
relatively
severe problem).  I think this has to do with some
introspection on
$0, but you won't run into that with this implementation.

Also, I mentioned the program called autopar we use at
Google that
does the same thing, and it also have a significant number
of weird
hacks in the shell header.  I think Thomas Wouters has also
worked on
another program to make an executable zip file.

Another example is that the behavior of the zip in your
example
depends on what else is in the current directory [1], which
isn't
desirable.  Nick pointed out this issue and I addressed it
in the
patch by removing "" from sys.path, since the -c
flag adds that.  If
lots of people reinvent this wheel (and they have), there
are going to
be other subtleties like this that will be missed.

The -z flag also eliminates starting an extra process -- you
invoke
the Python interpreter directly instead of starting a shell
which in
turn invokes the Python interpreter.

As mentioned, it's also a very tiny amount of code, and I
don't see
much potential for bad interactions with other things, the
way I've
written it.

Andy

1)
andychu test2$ ./foo_exe.zip
Traceback (most recent call last):
  File "<string>", line 1, in ?
  File "foo.py", line 16, in ?
    import outside
ImportError: No module named outside

andychu test2$ touch outside.py

andychu test2$ ./foo_exe.zip
main

andychu test2$
_______________________________________________
Python-Dev mailing list
Python-Devpython.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com

On 7/12/07, Thomas Wouters <thomaspython.org> wrote:
> On 7/12/07, "Martin v. Löwis" <martinv.loewis.de> wrote:
> > > Right, but it's supposed to be cross
platform, as mentioned in the
> > > patch.  This will work on Windows.
> >
> > But in the description, you said that you do the
same on Windows
> > by making a file that is both a zip file and a
batch file. So my
> > approach is also cross-platform, no?
>
> The approach is cross-platform, in that you can use the
approach on
> different platforms. The result of the approach,
however, is not
> cross-platform. You can't distribute your single
zip-as-executable to both
> Windows and bourne-shell-using platforms. The -z
argument does allow that.
>
> > Why is the shell script not robust?
>
> There are a lot of subtleties in figuring out which
python to execute,
> environment variables that you may or may not want to
tweak (admittedly
> Google's solution that Andy referenced is more
vulnerable to that, but it's
> not unique to Google by any means.) If you want any
kind of flexibility in
> the packaged-up program, you need a bunch of logic in
the shell script, and
> environment-tricks to pass information to the python
process, start the
> python process and provide a bunch more logic in Python
to boot. For
> instance, you need to set PYTHONPATH to include the
zipfile before you can
> import from it, but you don't want that PYTHONPATH to
be passed to
> subprocesses by accident.
>
> The -z argument makes it extremely simple: the user
decides which python to
> run, and the program is run directly just like it would
if it was unpacked
> and run that way.  It makes it extremely easy to create
'single executables'
> out of multiple Python files, in the form that single
.py files already are.
> It leaves building a more complex system (such as eggs)
ontop of it entirely
> open. The change is a good thing, IMHO. And I say this
not because we use a
> similar solution at Google -- we already solved it, and
we won't be using
> the -z argument anytime soon anyway. I say this because
I've had many
> requests from non-googlers for something exactly like
this 
>
>
> > "" should not be removed from sys.path .
It is *not* meant to be
> > the current directory, but the directory where the
main script
> > lives.
>
> Yes. "" should either be interpreted as the
zipfile, or be replaced by the
> zipfile. In the case of executing the zipfile, the main
script lives *in the
> zipfile*.
> > It's baggage that is rarely needed, and the
feature can be readily
> > implemented in a different way for people who need
it.
>
> I disagree with both statements. The bagage is much
less than zipimport
> itself, which has proven to be quite useful.
Nevertheless, zipimport built
> into the interpreter was by no means necessary; current
users of it could
> have readily implemented it themselves, with no changes
to Python. (In fact,
> Google's 'autopar' tool does exactly that to support
Python 2.2, which lacks
> zipimport.) This is a very small, logical and useful
extension to zipimport,
> and I believe you will find more uses for it than you
expect (although I do
> believe you yourself don't have a need for it. I just
don't think you're a
> typical Python programmer in this case 

+1.

(Hi Andy! 

-- 
--Guido van Rossum (home page: http://www.python.org/~
guido/)
_______________________________________________
Python-Dev mailing list
Python-Devpython.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com

On 08:41 am, guidopython.org wrote:
>On 7/12/07, Thomas Wouters <thomaspython.org> wrote:

>>I disagree with both statements. The bagage is much
less than 
>>zipimport
>>itself, which has proven to be quite useful.
Nevertheless, zipimport 
>>built
>>into the interpreter was by no means necessary;
current users of it 
>>could
>>have readily implemented it themselves, with no
changes to Python.

I wonder, is it even necessary to say anything, after:
>+1.

?

But, since I so often object to new features, and there is a
heavy 
Google bias in the existing survey sample, I would like to
say that I 
had a problem several months ago in a _radically_ different
environment 
(Twisted running on an embedded system, Zipfile of PYCs used
to shave 
off as much disk space and startup time as possible) where
having the 
subtleties of a "-z" flag figured out already
would have saved me a 
_ton_ of work.  I was already aware of the shell-header
trick, but 
discovering all the environment-setup details was tedious
and 
distracting enough to make me give up and switch to writing
a bunch of 
hard-to-test /bin/sh code.

It wasn't a bad project by any means, and Python worked out
even better 
than expected (we weren't even sure if it would be able to
load into the 
memory available, but it turns out that being able to do
everything in a 
single process helped a lot) but a -z option would have been
that much 
more impressive .

In fact, I distinctly remember thinking "You know, if
Python had an 
equivalent to Java's '-jar' option, this would be a whole
lot easier."

(Even better, on this _particular_ project, would have been
a generic 
"run this thing-which-looks-like-a-sys.path-entry"
standard format, 
which could have been switched for different deployments to
a directory, 
a zipfile, or the result of freezing.  Perhaps that's
starting to get 
too obscure, though.)
_______________________________________________
Python-Dev mailing list
Python-Devpython.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com

On 12/07/07, glyphdivmod.com <glyphdivmod.com> wrote:
> I wonder, is it even necessary to say anything, after:
> >+1.
[...]
> In fact, I distinctly remember thinking "You know,
if Python had an
> equivalent to Java's '-jar' option, this would be a
whole lot easier."

I'm also +1 on this, for exactly the same reason - I've
often thought
that an equivalent of -jar would be useful, but whenever
I've had a go
at implementing it myself, the fiddly bits needed have meant
that it
ended up not being cost effective to bother...

Paul.
_______________________________________________
Python-Dev mailing list
Python-Devpython.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com

On 12/07/07, "Martin v. Löwis" <martinv.loewis.de> wrote:
> > Right, but it's supposed to be cross platform, as
mentioned in the
> > patch.  This will work on Windows.
>
> But in the description, you said that you do the same
on Windows
> by making a file that is both a zip file and a batch
file. So my
> approach is also cross-platform, no?

Getting the details of such a batch file header right on
Windows is
not easy, not least because there is no "exec"
equivalent on Windows.
The following works, but (a) uses 2 processes, and (b)
doesn't
preserve the exit code. The first issue is minor, but the
second is a
big problem (and one I don't know how to fix).

Also, on Windows, zip-packaged GUI programs could be useful
- these
would be executed using "pythonw -z"

> How do you get the -z option to work on Windows? What
extension
> do you use, and how is the zipfile created?

The patch suggests using .pyz and adding a default
association to the
installer (much like .py and .pyw have). It also offers a
script for
building the zipfiles - either as sample code, or to be
included with
Python (it's not clear to me).

It's arguable that .pyz files should use pythonw -z, not
python -z, as
file extensions are more often useful for clickable programs
in the
GUI. You could have two extensions (.pyz and .pzw, maybe)
but I'm not
sure it's worth it.

The point here is that the fiddly part (setting sys.path,
locating the
main module, etc) is covered by the -z option, deployment
considerations are easier to handle (and hence the exact
defaults
supplied are less crucial) once -z is available.

Paul.
_______________________________________________
Python-Dev mailing list
Python-Devpython.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com

At 10:09 AM 7/12/2007 +0200, Martin v. Löwis wrote:
>"" should not be removed from sys.path. It is
*not* meant to be
>the current directory, but the directory where the main
script
>lives.

Right; it should be replaced with the zipfile path instead.

I would personally rather see this option defined as simply
placing a 
directory at the front of sys.path, and perhaps defining a
default -m 
value of __main__, unless overrridden.  Being able to use
the option 
more than once would be nice, too.  On Windows, you can't
set an 
environment variable on the same line as a command, so this
would 
give you a one-liner way of setting sys.path and running an
application.

I do not see a reason to make this option zipfile-specific
in any 
way, though; it's just as useful (and sometimes more so) to
be able 
to distribute an application as a directory, since that lets
you use 
.pyd, .so, .dll etc. without needing the egg cache system
for using those.


>Why that? Why do eggs fail to process $0 correctly,
whereas the
>-z option gets it correct? That just sounds like a bug
in eggs
>to me, that could be fixed - or, if not, I'd expect that
-z
>cannot fix it, either.
>
>My understanding of this note is that
>pkg_resources uses sys.argv[0] to determine the version
number
>of the egg; IIUC, -z won't help at all here because
sys.argv[0]
>will still be the name of the symlink.

That's correct; it will not help.  A change in the zipped
.egg format 
is required, but could be done.  If the option is added
(again, 
without being zipfile-specific!) then there is a reason for
me to 
make the change.

_______________________________________________
Python-Dev mailing list
Python-Devpython.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com

On 7/12/07, "Martin v. Löwis" <martinv.loewis.de> wrote:
> But in the description, you said that you do the same
on Windows
> by making a file that is both a zip file and a batch
file. So my
> approach is also cross-platform, no?
>
> How do you get the -z option to work on Windows? What
extension
> do you use, and how is the zipfile created?

Nick suggested using .pyz, and others seem to like that
solution
(possibly using pythonw) and that seems logical enough to
me.  If it's
agreed that that's the right solution on Windows, I can put
in the
work for that.

> Couldn't that also be achieved by documenting best
practice in
> the documentation? Why is the shell script not robust?

I think it's pretty clear that it's not robust, and there
have been
even more anecdotal examples on this thread.  Everyone does
it
slightly differently -- not for any particular reason, but
just
because the right thing isn't trivial.

As I pointed out, the example you came up with (which many
others
would come up with too) has a fairly serious problem, in
that it will
import things from outside the .zip file.  I could build my
.zip file
on my system, test it out, and then deploy it to another
machine and
it will break.  Ironically, this happened to *me* while
developing the
patch!

> Why that? Why do eggs fail to process $0 correctly,
whereas the
> -z option gets it correct? That just sounds like a bug
in eggs
> to me, that could be fixed - or, if not, I'd expect
that -z
> cannot fix it, either.
>
> My understanding of this note is that
> pkg_resources uses sys.argv[0] to determine the version
number
> of the egg; IIUC, -z won't help at all here because
sys.argv[0]
> will still be the name of the symlink.

OK, I could be mistaken here, I haven't actually repro'd
this bug.

> What are those weird hacks, why are they necessary, and
how does the
> -z option overcome the need for these hacks?
>
> That people fail to make it work with /bin/sh doesn't
automatically
> mean they succeed with -z. Either they are too
unexperienced to
> make the shell header correct (in which case
documenting best
> practice would help), or they have deeper problems with
that approach,
> in which case it isn't at all obvious that the proposed
change
> improves anything.

I don't think this is true at all.  I have provided the
sample code to
make one of these files, and so you basically have to run a
command
line, rather than write a shell header -- and the shell
header is
currently not documented anywhere.  As mentioned, this
approach also
prevents you from having to start the shell, and makes it
more
portable, since people might use #!/bin/myfavoriteshell or
use
#!/bin/sh and not realize they are using system-specific
features of
the shell.

> > Another example is that the behavior of the zip in
your example
> > depends on what else is in the current directory
[1], which isn't
> > desirable.  Nick pointed out this issue and I
addressed it in the
> > patch by removing "" from sys.path,
since the -c flag adds that.
>
> "" should not be removed from sys.path. It is
*not* meant to be
> the current directory, but the directory where the main
script
> lives.

Regardless of what "" *should* be interpretreted
as, the example you
gave has the problem mentioned (with current versions of
Python) --
that "" *is* the current directory and thus things
get imported
outside of the zip file when they are not found in the zip
file.

Right now "" is replaced with the zip file.  If
there's a better
implementation I'm willing to change it.

> > As mentioned, it's also a very tiny amount of
code, and I don't see
> > much potential for bad interactions with other
things, the way I've
> > written it.
>
> It's baggage that is rarely needed, and the feature can
be readily
> implemented in a different way for people who need it.

I also disagree with both statements. : )  I think others
have said
basically the exact same thing as I am saying: that it is
*commonly*
needed, it's not a lot of baggage in Python since it's so
little code,
and it's easy to get wrong.

Andy
_______________________________________________
Python-Dev mailing list
Python-Devpython.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com