PEP 370, open questions

The PEP 370 (http://www.py
thon.org/dev/peps/pep-0370) "per user site
packages directory" has several open questions:

* Are the directories for Windows, Mac and Unix fine?
* Mac: Should framework and non-framework builds of Python
use
  the same directories?
* The patch also adds a usecustomize hook to site. Is it
useful and
  should it stay?
* Should the site package directory also be ignored if
process
  gid != effective gid?
* Should the Windows installer add %APPDATA%/Python/Scripts
to PATH?
* Should the base directory be configurable with an
environment
  variable like PYTHONUSERHOME?

A CC of the mail goes to the authors of setuptools, virtual
python,
working env and virtual env. What's your opinion on the PEP?
Do you have
some input and advice for me?

Christian
_______________________________________________
Python-Dev mailing list
Python-Devpython.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com

On 17/01/2008, Christian Heimes <listscheimes.de> wrote:
> * Should the Windows installer add
%APPDATA%/Python/Scripts to PATH?

The Windows installers currently do not add the main
PythonScripts
directory to PATH, so they shouldn't add this one either.

Paul
_______________________________________________
Python-Dev mailing list
Python-Devpython.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com

On 07:55 am, listscheimes.de wrote:
>A CC of the mail goes to the authors of setuptools,
virtual python,
>working env and virtual env. What's your opinion on the
PEP? Do you 
>have
>some input and advice for me?

I wrote a similar tool called Combinator 
(http://d
ivmod.org/trac/wiki/DivmodCombinator) which Divmod uses
quite 
intensively.  I think a significant portion of the Twisted
community 
also uses it for development.  Luckily you don't need to CC
me specially 
as I'm here already .  One of
the features it provides is a user 
site-packages directory much in the same style that the PEP
proposes, so 
I'll comment on my experiences maintaining it.
>The PEP 370 (http://www.py
thon.org/dev/peps/pep-0370) "per user site
>packages directory" has several open questions:
>
>* Are the directories for Windows, Mac and Unix fine?
>* Mac: Should framework and non-framework builds of
Python use
>  the same directories?

One of the problems we've encountered over and over again
with 
Combinator is that MacOS has a dual personality.  Its
personality is not 
just an issue of framework vs. non-framework build, but a
fundamental 
question of "is this platform a UNIX or is it not a
UNIX".

An example: Let's say you have a GNU autotools project in C,
which we'll 
call "Xxx", and a set of Python bindings, PyXxx. 
Combinator deals with 
this by using ~/.local, and providing scripts to set up PATH
and 
LD_LIBRARY_PATH to point to ~/.local/bin and ~/.local/lib,
respectively. 
I'm not suggesting that Python come with such a tool
(although it might 
be handy, at least until this convention catches on with
distributors), 
but it should at least make it clear how one would achieve
the following 
straightforwardly:

  cd xxx-0.2.4
  ./configure --prefix ~/.local
  make
  make install
  cd ../pyxxx-0.0.1
  python setup.py install --prefix ~/.local

Using Combinator, the user is now fully set up to
"import xxx".  But 
only if they're using Linux, because I made the same mistake
(which we 
will probably be correcting at some point soon) of using
Python's 
*existing* per-user site-packages directory of
~/Library/Python on the 
mac, and not adding ~/.local.

On the Mac, our user now has a problem: given that
~/Library/Python/ 
doesn't follow the /usr or /usr/local style filesystem
layout, 
autotools-based projects will not build their libraries in
the right 
places using a single command-line option.  (I guess maybe
you could do 
something with --bindir and --sbindir and --exec-prefix or
whatever, I 
don't know.  I tend to get bored and wander off when I have
to pass more 
than 3 options to a configure script.)

A user familiar with the Mac alone might know what to do at
this point 
(to be honest, I don't!), but I do know that people familiar
with both 
platforms are confused by this apparently gratuitous
inconsistency. 
They follow familiar procedure on a Linux box, it works
great, then they 
do the same thing on a Mac (with the same shell, an
apparently similar 
version of Python) and it doesn't.  Keep in mind that
"./configure 
--prefix" is familiar procedure from a lot more places
than Combinator. 
For example, on shared hosting where I didn't have root,
I've used this 
same trick without Combinator, building Python itself with
--prefix 
~/.local and editing .bashrc to modify the appropriate env
vars.

My suggestion would be to have *both* ~/.local *and*
~/Library/Python be 
honored on the Mac, because there really isn't much harm in
doing so. 
Perhaps it would make sense for non-framework builds to not
honor 
~/Library/Python, but I am pretty certain, based on
experience fielding 
requests from confused users, that framework builds should
still honor 
~/.local/lib/python.../site-packages.

Windows has this problem less because everything has to be
done so 
completely differently.

In any event, the really important thing from my perspective
is that the 
PEP should explain how this very common use-case for
per-user 
installation of libraries can be accomplished on each of the
"big three" 
platforms.  This explanation should be put somewhere that
users can 
easily find it when they are building libraries.

I don't know what the "right way" to do this on
Windows is though, so I 
can't offer much help there.  Something to do with MinGW and
intense 
suffering, I would guess.
>* The patch also adds a usecustomize hook to site. Is it
useful and
>  should it stay?

Should this be "usercustomize"?  I thought it was
a typo but I see the 
same typo in the PEP.  I have often wished for something
like this for 
debugging, and it might have other uses, but there should be
a caution 
against abuse .
_______________________________________________
Python-Dev mailing list
Python-Devpython.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com

On 17 Jan, 2008, at 8:55, Christian Heimes wrote:

> The PEP 370 (http://www.py
thon.org/dev/peps/pep-0370) "per user site
> packages directory" has several open questions:
>
> * Are the directories for Windows, Mac and Unix fine?

The Mac directories look fine to me.

Is it worthwhile to note in the PEP that the directories for
the Mac  
are already used in Python 2.5?

>
> * Mac: Should framework and non-framework builds of
Python use
>  the same directories?

Yes, because that makes supporting users easier and keeps
things nice  
and simple (no need to ask which kind of Python they
installed to  
determine where their user directory is).  Distutils should
already  
link extensions in a way that allows you to have a python2.6
unix- 
build and framework-build that share the same site-packages
directory.

Ronald
_______________________________________________
Python-Dev mailing list
Python-Devpython.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com

Ronald Oussoren wrote:
> The Mac directories look fine to me.
> 
> Is it worthwhile to note in the PEP that the
directories for the Mac are
> already used in Python 2.5?

Good point!

>> * Mac: Should framework and non-framework builds of
Python use
>>  the same directories?
> 
> Yes, because that makes supporting users easier and
keeps things nice
> and simple (no need to ask which kind of Python they
installed to
> determine where their user directory is).  Distutils
should already link
> extensions in a way that allows you to have a python2.6
unix-build and
> framework-build that share the same site-packages
directory.

Have you read glyph's posting? He brought up some good
points why Python
on Mac should use ~/.local.

Framework builds have an additional import directory in
~/Library/Python

Christian
_______________________________________________
Python-Dev mailing list
Python-Devpython.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com

> * Are the directories for Windows, Mac and Unix fine?

Regarding Windows, I personally think its OK (but I also
personally think
the status-quo is OK too).  As has already been mentioned in
this thread,
Windows actually provides an API you can call to determine
various "well
known" folders.  I assume the PEP literally means
"the environment variable
APPDATA" instead of the result of
SHGetSpecialFolderPath(CSIDL_APPDATA); it
might be worth mentioning that (and ideally *why* the
env-var is preferred).
Similarly, a justification for why CSIDL_APPDATA/%APPDATA%
is preferred over
CSIDL_LOCAL_APPDATA - ie, why we are suggesting Python is a
"roaming"
application (eg, this implies that in certain Unix
environments, it is
likely that ~/.local/ will follow users as they log into
different machines
on the network - I've no idea how true that is) 

> * Should the Windows installer add
%APPDATA%/Python/Scripts to PATH?

I'd recommend not; IIUC, Python's main binary directory
isn't added to the
path, but even if it was, it will only be useful if .py is
also on PATHEXT.
Either way though, I'd suggest that particular decision is
outside the scope
of the PEP and a question for the windows installer assuming
the PEP be
accepted.  Maybe some general hand-waving along the lines of
"env vars on
Windows are left to the installer, where such global issues
are considered
in a wider context" would do 

Cheers,

Mark

_______________________________________________
Python-Dev mailing list
Python-Devpython.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com

Mark Hammond wrote:
>> * Are the directories for Windows, Mac and Unix
fine?
> 
> Regarding Windows, I personally think its OK (but I
also personally think
> the status-quo is OK too).  As has already been
mentioned in this thread,
> Windows actually provides an API you can call to
determine various "well
> known" folders.  I assume the PEP literally means
"the environment variable
> APPDATA" instead of the result of
SHGetSpecialFolderPath(CSIDL_APPDATA); it
> might be worth mentioning that (and ideally *why* the
env-var is preferred).
> Similarly, a justification for why
CSIDL_APPDATA/%APPDATA% is preferred over
> CSIDL_LOCAL_APPDATA - ie, why we are suggesting Python
is a "roaming"
> application (eg, this implies that in certain Unix
environments, it is
> likely that ~/.local/ will follow users as they log
into different machines
> on the network - I've no idea how true that is) 

Isn't SHGetSpecialFolderPath() XP and newer only? Python 2.6
is going to
support 2000 and newer. I've written a patch for the ntpath
module which
adds a getshellfolders() function. The function uses
SHGetFolderPathW to
build a dict of id to path mappings. The patch is available
at
http://bugs.python.
org/issue1763. Maybe you have some time to review it?

As far as I understand and remember the Windows API
SHGetFolderPath()
and friends are retrieving the information from the
registry. The shell
folder keys are (IIRC) REG_EXPAND_SZ keys which are expanded
with the
function ExpandEnvironmentStrings(). So at the end
CSIDL_APPDATA uses
the %APPDATA% env var -- unless I'm totally mistaken.

Or do you suggest that I'm using the win32 API calls instead
of the env var?

Point taken! I'll justify why I view Python as a roaming
app. All
company and university Linux boxes I've used in the past had
exported
$HOME via NFS. So ~/.local is roamed.

> I'd recommend not; IIUC, Python's main binary directory
isn't added to the
> path, but even if it was, it will only be useful if .py
is also on PATHEXT.
> Either way though, I'd suggest that particular decision
is outside the scope
> of the PEP and a question for the windows installer
assuming the PEP be
> accepted.  Maybe some general hand-waving along the
lines of "env vars on
> Windows are left to the installer, where such global
issues are considered
> in a wider context" would do 

*hehe* I love your idea.
It should be trivial to write a view lines of Python code
which modify
PATH in HKCU. I'm going to write a simple example and drop
it into
Tools/scripts/.

Christian

PS: Have you tested the new PCbuild directory for VS 2008?
Martin said
that you know how to create a new set of base addresses for
pre-linking.
dllbase_nt.txt is probably outdated.
_______________________________________________
Python-Dev mailing list
Python-Devpython.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com

On Thu, 17 Jan 2008 08:55:51 +0100, Christian Heimes
<listscheimes.de> wrote:
>
>* Should the site package directory also be ignored if
process
>  gid != effective gid?

If it should, I think the PEP should explain the attack this
defends
against in more detail.  The current brief mention of
"security issues"
is a bit hand-wavey.  For example, what is the relationship
between
security, this feature, and the PYTHONPATH environment
variable?  Isn't
the attack of putting malicious code into a user
site-packages directory
the same as the attack of putting it into a directory in
PYTHONPATH?

Jean-Paul
_______________________________________________
Python-Dev mailing list
Python-Devpython.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com

Jean-Paul Calderone wrote:
> If it should, I think the PEP should explain the attack
this defends
> against in more detail.  The current brief mention of
"security issues"
> is a bit hand-wavey.  For example, what is the
relationship between
> security, this feature, and the PYTHONPATH environment
variable?  Isn't
> the attack of putting malicious code into a user
site-packages directory
> the same as the attack of putting it into a directory
in PYTHONPATH?

The PYTHONPATH env var has the same security implications.
However a
user has multiple ways to avoid problems. For example the
user can use
the -E flag or set up sudo to ignore the environment.

The uid and gid tests aren't really required. They just
provide an extra
safety net if a user forgets to add the -s flag to a suid
app.

Christian

_______________________________________________
Python-Dev mailing list
Python-Devpython.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com

On 17 Jan, 2008, at 9:40, glyphdivmod.com wrote:

>
> On 07:55 am, listscheimes.de wrote:
>>
>
>> The PEP 370 (http://www.py
thon.org/dev/peps/pep-0370) "per user site
>> packages directory" has several open
questions:
>>
>> * Are the directories for Windows, Mac and Unix
fine?
>> * Mac: Should framework and non-framework builds of
Python use
>> the same directories?
>
> One of the problems we've encountered over and over
again with
> Combinator is that MacOS has a dual personality.  Its
personality is  
> not
> just an issue of framework vs. non-framework build, but
a fundamental
> question of "is this platform a UNIX or is it not
a UNIX".

IMHO the question is wrong. MacOS X is a UNIX platform (and
Apple has  
the paperwork to prove it), but at the same time it has a
non- 
traditional filesystem layout.

The "darwin" system at the foundation of OSX is a
classic unix system,  
with the usual filesystem layout, but the bits that make OSX
OSX use a  
completely diffent filesystem layout.

The framework build of Python definitly targets the upper
layer of the  
OSX stack, not just the Unix core. This sometimes causes
confusion,  
but mostly from developers wandering over from a Linux
system that  
complain that OSX isn't Linux.

Note that even Darwin is not Linux, there are several
differences that  
cause problems for naive porters. Two of those: Darwin uses
different  
binary formats for shared libraries and plugins;  the darwin
linker  
hardcodes the full path to shared libraries into executables
(without  
a runtime search path).

And that's only the start of fun, universal (fat) binaries
make live  
even more interesting.

>
>
> An example: Let's say you have a GNU autotools project
in C, which  
> we'll
> call "Xxx", and a set of Python bindings,
PyXxx.  Combinator deals  
> with
> this by using ~/.local, and providing scripts to set up
PATH and
> LD_LIBRARY_PATH to point to ~/.local/bin and
~/.local/lib,  
> respectively.

~/Library/Combinator would be a better installation root on
OSX, that  
location fits better into guidelines from Apple and also
avoids  
completely hiding the Combinator data from the user.

>
> I'm not suggesting that Python come with such a tool
(although it  
> might
> be handy, at least until this convention catches on
with  
> distributors),
> but it should at least make it clear how one would
achieve the  
> following
> straightforwardly:

This is probably off-topic for python-dev, but how is
combinator  
different from zc.buildout and virtualenv?

I'm particularly liking virtualenv at the moment as it
allows me to  
have seperate trees for projects without having several full
python  
installations.

>
>
>  cd xxx-0.2.4
>  ./configure --prefix ~/.local
>  make
>  make install
>  cd ../pyxxx-0.0.1
>  python setup.py install --prefix ~/.local
>
> Using Combinator, the user is now fully set up to
"import xxx".  But
> only if they're using Linux, because I made the same
mistake (which we
> will probably be correcting at some point soon) of
using Python's
> *existing* per-user site-packages directory of
~/Library/Python on the
> mac, and not adding ~/.local.

Why? Just because users can't remember on which platform
they are  
developing ? That
said, there's a libpython.a file in the framework  
build for basicly that reason: enough users complained about
there not  
being a python library they could link to that it was easier
to add a  
symlink then trying to educate everyone.

>
>
> On the Mac, our user now has a problem: given that
~/Library/Python/
> doesn't follow the /usr or /usr/local style filesystem
layout,
> autotools-based projects will not build their libraries
in the right
> places using a single command-line option.  (I guess
maybe you could  
> do
> something with --bindir and --sbindir and --exec-prefix
or whatever, I
> don't know.  I tend to get bored and wander off when I
have to pass  
> more
> than 3 options to a configure script.)

You could teach Combinator about running configure scripts
.


> A user familiar with the Mac alone might know what to
do at this point
> (to be honest, I don't!), but I do know that people
familiar with both
> platforms are confused by this apparently gratuitous
inconsistency.

This is not a gratuitous inconsistency.  The Mac developers
(mostly  
Jack Jansen, but I was involved as well) decided that
following the  
platform conventions is a good thing.

>
> They follow familiar procedure on a Linux box, it works
great, then  
> they
> do the same thing on a Mac (with the same shell, an
apparently similar
> version of Python) and it doesn't.  Keep in mind that
"./configure
> --prefix" is familiar procedure from a lot more
places than  
> Combinator.
> For example, on shared hosting where I didn't have
root, I've used  
> this
> same trick without Combinator, building Python itself
with --prefix
> ~/.local and editing .bashrc to modify the appropriate
env vars.
>
> My suggestion would be to have *both* ~/.local *and*
~/Library/ 
> Python be
> honored on the Mac, because there really isn't much
harm in doing so.
> Perhaps it would make sense for non-framework builds to
not honor
> ~/Library/Python, but I am pretty certain, based on
experience  
> fielding
> requests from confused users, that framework builds
should still honor
> ~/.local/lib/python.../site-packages.

As long as the default installation location is still
~/Library/Python/ 
X.Y for framework builds I wouldn't mind too much.


>
> Windows has this problem less because everything has to
be done so
> completely differently.
>
> In any event, the really important thing from my
perspective is that  
> the
> PEP should explain how this very common use-case for
per-user
> installation of libraries can be accomplished on each
of the "big  
> three"
> platforms.  This explanation should be put somewhere
that users can
> easily find it when they are building libraries.

I don't know if the PEP should mention this, but I do agree
that this  
should be in the Python documentation.


Ronald
_______________________________________________
Python-Dev mailing list
Python-Devpython.org
ht
tp://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/p
ython-dev/nessto%40sharedlog.com