python-central vs python-support

Hello,

now you know a bit better the policy (or at least the
generic idea, feel
free to discuss details further), it's time to think about
the
implementation (for modules and extensions).

References used:
- python-support source package in Debian
- python-central source package on http://peop
le.debian.org/~doko/pycentral/

1/ Making modules available to several python versions can
be done with
python-support or by python-central.

2/ Extensions can't be shared between several python
versions so they need to
be compiled once for each. The packaging needs to be
modified to do those
compilations. We really need a tool (maybe dh_python with a
special flag)
to generate dynamically the list of python version that the
package must
be compiled with. The .so files must be installed in
/usr/lib/python2.X/
and the associated .py files may be moved to a shared
directory (either
the python-support or python-central directory).

I agree with Matthias that it would be better to use only
one "helper
tool" but I would favor python-support instead of
python-central for the
reasons outlined below. Beware this is only *my* opinion.
I'll gladly
follow the consensus and would even accept to let both tools
coexist.

python-support:
* 247 lines of code
* maintained by Josselin Mouette (dh_python author)
* positive aspects
  * already in the archive, works, well tested
  * small, manageable
  * used by several packages in the python-modules team
* negative aspects
  * no control of which files is byte-compiled (compiles all
.py files of
    a directory given as argument to a update-python-modules
script)

pycentral:
* 1009 lines of code
* maintained by Matthias Klose (python maintainer)
* positive aspects
  * (more) fine-grained control of which file are
byte-compiled
  * already integrated in the python packaging
* negative aspects
  * integrate too tightly with dpkg's database (needed a
dpkg patch to
    make sure that the status file is up-to-date when
pycentral was
    scanning it!)
  * more complicated code, less manageable
  * not yet widely tested

python-central does more than python-support however. But
I'm not sure that
this "more" is useful. For example pycentral can
check that all installed
packages have support of a specific python-version.

Now, I would like to hear the opinion of others. Please take
some time to
check both tools and make your own opinion. Please tell here
the
advantages/disadvantages that you find to each tool. My list
is very
personal and certainly not exhaustive. 

In any case, the most important thing is that we need an
updated dh_python
which generates the right python dependencies (and provides)
based on the
new Python-version field, and which handles python-central
and
python-support in a sane way. I think dh_python should (by
default) not
move any file out of /usr/lib/python2.X/ but if it detects
either
/usr/share/python-central or /usr/share/python-support, then
it should add
the right postinst/postrm script snippet.

Josselin can you provide a dh_python following those rules?
Would anyone
else be ready to write it? (I could do it, I told so to doko
but in fact
my only interest here is to solve the current problems and
not to stay
heavily involved in the long term, so if anyone else has the
time that
would be great)

I also think that Josselin should provide the shell code
that needs to be
added to python itself to immediately bytecompile all
modules for any
newly installed python version.

Cheers,
-- 
Raphaël Hertzog

Premier livre français sur Debian GNU/Linux :
http://www.o
uaza.com/livre/admin-debian/


-- 
To UNSUBSCRIBE, email to debian-python-REQUESTlists.debian.org
with a subject of "unsubscribe". Trouble?
Contact listmasterlists.debian.org

On Sun, 2006-06-04 at 20:56 +0200, Raphael Hertzog wrote:
> Hello,
> 
> now you know a bit better the policy (or at least the
generic idea, feel
> free to discuss details further),

No. After the previous thread I am still in the dark on:
 - Tight upper dependencies. We you just incorrect, or are
they
   actually required?
 - python2.x-* packages -- are they needed? desirable?
   Steve and Matthias gave different answers, and if
they're present
   migrations end up just as fragile as they are now.

Until these two issues are clear, everything else is
irrelevant. The
goal shouldn't be to pick a tool, but to solve the problem.
If neither
tool solves the problem, or policy prevents us from using it
to solve
the problem, then this is just technical sophistry.

> 2/ Extensions can't be shared between several python
versions so they need to
> be compiled once for each. The packaging needs to be
modified to do those
> compilations. We really need a tool (maybe dh_python
with a special flag)
> to generate dynamically the list of python version that
the package must
> be compiled with. The .so files must be installed in
/usr/lib/python2.X/
> and the associated .py files may be moved to a shared
directory (either
> the python-support or python-central directory).

Was there consensus about whether or not extensions for all
versions
should be included in one binary package? This was not
mentioned in your
policy email. I'm ambivalent on the issue, I guess, but we
should choose
one way or the other.

> I agree with Matthias that it would be better to use
only one "helper
> tool" but I would favor python-support instead of
python-central for the
> reasons outlined below. Beware this is only *my*
opinion. I'll gladly
> follow the consensus and would even accept to let both
tools coexist.
> 
> python-support:
> * 247 lines of code
> * maintained by Josselin Mouette (dh_python author)
> * positive aspects
>   * already in the archive, works, well tested

This IMO is the most important part. It's here, and we're
using it.
python-central has "right arond the corner" for
months. Many modules are
already using -support, and it integrates very easily with
any project
using distutils. With proper debhelper integration it would
be even
simpler.
-- 
Joe Wreschnig <pimansacredchao.net>

Hi,

On Sun, 04 Jun 2006, Joe Wreschnig wrote:
> On Sun, 2006-06-04 at 20:56 +0200, Raphael Hertzog
wrote:
> > Hello,
> > 
> > now you know a bit better the policy (or at least
the generic idea, feel
> > free to discuss details further),
> 
> No. After the previous thread I am still in the dark
on:
>  - Tight upper dependencies. We you just incorrect, or
are they
>    actually required?

For extensions they are required. For arch:all modules
they're not
required since python-support / python-central can make them
instantly
available to a new python version. 

>  - python2.x-* packages -- are they needed? desirable?
>    Steve and Matthias gave different answers, and if
they're present
>    migrations end up just as fragile as they are now.

There's only python2.X-* _virtual_ packages. They are
desirable but not
needed unless a python application which doesn't use the
default python version
requires a specific python-version of a module.

However I don't see the problem of providing systematically
python2.X-foo
(in particular for extensions).

> Until these two issues are clear, everything else is
irrelevant. The
> goal shouldn't be to pick a tool, but to solve the
problem. If neither

Which problem? 

> > 2/ Extensions can't be shared between several
python versions so they need to
> > be compiled once for each. The packaging needs to
be modified to do those
> > compilations. We really need a tool (maybe
dh_python with a special flag)
> > to generate dynamically the list of python version
that the package must
> > be compiled with. The .so files must be installed
in /usr/lib/python2.X/
> > and the associated .py files may be moved to a
shared directory (either
> > the python-support or python-central directory).
> 
> Was there consensus about whether or not extensions for
all versions
> should be included in one binary package? This was not
mentioned in your
> policy email. I'm ambivalent on the issue, I guess,
but we should choose
> one way or the other.

Yes there was a clear consenus on that point. It's the only
way to have
sane inter-dependencies between modules and extensions
because otherwise 
modules depending on extensions do not know which package
they need to
depend on since it really depends on the running version of
python.

Cheers,
-- 
Raphaël Hertzog

Premier livre français sur Debian GNU/Linux :
http://www.o
uaza.com/livre/admin-debian/


-- 
To UNSUBSCRIBE, email to debian-python-REQUESTlists.debian.org
with a subject of "unsubscribe". Trouble?
Contact listmasterlists.debian.org

On Sun, 2006-06-04 at 23:05 +0200, Raphael Hertzog wrote:
> On Sun, 04 Jun 2006, Joe Wreschnig wrote:
> > On Sun, 2006-06-04 at 20:56 +0200, Raphael Hertzog
wrote:
> > > Hello,
> > > 
> > > now you know a bit better the policy (or at
least the generic idea, feel
> > > free to discuss details further),
> > 
> > No. After the previous thread I am still in the
dark on:
> >  - Tight upper dependencies. We you just
incorrect, or are they
> >    actually required?
> 
> For extensions they are required. For arch:all modules
they're not
> required since python-support / python-central can make
them instantly
> available to a new python version. 
> 
> >  - python2.x-* packages -- are they needed?
desirable?
> >    Steve and Matthias gave different answers, and
if they're present
> >    migrations end up just as fragile as they are
now.
> 
> There's only python2.X-* _virtual_ packages. They are
desirable but not
> needed unless a python application which doesn't use
the default python version
> requires a specific python-version of a module.
> 
> However I don't see the problem of providing
systematically python2.X-foo
> (in particular for extensions).
> 
> > Until these two issues are clear, everything else
is irrelevant. The
> > goal shouldn't be to pick a tool, but to solve
the problem. If neither
> 
> Which problem? 

The problem that Python migrations are currently very
fragile due to
strict dependencies and interconnectedness of many modules.

I'm finally beginning to piece together the conclusions of
the BoF, but
there were many missing details (and any other onlookers are
probably
even more confused than me). It's very difficult to discuss
this without
a clear statement of what policies were decided there, since
all emails
so far have only discussed part of the issue, or been vague.

It's hard to gauge whether python-support or python-central
is more
appropriate for the new policy without a clear statement of
what that
policy is. So here's *my* attempt at summarizing the BoF,
based on your
first mail and responses, and independent of the tools used:

1) Public modules and extensions should support all
available Python
versions, using a single binary package.

2) A new control field, XC-Python-Version will be used to
determine what
versions of Python a module supports.

3) The tight upper bound on module dependencies will be
removed,
provided the module actually works on future versions of
Python. The
upper bound on extension dependencies will not, because then
they
wouldn't work.

4) python2.x-* virtual packages are to be used only when
necessary, but
packages can provide them regardless.

5) Private modules and applications should use some way to
support more
than one Python version, if possible.

Is this accurate? 1), 3), and 4) contridict your original
email, but
match what you told me this time.

4) I am still unsure of, because Steve and Matthias gave
different
answers, and your answer is confusing -- if they're
desirable, let's
make them required; if not, let's not pollute the (virtual)
package
namespace unless we need them for a specific reason.

2) is suspect. It's not required for multiversion support
*in policy*.
The working python-support implementation in unstable, while
still
lacking such fields, proves that. I feel it's much more of
an
implementation issue than a policy one, and so if we don't
use
python-central (which I gather is the impetus for it), we
shouldn't
bother with the control field either.
-- 
Joe Wreschnig <pimansacredchao.net>

Joe Wreschnig writes:
> On Sun, 2006-06-04 at 20:56 +0200, Raphael Hertzog
wrote:
> > Hello,
> > 
> > now you know a bit better the policy (or at least
the generic idea, feel
> > free to discuss details further),
> 
> No. After the previous thread I am still in the dark
on:
>  - Tight upper dependencies. We you just incorrect, or
are they
>    actually required?

Tight upper dependency are required for packages containing
extension
modules.  For packages containing just modules, they not
strictly
required, but as soon as an application depending on a
non-standard
module, which the package doesn't yet provide, you have to
upload it
again to i.e. add the Provides: python2.5-foo. Note that we
can do
such an upload using a binary NMU.

>  - python2.x-* packages -- are they needed? desirable?
>    Steve and Matthias gave different answers, and if
they're present
>    migrations end up just as fragile as they are now.

No, not different answers. They may be needed, if an
extension get's
too big to be included in one package, or if the maintainer
of the
package does not want to package it that way. The
consequence is that
every module or extension depending directly or indirectly
on it must
be packages using the python2.x-* schema or else the
dependencies are
broken for non-default python versions.

> > 2/ Extensions can't be shared between several
python versions so they need to
> > be compiled once for each. The packaging needs to
be modified to do those
> > compilations. We really need a tool (maybe
dh_python with a special flag)
> > to generate dynamically the list of python version
that the package must
> > be compiled with. The .so files must be installed
in /usr/lib/python2.X/
> > and the associated .py files may be moved to a
shared directory (either
> > the python-support or python-central directory).
> 
> Was there consensus about whether or not extensions for
all versions
> should be included in one binary package? This was not
mentioned in your
> policy email. I'm ambivalent on the issue, I guess,
but we should choose
> one way or the other.

it was agreed to put them into one package, the advantage
beeing that
once you have support for the python version you switch from
and the
version you switch to, you do not have ro reupload the
package.

> >   * already in the archive, works, well tested
> 
> This IMO is the most important part. It's here, and
we're using it.
> python-central has "right arond the corner"
for months. Many modules are
> already using -support, and it integrates very easily
with any project
> using distutils. With proper debhelper integration it
would be even
> simpler.

it's incomplete, doesn't support extension modules, and
the author did
already refuse to support these, stating that this would be
too
complicated.  No status change in the last months. I'm
going to upload
python-central to unstable tomorrow, including a
dh_pycentral
debhelper, which could be included in debhelper as dh_python
as well.

  Matthias, mostly offline 'til Monday evening


-- 
To UNSUBSCRIBE, email to debian-python-REQUESTlists.debian.org
with a subject of "unsubscribe". Trouble?
Contact listmasterlists.debian.org

Joe Wreschnig writes:
> 1) Public modules and extensions should support all
available Python
> versions, using a single binary package.

s/available/supported/. we will try to keep the number of
supported
python versions/implementations minimal.

> 2) A new control field, XC-Python-Version will be used
to determine what
> versions of Python a module supports.

Plus XS-Python-Version, from which XC-Python-Version is
generated from.

> 3) The tight upper bound on module dependencies will be
removed,
> provided the module actually works on future versions
of Python. The
> upper bound on extension dependencies will not, because
then they
> wouldn't work.
> 
> 4) python2.x-* virtual packages are to be used only
when necessary, but
> packages can provide them regardless.

you don't know in advance, if they are necessary. Not a
problem, if
they are generated.

> 5) Private modules and applications should use some way
to support more
> than one Python version, if possible.

no, we should recommend or require that, or we can only
collapse the
pythonX-Y-foo packages into python-foo for a very small
number of
packages. Nothing new, see my analysis from February.

> 4) I am still unsure of, because Steve and Matthias
gave different
> answers, and your answer is confusing -- if they're
desirable, let's
> make them required; if not, let's not pollute the
(virtual) package
> namespace unless we need them for a specific reason.

you cannot predict if the package is used a dependency.

> 2) is suspect. It's not required for multiversion
support *in policy*.
> The working python-support implementation in unstable,
while still
> lacking such fields, proves that. I feel it's much
more of an
> implementation issue than a policy one, and so if we
don't use
> python-central (which I gather is the impetus for it),
we shouldn't
> bother with the control field either.

we agreed to use this information so that you can decide on
package
rebuilds based on the Sources and Packages files. See my
slides from
Debconf.

  Matthias


-- 
To UNSUBSCRIBE, email to debian-python-REQUESTlists.debian.org
with a subject of "unsubscribe". Trouble?
Contact listmasterlists.debian.org

Coin,

Matthias Klose <dokocs.tu-berlin.de> writes:

> s/available/supported/. we will try to keep the number
of supported
> python versions/implementations minimal.

Is there difference between available and supported versions
? What use
for an official python package if it is not supported ?

> Plus XS-Python-Version, from which XC-Python-Version is
generated from.

Why is this necessary ? You said you plan on having a
dh_pycentral
script, why can't this one do the job automagically ? Why
having the XC-
remain in package information then ?

>> 3) The tight upper bound on module dependencies
will be removed,
>> provided the module actually works on future
versions of Python. The
>> upper bound on extension dependencies will not,
because then they
>> wouldn't work.

I don't see any reason to keep this boundary, really, as
you plan to use
binNMUs on transitions. Either the extension does not work
on a newly
uploaded python version after being binNMU-ed, then this is
a bug (not
grave), or you use a boundary for an extension which would
have worked
perfectly and this is a bug too. As this is of the same
severity, i
would rather have to consider a bug for an upstream lack of
support for
a new version rather than for my extensions because of fear.

>> 4) python2.x-* virtual packages are to be used only
when necessary, but
>> packages can provide them regardless.
>
> you don't know in advance, if they are necessary. Not
a problem, if
> they are generated.

Yes i agree. But what a pollution. Now when Ruby and other
langages would
do the same, i'd like to know how long apt would be...

>> 2) is suspect. It's not required for multiversion
support *in policy*.
>> The working python-support implementation in
unstable, while still
>> lacking such fields, proves that. I feel it's much
more of an
>> implementation issue than a policy one, and so if
we don't use
>> python-central (which I gather is the impetus for
it), we shouldn't
>> bother with the control field either.
>
> we agreed to use this information so that you can
decide on package
> rebuilds based on the Sources and Packages files. See
my slides from
> Debconf.

Where are those slides ? Should we be aware where your
things are
stored ?

Don't you think this new field and Provides are redondant,
and that you
could decide which rebuilds are necessary from it ?

-- 
Marc Dequènes (Duck)

Em Dom, 2006-06-04 às 20:56 +0200, Raphael Hertzog
escreveu:
> I agree with Matthias that it would be better to use
only one "helper
> tool" but I would favor python-support instead of
python-central for the
> reasons outlined below. Beware this is only *my*
opinion. I'll gladly
> follow the consensus and would even accept to let both
tools coexist.

I'm with you on this. I believe python-support is the best
option
because it's simple and straight to the point. We can add
more
sophistication to it when, and if needed.

> python-central does more than python-support however.
But I'm not sure that
> this "more" is useful. For example
pycentral can check that all installed

Yeah; I don't think that more is useful at all right now. I
won't have
problems with transitioning my packages to use pycentral if
that's the
consensus, but I don't really see a need of doing so.

See you,

-- 
Gustavo Noronha Silva <kovdebian.org>
http://people.debian.o
rg/~kov/

On Mon, Jun 05, 2006 at 12:24:19AM +0200, Marc Dequènes
wrote:
> >> 3) The tight upper bound on module
dependencies will be removed,
> >> provided the module actually works on future
versions of Python. The
> >> upper bound on extension dependencies will
not, because then they
> >> wouldn't work.

> I don't see any reason to keep this boundary, really,
as you plan to use
> binNMUs on transitions. Either the extension does not
work on a newly
> uploaded python version after being binNMU-ed, then
this is a bug (not
> grave), or you use a boundary for an extension which
would have worked
> perfectly and this is a bug too.

No.  An extension has to be separately *compiled* for each
python version
it's to support.  A python-foo package containing
/usr/lib/python2.3/site-packages/foo/foo.so and
/usr/lib/python2.4/site-packages/foo/foo.so must not claim
to be compatible
with python (>= 2.5).

However, it *should* be possible to provide a toolchain such
that this
python-foo can be binNMUed when python-2.5 becomes available
and
automatically pick up support for it.

-- 
Steve Langasek                   Give me a lever long enough
and a Free OS
Debian Developer                   to set it on, and I can
move the world.
vorlondebian.org                                   http://www.debian.org/

On Sun, Jun 04, 2006 at 11:44:00PM +0200, Matthias Klose
wrote:
> > 4) python2.x-* virtual packages are to be used
only when necessary, but
> > packages can provide them regardless.

> you don't know in advance, if they are necessary. Not
a problem, if
> they are generated.

No, but they can be added on demand.  The difference is
merely a pair of
additional substitutions in the Depends and Provides fields.

Adding them where they *aren't* known to be needed by
reverse dependencies
does make transitions more brittle than they need to be --
at least in the
possible case of an overlap between a removal of one python
version and an
addition of another.

-- 
Steve Langasek                   Give me a lever long enough
and a Free OS
Debian Developer                   to set it on, and I can
move the world.
vorlondebian.org                                   http://www.debian.org/