PyPI Enhancements (was: permissive trove classification)

Hi, 

As discussed earlier, I have started to gather in a wiki
page the ideas on
enhancing multiple servers support in Disutils, and the
changes needed in
Distutils and PyPI code.

http://wiki.
python.org/moin/EnhancedPyPI

I will continue the work in it after it has been commented,


Please comment !

Regards
Tarek 
-- 
View this message in context: htt
p://www.nabble.com/PyPI-Enhancements-%28was%3A-permissive-tr
ove-classification%29-tp14614933p14614933.html
Sent from the Python - catalog-sig mailing list archive at
Nabble.com.

_______________________________________________
Catalog-SIG mailing list
Catalog-SIGpython.org
h
ttp://mail.python.org/mailman/listinfo/catalog-sig

> As discussed earlier, I have started to gather in a
wiki page the ideas on
> enhancing multiple servers support in Disutils, and the
changes needed in
> Distutils and PyPI code.
> 
> http://wiki.
python.org/moin/EnhancedPyPI
> 
> I will continue the work in it after it has been
commented, 
> 
> Please comment !

Here you go:

"when PyPI is down or slow" - I believe I have
worked a lot
and successfully in making sure that this actually doesn't
happen.
For the last several months, PyPI wasn't slow, and it
wasn't
down for any significant period of time.

"several sections" - how do you use it from
distutils, or
setuptools?

"When a user call register or uplad with such a file,
it will be asked
for each server wheter he wants to perform the action over
it."
I don't understand that sentence. What is "such" a
file?
(wheter->whether, it will -> he will)

"Making PyPI permissive" - it's not clear whether
you want PyPI to
store these additional classifiers or not. If PyPI is
required
to store them, be prepared that the changes to PyPI will be
*very* difficult to implement. If PyPI is allowed to discard
them,
I see a number of alternatives:
a) make a list of "extra_classifiers" in setup.py,
which a server
   is free to discard.
b) give the list of extra classifiers in .pypirc, per
server.
   distutil's register command would then drop all
classifiers
   which are listed as server specific, unless registering
with
   that selected server

"Providing a base layer". It's not clear to me
what that actually
means. PyPI *does* have a separation of webui and store, so
it
is layered. Not sure what you are asking for.

Regards,
Martin
_______________________________________________
Catalog-SIG mailing list
Catalog-SIGpython.org
h
ttp://mail.python.org/mailman/listinfo/catalog-sig

Thanks for the feedback


"Martin v. Löwis" wrote:
> 
> "when PyPI is down or slow" - I believe I
have worked a lot
> and successfully in making sure that this actually
doesn't happen.
> For the last several months, PyPI wasn't slow, and it
wasn't
> down for any significant period of time.
> 

Yes this was a bit of a shortcut, as I didn't really mean to
say that PyPI
is slow.
It works pretty well. But having an alternative server can
be useful:

In my company, it happens that the network is really slow,
and some timeouts
may occurs when we work on building some software that calls
PyPI. These
builds occurs
all the time with the builder we use (zc.buildout). Having
an alternative
server that can be located in our network is useful in that
case. We could
use some
caching, but a 100% functional PyPI server is a better pick.

The typical configuration is to have a private PyPI -like
server that
contains all packages
needed to build our software (public or private packages)
and to use PyPI
itself.


"Martin v. Löwis" wrote:
> 
> "several sections" - how do you use it from
distutils, or
> setuptools?
> 
> "When a user call register or uplad with such a
file, it will be asked
> for each server wheter he wants to perform the action
over it."
> I don't understand that sentence. What is
"such" a file?
> (wheter->whether, it will -> he will)
> 

When a user call the register or the upload command with
such a file, it
will loop through every section, and ask the used at the
prompt if he wants
to perform the action over the given server.
I have changed the document to make it clearer. See
"How sections in .pypirc
are used"

The iw.dist module implements a prototype of such a change,
with two
commands called "mupload"
and "mregister", that are subclasses of
setuptools' upload and regsiter at
this time.


"Martin v. Löwis" wrote:
> 
> "Making PyPI permissive" - it's not clear
whether you want PyPI to
> store these additional classifiers or not. If PyPI is
required
> to store them, be prepared that the changes to PyPI
will be
> *very* difficult to implement. I
> 

No, the server wouldn't store them. It will just ignore
them. That
would be done by returning a simple warning when a category
in 
the received classifiers is unknown.


"Martin v. Löwis" wrote:
> 
> If PyPI is allowed to discard them,
> I see a number of alternatives:
> a) make a list of "extra_classifiers" in
setup.py, which a server
>    is free to discard.
> b) give the list of extra classifiers in .pypirc, per
server.
>    distutil's register command would then drop all
classifiers
>    which are listed as server specific, unless
registering with
>    that selected server
> 

An extra_classifiers would be interesting, but I don't think
storing these classifiers on client-side would be
maintainable, 
since it's up to the server to make this list evolve.


"Martin v. Löwis" wrote:
> 
> "Providing a base layer". It's not clear to
me what that actually
> means. PyPI *does* have a separation of webui and
store, so it
> is layered. Not sure what you are asking for.
> 

Yes, that was not clear at all, I included an example the
document.
I was just talking about having a separate interface that
exposes
the API for the storage and the web interface. This
interface
could be used as a common base for all PyPI-like server,
even if they don't use SQL to store data


Regards
Tarek

-- 
View this message in context: htt
p://www.nabble.com/PyPI-Enhancements-%28was%3A-permissive-tr
ove-classification%29-tp14614933p14646918.html
Sent from the Python - catalog-sig mailing list archive at
Nabble.com.

_______________________________________________
Catalog-SIG mailing list
Catalog-SIGpython.org
h
ttp://mail.python.org/mailman/listinfo/catalog-sig

On Jan 6, 2008 6:03 AM, Tarek Ziadé <ziade.tarekgmail.com> wrote:
> When a user call the register or the upload command
with such a file, it
> will loop through every section, and ask the used at
the prompt if he wants
> to perform the action over the given server.

This doesn't sound quite right to me.  The set of servers
used should
be explicit, and there shouldn't be an assumption that any
section
found is necessarily a server.

Perhaps something more like this would be appropriate:

  [distutils]
  index-servers = pypi mycompany

  [pypi]
  url = http://pypi.python.org/p
ypi/
  username = mypubliclogin
  password = mypassword

  [mycompany]
  url = http://pypi.example.com
/pypi/
  username = mycompanylogin
  password = mypassword

The default for distutils:index-servers could be pypi (and
the
pypi:url value would be defaulted as well).  (In this
structure, it's
almost unfortunate that the section storing the username and
password
is "server-login" instead of "pypi";
something would need to be done
to finesse that, or perhaps that section could be used to
provide
defaults for the username and password values for the
specific server
sections.

My main point, though, is that detecting the set of servers
shouldn't
be magical.

I don't really like the sort of interactivity you're
proposing, but I
suspect there are a variety of ways to deal with that as
well.  I've
some thoughts on that, but they're not horribly clear just
yet.


  -Fred

-- 
Fred L. Drake, Jr.    <fdrake at gmail.com>
"Chaos is the score upon which reality is
written." --Henry Miller
_______________________________________________
Catalog-SIG mailing list
Catalog-SIGpython.org
h
ttp://mail.python.org/mailman/listinfo/catalog-sig

> When a user call the register or the upload command
with such a file, it
> will loop through every section, and ask the used at
the prompt if he wants
> to perform the action over the given server.

I'm with Fred here. The distutils command line interface
should avoid
interactivity, and receive all input through command line
options.

If people still want interactive tools, they should build on
top of
distutils commands (or perhaps be additional commands).

> No, the server wouldn't store them. It will just ignore
them. That
> would be done by returning a simple warning when a
category in 
> the received classifiers is unknown.

Ok, that's fine.

>> "Providing a base layer". It's not clear
to me what that actually
>> means. PyPI *does* have a separation of webui and
store, so it
>> is layered. Not sure what you are asking for.
>>
> 
> Yes, that was not clear at all, I included an example
the document.
> I was just talking about having a separate interface
that exposes
> the API for the storage and the web interface. This
interface
> could be used as a common base for all PyPI-like
server,
> even if they don't use SQL to store data

Ok, so you are after separate storage mechanisms. I'm
opposed, as
I don't see why anybody would want to have that -
specifically,
what else do you want to store it in, and why? What's wrong
with
PostgreSQL?

It looks like a major complication in the code, for perhaps
just
one user (i.e. you).

Also, and again: pypi *already* has store.py. Is it not good
enough
to provide an alternative implementation of that class?
Where you
have list_classifiers(), store.Store has get_classifiers.
Where you
have file_upload, store.Store has add_file. So it seems to
me that
this request is already implemented, except that it doesn't
use
Zope interfaces.

Regards,
Martin
_______________________________________________
Catalog-SIG mailing list
Catalog-SIGpython.org
h
ttp://mail.python.org/mailman/listinfo/catalog-sig

Fred Drake wrote:
> 
> Perhaps something more like this would be appropriate:
> 
>   [distutils]
>   index-servers = pypi mycompany
> 
>   [pypi]
>   url = http://pypi.python.org/p
ypi/
>   username = mypubliclogin
>   password = mypassword
> 
>   [mycompany]
>   url = http://pypi.example.com
/pypi/
>   username = mycompanylogin
>   password = mypassword
> 
> The default for distutils:index-servers could be pypi
(and the
> pypi:url value would be defaulted as well).  (In this
structure, it's
> almost unfortunate that the section storing the
username and password
> is "server-login" instead of
"pypi"; something would need to be done
> to finesse that, or perhaps that section could be used
to provide
> defaults for the username and password values for the
specific server
> sections.
> 
> My main point, though, is that detecting the set of
servers shouldn't
> be magical.
> 

Yes I was wondering how it could look like, based on what we
have now.
I have change the proposal with your remarks.
For the [server-login] section, maybe we could drop it and
just
have a backward compatibilty mechanism. (see updated doc)


Fred Drake wrote:
> 
> I don't really like the sort of interactivity you're
proposing, but I
> suspect there are a variety of ways to deal with that
as well.  I've
> some thoughts on that, but they're not horribly clear
just yet.
> 

Ok, I'll answer on Martin mail on that 

Tarek

-- 
View this message in context: htt
p://www.nabble.com/PyPI-Enhancements-%28was%3A-permissive-tr
ove-classification%29-tp14614933p14653061.html
Sent from the Python - catalog-sig mailing list archive at
Nabble.com.

_______________________________________________
Catalog-SIG mailing list
Catalog-SIGpython.org
h
ttp://mail.python.org/mailman/listinfo/catalog-sig

"Martin v. Löwis" wrote:
> 
>> When a user call the register or the upload command
with such a file, it
>> will loop through every section, and ask the used
at the prompt if he
>> wants
>> to perform the action over the given server.
> 
> I'm with Fred here. The distutils command line
interface should avoid
> interactivity, and receive all input through command
line options.
> 
> If people still want interactive tools, they should
build on top of
> distutils commands (or perhaps be additional commands).
> 

Ok, sounds right, I have changed to doc accordingly


"Martin v. Löwis" wrote:
> 
> Ok, so you are after separate storage mechanisms. I'm
opposed, as
> I don't see why anybody would want to have that -
specifically,
> what else do you want to store it in, and why? What's
wrong with
> PostgreSQL?
> ...
> Also, and again: pypi *already* has store.py. Is it not
good enough
> to provide an alternative implementation of that class?
Where you
> have list_classifiers(), store.Store has
get_classifiers. Where you
> have file_upload, store.Store has add_file. So it seems
to me that
> this request is already implemented, except that it
doesn't use
> Zope interfaces.
> 
 
Ok, I dropped that part

So what remains in the document is:

- change the PyPI code so it warns on unexisting classifier
- change the .pypirc file so it has several sections (Fred's
model)
- change the distutils register and upload commands so it
works with the new
.pypirc file,
  and let the user choose the server in the command line 

Regards
Tarek
-- 
View this message in context: htt
p://www.nabble.com/PyPI-Enhancements-%28was%3A-permissive-tr
ove-classification%29-tp14614933p14653267.html
Sent from the Python - catalog-sig mailing list archive at
Nabble.com.

_______________________________________________
Catalog-SIG mailing list
Catalog-SIGpython.org
h
ttp://mail.python.org/mailman/listinfo/catalog-sig

> So what remains in the document is:
> 
> - change the PyPI code so it warns on unexisting
classifier
> - change the .pypirc file so it has several sections
(Fred's model)
> - change the distutils register and upload commands so
it works with the new
> .pypirc file,
>   and let the user choose the server in the command
line 

As a feature list, this sounds right. I'm curious how you'll
implement
the warning mechanism, though, through XML-RPC.

Regards,
Martin
_______________________________________________
Catalog-SIG mailing list
Catalog-SIGpython.org
h
ttp://mail.python.org/mailman/listinfo/catalog-sig

"Martin v. Löwis" wrote:
> 
>> So what remains in the document is:
>> 
>> - change the PyPI code so it warns on unexisting
classifier
>> - change the .pypirc file so it has several
sections (Fred's model)
>> - change the distutils register and upload commands
so it works with the
>> new
>> .pypirc file,
>>   and let the user choose the server in the command
line 
> 
> As a feature list, this sounds right. I'm curious how
you'll implement
> the warning mechanism, though, through XML-RPC.
> 

I have to look how XML-RPC is implemented in PyPI today,
but in regular mode the warning message will probably be a
simple
line like what we have right now, added before the OK in the
stream. 
For instance in file_upload:

  self.handler.send_response(200, 'OK')
  self.handler.set_content_type('text/plain')
  self.handler.end_headers()
  self.wfile.write('WARNING: ....n')
  self.wfile.write('OKn')

then make some xmlrpclibs.dumps for the xmlrpc version
but this is my first guess, as I need to look deeper in the
code. I guess it's more complicated than that 
if you are saying that

Regards
Tarek



-- 
View this message in context: htt
p://www.nabble.com/PyPI-Enhancements-%28was%3A-permissive-tr
ove-classification%29-tp14614933p14707904.html
Sent from the Python - catalog-sig mailing list archive at
Nabble.com.

_______________________________________________
Catalog-SIG mailing list
Catalog-SIGpython.org
h
ttp://mail.python.org/mailman/listinfo/catalog-sig

> then make some xmlrpclibs.dumps for the xmlrpc version
> but this is my first guess, as I need to look deeper in
the
> code. I guess it's more complicated than that 
> if you are saying that

Perhaps not - this sounds feasible. I was just puzzled by
your spec suggesting that the warning gets output before
the 200 status - that can't work on the wire in that order.

Regards,
Martin
_______________________________________________
Catalog-SIG mailing list
Catalog-SIGpython.org
h
ttp://mail.python.org/mailman/listinfo/catalog-sig