Until the problems with "dumb spiders"[1] are
worked around, couldn't
something simple
be done such as using a spider trap[2]?
Add this to robots.txt:
User-agent: *
Disallow: /pypi?:action=bot_ban
Add a hidden link on the front page, then grab the ip of the
offending
spider and block it by adding
it to .htacess
Just a thought, as PyPI has been painfully slow for hours at
a time lately.
[1] http://mail.python.org/pipermail/catalog-sig/
2007-April/001055.html
[2] http://daniel
webb.us/software/bot-trap/
_______________________________________________
Catalog-sig mailing list
Catalog-sig python.org
h
ttp://mail.python.org/mailman/listinfo/catalog-sig
|