List Info

Thread: robots.txt




robots.txt
user name
2006-09-25 21:53:15
Hi All,

Wikipedia's robots.txt file (http://www.wikipe
dia.org/robots.txt) 
excludes robots from action pages (edit, history, etc.) with
this:

User-agent: *
Disallow: /w/

But in the interest of short URLs, I serve my MediaWiki
directly from 
site / without any /wiki/ or /w/ directories. So above
meathod would 
not work on my installation.

Any ideas how I can exclude robots from crawling all my
wiki's edit, 
history, talk, etc, pages *without* excluding its article
pages?

Thanks,

Roger Chrisman
http://Wikigogy.org
(MediaWiki 1.6.7)
_______________________________________________
MediaWiki-l mailing list
MediaWiki-lWikimedia.org
http://mail.wikipedia.org/mailman/listinfo/mediawiki-l

[1]

about | contact  Other archives ( Real Estate discussion Medical topics )