yazproxy and robots.txt

Hi,

is there any way of getting yazproxy to deliver a robots.txt
file when asked for by a client? I've noticed that our
instance of yazproxy (though pretty much unused at present)
is
getting trawled by the likes of Google. The Google bot
always
requests robots.txt before trying for anything else.

Thanks,

Ashley.
-- 
Ashley Sanders a.sandersmanchester.ac.uk
Copac http://copac.ac.uk --
A MIMAS service funded by JISC

_______________________________________________
Yazlist mailing list
Yazlistlists.indexdata.dk
http://lists.indexdata.dk/cgi-bin/mailman/listinfo/yaz
list

Ashley Sanders wrote:
> Hi,
> 
> is there any way of getting yazproxy to deliver a
robots.txt
> file when asked for by a client? I've noticed that our
> instance of yazproxy (though pretty much unused at
present) is
> getting trawled by the likes of Google. The Google bot
always
> requests robots.txt before trying for anything else.
> 
> Thanks,
> 
> Ashley.

Hi Ashley

No, unfortunately, not.  But I see your point.
This might be a very valuable and sensible thing to do ..
We see if we can make it in near future.

Marc

-- 

Marc Cromme, cand. polyt, Ph.D
Senior Developer, Project Manager

Index Data Aps
Købmagergade 43, 2
1150 Copenhagen K.
Denmark

tel: +45 3341 0100
fax: +45 3341 0101

http://www.indexdata.com

INDEX DATA Means Business
for Open Source and Open Standards





_______________________________________________
Yazlist mailing list
Yazlistlists.indexdata.dk
http://lists.indexdata.dk/cgi-bin/mailman/listinfo/yaz
list

Hi Ashley and Marc!

probably it is not part of design of yazproxy, but there is
a way to get
robots.txt file. ANY file, which located in the work
directory and open for
reading is allowed for anybody via HTTP.

% cd ~/tmp
% echo "User-agent: *" > robots.txt
% /usr/local/sbin/yazproxy -c config.xml -u $USER -l
yazproxy.log localgost:9210 &
% GET http://localhost:921
0/robots.txt
20:44:49-20/02 [log] 0 Set the proxy negotiation: charset to
'none', language to 'none'
20:44:49-20/02 [log] 1140446689:1 New session tcp:127.0.0.1
20:44:49-20/02 [log] file_access
User-agent
20:44:49-20/02 [log] 1140446689:1 2 Connection closed by
client
20:44:49-20/02 [log] 1140446689:1 2 Shutdown (client to
proxy)
20:44:49-20/02 [log] 1140446689:1 2 Closed 0/127 sent/recv
bytes total

I have a little question - do you have interest to apply
robots.txt to
say a bot about SRW/U databases?

Thanks,
Oleg


On Thu, Feb 16, 2006 at 04:48:19PM +0100, marc wrote:
> Ashley Sanders wrote:
> >Hi,
> >
> >is there any way of getting yazproxy to deliver a
robots.txt
> >file when asked for by a client? I've noticed that
our
> >instance of yazproxy (though pretty much unused at
present) is
> >getting trawled by the likes of Google. The Google
bot always
> >requests robots.txt before trying for anything
else.
> >
> >Thanks,
> >
> >Ashley.
> 
> Hi Ashley
> 
> No, unfortunately, not.  But I see your point.
> This might be a very valuable and sensible thing to do
..
> We see if we can make it in near future.
> 
> Marc
> 
> -- 
> 
> Marc Cromme, cand. polyt, Ph.D
> Senior Developer, Project Manager
> 
> Index Data Aps
> K?bmagergade 43, 2
> 1150 Copenhagen K.
> Denmark
> 
> tel: +45 3341 0100
> fax: +45 3341 0101
> 
> http://www.indexdata.com
> 
> INDEX DATA Means Business
> for Open Source and Open Standards
> 
> 
> 
> 
> 
> _______________________________________________
> Yazlist mailing list
> Yazlistlists.indexdata.dk
> http://lists.indexdata.dk/cgi-bin/mailman/listinfo/yaz
list

-- 
Oleg Kolobov                              | oleg (at)
lib.tpu.ru

_______________________________________________
Yazlist mailing list
Yazlistlists.indexdata.dk
http://lists.indexdata.dk/cgi-bin/mailman/listinfo/yaz
list

Hi Oleg,

> probably it is not part of design of yazproxy, but
there is a way to get
> robots.txt file. ANY file, which located in the work
directory and open for
> reading is allowed for anybody via HTTP.

You're right; thanks!

Worryingly it also allows anyone to download the yazproxy
config
file. I guess I need to move the config to a different
directory.

Regards,

Ashley.

-- 
Ashley Sanders a.sandersmanchester.ac.uk
Copac http://copac.ac.uk --
A MIMAS service funded by JISC

_______________________________________________
Yazlist mailing list
Yazlistlists.indexdata.dk
http://lists.indexdata.dk/cgi-bin/mailman/listinfo/yaz
list