Re: Surgeplus url root folder

Thread: Re: Surgeplus url root folder

Search this list only Search all lists
Re: Surgeplus url root folder
Germany	2007-08-27 14:41:47
Hi Barry, > > Put the robots.txt into /surgemail/www/robots.txt - which will protect > (supposedly) the whole server. Unfortunately this doesn't work. > > This brings up an interesting question.... > Do our customers wish their Blogs and SurgePlus public directories to > be indexed by the search engines? A suggestion to NetWin is to place a > new setting on the Blogs creation page and the Public directory page > asking "Do you wish this directory to be indexed by the search > engines? Yes | No Obviously, admins who put the robots.txt in the root > www dir will circumventthe customer's request -- for the legit search > engines anyway. For me this setting would be very helpful. The Random folder is not a public folder. It is neither searchable by bots nor by humans without prior knowledge of the exact URL. This URL is created automatically in a random manner when you place some files in the Random folder (something like this ...../Random/1234567890-5678901234/file). So it is intended to hide files from prying eyes. You just share this URL with people of you choice. But I have a simlple workaround until Netwin provides us with a better solution: I told my user to replace the hyphen in the Random folder by an underscore prior to sending the URL and to instruct her colleagues to replace it again with a hyphen. This is easy for humans, but impossible for stupd bots. > > 'Yer welcome. Glad to be of assistance. > BarryZ > 1USA > > > ----- Original Message ----- From: kvisserfz-borstel.de > To: surgemail-listnetwinsite.com > Sent: Saturday, August 25, 2007 2:53 PM > Subject: Re: [SurgeMail List] Surgeplus url root folder > > Hi, > > Barry, thank you for your response. But this did not answer > my question. Maybe I should be more precise and explain what > I am looking for. > > >Each customer account has their own files directories.One > >for sharing and another for Private. On our server, It's > >under /mbox/<account name>/mdir/xfile/web/image.gif That > >way, when the account is deleted, all the customer's files > >are deleted too. > > > >To find out for sure, login to Webmail as the customer, > >then upload a uniquely-named file, such as > >corvette-clock.gif or something like that. Then goto the > >mbox directory and do a Find for that filename. That will > >show you where the files are kept. > > > > I know where I can find the user files but in this case I > wanted to add a robots.txt file to the root directory of the > website (http://surgemailserver.domain/robots.txt) where it > can be accessed by search bots. The robots.txt should just > contain these two records > > User-agent: * > Disallow: / > > to prevent any search bots from crawling the Surgeplus > Random directory. I tried all the different web directories > under /usr/local/surgemail ( on Solaris): web, www, work, > web_work, web_latest, but I always get: > > 'The file you requested does not exist. The url may be > incorrect. > Requested File: (robots.txt)' > > As the Surgemail/Webmail http engine is not a full-fledged > web server, this might be impossible to do, I don't know. > Maybe Surgemail/Webmail uses some rewriting rules for server > access. (When using Apache it works fine if you place the > robots.txt into the htdocs directory) > The reason why I want to stop search bots: When a user loads > some files into her Random folder via Surgeplus and sends > the url to some colleagues with accounts e.g. at yahoo.com > , seconds later this very > url is crawled by a search bot. This means that the emails > have been parsed and the urls found are used for crawling > the website immediately. > And the bots come back almost every day. > I have observed several search bots now via our firewall > logs, from Yahoo as well as from MSN. The files in the > Random directory are not intended for unauthorized access. > The bots search for the robots.txt first before they start > crawling as I have observed, and so I hope I can stop them. > > > >Hi, > > > >can someone point me to the root folder of the Surgeplus > >filesharing url > >(http://surgemailserver.domain:80/users/username/Random/... > >..) to place a robots.txt file in it ? > > > >Thanks in advance. > > > > Any help and advice is very much appreciated. Klaus Visser Research Center Borstel Center for Medicine and Biosciences Germany ------------------------------------- Dr. Klaus Visser Research Center Borstel Parkallee 1-40 23845 Borstel, Germany Tel/Fax +49-4537-188-605/783 -------------------------------------