List Info

Thread: Searchbots create hundreds of empty folders




Searchbots create hundreds of empty folders
user name
2007-03-23 18:44:19
I have been having a heck of a time with searchbots creating
hundreds
of empty folders in my "Pages" folder. These are
named pages of
obsolete URLs that were indexed months ago and deleted or
moved.

I reduced the problem by blocking many searchbots with
rewrite like this:

RewriteCond % 60.191.80.29 [OR]
RewriteCond % 74.6.73.98 [OR]
RewriteCond % 74.6.86.151 [OR]
RewriteCond % 74.6.87.123 [OR]
RewriteCond % 65.55.208..* [OR]
RewriteCond % 65.214.44.173 [OR]
RewriteCond % 66.231.189.78 [OR]
RewriteCond % 66.249.65.81 [OR]
RewriteCond % 71.168.107.138 [OR]
RewriteCond % 72.30.177.87 [OR]
RewriteCond % 72.30.177.93 [OR]
RewriteCond % 72.30.177.98 [OR]
RewriteCond % 74.6.67.212 [OR]
RewriteCond % 74.6.72.56 [OR]
RewriteCond % 74.6.70.58 [OR]
RewriteCond % 74.6.73.98 [OR]
RewriteCond % 74.6.86.151 [OR]
RewriteCond % 74.6.87.123 [OR]
RewriteCond % 81.52.143.15 [OR]
RewriteCond % 81.52.143.16 [OR]
RewriteCond % 82.165.167.199 [OR]
RewriteCond % 122.152.128.48 [OR]
RewriteCond % 128.194.135.94 [OR]
RewriteCond % 139.18.13.204 [OR]
RewriteCond % 193.47.80.49 [OR]
RewriteCond % 194.50.163.175 [OR]
RewriteCond % 202.160.179.131 [OR]
RewriteCond % 202.160.179.168 [OR]
RewriteCond % 202.160.180.184 [OR]
RewriteCond % 209.22.78.65 [OR]
RewriteCond % 220.181.19.173 [OR]
RewriteCond % 220.181.19.189 [OR]
RewriteCond % 220.181.34.182
RewriteRule .* - [F,L]


Unfortunately this does not solve the problem, because while
Apache
does not serve up a page the empty folders are still
created. I don't
understand why, because Apache should intercept the URL
before the
script runs.

This is Moin Version 1.5.5a, would a newer version put an
end to the
empty folders, or is there a configuration way to stop it?

-- Gnarlie
http://Gnarlodious.com/

------------------------------------------------------------
-------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the
chance to share your
opinions on IT & business topics through brief
surveys-and earn cash
http://www.techsay.com/default.
php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Moin-user mailing list
Moin-userlists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/moin-user


Re: Searchbots create hundreds of empty folders
user name
2007-03-23 19:11:45
Stranger and stranger. It turns out moinmoin page is
creating all
those empty folders, not a searchbot at all! I can load any
page
(apparently), but when I load my home page it does this. I
can delete
all the empties with a shell script. Then when I reload the
home page
it waits about 10 seconds, presumable while hundreds of
empty folders
are being created. Then it loads the page and renders
normally.

What in the world is causing this? Page caching? A wayward
Macro?

-- Gnarlie


On 3/23/07, Gnarlodious <lists.gnarlodiousgmail.com> wrote:
> I have been having a heck of a time with searchbots
creating hundreds
> of empty folders in my "Pages" folder. These
are named pages of
> obsolete URLs that were indexed months ago and deleted
or moved.
>
> I reduced the problem by blocking many searchbots with
rewrite like this:
>
> RewriteCond % 60.191.80.29 [OR]
> RewriteCond % 74.6.73.98 [OR]
> RewriteCond % 74.6.86.151 [OR]
> RewriteCond % 74.6.87.123 [OR]
> RewriteCond % 65.55.208..* [OR]
> RewriteCond % 65.214.44.173 [OR]
> RewriteCond % 66.231.189.78 [OR]
> RewriteCond % 66.249.65.81 [OR]
> RewriteCond % 71.168.107.138 [OR]
> RewriteCond % 72.30.177.87 [OR]
> RewriteCond % 72.30.177.93 [OR]
> RewriteCond % 72.30.177.98 [OR]
> RewriteCond % 74.6.67.212 [OR]
> RewriteCond % 74.6.72.56 [OR]
> RewriteCond % 74.6.70.58 [OR]
> RewriteCond % 74.6.73.98 [OR]
> RewriteCond % 74.6.86.151 [OR]
> RewriteCond % 74.6.87.123 [OR]
> RewriteCond % 81.52.143.15 [OR]
> RewriteCond % 81.52.143.16 [OR]
> RewriteCond % 82.165.167.199 [OR]
> RewriteCond % 122.152.128.48 [OR]
> RewriteCond % 128.194.135.94 [OR]
> RewriteCond % 139.18.13.204 [OR]
> RewriteCond % 193.47.80.49 [OR]
> RewriteCond % 194.50.163.175 [OR]
> RewriteCond % 202.160.179.131 [OR]
> RewriteCond % 202.160.179.168 [OR]
> RewriteCond % 202.160.180.184 [OR]
> RewriteCond % 209.22.78.65 [OR]
> RewriteCond % 220.181.19.173 [OR]
> RewriteCond % 220.181.19.189 [OR]
> RewriteCond % 220.181.34.182
> RewriteRule .* - [F,L]
>
>
> Unfortunately this does not solve the problem, because
while Apache
> does not serve up a page the empty folders are still
created. I don't
> understand why, because Apache should intercept the URL
before the
> script runs.
>
> This is Moin Version 1.5.5a, would a newer version put
an end to the
> empty folders, or is there a configuration way to stop
it?
>
> -- Gnarlie
> http://Gnarlodious.com/
>

------------------------------------------------------------
-------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the
chance to share your
opinions on IT & business topics through brief
surveys-and earn cash
http://www.techsay.com/default.
php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Moin-user mailing list
Moin-userlists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/moin-user


Re: Searchbots create hundreds of empty folders
user name
2007-03-23 20:28:02
It turns out the macro

[[RecentlyCreatedPages(numberOfPages=7,maxDaysOld=14)]]

creates hundreds of empty folders when the script is run. I
have
commented out the line and it works normally. The
misbehaving script
is on this page:
http:/
/moinmoin.wikiwikiweb.de/MacroMarket?action=AttachFile&d
o=get&target=RecentlyCreatedPages.py

-- Gnarlie


On 3/23/07, Gnarlodious <lists.gnarlodiousgmail.com> wrote:
> Stranger and stranger. It turns out moinmoin page is
creating all
> those empty folders, not a searchbot at all! I can load
any page
> (apparently), but when I load my home page it does
this. I can delete
> all the empties with a shell script. Then when I reload
the home page
> it waits about 10 seconds, presumable while hundreds of
empty folders
> are being created. Then it loads the page and renders
normally.
>
> What in the world is causing this? Page caching? A
wayward Macro?
>
> -- Gnarlie
>
>
> On 3/23/07, Gnarlodious <lists.gnarlodiousgmail.com> wrote:
> > I have been having a heck of a time with
searchbots creating hundreds
> > of empty folders in my "Pages" folder.
These are named pages of
> > obsolete URLs that were indexed months ago and
deleted or moved.
> >
> > I reduced the problem by blocking many searchbots
with rewrite like this:
> >
> > RewriteCond % 60.191.80.29 [OR]
> > RewriteCond % 74.6.73.98 [OR]
> > RewriteCond % 74.6.86.151 [OR]
> > RewriteCond % 74.6.87.123 [OR]
> > RewriteCond % 65.55.208..* [OR]
> > RewriteCond % 65.214.44.173 [OR]
> > RewriteCond % 66.231.189.78 [OR]
> > RewriteCond % 66.249.65.81 [OR]
> > RewriteCond % 71.168.107.138 [OR]
> > RewriteCond % 72.30.177.87 [OR]
> > RewriteCond % 72.30.177.93 [OR]
> > RewriteCond % 72.30.177.98 [OR]
> > RewriteCond % 74.6.67.212 [OR]
> > RewriteCond % 74.6.72.56 [OR]
> > RewriteCond % 74.6.70.58 [OR]
> > RewriteCond % 74.6.73.98 [OR]
> > RewriteCond % 74.6.86.151 [OR]
> > RewriteCond % 74.6.87.123 [OR]
> > RewriteCond % 81.52.143.15 [OR]
> > RewriteCond % 81.52.143.16 [OR]
> > RewriteCond % 82.165.167.199 [OR]
> > RewriteCond % 122.152.128.48 [OR]
> > RewriteCond % 128.194.135.94 [OR]
> > RewriteCond % 139.18.13.204 [OR]
> > RewriteCond % 193.47.80.49 [OR]
> > RewriteCond % 194.50.163.175 [OR]
> > RewriteCond % 202.160.179.131 [OR]
> > RewriteCond % 202.160.179.168 [OR]
> > RewriteCond % 202.160.180.184 [OR]
> > RewriteCond % 209.22.78.65 [OR]
> > RewriteCond % 220.181.19.173 [OR]
> > RewriteCond % 220.181.19.189 [OR]
> > RewriteCond % 220.181.34.182
> > RewriteRule .* - [F,L]
> >
> >
> > Unfortunately this does not solve the problem,
because while Apache
> > does not serve up a page the empty folders are
still created. I don't
> > understand why, because Apache should intercept
the URL before the
> > script runs.
> >
> > This is Moin Version 1.5.5a, would a newer version
put an end to the
> > empty folders, or is there a configuration way to
stop it?
> >
> > -- Gnarlie
> > http://Gnarlodious.com/
> >
>

------------------------------------------------------------
-------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the
chance to share your
opinions on IT & business topics through brief
surveys-and earn cash
http://www.techsay.com/default.
php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Moin-user mailing list
Moin-userlists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/moin-user


[1-3]

about | contact  Other archives ( Real Estate discussion Medical topics )