Photoblogs.org Down

Hey Everybody,

I know several of you have noticed that Photoblogs.org has
been going
down from time to time over the past few days.  I've just
figured out
that this is apparently being caused by some pretty bad
spammers.

Just wanted you to know that I've been able to pinpoint
where the
problem is and now I'm working on a solution.

Thanks for your patience and understanding.  I'll give
further updates
as they come.

Thanks,

-b

-- 
Brandon Stone
http://www.brandonstone.
com/

--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the
Google Groups "Photoblogs" group.
To post to this group, send email to photoblogsgooglegroups.com
To unsubscribe from this group, send email to
photoblogs-unsubscribegooglegroups.com
For more options, visit this group at 
http://groups-beta.google.com/group/photoblogs?hl=en
-~----------~----~----~----~------~----~------~--~---

Thanks for the heads-up Brandon - I was wondering what the
problem was.
G;ad to know you're on it 

Catherine
___________________________
http://www.catherinebuca
.com


--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the
Google Groups "Photoblogs" group.
To post to this group, send email to photoblogsgooglegroups.com
To unsubscribe from this group, send email to
photoblogs-unsubscribegooglegroups.com
For more options, visit this group at 
http://groups-beta.google.com/group/photoblogs?hl=en
-~----------~----~----~----~------~----~------~--~---

Hey All,

There are currently a couple threads in this discussion
group about
downtime and errors on Photoblogs.org.  Here's an update on
what's
happening with this...

Over the past week or so Photoblogs.org has been going down
a few
times a day for a few minutes at a time.  I've been talking
repeatedly
with Rackspace about fixing the intermittent downtime that
we're
seeing on Photoblogs.org, etc. (Rackspace is the company
that hosts
Photoblogs.org and related websites.)

What we've found is that Photoblogs.org is being attacked by
a botnet
that is activated every several hours at random intervals. 
There are
literally thousands of computers that come together all at
once and
attack just one URL on one of our sites. It looks like all
they're
trying to do is post spam to blog.photoblogs.org, but for
some reason
they're coordinating thousands of computers together and
basically
performing a DDOS attack which brings down the server.

Right now these are the options that are being presented to
me:
1)  Try to collect all the IP addresses in the botnet and
stop them
from reaching the server.
2)  Buy more memory for the server so that it can withstand
the attack
for longer.
3)  Get some kind of enterprise firewall solution that will
automatically filter out the botnet.
4)  Get another server and spread out the load, so that all
of the
sites will be less vulnerable to attacks.
5)  Try to optimize all the programming code as much as
possible, so
that when we are attacked the server won't become slow and
eventually
go down.

We've tried to do #1 (block the IPs) as much as possible,
but there
are a lot of machines in this botnet and it looks like we're
not able
to catch everything.  (It's also possible that they're
adding new
machines regularly and it may be difficult to stay ahead of
them.)

I may be able to do #2 (get more memory) but it will cost me
more
monthly server fees.

I'm not sure if #3 (get an enterprise firewall) is even
possible...
Rackspace said we don't have the right server configuration
for it.
(We need to be behind a load-balancer, which seems like
overkill to
me.)

Right now I can't afford to do #4 (get an additional
server.)

I'm actively working on #5 (optimizing my program code) but,
quite
honestly, a lot of the programming is already very
optimized, so I'm
not sure where I can even fix things.  The only thing I can
do is
resort to doing some caching that will make the site a
little less
dynamic and will require some real, solid programming time
to
implement.

At this point we've been able to hold off some of the
attacks, but not
all of them.  (We're going down less, but we're still going
down.)

I really appreciate your patience throughout all this and
I'm sorry it
has been going on for so many days...  Even though it's a
pretty
tricky problem to fix, I'm confident we'll come up with a
solution.

Let me know if any of you have thoughts on this.  I'd be
happy to hear
your opinions on how best to proceed.

Thanks,

Brandon

-- 
Brandon Stone
http://www.brandonstone.
com/

--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the
Google Groups "Photoblogs" group.
To post to this group, send email to photoblogsgooglegroups.com
To unsubscribe from this group, send email to
photoblogs-unsubscribegooglegroups.com
For more options, visit this group at 
http://groups-beta.google.com/group/photoblogs?hl=en
-~----------~----~----~----~------~----~------~--~---

I believe Lunarpages.com has a firewall that takes care of
DDOS
attacks on their servers. They are also very reasonable in
price. I
use them and I'm very happy!

Also this might help... an interesting article
http://bobcares.co
m/article33.html
also good info:
http:
//staff.washington.edu/dittrich/misc/ddos/

good luck - Joey

On 12/22/06, Brandon Stone <stonelbstone.com> wrote:
>
> Hey All,
>
> There are currently a couple threads in this discussion
group about
> downtime and errors on Photoblogs.org.  Here's an
update on what's
> happening with this...
>
> Over the past week or so Photoblogs.org has been going
down a few
> times a day for a few minutes at a time.  I've been
talking repeatedly
> with Rackspace about fixing the intermittent downtime
that we're
> seeing on Photoblogs.org, etc. (Rackspace is the
company that hosts
> Photoblogs.org and related websites.)
>
> What we've found is that Photoblogs.org is being
attacked by a botnet
> that is activated every several hours at random
intervals.  There are
> literally thousands of computers that come together all
at once and
> attack just one URL on one of our sites. It looks like
all they're
> trying to do is post spam to blog.photoblogs.org, but
for some reason
> they're coordinating thousands of computers together
and basically
> performing a DDOS attack which brings down the server.
>
> Right now these are the options that are being
presented to me:
> 1)  Try to collect all the IP addresses in the botnet
and stop them
> from reaching the server.
> 2)  Buy more memory for the server so that it can
withstand the attack
> for longer.
> 3)  Get some kind of enterprise firewall solution that
will
> automatically filter out the botnet.
> 4)  Get another server and spread out the load, so that
all of the
> sites will be less vulnerable to attacks.
> 5)  Try to optimize all the programming code as much as
possible, so
> that when we are attacked the server won't become slow
and eventually
> go down.
>
> We've tried to do #1 (block the IPs) as much as
possible, but there
> are a lot of machines in this botnet and it looks like
we're not able
> to catch everything.  (It's also possible that they're
adding new
> machines regularly and it may be difficult to stay
ahead of them.)
>
> I may be able to do #2 (get more memory) but it will
cost me more
> monthly server fees.
>
> I'm not sure if #3 (get an enterprise firewall) is even
possible...
> Rackspace said we don't have the right server
configuration for it.
> (We need to be behind a load-balancer, which seems like
overkill to
> me.)
>
> Right now I can't afford to do #4 (get an additional
server.)
>
> I'm actively working on #5 (optimizing my program code)
but, quite
> honestly, a lot of the programming is already very
optimized, so I'm
> not sure where I can even fix things.  The only thing I
can do is
> resort to doing some caching that will make the site a
little less
> dynamic and will require some real, solid programming
time to
> implement.
>
> At this point we've been able to hold off some of the
attacks, but not
> all of them.  (We're going down less, but we're still
going down.)
>
> I really appreciate your patience throughout all this
and I'm sorry it
> has been going on for so many days...  Even though it's
a pretty
> tricky problem to fix, I'm confident we'll come up with
a solution.
>
> Let me know if any of you have thoughts on this.  I'd
be happy to hear
> your opinions on how best to proceed.
>
> Thanks,
>
> Brandon
>
> --
> Brandon Stone
> http://www.brandonstone.
com/
>
> >
>

--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the
Google Groups "Photoblogs" group.
To post to this group, send email to photoblogsgooglegroups.com
To unsubscribe from this group, send email to
photoblogs-unsubscribegooglegroups.com
For more options, visit this group at 
http://groups-beta.google.com/group/photoblogs?hl=en
-~----------~----~----~----~------~----~------~--~---

A big, big thanks to everyone who contacted me off list with
different
suggestions and well wishes...  It's not possible for me to
respond to
everyone directly, but I hope you all know how much I
appreciate your
help and kind words.

I've been sorting through the existing problems and
suggested
solutions and I've come up with a plan that should stabilize
things
for us again.  I've decided to rearrange the server setup
that we have
for Photoblogs.org and it's related sites.  (This will make
it so that
the entire network of sites won't go down when there's just
a problem
with one site.)  We've basically outgrown our current
infrastructure,
so now it's time to move to the next level.

In order to accomplish this I'm going to set up a donation
system on
Photoblogs.org that should allow us to move forward with
much needed
hardware and additional programming resources.  I have big
plans for
Photoblogs.org in 2007 and we simply need more resources to
make this
a reality.

My goal is to get the servers set up and the donation system
working
within the next couple of weeks.  I'll send updates as
progress is
made.

Thanks again, everyone, for your support while we make it
through this
next stage of growing pains.

-b

On 12/22/06, Brandon Stone <stonelbstone.com> wrote:
> Hey All,
>
> There are currently a couple threads in this discussion
group about
> downtime and errors on Photoblogs.org.  Here's an
update on what's
> happening with this...
>
> Over the past week or so Photoblogs.org has been going
down a few
> times a day for a few minutes at a time.  I've been
talking repeatedly
> with Rackspace about fixing the intermittent downtime
that we're
> seeing on Photoblogs.org, etc. (Rackspace is the
company that hosts
> Photoblogs.org and related websites.)
>
> What we've found is that Photoblogs.org is being
attacked by a botnet
> that is activated every several hours at random
intervals.  There are
> literally thousands of computers that come together all
at once and
> attack just one URL on one of our sites. It looks like
all they're
> trying to do is post spam to blog.photoblogs.org, but
for some reason
> they're coordinating thousands of computers together
and basically
> performing a DDOS attack which brings down the server.
>
> Right now these are the options that are being
presented to me:
> 1)  Try to collect all the IP addresses in the botnet
and stop them
> from reaching the server.
> 2)  Buy more memory for the server so that it can
withstand the attack
> for longer.
> 3)  Get some kind of enterprise firewall solution that
will
> automatically filter out the botnet.
> 4)  Get another server and spread out the load, so that
all of the
> sites will be less vulnerable to attacks.
> 5)  Try to optimize all the programming code as much as
possible, so
> that when we are attacked the server won't become slow
and eventually
> go down.
>
> We've tried to do #1 (block the IPs) as much as
possible, but there
> are a lot of machines in this botnet and it looks like
we're not able
> to catch everything.  (It's also possible that they're
adding new
> machines regularly and it may be difficult to stay
ahead of them.)
>
> I may be able to do #2 (get more memory) but it will
cost me more
> monthly server fees.
>
> I'm not sure if #3 (get an enterprise firewall) is even
possible...
> Rackspace said we don't have the right server
configuration for it.
> (We need to be behind a load-balancer, which seems like
overkill to
> me.)
>
> Right now I can't afford to do #4 (get an additional
server.)
>
> I'm actively working on #5 (optimizing my program code)
but, quite
> honestly, a lot of the programming is already very
optimized, so I'm
> not sure where I can even fix things.  The only thing I
can do is
> resort to doing some caching that will make the site a
little less
> dynamic and will require some real, solid programming
time to
> implement.
>
> At this point we've been able to hold off some of the
attacks, but not
> all of them.  (We're going down less, but we're still
going down.)
>
> I really appreciate your patience throughout all this
and I'm sorry it
> has been going on for so many days...  Even though it's
a pretty
> tricky problem to fix, I'm confident we'll come up with
a solution.
>
> Let me know if any of you have thoughts on this.  I'd
be happy to hear
> your opinions on how best to proceed.
>
> Thanks,
>
> Brandon
>
> --
> Brandon Stone
> http://www.brandonstone.
com/
>


-- 
Brandon Stone
http://www.brandonstone.
com/

--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the
Google Groups "Photoblogs" group.
To post to this group, send email to photoblogsgooglegroups.com
To unsubscribe from this group, send email to
photoblogs-unsubscribegooglegroups.com
For more options, visit this group at 
http://groups-beta.google.com/group/photoblogs?hl=en
-~----------~----~----~----~------~----~------~--~---