Hot weather and power outages continue

Due to the hot weather in many parts of the US, there have
been various
power outages.  Some large outages have been caused by
severe storms, but
mostly the heat has just overloaded power distribution
equipment.

The good news so far is the Net has shown very few
disruptions due to
the heat and some multi-day power outages.  The major ISP
access providers
have been indicating about average numbers of outages in
their networks,
i.e. even during "normal" times, there are
usually a few tens of
thousands of lines down nationwide.

While its expected for individual customers to go down
during power
outages, usually because the customer does not have local
backup power, it
is less common for major web sites and co-location centers
to experience
downtime during power outages.

MySpace.com reported it was down tonight due to power
problems, with a
temporary page up now. MySpace.com is hosted at a few
Internet
co-location data centers (Equinix and CWIE).

On Mon, Jul 24, 2006 at 02:22:26AM -0400, Sean Donelan
wrote:
> 
> Due to the hot weather in many parts of the US, there
have been various
> power outages.  Some large outages have been caused by
severe storms, but
> mostly the heat has just overloaded power distribution
equipment.
> 
> The good news so far is the Net has shown very few
disruptions due to
> the heat and some multi-day power outages.  The major
ISP access providers
> have been indicating about average numbers of outages
in their networks,
> i.e. even during "normal" times, there are
usually a few tens of
> thousands of lines down nationwide.

I'm surprised nobody said anything about the (apparently
regional) utility 
outage in NoVA on Saturday. Equinix Ashburn was running on
generator for 
several hours, but apparently the SAVVIS facility down the
road a few 
miles in Sterling (old Exodus facility) didn't fare so
well. The latest 
story I heard was that they lost 14 out of 16 chillers and
customers had 
to send techs in in shifts because it reached over 130F
inside.

Come on Sean, this "very few disruptions" stuff
is below your usual 
standards. The least you can do to help us pass the time in
this damn heat 
is to recount a few good stories about routers you could
scramble eggs on. 


-- 
Richard A Steenbergen <rase-gerbil.net>       http://www.e-gerbil.net/r
as
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41
5ECA F8B1 2CBC)

On Mon, 24 Jul 2006, Richard A Steenbergen wrote:
> I'm surprised nobody said anything about the
(apparently regional) utility
> outage in NoVA on Saturday.

There have been several around the country.  Secaucus, NJ,
various
cities near Los Angeles and the SF Bay Area, not to mention
the power
quality goes to hell when the grid gets very hot.  I've
been a bit
surprised we haven't heard about more equipment failures.

> Equinix Ashburn was running on generator for
> several hours, but apparently the SAVVIS facility down
the road a few
> miles in Sterling (old Exodus facility) didn't fare so
well. The latest
> story I heard was that they lost 14 out of 16 chillers
and customers had
> to send techs in in shifts because it reached over 130F
inside.

I've always been a fan of being able to force 100%
economizer and chiller
loop bypass emergency operation; it won't keep you
"cool" but will help
keep your data center from turning into an Easy-Bake
Oven(tm). But that
failure operating mode is rarely part of the standard HVAC
programming.
You are probably still going to need to replace some toasted
disks
and chips afterwards.  There is a reason why NEBS equipment
costs more.

> Come on Sean, this "very few disruptions"
stuff is below your usual
> standards. The least you can do to help us pass the
time in this damn heat
> is to recount a few good stories about routers you
could scramble eggs on.
> 

I remember back in the old days, when all we had were AGS+
routers in
wiring closets without any air conditioning .... We had to
carry them
uphill, both ways.

On Mon, 24 Jul 2006, Richard A Steenbergen wrote:
>
> Come on Sean, this "very few disruptions"
stuff is below your usual
> standards. The least you can do to help us pass the
time in this damn heat
> is to recount a few good stories about routers you
could scramble eggs on.
> 

there is a funny story of some dial devices on fire, and
still passing
packets...

"Christopher L. Morrow"
<christopher.morrowverizonbusiness.com> writes:

> On Mon, 24 Jul 2006, Richard A Steenbergen wrote:
>>
>> Come on Sean, this "very few
disruptions" stuff is below your usual
>> standards. The least you can do to help us pass the
time in this damn heat
>> is to recount a few good stories about routers you
could scramble eggs on.
>> 
>
> there is a funny story of some dial devices on fire,
and still passing
> packets...

and an equally funny story of said devices being held up in
customs in
a particular european country because they said
"TNT" on the outside
of their crates...

                                        ---rob

Robert E.Seastrom wrote:
> 
> "Christopher L. Morrow"
<christopher.morrowverizonbusiness.com> writes:
> 
>> On Mon, 24 Jul 2006, Richard A Steenbergen wrote:
>>> Come on Sean, this "very few
disruptions" stuff is below your usual
>>> standards. The least you can do to help us pass
the time in this damn heat
>>> is to recount a few good stories about routers
you could scramble eggs on.
>>> 
>> there is a funny story of some dial devices on
fire, and still passing
>> packets...
> 
> and an equally funny story of said devices being held
up in customs in
> a particular european country because they said
"TNT" on the outside
> of their crates...

I ordered a new personal PC back in March(?) from Lenovo
(discount 
overstock offering).  Everything shipped immediately but was
delayed in 
transit, due to a "Live Entity" inspection hold
placed on it by the US 
FDA.  The packing list included an item identified as
"mouse" (it was 
right under the item "keyboard").  I'm waiting
for nVidia or ATI to come 
out with a next-gen product named "Nuclear
XForce" or "Plutonium 
Wonder".  

-Jim P.

--On July 24, 2006 2:22:26 AM -0400 Sean Donelan
<seandonelan.com> wrote:

> While its expected for individual customers to go down
during power
> outages, usually because the customer does not have
local backup power, it
> is less common for major web sites and co-location
centers to experience
> downtime during power outages.

Except if you're in Qwest territory.  Apparently they
don't put any battery 
backup at their mini-DSLAMs and such.  Every time we lose
power, I'm still 
up, but the DSL signal goes away.  Haven't checked
dialtone, but I keep 
meaning too during the next outage.

Now I know it's not exactly fair singling out Qwest,
because I'll bet 
Verizon and others share the same thing, and I'm pretty
sure it's just 
their ADSL service and not the  voice service (I haven't
checked though) 
it's still becoming more and more common that as an
individual user your 
connection to the internet, unless you're paying for
something other than 
ADSL or Cable, will be just as affected by local power
outages.

Indeed, my RoadRunner connection is the same way.  All of my
stuff stays up,
but "teh Interweb is broken." I'm guessing that
they (DSL/CableCo's) find it
too cost-prohibitive to roll out UPSes to the customer
aggregation points.
Suprisingly, my cable TV goes out as well when the power
goes, so it might
just be more than the CMTS that's going out.

-----Original Message-----
From: owner-nanogmerit.edu [mailto:owner-nanogmerit.edu] On Behalf Of
Michael Loftis
Sent: Monday, July 24, 2006 16:20
To: nanogmerit.edu
Subject: Re: Hot weather and power outages continue




--On July 24, 2006 2:22:26 AM -0400 Sean Donelan
<seandonelan.com> wrote:

> While its expected for individual customers to go down
during power 
> outages, usually because the customer does not have
local backup 
> power, it is less common for major web sites and
co-location centers 
> to experience downtime during power outages.

Except if you're in Qwest territory.  Apparently they
don't put any battery
backup at their mini-DSLAMs and such.  Every time we lose
power, I'm still
up, but the DSL signal goes away.  Haven't checked
dialtone, but I keep
meaning too during the next outage.

Now I know it's not exactly fair singling out Qwest,
because I'll bet
Verizon and others share the same thing, and I'm pretty
sure it's just their
ADSL service and not the  voice service (I haven't checked
though) it's
still becoming more and more common that as an individual
user your
connection to the internet, unless you're paying for
something other than
ADSL or Cable, will be just as affected by local power
outages.

On Mon, 24 Jul 2006, Robert E. Seastrom wrote:

>
> "Christopher L. Morrow"
<christopher.morrowverizonbusiness.com> writes:
>
> > On Mon, 24 Jul 2006, Richard A Steenbergen wrote:
> >>
> >> Come on Sean, this "very few
disruptions" stuff is below your usual
> >> standards. The least you can do to help us
pass the time in this damn heat
> >> is to recount a few good stories about routers
you could scramble eggs on.
> >> 
> >
> > there is a funny story of some dial devices on
fire, and still passing
> > packets...
>
> and an equally funny story of said devices being held
up in customs in
> a particular european country because they said
"TNT" on the outside
> of their crates...

Now, where was youtube and portable digital camcorders back
when these
things were going on? This is an excellent application for
all that super
surveillance the DHS wants to do all the time! 
life-bloopers.com!

> -----Original Message-----
> From: owner-nanogmerit.edu [mailto:owner-nanogmerit.edu] On Behalf Of
> Sean Donelan
> Sent: 24 July 2006 13:27
> To: nanogmerit.edu
> Subject: Re: Hot weather and power outages continue
>
>
> I've always been a fan of being able to force 100%
economizer and chiller
> loop bypass emergency operation; it won't keep you
"cool" but will help
> keep your data center from turning into an Easy-Bake
Oven(tm). But that
> failure operating mode is rarely part of the standard
HVAC programming.

Sean,

Can you elaborate on what you mean by " force 100%
economizer and chiller
loop bypass emergency operation"

Thanks,

Sam