San Francisco Power Outage

Just a heads up to anyone on list that PG&E has just
sustained a large
outage in San Francisco that has caused a few hiccups (both
network,
electrical, infrastructural, etc.) around the city.

I've confirmed that both customers in 365 Main and parts of
telecom 1
have both sustained brief blackouts. No word yet form 200
Paul.

Anyone in the area that could use a hand with anything, I'll
probably
be wrapping up fixes for my stuff soon, and would be glad to
help
however I can.

Cheers,
jonathan

-- 
Jonathan Lassoff
echo thejof | sed 's/^/jof/;s/$/.com/'
http://thejof.com
415-215-2464
GPG: 0xC8579EE5

Jonathan Lassoff wrote:
> 
> Just a heads up to anyone on list that PG&E has
just sustained a large
> outage in San Francisco that has caused a few hiccups
(both network,
> electrical, infrastructural, etc.) around the city.
> 
> I've confirmed that both customers in 365 Main and
parts of telecom 1
> have both sustained brief blackouts. No word yet form
200 Paul.
> 
> Anyone in the area that could use a hand with anything,
I'll probably
> be wrapping up fixes for my stuff soon, and would be
glad to help
> however I can.
> 

I have a question: does anyone seriously accept "oh,
power trouble" as a 
reason your servers went offline? Where's the generators?
UPS? Testing 
said combination of UPS and generators? What if it was
important? I 
honestly find it hard to believe anyone runs a facility like
that and 
people actually *pay* for it.

If you do accept this is a good reason for failure, why?

~Seth

On Tue, Jul 24, 2007, Seth Mattinen wrote:

> I have a question: does anyone seriously accept
"oh, power trouble" as a 
> reason your servers went offline? Where's the
generators? UPS? Testing 
> said combination of UPS and generators? What if it was
important? I 
> honestly find it hard to believe anyone runs a facility
like that and 
> people actually *pay* for it.

> If you do accept this is a good reason for failure,
why?

Didn't you read? He paid extra for super-reliable power from
his
electricity provider..



Adrian

On Jul 24, 2007, at 6:54 PM, Seth Mattinen wrote:
>
> I have a question: does anyone seriously accept
"oh, power trouble"  
> as a reason your servers went offline? Where's the
generators? UPS?  
> Testing said combination of UPS and generators? What if
it was  
> important? I honestly find it hard to believe anyone
runs a  
> facility like that and people actually *pay* for it.
>

Sad that the little Telcove DC here in Lancaster, PA, that
Level3  
bought a few months ago, has weekly full-on generator tests
where  
100% of the load is transferred to the generator, while
apparently  
large DCs that are charging premium rates, do not.

Cordially

Patrick Giagnocavo
patrickzill.net

sethmrollernet.us (Seth Mattinen) writes:

> I have a question: does anyone seriously accept
"oh, power trouble" as a 
> reason your servers went offline? Where's the
generators? UPS? Testing 
> said combination of UPS and generators? What if it was
important? I 
> honestly find it hard to believe anyone runs a facility
like that and 
> people actually *pay* for it.
> 
> If you do accept this is a good reason for failure,
why?

sometimes the problem is in the redundancy gear itself. 
PAIX lost power
twice during its first five years of operation, and both
times it was due
to faulty GFI in the UPS+redundancy gear.  which had passed
testing during
construction and subsequently, but eventually some component
just wore out.
-- 
Paul Vixie

They should have generators running...I can't foresee any
good
datacenter not having multiple generators to keep their
customers
servers online with UPS.

-Ray

-----Original Message-----
From: owner-nanogmerit.edu [mailto:owner-nanogmerit.edu] On Behalf Of
Adrian Chadd
Sent: Tuesday, July 24, 2007 7:54 PM
To: Seth Mattinen
Cc: nanog list
Subject: Re: San Francisco Power Outage


On Tue, Jul 24, 2007, Seth Mattinen wrote:

> I have a question: does anyone seriously accept
"oh, power trouble" as
a 
> reason your servers went offline? Where's the
generators? UPS? Testing

> said combination of UPS and generators? What if it was
important? I 
> honestly find it hard to believe anyone runs a facility
like that and 
> people actually *pay* for it.

> If you do accept this is a good reason for failure,
why?

Didn't you read? He paid extra for super-reliable power from
his
electricity provider..



Adrian

On Tue, 2007-07-24 at 19:57 -0400, Patrick Giagnocavo
wrote:
> 
> On Jul 24, 2007, at 6:54 PM, Seth Mattinen wrote:
> >
> > I have a question: does anyone seriously accept
"oh, power trouble"  
> > as a reason your servers went offline? Where's the
generators? UPS?  
> > Testing said combination of UPS and generators?
What if it was  
> > important? I honestly find it hard to believe
anyone runs a  
> > facility like that and people actually *pay* for
it.
> >
> 
> Sad that the little Telcove DC here in Lancaster, PA,
that Level3  
> bought a few months ago, has weekly full-on generator
tests where  
> 100% of the load is transferred to the generator, while
apparently  
> large DCs that are charging premium rates, do not.

Perhaps they do.  Wouldn't have mattered in this case if
the
big-red-button rumor is real.  

-Jim P.

We also have weekly backups where 100% of the load for our
entire
company is put on the three generators. Everything inside
the building
is put onto the generators power, this way we can test for
faulty UPS's
etc and ensure the generators are working etc. I don't
believe that they
don't have a similar setup.

Ray Corbin

rcorbinhostmysite.com

-----Original Message-----
From: owner-nanogmerit.edu [mailto:owner-nanogmerit.edu] On Behalf Of
Patrick Giagnocavo
Sent: Tuesday, July 24, 2007 7:57 PM
To: nanognanog.org
Subject: Re: San Francisco Power Outage



On Jul 24, 2007, at 6:54 PM, Seth Mattinen wrote:
>
> I have a question: does anyone seriously accept
"oh, power trouble"  
> as a reason your servers went offline? Where's the
generators? UPS?  
> Testing said combination of UPS and generators? What if
it was  
> important? I honestly find it hard to believe anyone
runs a  
> facility like that and people actually *pay* for it.
>

Sad that the little Telcove DC here in Lancaster, PA, that
Level3  
bought a few months ago, has weekly full-on generator tests
where  
100% of the load is transferred to the generator, while
apparently  
large DCs that are charging premium rates, do not.

Cordially

Patrick Giagnocavo
patrickzill.net

On Jul 24, 2007, at 4:57 PM, Patrick Giagnocavo wrote:

>
>
> On Jul 24, 2007, at 6:54 PM, Seth Mattinen wrote:
>>
>> I have a question: does anyone seriously accept
"oh, power  
>> trouble" as a reason your servers went
offline? Where's the  
>> generators? UPS? Testing said combination of UPS
and generators?  
>> What if it was important? I honestly find it hard
to believe  
>> anyone runs a facility like that and people
actually *pay* for it.
>>
>
> Sad that the little Telcove DC here in Lancaster, PA,
that Level3  
> bought a few months ago, has weekly full-on generator
tests where  
> 100% of the load is transferred to the generator, while
apparently  
> large DCs that are charging premium rates, do not.


I am not familiar with the operational details of 365 Main,
but, I  
suspect that
they, like most datacenters, probably do have weekly
generator and  
transfer
test procedures.

However, there are lots of things that can go wrong that are
not  
covered by
generators and transfer tests:

It is possible to cascade fail a power distribution system
in a  
number of
ways. It is possible for someone to connect things out of
phase during a
maintenance procedure in such a way that everything is fine
until a
transfer occurs, then, all hell breaks loose (ever seen what
happens
when a large CRAC unit starts trying to run backwards
because the
3 Phase rotation is out of order?)

There are also things that can go wrong in the transfer
process (like
putting the UPS and Generators on the bus together some
degrees
out of phase).

Most of these things become far more likely and far harder
to avoid as
the amount of power and the number of units in the system
increases.

I'm not defending the situation at 365 Main. I don't have
any first hand
knowledge.  I'm just saying that the mere fact that they are
dark for
several hours today does not necessarily mean that they
don't do
weekly full-on generator tests.

I have no idea what the root cause of today's outage is.  I
will be
interested in hearing from any credible source as to any
actual details,
but, I'm betting that right now, any such credible source is
a bit busy.

Owen