Bee attack, fiber cut, 7-hour outage

http://www.chron.com/disp/story.mpl/ap/tx/5148125.html


(AUSTIN) Telephone service was out for seven hours in rural
Central Texas 
after bees attacked a construction worker, causing him to
jump off his 
tractor and hit a lever that lowered an auger that sliced a
fiber-optic 
line.

On Sep 21, 2007, at 2:38 PM, Deepak Jain wrote:

> Anytime you talk about "rural" I'm impressed
with 7 hours, however  
> -- isn't SONET supposed to make this better?

We had a customer hit by this, and actually saw services
restored for  
a few minutes in just four hours, but then they went back
down.

--Chris

On Fri, 21 Sep 2007 15:38:30 EDT, Deepak Jain said:

> Anytime you talk about "rural" I'm impressed
with 7 hours, however -- 
> isn't SONET supposed to make this better?

I'm not in Texas, but I am rural - there's plenty of places
around here
where it's just not economically feasible to run 2 diverse
fiber paths
to a town.  Heck, a lot of these places didn't get their
*first* fiber
until fairly recently.

http://maps.google.com/maps?f=q&hl=en&
;geocode=&q=grundy,+virgina&ie=UTF8&ll=37.27
9107,-82.099457&spn=0.414686,0.782776&t=h&z=11&a
mp;om=1

Not a place you'll find a redundant SONET ring. ;)

Deepak Jain wrote:
> 
> 
> 
> Sean Donelan wrote:
>>
>>
>> http://www.chron.com/disp/story.mpl/ap/tx/5148125.html

>>
>> (AUSTIN) Telephone service was out for seven hours
in rural Central 
>> Texas after bees attacked a construction worker,
causing him to jump 
>> off his tractor and hit a lever that lowered an
auger that sliced a 
>> fiber-optic line.
>>
>>
> 
> Is this a 7 hour outage a comment on rural Central
Texas availability of 
> fiber splicers or novel ways fiber gets cut?
> 
I'm thinking that getting hit by an auger might put the
fiber more into 
the "mangled" category rather than simply cut.

> Anytime you talk about "rural" I'm impressed
with 7 hours, however -- 
> isn't SONET supposed to make this better?

SONET... yeah, right. We had a fiber-seeking backhoe take
out some fiber 
in the mid-Willamette Valley recently - it took out long
distance for 
several smaller local phone companies for about that long as
well. 911 
service for at least some people too. I'm interestedly
awaiting the 
final word on what happened there...

Jeff

On Fri, 21 Sep 2007, Deepak Jain wrote:

> Is this a 7 hour outage a comment on rural Central
Texas availability of 
> fiber splicers or novel ways fiber gets cut?
>
> Anytime you talk about "rural" I'm impressed
with 7 hours, however -- isn't 
> SONET supposed to make this better?

Sure, if:
1. the protect path is configured and enabled
2. both the working and protect paths don't run through the
same 
conduit/duct/buffer

jms

On Fri, Sep 21, 2007 at 04:49:22PM -0400, Justin M. Streiner
wrote:
> >Anytime you talk about "rural" I'm
impressed with 7 hours, however -- 
> >isn't SONET supposed to make this better?
> 
> Sure, if:
> 1. the protect path is configured and enabled
> 2. both the working and protect paths don't run through
the same 
> conduit/duct/buffer

I am continually amazed at how often this is the case.

I realize that it's expensive to run these lines but when
you put your
working and protect in the same cable or different cables in
the same
trench (not even a trench a few feet apart, but the same
trench and
same innerduct), you have to EXPECT that you're gonna have
angry
customers. And yet when telco folks learn that this has
occured, they
often fein being as surprised as the customers.

Truely amazing.

---
Wayne Bouchard
webtypo.org
Network Dude
http://www.typo.org/~web/

On 9/22/07, Wayne E. Bouchard <webtypo.org> wrote:

> I realize that it's expensive to run these lines but
when you put your
> working and protect in the same cable or different
cables in the same
> trench (not even a trench a few feet apart, but the
same trench and
> same innerduct), you have to EXPECT that you're gonna
have angry
> customers. And yet when telco folks learn that this has
occured, they
> often fein being as surprised as the customers.

.. and as long as they are the only telco with copper in the
area,
they could care less, I guess?

> jump off his tractor and hit a lever that lowered an
auger that sliced
> a fiber-optic line.

ps: That story had a kind of  "... that killed the rat
that ate the
malt that made the house that jack built" feel to it.

srs

> I'm forking this thread to complain about vendor L's 
> international long haul network. Protected Sonet
service 
> (T3). DC to UK.

I wonder if anyone is using PWE3 for this kind of service,
perhaps in an academic/research environment? It would be
interesting to compare notes on outages, latency/jitter,
etc.

--Michael Dillon

>Has anyone calculated what the cost of doing this
correctly once vs the ongoing support/SLA/etc issues of
repairing it when it goes boom is? I've gotta believe that
for >90% of the situations where diverse routes exist,
just being used as dual linear paths, its cheaper in the
long term to do it "right" and cut the size of
your outside plant crew (assigned to break/fix) by 90%. 

It is definitely cheaper to "do it right" the
first time, but to
"keep it right", the operator needs to pay very
close attention
to each and every circuit groom that they perform, less
they
end up with a degenerate loop somewhere.  Depending on the
state of their circuit routing database(s), the exercise of
checking
for overlap against "the other half" of the ring
can anywhere
from trivial to impossible.

I don't think operators intentionally foul this up, but it's
real
easy to get wrong, particular in the fallout after
accumulating
a bunch of different companies fiber plants and circuit
route
systems and trying to consolidate everything for savings. 
I'm
just providing this in answer to "why can't telcos get
this right",
as there's no reason to think that grooming activity was at
all
involved in why this particular carrier got stung... 


/John

On 9/21/07, Deepak Jain <deepakai.net> wrote:
> However, when I see "Location of Maintenance:
France" and a 5 minute
> outage for a protected SONET service on a supposedly
redundant, high
> quality International voice/data network... well, let's
just say I'm not
> impressed -- on 36 hrs notice, no less.
>
> I can't do anything with respect to an SLA since there
is advanced
> notice, but isn't it reasonable to assume that in this
day-and-age
> running a properly protected T3 isn't *that hard*
anymore????
> Especially in advance -- you know, shunt the traffic to
one your other
> circuits because, you know, you are supposed to have
this massive network.

Typically for a service like that, the carrier would have
one or more
long-haul rings,
either SONET or DWDM with the SONET on top of it or
something,
interconnected to access rings from a local access carrier
at each end,
either close to the customer endpoint or possibly back at
regional
interconnects.
The long-haul carrier might not control the local access
carrier
except through an SLA,
so it could be that the local carrier messed up.  One hopes
that the
interconnections
between the networks are also at diverse POPs (at least for
big
countries like France),
but it is possible for the interconnections to fail
clumsily.

The "shunt the traffic to other circuits" approach
is naively correct,
but I've seen more than one case of "discover that one
side of an
access ring has failed
by shunting the traffic from the other side to do
maintenance and
having a customer call you
a couple of minutes later to report an outage at their
headquarters",
which is absolutely never supposed to happen, of course 

What a 5-minute outage sounds like to me is "The
early-90s T3-based
restoration system
recovered the circuit on an end-to-end basis using DACSs
instead of
SONET restoring it",
but I'd be surprised to see than happening at a newer
carrier like
Vendor L who built most of their network after SONET
technology became
affordable.

----
             Thanks;     Bill

Note that this isn't my regular email account - It's still
experimental so far.
And Google probably logs and indexes everything you send
it.