24x7 Support Strategies

Hi,

I'm wondering how different organisations structure their
24x7 network 
operations? We are undergoing some restructuring here and it
would be 
interesting for us to know how other large enterprises and
service 
providers arrange this. We are particulary interested in
service 
providers. (Currently we have an enterprise that is slowly
morphing into 
more of a service provider setup). I'll summarise back to
the list, 
after removing any identifying details.

These questions specifically refer to network staff, as
opposed to any 
general Ops team.

Do you have 24x7 staff on site?
What level of technical ability do the on-site staff have?
What shift patterns do the 24x7 staff use?

Do you have a response time for on-call staff, by which time
they must 
be VPN'ed into the network?
What level of techincal ability do the first line on-call
staff have?
Do you have an official escalation system if the first-line
on-call 
staff do not have the required techincal ability?
Do the staff on on-call escalation have a required response
time, by 
which time they must be VPN'ed into the network?
Do the staff on on-call escalation rota the on-call
responsibilities?
Do the on-call staff receive additional benefits or
compensation for 
being on-call?

All,

Thanks for the replies that have started rolling in. They've
made me 
realise I should have added an additional question for
clarity.

Does anyone have any CCIE (or equivalent technical ability)
staff on a 
24x7 shift? What about CCIE level staff on an on-call rota
with a 
garanteed response time? How about CCNP?

If people could also give an identication of the size of
their 
organisation/network it would be useful.

Sam

Sam Stickland wrote:
>
> Hi,
>
> I'm wondering how different organisations structure
their 24x7 network 
> operations? We are undergoing some restructuring here
and it would be 
> interesting for us to know how other large enterprises
and service 
> providers arrange this. We are particulary interested
in service 
> providers. (Currently we have an enterprise that is
slowly morphing 
> into more of a service provider setup). I'll summarise
back to the 
> list, after removing any identifying details.
>
> These questions specifically refer to network staff, as
opposed to any 
> general Ops team.
>
> Do you have 24x7 staff on site?
> What level of technical ability do the on-site staff
have?
> What shift patterns do the 24x7 staff use?
>
> Do you have a response time for on-call staff, by which
time they must 
> be VPN'ed into the network?
> What level of techincal ability do the first line
on-call staff have?
> Do you have an official escalation system if the
first-line on-call 
> staff do not have the required techincal ability?
> Do the staff on on-call escalation have a required
response time, by 
> which time they must be VPN'ed into the network?
> Do the staff on on-call escalation rota the on-call
responsibilities?
> Do the on-call staff receive additional benefits or
compensation for 
> being on-call?

This topic interests me very much, and I had a BOF about
staff development
at the Montreal meeting in 1999.  I remember some of the
details, and, while
I am no longer generally doing course development, I have
some pretty strong
ideas of what reasonably constitutes a proper training
sandbox for a major
ISP.

If anyone would like to discuss this, pleae feel free to
contact me offline.
If there's a use for a separate mailing list or summaries to
NANOG, I'd be
happy to try to organize it.

-----Original Message-----
From: owner-nanogmerit.edu [mailto:owner-nanogmerit.edu] On Behalf Of Sam
Stickland
Sent: Thursday, June 14, 2007 5:33 AM
Cc: NANOG list
Subject: Re: 24x7 Support Strategies


All,

Thanks for the replies that have started rolling in. They've
made me 
realise I should have added an additional question for
clarity.

Does anyone have any CCIE (or equivalent technical ability)
staff on a 
24x7 shift? What about CCIE level staff on an on-call rota
with a 
garanteed response time? How about CCNP?

If people could also give an identication of the size of
their 
organisation/network it would be useful.

Sam

Sam Stickland wrote:
>
> Hi,
>
> I'm wondering how different organisations structure
their 24x7 network 
> operations? We are undergoing some restructuring here
and it would be 
> interesting for us to know how other large enterprises
and service 
> providers arrange this. We are particulary interested
in service 
> providers. (Currently we have an enterprise that is
slowly morphing 
> into more of a service provider setup). I'll summarise
back to the 
> list, after removing any identifying details.
>
> These questions specifically refer to network staff, as
opposed to any 
> general Ops team.
>
> Do you have 24x7 staff on site?
> What level of technical ability do the on-site staff
have?
> What shift patterns do the 24x7 staff use?
>
> Do you have a response time for on-call staff, by which
time they must 
> be VPN'ed into the network?
> What level of techincal ability do the first line
on-call staff have?
> Do you have an official escalation system if the
first-line on-call 
> staff do not have the required techincal ability?
> Do the staff on on-call escalation have a required
response time, by 
> which time they must be VPN'ed into the network?
> Do the staff on on-call escalation rota the on-call
responsibilities?
> Do the on-call staff receive additional benefits or
compensation for 
> being on-call?

On 14-Jun-2007, at 02:32, Sam Stickland wrote:

> Does anyone have any CCIE (or equivalent technical
ability) staff  
> on a 24x7 shift? What about CCIE level staff on an
on-call rota  
> with a garanteed response time? How about CCNP?

Does anybody actually put any stock in the presence or
absence of  
vendor certifications on a resume when judging the
capabilities of an  
engineer?

There's no correlation between certification and capability,
in my  
experience.


Joe

I doubt it maybe training companies!

A number of vendors have grades to meet. So X number of
certified experts
mean better deals from said vendor.

Regards,
Neil.

> Does anybody actually put any stock in the presence or
absence of  
> vendor certifications on a resume when judging the
capabilities of an  
> engineer?

> There's no correlation between certification and
capability, in my  
> experience.

Joe Abley wrote:
>
> On 14-Jun-2007, at 02:32, Sam Stickland wrote:
>
>> Does anyone have any CCIE (or equivalent technical
ability) staff on 
>> a 24x7 shift? What about CCIE level staff on an
on-call rota with a 
>> garanteed response time? How about CCNP?
>
> Does anybody actually put any stock in the presence or
absence of 
> vendor certifications on a resume when judging the
capabilities of an 
> engineer?
>
> There's no correlation between certification and
capability, in my 
> experience.
I fully agree with you Joe, but I needed to quantify the
level of 
technical expertise somehow. I think most people have some
kind of a 
feel for what level we are talking about if we say
"equivalent techincal 
ability to CCIE", even if there are CCIEs out there who
are useless ;)

S

Even my recent experience says no here. I had a CCIE
(written..!) in for 
an interview and, well, I am not sure how he managed to get
CCIE written 
but he sure as hell didn't know much.

There are some useful qualifications for ISP potential
employees that 
the LINX provide in conjunction with some training
companies, these are 
good for NOC or junior engineers but at the end of the
routing table 
there's no substitute for getting people who have had a few
years to 
screw up other people's networks and learn what not to do..


--
Leigh


Neil J. McRae wrote:
> I doubt it maybe training companies!
>
> A number of vendors have grades to meet. So X number of
certified experts
> mean better deals from said vendor.
>
> Regards,
> Neil.
>
>   
>> Does anybody actually put any stock in the presence
or absence of  
>> vendor certifications on a resume when judging the
capabilities of an  
>> engineer?
>>     
>
>   
>> There's no correlation between certification and
capability, in my  
>> experience.
>>     
>
>   

I find that within 15 mins of any interview I have
accurately judged the 
technical competence of most candidates...


Sam Stickland wrote:
>
> Joe Abley wrote:
>>
>> On 14-Jun-2007, at 02:32, Sam Stickland wrote:
>>
>>> Does anyone have any CCIE (or equivalent
technical ability) staff on 
>>> a 24x7 shift? What about CCIE level staff on an
on-call rota with a 
>>> garanteed response time? How about CCNP?
>>
>> Does anybody actually put any stock in the presence
or absence of 
>> vendor certifications on a resume when judging the
capabilities of an 
>> engineer?
>>
>> There's no correlation between certification and
capability, in my 
>> experience.
> I fully agree with you Joe, but I needed to quantify
the level of 
> technical expertise somehow. I think most people have
some kind of a 
> feel for what level we are talking about if we say
"equivalent 
> techincal ability to CCIE", even if there are
CCIEs out there who are 
> useless ;)
>
> S

On Thu, 2007-06-14 at 12:57 +0100, Leigh Porter wrote:
> 
> Even my recent experience says no here. I had a CCIE
(written..!) in for 
> an interview and, well, I am not sure how he managed to
get CCIE written 
> but he sure as hell didn't know much.

There is no such thing as a "CCIE (written)". 
That would be kind of
like PhD ABD (All But Dissertation), but without the
coursework, too.

-- 
Daniel J McDonald, CCIE # 2495, CISSP # 78281, CNX
Austin Energy
http://www.austinenergy.c
om

Daniel J McDonald wrote:
> On Thu, 2007-06-14 at 12:57 +0100, Leigh Porter wrote:
>   
>> Even my recent experience says no here. I had a
CCIE (written..!) in for 
>> an interview and, well, I am not sure how he
managed to get CCIE written 
>> but he sure as hell didn't know much.
>>     
>
> There is no such thing as a "CCIE (written)".
 That would be kind of
> like PhD ABD (All But Dissertation), but without the
coursework, too.
>
>   

Indeed. I can send you his details if you're looking for
somebody 

--
Leigh