Network Operations Guide

 
> My customer grew from a small enterprise to an
SP/Mobile operator
network very quickly.
> I want to guide them on how to operate their network,
so I am looking
for a document for this.

I've found the following websites quite useful for this:

http://www.google.com/search?hl=en&q=Change+Manage
ment
http://www.google.com/search?hl=en&q=Network+Mana
gement
http://www.google.com/search?hl=en&q=
Daily-Operations-tasks+network

http://www.google.com/search?hl=en&q=Change+Order
http://www.google.com/search?hl=en&q=Troubl
e-Ticketing-system
http://www.google.com/search?hl=en&q=escalatio
n+procedures
http://www.google.com/search?hl=en&q=troub
leshooting+processes
http://www.google.com/search?hl=en&q=backup+proced
ures
http://www.google.com/search?hl=en&q=Network-Inven
tory

No, this is not a joke. I checked every one of the above
URLs and you
*WILL* find useful information at each one.

And here is an important tip. Install a database server
(Oracle,
PostgreSQL) for general use and make sure that everyone has
ODBC/JDBC
access to it. Then ban the storage of information in
spreadsheets and
make it a habit to ask pointed questions about where data is
stored when
you are in meetings. Order the DB Admins to create any
tables that
people ask for in the "general use" database but
to advise people on
table design, for instance field types (IP addresses are not
VARCHAR).
Some attempt should be made to ensure that important fields
are
consistent between users since at some future date, tables
may need to
be joined. But these rules need to be applied with a light
touch because
the goal is to KEEP DATA OUT OF SPREADSHEETS.

Many modern companies operate using 1950's style processes
shuffling
spreadsheets through email instead of shuffling paper
through the mail
carts. That is counterproductive. It is far better to
emulate
1970's/80's companies who could only justify the vast
expense of
computers by simplifying processes and centralizing data on
shared
databases.

Once a spreadsheet starts to become a valuable source of
data, the data
should be stuffed into the database where it can be shared,
kept up to
date, be viewed consistently by all parts of the business,
and joined
with other data for reporting purposes. Then, when it makes
sense to
build or buy an application centered around a database, you
already have
a source of clean consistent data ready and waiting.

--Michael Dillon

> I'm guessing there is a tool somewhere that will take a
set 
> of data from a database and present it like a
spreadsheet for 
> import/export of updates. Anyone have a pointer?

Keeping data clean and consistent is so important to network
operations
that I think this is relevant to discuss for a bit.

First of all, assuming Windows is the client OS and the data
is on a
server somewhere, you may find it easier to use Access to
get at the
data, join tables, sort it and filter out unwanted rows.
Then export it
to an Excel spreadsheet. But not all corporate setups
include Access, so
you can do much the same directly from Excel. Look in
Data->Import
External Data->New Database Query, then select your data
source,
table/query/view, columns, etc. At the end you can select
"View or edit
query in Microsoft Query" and you will get a query
builder than can help
you sift out only the rows and columns that you need to
import. These
queries can be saved so that in future you simply rerun the
query
Data->Refresh Data and get fresh up-to-date data.

For updating a central database, you either need to develop
applications
or use a general purpose tool like MS Access. Usually
spreadsheets are
used to store fairly straightforward tables so building an
update
application is not necessarily that complex. For instance an
Excel
spreadsheet template can contain a VB subroutine that takes
a row of
data and turns it into an SQL UPDATE or INSERT statement.

To an IT person this all may sound rather crude and hardly
any better
than just keeping a bunch of spreadsheets, but they probably
never have
to deal with the consequences of dirty and inconsistent
data.
Spreadsheets tend to breed. They get copied around in email
and pretty
soon, people make mistakes and throw away the updated
version, not the
old one. Or other people, building a new spreadsheet think
that person X
has the definitive spreadsheet with all the latest IP
address
allocations, not realizing that this is a second hand copy
of another
master spreadsheet, and person X only gets a copy whenever
an upgrade
project completes, every few months. Meanwhile, operations
is busy
rationalising PoP layout and all the subnetting changes but
nobody
writes it down except in the one project managers planning
spreadsheet. 

The key principle here is to keep all important data in
tables on a
database server, and make sure that everybody understands
that these
tables are the one and only true source for this data. And,
of course,
make sure that server is backed up properly and you have a
disaster-recovey clone server ready for action when needed.

As long as the data is on a DB server, non-Windows systems
should have
no problems with accessing it. Scripting languages, Open
Office, web
servers and so on, can all share the same data.

Note that I am suggesting this be done, separate from the
kind of
OFFICIAL corporate databases that run financial, ordering
and billing
applications. Those databases are always locked down by the
DBAs and no
tables are added to them without being properly designed and
approved by
DBAs, data architects, app developers, etc. I am suggesting
that you run
a separate DB server to attract all the data that usually
gets
squirreled away in spreadsheets, to entice employees to
share data and
cooperate, without a lot of bureaucracy in the way. DBAs can
help a bit,
advise a bit, but they should not be able to forbid people
to set up a
table or index or query/view. This suggestion is to treat
the DB server
like a general service to all employees, like telephones or
meeting
rooms. 

--Michael Dillon

Deepak Jain wrote:
> 
>> Once a spreadsheet starts to become a valuable
source of data, the data
>> should be stuffed into the database where it can be
shared, kept up to
>> date, be viewed consistently by all parts of the
business, and joined
>> with other data for reporting purposes. Then, when
it makes sense to
>> build or buy an application centered around a
database, you already have
>> a source of clean consistent data ready and
waiting.
> 
> I'm guessing there is a tool somewhere that will take a
set of data from
> a database and present it like a spreadsheet for
import/export of
> updates. Anyone have a pointer?

This used to be the key feature of relational databases...

In a desktop computing context people do this with microsoft
access and
filemaker routinely...

These days however people have gotten so used to doing web
interfaces to
tables in sql databases that it's pretty trivial even with a
cookbook to
whip something up in php that will allow you view a table,
do a select,
and perform an insert update or delete on the changed or
added row.

> Is anyone doing anything fancy like exporting their
Visio to CSV some
> other sort for inclusion into a database as meta data?
(for viewing one
> would grab the data and then ask Visio to render it)
> 
> DJ
> 

On Wed, 22 Aug 2007, Joel Jaeggli wrote:

> > I'm guessing there is a tool somewhere that will
take a set of data from
> > a database and present it like a spreadsheet for
import/export of
> > updates. Anyone have a pointer?
> 
> This used to be the key feature of relational
databases...
> 
> In a desktop computing context people do this with
microsoft access and
> filemaker routinely...
> 
> These days however people have gotten so used to doing
web interfaces to
> tables in sql databases that it's pretty trivial even
with a cookbook to
> whip something up in php that will allow you view a
table, do a select,
> and perform an insert update or delete on the changed
or added row.
> 

Much of the advice I'm about to offer depends on scale, a
lot. If you're 
got a small network, you'd be fine with Cacti and similiar
OSS toolkits. 
If you're running more than a couple devices, MRTG will
always look 
attractive as long as you're not responsible for
administrating it. 
Please, don't try to scale it. There are lots of decent
OSS/OTS toolkits 
in the wild, take the time to find one you're comfortable
with and you'll 
save yourself some pain. 

If you don't have anyone in-house with the cluepower to
install, maintain, 
and understand one of these, consider buying, with support.
I've been 
spending a lot of time elbow-deep in the Monolith platform.
Even though I 
can build all my own tools, finding a good organized, full
featured 
platform that I can hack to hell and back has been a pretty
big boon. 

If you're going to roll your own:
Even if only a contract basis, I'd recommend tapping the
skillset of a 
good DBA or a professional network toolsmith for advice on
organizing the 
sheer dearth of data involved here. Getting off on a good
footing is the 
single most important part of a task like this, to minimize
how much time 
you spend going back and redoing things because they didn't
scale or 
simply don't apply generally enough.

The hard part in building management tools from scratch is
coming up with 
a good schema for standardizing how you organize your data.
A scheme that 
seems to make great sense will be completely obliterated
when it first 
encounters SNMP based conventions, and heaven help you if
you standardize 
on the lingo of a single vendor. Take Cisco vs *, for
example. The Cisco 
standard way of describing things is decent enough, and
that's fine until 
you decide to bring another vendor into the relationship.

Another reason I support a good standard is having multiple
hands in the 
cookpot. If you have a team of people working on tools,
versus one 
dedicated snmp ninja, you can't have multiple designers. It
just doesn't 
work. This is an area of networking that needs sunlight at
all times, to 
keep evil things from growing in the code. Ugly hacks and
stupid shell 
scripts are all well and good, if they're in your personal
toolbox bin 
directory. You don't want them in your enterprise/production
grade tools. 
There's no telling what will happen three years down the
road.

Organization of network data is usually pretty simple:
Devices contain 
interfaces and sessions, interfaces contain counters,
states, and 
descriptions. 

Smart pollers won't bother with counters on a down/shut
interfaces, and 
will only check descriptions/labels every so often. Your
only frequent 
polls should be in/out/error counters. Use 64 bit counters,
and account 
for rollover even if it's less of an issue at 64 bits.

If you're a hosting company or have high traffic on servers,
the ARP cache 
and the bridge tables are your friend.

The single most important piece of advice I can offer when
building your 
own tools: Never poll the routing table with SNMP. Ever. Any
OTS tool that 
says it can, as a feature, well, it's a witch, burn it. (If
you need 
routes, build something that can speak BGP. It's not that
hard, last time 
I did it was maybe 50 lines of code plus perl modules.)

- billn

Bill Nash wrote:
> The single most important piece of advice I can offer
when building your 
> own tools: Never poll the routing table with SNMP.
Ever. Any OTS tool that 
> says it can, as a feature, well, it's a witch, burn it.
(
>   
You mean like the Cisco Route Manager?

*http://www.cisco.com/en/US/products/ps6336/index.html
*
I not used it myself but the datasheet does say it collects
the routing 
tables via SNMP, with the one cavet:

"Prior to a scan, the Cisco Route Manager checks the
router's CPU and if 
the percentage is greater than a configured threshold then
it does not 
scan the route table and moves onto the next router."

Sam

On Thu, 23 Aug 2007, Sam Stickland wrote:

> Bill Nash wrote:
> > The single most important piece of advice I can
offer when building your own
> > tools: Never poll the routing table with SNMP.
Ever. Any OTS tool that says
> > it can, as a feature, well, it's a witch, burn it.
(
> >   
> You mean like the Cisco Route Manager?
> 
> *http://www.cisco.com/en/US/products/ps6336/index.html
> *
> I not used it myself but the datasheet does say it
collects the routing tables
> via SNMP, with the one cavet:
> 
> "Prior to a scan, the Cisco Route Manager checks
the router's CPU and if the
> percentage is greater than a configured threshold then
it does not scan the
> route table and moves onto the next router."

That 'feature' right there is a great indicator as to just
how unhealthy 
such an action is. Even if your CPU load is below your
threshold, you 
can't guarantee that the polling action isn't going to put
it over that 
threshold to the point of impacting performance. 

Break out the Zebra/Quagga install and offload that CPU
load. Don't use 
SNMP to read routing information.

Ok, I shouldn't be so declarative about it. You can do
whatever you want 
with your network. I will continue to beat people if/when I
catch them 
doing it. =)

- billn

On Fri, 24 Aug 2007, Greg VILLAIN wrote:

> Thing is, Zebra/Quagga doesn't seem to have any sort of
"external connector"
> to withdraw the routes in a decent format for analysis
purposes (xml, csv,
> plain list...).
> Last time we tried, Zebra/Quagga broke down when we
installed SNMP support (to
> locally query the routing table on the soft router).
> 
> Anyone know about any solution for such BGP data
collection ? (OpenBGPd ?)
> Thanks in advance for any hint.

I built a perl daemon using Net::BGP and DBI that inserted
and removed 
routes, on update, into an SQL db. I could then query to my
hearts 
content, beating up a db with full routes with all the
efficiency of SQL. 
It's simple as hell and works great.

- billn

On Fri, 24 Aug 2007, Bill Nash wrote:
> I built a perl daemon using Net::BGP and DBI that
inserted and removed 
> routes, on update, into an SQL db. I could then query
to my hearts 
> content, beating up a db with full routes with all the
efficiency of SQL. 
> It's simple as hell and works great.

Just to answer a bunch of off-list mails en masse, no, I do
not have a 
copy of this code for distribution. I wrote it originally
for an 
ex-employer, so I don't have a copy. I plan on rebuilding
it, it's fairly 
simple, so if you have use cases for a tool like this, shoot
me an email 
off list with your thoughts. Time permitting, I'll rebuild
and post. I 
need it for some of my own purposes anyway, might as well
make it suck 
less.

- billn