List Info

Thread: RE: Cacti 0.8.6j Released (fwd)
RE: Cacti 0.8.6j Released (fwd)
user name
 2007-01-25 05:06:47 

  


  

    
>  how do you define your schema?
> how long does it take to insert/index/whatnot the data?

This is a much bigger deal than most people realize.
Poor schema design will cause your system to choke
bade when you try to scale it. In fact, relational 
databases are not the ideal way to store this kind
of data so when you design your schema, you are really
fighting against the database to wrestle it into something

that will work.

> 	this is a huge burden to figure it all out, implement 
> and then monitor/operate 24x7.  miss enough samples or data 
> and you end up billing too little.  this is why most folks 
> have either cooked their own, or use some expensive suite of 
> tools, leaving just a little bit of other stuff out there.

Personally, I doubt that it is possible to build a
workable system, even with plugins, that will do the
job for a significant percentage of service providers.
Different companies have different needs, different

hot button items, etc. This is an area where breaking
the problem down into well-defined separate problems
with a well-defined linkage, will go a long way.

But to start with, just solving the data storage problem
is a good place to start. If someone can create a specialized
network monitoring database that scales, then the rest of
the toolkit will be much easier to deal with. Note that 
people have done a lot of research on this sort of
time-series database. People working in high-energy physics
also have to deal with massive sets of time-series data.
There is plenty of literature out there to help guide
a design effort. But Open-Source developers don't usually
do this kind of up-front research before starting coding.
Money and manpower won't solve that kind of problem.

--Michael Dillon



  

  
  
   
     
    

  

    RE:  Cacti 0.8.6j Released (fwd)
  


  

     user name

     2007-01-25 06:10:33
  


  

    

  


  

    

> 
> But to start with, just solving the data storage problem is a 
> good place to start. If someone can create a specialized 
> network monitoring database that scales, then the rest of the 
> toolkit will be much easier to deal with. Note that people 
> have done a lot of research on this sort of time-series 
> database. People working in high-energy physics also have to 
> deal with massive sets of time-series data.
> There is plenty of literature out there to help guide a 
> design effort. But Open-Source developers don't usually do 
> this kind of up-front research before starting coding.
> Money and manpower won't solve that kind of problem.
> 

How about something like:
http://www.hdfgroup.org/whatishdf5.html


-- 
Scanned for viruses and dangerous content at 
http://www.oneunified.net and is believed to be clean.



  

  
  
   
     
    

  

    Re:  Cacti 0.8.6j Released (fwd)
  


  

     user name

     2007-01-25 06:51:15
  


  

    

  


  

    
This is where dbms' designed for data warehouses might come into play, 
something like SybaseIQ.  It is adapted for long term storage and retrieval.

michael.dillonbt.com wrote:
>>  how do you define your schema?
>> how long does it take to insert/index/whatnot the data?
>>     
>
> This is a much bigger deal than most people realize.
> Poor schema design will cause your system to choke
> bade when you try to scale it. In fact, relational 
> databases are not the ideal way to store this kind
> of data so when you design your schema, you are really
> fighting against the database to wrestle it into something

> that will work.
>
>   
>> 	this is a huge burden to figure it all out, implement 
>> and then monitor/operate 24x7.  miss enough samples or data 
>> and you end up billing too little.  this is why most folks 
>> have either cooked their own, or use some expensive suite of 
>> tools, leaving just a little bit of other stuff out there.
>>     
>
> Personally, I doubt that it is possible to build a
> workable system, even with plugins, that will do the
> job for a significant percentage of service providers.
> Different companies have different needs, different

> hot button items, etc. This is an area where breaking
> the problem down into well-defined separate problems
> with a well-defined linkage, will go a long way.
>
> But to start with, just solving the data storage problem
> is a good place to start. If someone can create a specialized
> network monitoring database that scales, then the rest of
> the toolkit will be much easier to deal with. Note that 
> people have done a lot of research on this sort of
> time-series database. People working in high-energy physics
> also have to deal with massive sets of time-series data.
> There is plenty of literature out there to help guide
> a design effort. But Open-Source developers don't usually
> do this kind of up-front research before starting coding.
> Money and manpower won't solve that kind of problem.
>
> --Michael Dillon
>

>   



  

  
  
   
     
    






 [1-3]











   
 





about | contact  Other archives ( Real Estate discussion Medical topics )