High Availability Design - Thoughts?

Looking for thoughts/feedback/etc. on the path below:

What I am looking to resolve is a single point of failure in
the Prelude 
Manager.  So, two ideas come to mind.

1.  Have two servers, built adequately as database servers. 
Each run an 
instance of MySQL with a Prelude database and Prelude
Manager.  Have 
client sensors' configurations have IP addresses of both
Managers, so if 
they can't reach one Manager, they would contact the other. 
On both 
Managers, use two [db] output plugins, one for the MySQL
prelude 
database that is local to them, and one that is located on
the other 
DB/Manager server.  So at any one time, both DBs/Managers
would have the 
same data - within reason (time considerations, network
latency, etc.).

or....

2.  Have two servers, both with Prelude-Manager installed,
only one 
active - with a heartbeat of some sort, so the secondary
would take over 
if the first failed.  But with the twist of using shared
storage for the 
location of the database.  Additionally, to solve the
database single 
instance, having a backup replication of it somewhere or
whatnot.

Anybody have thoughts on either of the two ideas above, or
maybe a 
different solution that you are using?

Thanks,

ScottO
_______________________________________________
Prelude-user site list
Prelude-userprelude-ids.org
http://www.prelude-ids.org/mailman/listinfo/prelude-user

Hi Scott,
Some time ago I wrote up some design philosophy on scaling
Prelude,
including HA/LB based on my work with commercial SEIM
products and how
they do it.
The way it works ---

You can break the Prelude Architecture down into phases:
*  Data Acquisition (syslog, Prelude-LML, other
Prelude-native sensors):
You can have as many of these as you want based on your
acquisition
needs.  Additionally, in the case of LML (or anything using
a
lightweight protocol such as syslog), you can load-balance
the acquiring
hosts to get your first layer of HA.
*  Correlation (Prelude-Manager, Prelude-Correlator):  There
are a
couple of ways that you can establish HA on this platform. 
The simplest
comprehensive solution is network load balancing, however
you could also
do host-based load balancing using an IP that is floated
between
multiple hosts.  Regardless of what you do, you'll want to
do hot
failover using an active-passive approach, otherwise
Prelude-Correlator
will get confused.
*  Data Warehousing (MySQL, PostgreSQL):  You should refer
to the
documentation on the product to establish HA.
*  Presentation (Prewikka):  No more difficult than load
balancing a web
farm.

Breaking it down to this many servers would represent a
LARGE deployment
(although, not so large that I haven't seen and worked on
one of this
size before).  If you need to do this cheaply, you can
consolidate the
DA and Corr tiers, have the Correlation and Presentation
tiers provide
HA for one another, and run an external MySQL cluster
(resulting in four
servers).
You can probably consolidate it more than that, as well, but
I haven't
put much time into testing or using a smaller architecture.

- Ramon

-----Original Message-----
From: prelude-user-bouncesprelude-ids.org
[mailto:prelude-user-bouncesprelude-ids.org] On Behalf
Of ScottO
Sent: Friday, April 06, 2007 6:45 AM
To: prelude-userprelude-ids.org
Subject: [prelude-user] High Availability Design -
Thoughts?

Looking for thoughts/feedback/etc. on the path below:

What I am looking to resolve is a single point of failure in
the Prelude
Manager.  So, two ideas come to mind.

1.  Have two servers, built adequately as database servers. 
Each run an
instance of MySQL with a Prelude database and Prelude
Manager.  Have
client sensors' configurations have IP addresses of both
Managers, so if
they can't reach one Manager, they would contact the other. 
On both
Managers, use two [db] output plugins, one for the MySQL
prelude
database that is local to them, and one that is located on
the other
DB/Manager server.  So at any one time, both DBs/Managers
would have the
same data - within reason (time considerations, network
latency, etc.).

or....

2.  Have two servers, both with Prelude-Manager installed,
only one 
active - with a heartbeat of some sort, so the secondary
would take over

if the first failed.  But with the twist of using shared
storage for the

location of the database.  Additionally, to solve the
database single 
instance, having a backup replication of it somewhere or
whatnot.

Anybody have thoughts on either of the two ideas above, or
maybe a 
different solution that you are using?

Thanks,

ScottO
_______________________________________________
Prelude-user site list
Prelude-userprelude-ids.org
http://www.prelude-ids.org/mailman/listinfo/prelude-user

_______________________________________________
Prelude-user site list
Prelude-userprelude-ids.org
http://www.prelude-ids.org/mailman/listinfo/prelude-user

Thanks for the input Ramon, much appreciated!

Does anyone do any HA with MySQL for the Prelude DB?  Have
you found that
clustering or replication works better in regards to the
Prelude database?

Thanks,

ScottO

On 4/6/07, Gene Ramon Gomez <ggomezragingwire.com> wrote:
>
> Hi Scott,
> Some time ago I wrote up some design philosophy on
scaling Prelude,
> including HA/LB based on my work with commercial SEIM
products and how
> they do it.
> The way it works ---
>
> You can break the Prelude Architecture down into
phases:
> *  Data Acquisition (syslog, Prelude-LML, other
Prelude-native sensors):
> You can have as many of these as you want based on your
acquisition
> needs.  Additionally, in the case of LML (or anything
using a
> lightweight protocol such as syslog), you can
load-balance the acquiring
> hosts to get your first layer of HA.
> *  Correlation (Prelude-Manager, Prelude-Correlator): 
There are a
> couple of ways that you can establish HA on this
platform.  The simplest
> comprehensive solution is network load balancing,
however you could also
> do host-based load balancing using an IP that is
floated between
> multiple hosts.  Regardless of what you do, you'll want
to do hot
> failover using an active-passive approach, otherwise
Prelude-Correlator
> will get confused.
> *  Data Warehousing (MySQL, PostgreSQL):  You should
refer to the
> documentation on the product to establish HA.
> *  Presentation (Prewikka):  No more difficult than
load balancing a web
> farm.
>
> Breaking it down to this many servers would represent a
LARGE deployment
> (although, not so large that I haven't seen and worked
on one of this
> size before).  If you need to do this cheaply, you can
consolidate the
> DA and Corr tiers, have the Correlation and
Presentation tiers provide
> HA for one another, and run an external MySQL cluster
(resulting in four
> servers).
> You can probably consolidate it more than that, as
well, but I haven't
> put much time into testing or using a smaller
architecture.
>
> - Ramon
>
> -----Original Message-----
> From: prelude-user-bouncesprelude-ids.org
> [mailto:prelude-user-bouncesprelude-ids.org] On Behalf
Of ScottO
> Sent: Friday, April 06, 2007 6:45 AM
> To: prelude-userprelude-ids.org
> Subject: [prelude-user] High Availability Design -
Thoughts?
>
> Looking for thoughts/feedback/etc. on the path below:
>
> What I am looking to resolve is a single point of
failure in the Prelude
> Manager.  So, two ideas come to mind.
>
> 1.  Have two servers, built adequately as database
servers.  Each run an
> instance of MySQL with a Prelude database and Prelude
Manager.  Have
> client sensors' configurations have IP addresses of
both Managers, so if
> they can't reach one Manager, they would contact the
other.  On both
> Managers, use two [db] output plugins, one for the
MySQL prelude
> database that is local to them, and one that is located
on the other
> DB/Manager server.  So at any one time, both
DBs/Managers would have the
> same data - within reason (time considerations, network
latency, etc.).
>
> or....
>
> 2.  Have two servers, both with Prelude-Manager
installed, only one
> active - with a heartbeat of some sort, so the
secondary would take over
>
> if the first failed.  But with the twist of using
shared storage for the
>
> location of the database.  Additionally, to solve the
database single
> instance, having a backup replication of it somewhere
or whatnot.
>
> Anybody have thoughts on either of the two ideas above,
or maybe a
> different solution that you are using?
>
> Thanks,
>
> ScottO
> _______________________________________________
> Prelude-user site list
> Prelude-userprelude-ids.org
> http://www.prelude-ids.org/mailman/listinfo/prelude-user

>
_______________________________________________
Prelude-user site list
Prelude-userprelude-ids.org
http://www.prelude-ids.org/mailman/listinfo/prelude-user