Strange issue with Cluster XL

Hello,

I have a customer that has two firewall modules running on
SPLAT R60 HFA04,
those are in HA config (active/standby) and recently,
precisely after the
HFA upgrade, module 2 was left as active, but the customer
noticed one day
something had happened and the cluster had switched to
module 1 as active,
he noticed some strange logs but did not think about it too
much, the issue
is that it happened again a couple of days ago and now he
reported the
situation and sent me a screen-shot of the logs shown, which
look something
like this:

8:17:29    cluster-1    cluster_info: Stopping Cluster XL
8:17:29    cluster-1    cluster_info: Starting Cluster XL
8:17:29    cluster-1    cluster_info: (cluster XL) member 1
is up
8:17:29    cluster-1    cluster_info: (cluster XL) member 1
is initializing
8:17:29    cluster-1    cluster_info: (cluster XL) member 1
is active
8:17:29    cluster-1    cluster_info: (cluster XL) member 1
is initializing
8:17:29    cluster-1    cluster_info: (cluster XL) member 1
is active
8:40:26    cluster-2    cluster_info: (cluster XL) member 2
is standby

If you notice, the first 7 logs have exactly the same time
stamp, so this
must be happenning really quick, a few seconds later, module
2 reports as
standby, even when it was active until that day.
In the cluster configuration they have the following option
configured :

Upon gateway recovery: Maintain current active gateway

So, there is no reason to expect an automatic switch between
members due to
a priority matter.
SmartView Monitor has not shown any Cluster XL problems and
my customer
followed some tests stoping one member and letting the other
become active
back and forward, with no problems.
Even when no traffic issues have been seen, my customer is
concerned there
might be some sort of small issue that could become really
big at some
point.

Thanks in advance for your help.

Regards


-- 
Sergio Alvarez
(506)8301342

=================================================
To set vacation, Out-Of-Office, or away messages,
send an email to LISTSERVamadeus.us.checkpoint.com
in the BODY of the email add:
set fw-1-mailinglist nomail
=================================================
To unsubscribe from this mailing list,
please see the instructions at
http:
//www.checkpoint.com/services/mailing.html
=================================================
If you have any questions on how to change your
subscription options, email
fw-1-ownerts.checkpoint.com
=================================================

On Fri, 24 Nov 2006, Sergio Alvarez wrote:

> I have a customer that has two firewall modules running
on SPLAT R60 HFA04,
> those are in HA config (active/standby) and recently,
precisely after the
> HFA upgrade, module 2 was left as active, but the
customer noticed one day
> something had happened and the cluster had switched to
module 1 as active,
> he noticed some strange logs but did not think about it
too much, the issue
> is that it happened again a couple of days ago and now
he reported the
> situation and sent me a screen-shot of the logs shown,
which look something
> like this:
> 
> 8:17:29    cluster-1    cluster_info: Stopping Cluster
XL
> 8:17:29    cluster-1    cluster_info: Starting Cluster
XL
> 8:17:29    cluster-1    cluster_info: (cluster XL)
member 1 is up
> 8:17:29    cluster-1    cluster_info: (cluster XL)
member 1 is initializing
> 8:17:29    cluster-1    cluster_info: (cluster XL)
member 1 is active
> 8:17:29    cluster-1    cluster_info: (cluster XL)
member 1 is initializing
> 8:17:29    cluster-1    cluster_info: (cluster XL)
member 1 is active
> 8:40:26    cluster-2    cluster_info: (cluster XL)
member 2 is standby

Funny. This is the first time I heard of this on a
non-Solaris firewall.

On Solaris this is usually caused by CPU starvation. Somehow
the traffic 
handling in kernel takes up such a percentage that it will
not send a 
timely notification to the watchdog and the watchdog decides
to transfer 
control to the other node.

If traffic remains high this will repeat itself on the other
node and you 
will have a flapping cluster.

There are some notes about this in the knowledgebase.

You can see if this might be the case if you can babysit the
unit using 
vmstat. On Solaris it seems to occur if you spend over 50%
in kernel.
Also check all your *.elg files for additional details.

Hugo.

-- 
	hvdkooijvanderkooij.org	http://hvdkooij.xs4all.nl/

	    This message is using 100% recycled electrons.

=================================================
To set vacation, Out-Of-Office, or away messages,
send an email to LISTSERVamadeus.us.checkpoint.com
in the BODY of the email add:
set fw-1-mailinglist nomail
=================================================
To unsubscribe from this mailing list,
please see the instructions at
http:
//www.checkpoint.com/services/mailing.html
=================================================
If you have any questions on how to change your
subscription options, email
fw-1-ownerts.checkpoint.com
=================================================

Thanks a lot Hugo.

On 11/24/06, Hugo van der Kooij <hvdkooijvanderkooij.org> wrote:
>
> On Fri, 24 Nov 2006, Sergio Alvarez wrote:
>
> > I have a customer that has two firewall modules
running on SPLAT R60
> HFA04,
> > those are in HA config (active/standby) and
recently, precisely after
> the
> > HFA upgrade, module 2 was left as active, but the
customer noticed one
> day
> > something had happened and the cluster had
switched to module 1 as
> active,
> > he noticed some strange logs but did not think
about it too much, the
> issue
> > is that it happened again a couple of days ago and
now he reported the
> > situation and sent me a screen-shot of the logs
shown, which look
> something
> > like this:
> >
> > 8:17:29    cluster-1    cluster_info: Stopping
Cluster XL
> > 8:17:29    cluster-1    cluster_info: Starting
Cluster XL
> > 8:17:29    cluster-1    cluster_info: (cluster XL)
member 1 is up
> > 8:17:29    cluster-1    cluster_info: (cluster XL)
member 1 is
> initializing
> > 8:17:29    cluster-1    cluster_info: (cluster XL)
member 1 is active
> > 8:17:29    cluster-1    cluster_info: (cluster XL)
member 1 is
> initializing
> > 8:17:29    cluster-1    cluster_info: (cluster XL)
member 1 is active
> > 8:40:26    cluster-2    cluster_info: (cluster XL)
member 2 is standby
>
> Funny. This is the first time I heard of this on a
non-Solaris firewall.
>
> On Solaris this is usually caused by CPU starvation.
Somehow the traffic
> handling in kernel takes up such a percentage that it
will not send a
> timely notification to the watchdog and the watchdog
decides to transfer
> control to the other node.
>
> If traffic remains high this will repeat itself on the
other node and you
> will have a flapping cluster.
>
> There are some notes about this in the knowledgebase.
>
> You can see if this might be the case if you can
babysit the unit using
> vmstat. On Solaris it seems to occur if you spend over
50% in kernel.
> Also check all your *.elg files for additional details.
>
> Hugo.
>
> --
>         hvdkooijvanderkooij.org        http://hvdkooij.xs4all.nl/

>             This message is using 100% recycled
electrons.
>
> =================================================
> To set vacation, Out-Of-Office, or away messages,
> send an email to LISTSERVamadeus.us.checkpoint.com
> in the BODY of the email add:
> set fw-1-mailinglist nomail
> =================================================
> To unsubscribe from this mailing list,
> please see the instructions at
> http:
//www.checkpoint.com/services/mailing.html
> =================================================
> If you have any questions on how to change your
> subscription options, email
> fw-1-ownerts.checkpoint.com
> =================================================
>



-- 
Sergio Alvarez
(506)8301342

=================================================
To set vacation, Out-Of-Office, or away messages,
send an email to LISTSERVamadeus.us.checkpoint.com
in the BODY of the email add:
set fw-1-mailinglist nomail
=================================================
To unsubscribe from this mailing list,
please see the instructions at
http:
//www.checkpoint.com/services/mailing.html
=================================================
If you have any questions on how to change your
subscription options, email
fw-1-ownerts.checkpoint.com
=================================================