pfsync after reboot does not synchronize

I tried posting some messages about PF to the freebsd-net
mailing list,
but they seemed to be ignored.  So I thought I would try
sending my
questions here.

I am trying to figure out why pfsync does not seem to work
correctly
when one of my cluster nodes reboots.

When I reboot one of the cluster members, the state tables
do appear to
synchronize, sort of, and populate with some of the same
connection
states, but not all of them.

That is "pfctl -ss" on both cluster members will
show a different number
of state entries.  Vastly different if the new member has
only been up
for a minute or two.

In particular, long-lived, extant connections (such as IRC
server
connections) seem to never show up in the rebooted member's
state table,
even though the connections continue to update their state
on the
current carp master.

I figured that doing ifconfig down/up would send some sort
of "full
sync" message between the two members, to cause the
entire state table
to be sent in bulk.  Eventually I learned that the method to
do this is
to use "ifconfig syncdev" to force a bulk
update:

    ifconfig pfsync0 syncdev fxp0   # $pfsync_syncdev

When I perform the above command, I see the following debug
output (when
PF is configured at "misc" or "loud"
debug level):

    On the cluster member receiving the requests:

	pfsync: received bulk update request
	pfsync: received bulk update request
	pfsync: received bulk update request
	pfsync: received bulk update request
	pfsync: received bulk update request
	pfsync: received bulk update request
	pfsync: received bulk update request
	pfsync: received bulk update request
	pfsync: received bulk update request
	pfsync: received bulk update request
	pfsync: received bulk update request
	pfsync: received bulk update request
	pfsync: received bulk update request

    On the cluster member making the request (where syncdev
was just
    ifconfig'd):

	pfsync: requesting bulk update
	pfsync: received bulk update start
	pfsync: received bulk update start
	pfsync: received bulk update start
	pfsync: received bulk update start
	pfsync: received bulk update start
	pfsync: received bulk update start
	pfsync: received bulk update start
	pfsync: received bulk update start
	pfsync: received bulk update start
	pfsync: received bulk update start
	pfsync: received bulk update start
	pfsync: received bulk update start
	pfsync: received bulk update start
	pfsync: failed to receive bulk update status

After performing this manual action, I find the state table
is much
better populated, and the two firewalls appear to be
synchronized. 
However, the messages above bother me.  It looks to me like
the cluster
member making the request repeats it over and over again,
and finally
gives up after PFSYNC_MAX_BULKTRIES (12) attempts. 
Shouldn't that be
something that only happens in exceptional conditions?  Yet,
I can make
it happen every time, even on a test cluster with no traffic
(and thus
an almost empty state table).

Does anyone have any insight as to why I see these problems?

1.  Why does pfsync synchronize the state tables when I use
the
    "ifconfig syncdev" trick to force a bulk
update, yet it does
    not do this when the system is booting up?

2.  Why does pfsync keep repeating the bulk update request
and then give
    up?  What message is not getting through?


The two cluster members have a direct cross-cable between
them.  My PF
policy has these settings:

    set skip on pfsync0

    pass quick on fxp0 proto pfsync	# $pfsync_syncdev

-- 
David DeSimone == Network Admin == foxverio.net
  "It took me fifteen years to discover that I had no
   talent for writing, but I couldn't give it up because
   by that time I was too famous.  -- Robert Benchley
_______________________________________________
freebsd-pffreebsd.org mailing list

http://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to
"freebsd-pf-unsubscribefreebsd.org"

On 6/5/06, David DeSimone <foxverio.net> wrote:
> I tried posting some messages about PF to the
freebsd-net mailing list,
> but they seemed to be ignored.  So I thought I would
try sending my
> questions here.
>
> I am trying to figure out why pfsync does not seem to
work correctly
> when one of my cluster nodes reboots.
>
> When I reboot one of the cluster members, the state
tables do appear to
> synchronize, sort of, and populate with some of the
same connection
> states, but not all of them.
>
> That is "pfctl -ss" on both cluster members
will show a different number
> of state entries.  Vastly different if the new member
has only been up
> for a minute or two.
>
> In particular, long-lived, extant connections (such as
IRC server
> connections) seem to never show up in the rebooted
member's state table,
> even though the connections continue to update their
state on the
> current carp master.
>
> I figured that doing ifconfig down/up would send some
sort of "full
> sync" message between the two members, to cause
the entire state table
> to be sent in bulk.  Eventually I learned that the
method to do this is
> to use "ifconfig syncdev" to force a bulk
update:
>
>     ifconfig pfsync0 syncdev fxp0   # $pfsync_syncdev
>
> When I perform the above command, I see the following
debug output (when
> PF is configured at "misc" or
"loud" debug level):
>
>     On the cluster member receiving the requests:
>
>         pfsync: received bulk update request
>         pfsync: received bulk update request
>         pfsync: received bulk update request
>         pfsync: received bulk update request
>         pfsync: received bulk update request
>         pfsync: received bulk update request
>         pfsync: received bulk update request
>         pfsync: received bulk update request
>         pfsync: received bulk update request
>         pfsync: received bulk update request
>         pfsync: received bulk update request
>         pfsync: received bulk update request
>         pfsync: received bulk update request
>
>     On the cluster member making the request (where
syncdev was just
>     ifconfig'd):
>
>         pfsync: requesting bulk update
>         pfsync: received bulk update start
>         pfsync: received bulk update start
>         pfsync: received bulk update start
>         pfsync: received bulk update start
>         pfsync: received bulk update start
>         pfsync: received bulk update start
>         pfsync: received bulk update start
>         pfsync: received bulk update start
>         pfsync: received bulk update start
>         pfsync: received bulk update start
>         pfsync: received bulk update start
>         pfsync: received bulk update start
>         pfsync: received bulk update start
>         pfsync: failed to receive bulk update status
>
> After performing this manual action, I find the state
table is much
> better populated, and the two firewalls appear to be
synchronized.
> However, the messages above bother me.  It looks to me
like the cluster
> member making the request repeats it over and over
again, and finally
> gives up after PFSYNC_MAX_BULKTRIES (12) attempts. 
Shouldn't that be
> something that only happens in exceptional conditions? 
Yet, I can make
> it happen every time, even on a test cluster with no
traffic (and thus
> an almost empty state table).
>
> Does anyone have any insight as to why I see these
problems?
>
> 1.  Why does pfsync synchronize the state tables when I
use the
>     "ifconfig syncdev" trick to force a
bulk update, yet it does
>     not do this when the system is booting up?
>
> 2.  Why does pfsync keep repeating the bulk update
request and then give
>     up?  What message is not getting through?
>
>
> The two cluster members have a direct cross-cable
between them.  My PF
> policy has these settings:
>
>     set skip on pfsync0
>
>     pass quick on fxp0 proto pfsync     #
$pfsync_syncdev

I have also seen this problem with pfSense.  To get around
the problem
I set the advskew to 200 on the host and wait 30 seconds to
give
everything time to sync.  I am really not sure what is
causing it but
it may be related to the pfsync hold down timer?   At any
rate we
worked around the problem and I wanted to readdress it after
our 1.0
release.  I am glad someone else is also seeing the problem.

Let me know if anyone needs more information.

Scott
_______________________________________________
freebsd-pffreebsd.org mailing list

http://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to
"freebsd-pf-unsubscribefreebsd.org"

1.  Why does pfsync synchronize the state tables when I use
the

>     "ifconfig syncdev" trick to force a
bulk update, yet it does
>     not do this when the system is booting up?



What does your rc.conf look like?


2.  Why does pfsync keep repeating the bulk update request
and then give
>     up?  What message is not getting through?



Are you running the same versions of everything on all
nodes?  Different
versions of pfsync can sometimes not keep state with
eachother (3.8 ->
3.9comes to mind).


The two cluster members have a direct cross-cable between
them.  My PF
> policy has these settings:
>
>     set skip on pfsync0
>
>     pass quick on fxp0 proto pfsync     #
$pfsync_syncdev



Won't fix your problem, but if you 'set skip' on that
interface, you don't
need to 'pass quick' as filtering isn't applied.


Kian



_______________________________________________
> freebsd-pffreebsd.org mailing list
> 
http://lists.freebsd.org/mailman/listinfo/freebsd-pf
> To unsubscribe, send any mail to
"freebsd-pf-unsubscribefreebsd.org"
>
_______________________________________________
freebsd-pffreebsd.org mailing list

http://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to
"freebsd-pf-unsubscribefreebsd.org"

Kian Mohageri <kian.mohagerigmail.com> wrote:
>
> > Why does pfsync synchronize the state tables when
I use the
> > "ifconfig syncdev" trick to force a
bulk update, yet it does
> > not do this when the system is booting up?
> 
> What does your rc.conf look like?

    gateway_enable="YES"
    pf_enable="YES"
    pf_rules="/usr/local/etc/pf.conf"
    pflog_enable="YES"
    pfsync_enable="YES"
    pfsync_syncdev="fxp0"

    defaultrouter="192.168.40.254"

    cloned_interfaces="carp0 carp1"

    ifconfig_dc0="inet 192.168.40.231 netmask
255.255.255.224"
    ifconfig_dc1="inet 172.16.30.2 netmask
255.255.255.0"

    ifconfig_fxp0="up"

    ifconfig_carp0="inet 192.168.40.230 netmask
255.255.255.224 vhid 230"
    ifconfig_carp1="inet 172.16.30.1 netmask
255.255.255.0 vhid 11"

As you can see, no IP is put on the sync interface; it is
merely
configured up.  Auto-negotiation succeeds on both ends of
the cross
cable:

    media: Ethernet autoselect (100baseTX
<full-duplex>)

> > Why does pfsync keep repeating the bulk update
request and then give
> > up?  What message is not getting through?
> 
> Are you running the same versions of everything on all
nodes? 
> Different versions of pfsync can sometimes not keep
state with
> eachother (3.8 -> 3.9comes to mind).

Both are FreeBSD 6.0-RELEASE cloned from the same disk.

> >    set skip on pfsync0
> >
> >    pass quick on fxp0 proto pfsync     #
$pfsync_syncdev
> 
> Won't fix your problem, but if you 'set skip' on
that interface, you
> don't need to 'pass quick' as filtering isn't
applied.

Note that the "set skip" is on the pfsync0
pseudo interface, while the
"pass quick" is on the actual fxp0 interface.

Is there a protocol other than pfsync that should be
permitted on that
interface?  I didn't expect I'd see any other protocol
there, so I
didn't bother to allow anything else.

-- 
David DeSimone == Network Admin == foxverio.net
  "It took me fifteen years to discover that I had no
   talent for writing, but I couldn't give it up because
   by that time I was too famous.  -- Robert Benchley
_______________________________________________
freebsd-pffreebsd.org mailing list

http://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to
"freebsd-pf-unsubscribefreebsd.org"

> As you can see, no IP is put on the sync interface; it
is merely
> configured up.  Auto-negotiation succeeds on both ends
of the cross
> cable:


All the examples I've seen give the syncdev an IP address,
my setup
included.  I'd try that.  It's strange that it works
partially without
having done that, though.  Maybe someone with more pfsync
knowledge could
explain why for me?  I'd imagine it has something to do
with sending out via
multicast.

http://www.
benzedrine.cx/pf/msg04781.html


>
> Note that the "set skip" is on the pfsync0
pseudo interface, while the
> "pass quick" is on the actual fxp0
interface.



My apologies, I misread that 

Is there a protocol other than pfsync that should be
permitted on that
> interface?  I didn't expect I'd see any other
protocol there, so I
> didn't bother to allow anything else.


You don't need anything other than pfsync to pass on fxp0
for this to work.

Kian
_______________________________________________
freebsd-pffreebsd.org mailing list

http://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to
"freebsd-pf-unsubscribefreebsd.org"