pfsync errors

For the last two days I've been troubleshooting a wierd
issue where my
secondary firewall in a pfsync/carp cluster isn't
maintaining a state
table similar in size to the primary - it's slowly
increasing to the
max size.  I think I've finally tracked it down to
ip_output()
returning an error, but at this point I'm lost.  The
interfaces show
no errors, this box happily ran OpenBSD for the last three
years with
no similar errors and has only started exhibiting this
behavior after
converting it.  I'm seeing this on multiple boxes, but am
spending my
time troubleshooting just one.  Any advice/assistance would
be greatly
appreciated, I'm at a loss and this is affecting my
production
environment.

We're running RELENG_6_2, nics are Intel PRO/1000's (copper,
but the
cat-5e cable is a direct run to the 6513 switch one cabinet
over -
15ft cable).

This is a netstat from the primary machine, the secondary
has been
failed over to a couple times and looks similar (although
interestingly the cluster seems to handle being on the
secondary box
better)
# netstat -s -p pfsync
pfsync:
        409302985 packets received (IPv4)
        0 packets received (IPv6)
                0 packets discarded for bad interface
                0 packets discarded for bad ttl
                0 packets shorter than header
                0 packets discarded for bad version
                0 packets discarded for bad HMAC
                0 packets discarded for bad action
                0 packets discarded for short packet
                0 states discarded for bad values
                0 stale states
                16980281 failed state lookup/inserts
        1541416698 packets sent (IPv4)
        0 packets sent (IPv6)
                0 send failed due to mbuf memory error
                182754275 send error
# netstat -i -Iem2
Name    Mtu Network       Address              Ipkts Ierrs  
 Opkts Oerrs  Coll
em2    1500 <Link#3>      00:04:23:a6:b7:be 409328713 
  27 1359271127
    0     0
em2    1500 192.168.100.2 l4dupfw140-sync   409327567     -
1359270884
    -     -



Thanks

--Bill
_______________________________________________
freebsd-pffreebsd.org mailing list

http://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to
"freebsd-pf-unsubscribefreebsd.org"

On 8/22/07, Max Laier <maxlove2party.net> wrote:
> There are two reasons why we increase the send error
counter.  Either the
> internal deferred work queue is full or ip_output
fails.  Could you
> locate "pfsyncstats.pfsyncs_oerrors++" in
your source code and replace
> either occurrence with a printf().  Maybe use the
attached.  This way we
> will know what exactly fails and if it is ip_output,
why.

Thanks, we'll give it a shot shortly.

--Bill
_______________________________________________
freebsd-pffreebsd.org mailing list

http://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to
"freebsd-pf-unsubscribefreebsd.org"

On 8/22/07, Max Laier <maxlove2party.net> wrote:
> There are two reasons why we increase the send error
counter.  Either the
> internal deferred work queue is full or ip_output
fails.  Could you
> locate "pfsyncstats.pfsyncs_oerrors++" in
your source code and replace
> either occurrence with a printf().  Maybe use the
attached.  This way we
> will know what exactly fails and if it is ip_output,
why.

Here's what we get with the patch:
pfsync_senddef: ip_output 64

# netstat -s -p pfsync
pfsync:
        1264507 packets received (IPv4)
        0 packets received (IPv6)
                0 packets discarded for bad interface
                0 packets discarded for bad ttl
                0 packets shorter than header
                0 packets discarded for bad version
                0 packets discarded for bad HMAC
                0 packets discarded for bad action
                0 packets discarded for short packet
                0 states discarded for bad values
                0 stale states
                115608 failed state lookup/inserts
        86591 packets sent (IPv4)
        0 packets sent (IPv6)
                0 send failed due to mbuf memory error
                37231 send error

and a dmesg
# dmesg
Copyright (c) 1992-2007 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991,
1992, 1993, 1994
        The Regents of the University of California. All
rights reserved.
FreeBSD is a registered trademark of The FreeBSD
Foundation.
FreeBSD 6.2-RELEASE-p6 #0: Mon Aug 27 17:24:24 UTC 2007
    roothostname.somedomain.com:/usr/obj/usr/src/sys/HA-PFOPT
S-SYNCDEBUG
Timecounter "i8254" frequency 1193182 Hz quality
0
CPU: Intel(R) Xeon(TM) CPU 2.80GHz (2790.97-MHz 686-class
CPU)
  Origin = "GenuineIntel"  Id = 0xf27  Stepping =
7
 
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,AP
IC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR
,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x4400<CNTX-ID,<b14>>
  Logical CPUs per core: 2
real memory  = 1073717248 (1023 MB)
avail memory = 1041539072 (993 MB)
ACPI APIC Table: <COMPAQ 00000083>
MADT: Forcing active-low polarity and level trigger for SCI
ioapic0 <Version 1.1> irqs 0-15 on motherboard
ioapic1 <Version 1.1> irqs 16-31 on motherboard
ioapic2 <Version 1.1> irqs 32-47 on motherboard
ioapic3 <Version 1.1> irqs 48-63 on motherboard
kbd1 at kbdmux0
ath_hal: 0.9.17.2 (AR5210, AR5211, AR5212, RF5111, RF5112,
RF2413, RF5413)
acpi0: <COMPAQ P29> on motherboard
acpi0: Power Button (fixed)
Timecounter "ACPI-safe" frequency 3579545 Hz
quality 1000
acpi_timer0: <32-bit timer at 3.579545MHz> port
0x920-0x923 on acpi0
cpu0: <ACPI CPU> on acpi0
pcib0: <ACPI Host-PCI bridge> on acpi0
pci0: <ACPI PCI bus> on pcib0
pci0: <display, VGA> at device 3.0 (no driver
attached)
pci0: <base peripheral> at device 4.0 (no driver
attached)
pci0: <base peripheral> at device 4.2 (no driver
attached)
isab0: <PCI-ISA bridge> at device 15.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <ServerWorks CSB5 UDMA100 controller> port
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x2000-0x200f at device
15.1 on
pci0
ata0: <ATA channel 0> on atapci0
ata1: <ATA channel 1> on atapci0
ohci0: <OHCI (generic) USB controller> mem
0xf5ef0000-0xf5ef0fff irq 7
at device 15.2 on pci0
ohci0: [GIANT-LOCKED]
usb0: OHCI version 1.0, legacy support
usb0: SMM does not respond, resetting
usb0: <OHCI (generic) USB controller> on ohci0
usb0: USB revision 1.0
uhub0: (0x1166) OHCI root hub, class 9/0, rev 1.00/1.00,
addr 1
uhub0: 4 ports with 4 removable, self powered
pcib1: <ACPI Host-PCI bridge> on acpi0
pci1: <ACPI PCI bus> on pcib1
ciss0: <Compaq Smart Array 5i> port 0x3000-0x30ff mem
0xf7cc0000-0xf7cfffff,0xf7bf0000-0xf7bf3fff irq 30 at device
3.0 on
pci1
ciss0: [GIANT-LOCKED]
pcib2: <ACPI Host-PCI bridge> on acpi0
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI Host-PCI bridge> on acpi0
pci3: <ACPI PCI bus> on pcib3
em0: <Intel(R) PRO/1000 Network Connection Version -
6.2.9> port
0x4000-0x403f mem 0xf7de0000-0xf7dfffff irq 20 at device 1.0
on pci3
em0: Ethernet address: 00:07:e9:10:d6:58
em1: <Intel(R) PRO/1000 Network Connection Version -
6.2.9> port
0x4040-0x407f mem 0xf7dc0000-0xf7ddffff irq 21 at device 1.1
on pci3
em1: Ethernet address: 00:07:e9:10:d6:59
pcib4: <ACPI Host-PCI bridge> on acpi0
pci6: <ACPI PCI bus> on pcib4
em2: <Intel(R) PRO/1000 Network Connection Version -
6.2.9> port
0x5000-0x503f mem
0xf7fe0000-0xf7ffffff,0xf7f80000-0xf7fbffff irq 24
at device 1.0 on pci6
em2: Ethernet address: 00:04:23:a7:0f:0e
em3: <Intel(R) PRO/1000 Network Connection Version -
6.2.9> port
0x5040-0x507f mem
0xf7f60000-0xf7f7ffff,0xf7f00000-0xf7f3ffff irq 25
at device 1.1 on pci6
em3: Ethernet address: 00:04:23:a7:0f:0f
pci6: <base peripheral, PCI hot-plug controller> at
device 30.0 (no
driver attached)
acpi_tz0: <Thermal Zone> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64
irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model Generic PS/2 mouse, device ID 0
sio0: <Standard PC COM port> port 0x3f8-0x3ff irq 4
flags 0x10 on acpi0
sio0: type 16550A
fdc0: <floppy drive controller (FDE)> port 0x3f2-0x3f5
irq 6 drq 2 on acpi0
fdc0: [FAST]
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
pmtimer0 on isa0
orm0: <ISA Option ROMs> at iomem
0xc0000-0xc7fff,0xc8000-0xcbfff,0xee000-0xeffff on isa0
ppc0: parallel port not found.
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem
0xa0000-0xbffff on isa0
Timecounter "TSC" frequency 2790968808 Hz quality
800
Timecounters tick every 1.000 msec
acd0: CDROM <CD-224E/9.9A> at ata0-master PIO4
da0 at ciss0 bus 0 target 0 lun 0
da0: <COMPAQ RAID 1  VOLUME OK> Fixed Direct Access
SCSI-0 device
da0: 135.168MB/s transfers
da0: 34727MB (71122560 512 byte sectors: 255H 32S/T 8716C)
Trying to mount root from ufs:/dev/da0s2a
pfsync_senddef: ip_output 64
_______________________________________________
freebsd-pffreebsd.org mailing list

http://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to
"freebsd-pf-unsubscribefreebsd.org"

On Tuesday 28 August 2007, Bill Marquette wrote:
> On 8/22/07, Max Laier <maxlove2party.net> wrote:
> > There are two reasons why we increase the send
error counter.  Either
> > the internal deferred work queue is full or
ip_output fails.  Could
> > you locate
"pfsyncstats.pfsyncs_oerrors++" in your source
code and
> > replace either occurrence with a printf().  Maybe
use the attached. 
> > This way we will know what exactly fails and if it
is ip_output, why.
>
> Here's what we get with the patch:
> pfsync_senddef: ip_output 64

that's EHOSTDOWN ... that's strange.  Are you using
syncpeer?

> # netstat -s -p pfsync
> pfsync:
>         1264507 packets received (IPv4)
>         0 packets received (IPv6)
>                 0 packets discarded for bad interface
>                 0 packets discarded for bad ttl
>                 0 packets shorter than header
>                 0 packets discarded for bad version
>                 0 packets discarded for bad HMAC
>                 0 packets discarded for bad action
>                 0 packets discarded for short packet
>                 0 states discarded for bad values
>                 0 stale states
>                 115608 failed state lookup/inserts
>         86591 packets sent (IPv4)
>         0 packets sent (IPv6)
>                 0 send failed due to mbuf memory error
>                 37231 send error
>

But since the send error still increases it seems as if the
internal queue 
is overflowing, too.  This is something that must be fixed
as well, but I 
think the EHOSTDOWN from ip_output is more serious.

-- 
/"  Best regards,                      | mlaierfreebsd.org
 /  Max Laier                          | ICQ #67774661
 X   http://pf4freebsd.l
ove2party.net/  | mlaierEFnet
/   ASCII Ribbon Campaign              | Against HTML Mail
and News

On 8/27/07, Max Laier <maxlove2party.net> wrote:
> On Tuesday 28 August 2007, Bill Marquette wrote:
> > On 8/22/07, Max Laier <maxlove2party.net> wrote:
> > > There are two reasons why we increase the
send error counter.  Either
> > > the internal deferred work queue is full or
ip_output fails.  Could
> > > you locate
"pfsyncstats.pfsyncs_oerrors++" in your source
code and
> > > replace either occurrence with a printf(). 
Maybe use the attached.
> > > This way we will know what exactly fails and
if it is ip_output, why.
> >
> > Here's what we get with the patch:
> > pfsync_senddef: ip_output 64
>
> that's EHOSTDOWN ... that's strange.  Are you using
syncpeer?

I'll double check this, but with the reboot this box
shouldn't be.  I
had configured them to use syncpeer during troubleshooting
(we'd had
issues prior to setting this).

> But since the send error still increases it seems as if
the internal queue
> is overflowing, too.  This is something that must be
fixed as well, but I
> think the EHOSTDOWN from ip_output is more serious.

OK, I'll spend some more time on this in the morning and
confirm all
our settings.  It's most interesting to me that those stats
came from
the secondary unit, not the primary (which isn't quite as
easy to take
down - even more so with our state tables not quite in
sync).

--Bill
_______________________________________________
freebsd-pffreebsd.org mailing list

http://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to
"freebsd-pf-unsubscribefreebsd.org"

On 8/27/07, Bill Marquette <bill.marquettegmail.com> wrote:
> > > Here's what we get with the patch:
> > > pfsync_senddef: ip_output 64
> >
> > that's EHOSTDOWN ... that's strange.  Are you
using syncpeer?

After converting both sides to using syncpeer, I now got
this in my
dmesg after reboot:
<snip most of dmesg you've already seen>
Trying to mount root from ufs:/dev/da0s2a
pfsync_senddef: ip_output 65
pfsync_senddef: ip_output 65

$ netstat -s -ppfsync && sleep 1 && netstat
-s -ppfsync
pfsync:
        699811 packets received (IPv4)
        0 packets received (IPv6)
                0 packets discarded for bad interface
                0 packets discarded for bad ttl
                0 packets shorter than header
                0 packets discarded for bad version
                0 packets discarded for bad HMAC
                0 packets discarded for bad action
                0 packets discarded for short packet
                0 states discarded for bad values
                0 stale states
                61740 failed state lookup/inserts
        46728 packets sent (IPv4)
        0 packets sent (IPv6)
                0 send failed due to mbuf memory error
                23020 send error
pfsync:
        704677 packets received (IPv4)
        0 packets received (IPv6)
                0 packets discarded for bad interface
                0 packets discarded for bad ttl
                0 packets shorter than header
                0 packets discarded for bad version
                0 packets discarded for bad HMAC
                0 packets discarded for bad action
                0 packets discarded for short packet
                0 states discarded for bad values
                0 stale states
                62032 failed state lookup/inserts
        47070 packets sent (IPv4)
        0 packets sent (IPv6)
                0 send failed due to mbuf memory error
                23221 send error

The other pfsyncstats.pfsyncs_oerrors++ is after a
IF_HANDOFF macro, I
traced it back to if_handoff() in if.c, but I'm not sure how
to track
it down any further as it look like it's _IF_QFULL(ifq)
that's
triggering the return 0.

I'm going to toy with some settings for the em(4) driver in
loader.conf and see if I can raise the txd and rxd
descriptors since
we're not running on 82542 or 82543 chipped hardware we can
go above
256 descriptors.

--Bill
_______________________________________________
freebsd-pffreebsd.org mailing list

http://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to
"freebsd-pf-unsubscribefreebsd.org"

On Tuesday 28 August 2007, Bill Marquette wrote:
> On 8/27/07, Bill Marquette <bill.marquettegmail.com> wrote:
> > > > Here's what we get with the patch:
> > > > pfsync_senddef: ip_output 64
> > >
> > > that's EHOSTDOWN ... that's strange.  Are you
using syncpeer?
>
> After converting both sides to using syncpeer, I now
got this in my
> dmesg after reboot:
> <snip most of dmesg you've already seen>
> Trying to mount root from ufs:/dev/da0s2a
> pfsync_senddef: ip_output 65
> pfsync_senddef: ip_output 65
>
> $ netstat -s -ppfsync && sleep 1 &&
netstat -s -ppfsync
> pfsync:
>         699811 packets received (IPv4)
>         0 packets received (IPv6)
>                 0 packets discarded for bad interface
>                 0 packets discarded for bad ttl
>                 0 packets shorter than header
>                 0 packets discarded for bad version
>                 0 packets discarded for bad HMAC
>                 0 packets discarded for bad action
>                 0 packets discarded for short packet
>                 0 states discarded for bad values
>                 0 stale states
>                 61740 failed state lookup/inserts
>         46728 packets sent (IPv4)
>         0 packets sent (IPv6)
>                 0 send failed due to mbuf memory error
>                 23020 send error
> pfsync:
>         704677 packets received (IPv4)
>         0 packets received (IPv6)
>                 0 packets discarded for bad interface
>                 0 packets discarded for bad ttl
>                 0 packets shorter than header
>                 0 packets discarded for bad version
>                 0 packets discarded for bad HMAC
>                 0 packets discarded for bad action
>                 0 packets discarded for short packet
>                 0 states discarded for bad values
>                 0 stale states
>                 62032 failed state lookup/inserts
>         47070 packets sent (IPv4)
>         0 packets sent (IPv6)
>                 0 send failed due to mbuf memory error
>                 23221 send error
>
> The other pfsyncstats.pfsyncs_oerrors++ is after a
IF_HANDOFF macro, I
> traced it back to if_handoff() in if.c, but I'm not
sure how to track
> it down any further as it look like it's _IF_QFULL(ifq)
that's
> triggering the return 0.
>
> I'm going to toy with some settings for the em(4)
driver in
> loader.conf and see if I can raise the txd and rxd
descriptors since
> we're not running on 82542 or 82543 chipped hardware we
can go above
> 256 descriptors.

No that's the internal work deferral queue.  Try something
like the 
following in contrib/pf/net/if_pfsync.c:

 -229,7
+229,7 
        callout_init(&sc->sc_bulk_tmo,
NET_CALLOUT_MPSAFE);
        callout_init(&sc->sc_bulkfail_tmo,
NET_CALLOUT_MPSAFE);
        callout_init(&sc->sc_send_tmo,
NET_CALLOUT_MPSAFE);
-       sc->sc_ifq.ifq_maxlen = ifqmaxlen;
+       sc->sc_ifq.ifq_maxlen = 2 * ifqmaxlen;
        mtx_init(&sc->sc_ifq.ifq_mtx,
ifp->if_xname, "pfsync send queue",
            MTX_DEF);
        if_attach(ifp);

But there might be other reasons like timing wrt the locks. 
I'll have to 
check for details.  It might also be a good idea to MFC the
taskqueue 
approach from CURRENT, rather than using the callout ...
that's a bit of 
work however.

-- 
/"  Best regards,                      | mlaierfreebsd.org
 /  Max Laier                          | ICQ #67774661
 X   http://pf4freebsd.l
ove2party.net/  | mlaierEFnet
/   ASCII Ribbon Campaign              | Against HTML Mail
and News

On 8/28/07, Max Laier <maxlove2party.net> wrote:
> > I'm going to toy with some settings for the em(4)
driver in
> > loader.conf and see if I can raise the txd and rxd
descriptors since
> > we're not running on 82542 or 82543 chipped
hardware we can go above
> > 256 descriptors.

Raised it to 4096 for both txd and rxd with no impact (may
help
something else, who knows).

> No that's the internal work deferral queue.  Try
something like the
> following in contrib/pf/net/if_pfsync.c:
>
>  -229,7 +229,7 
>         callout_init(&sc->sc_bulk_tmo,
NET_CALLOUT_MPSAFE);
>         callout_init(&sc->sc_bulkfail_tmo,
NET_CALLOUT_MPSAFE);
>         callout_init(&sc->sc_send_tmo,
NET_CALLOUT_MPSAFE);
> -       sc->sc_ifq.ifq_maxlen = ifqmaxlen;
> +       sc->sc_ifq.ifq_maxlen = 2 * ifqmaxlen;
>         mtx_init(&sc->sc_ifq.ifq_mtx,
ifp->if_xname, "pfsync send queue",
>             MTX_DEF);
>         if_attach(ifp);
>
> But there might be other reasons like timing wrt the
locks.  I'll have to
> check for details.  It might also be a good idea to MFC
the taskqueue
> approach from CURRENT, rather than using the callout
... that's a bit of
> work however.

That seems to have halved the error rate, not sure if it's
not just
the time of day though (we're now past our peak traffic load
by a few
hours).

$ netstat -s -ppfsync && sleep 1 && netstat
-s -ppfsync
pfsync:
        383169 packets received (IPv4)
        0 packets received (IPv6)
                0 packets discarded for bad interface
                0 packets discarded for bad ttl
                0 packets shorter than header
                0 packets discarded for bad version
                0 packets discarded for bad HMAC
                0 packets discarded for bad action
                0 packets discarded for short packet
                0 states discarded for bad values
                0 stale states
                35622 failed state lookup/inserts
        23635 packets sent (IPv4)
        0 packets sent (IPv6)
                0 send failed due to mbuf memory error
                5580 send error
pfsync:
        387782 packets received (IPv4)
        0 packets received (IPv6)
                0 packets discarded for bad interface
                0 packets discarded for bad ttl
                0 packets shorter than header
                0 packets discarded for bad version
                0 packets discarded for bad HMAC
                0 packets discarded for bad action
                0 packets discarded for short packet
                0 states discarded for bad values
                0 stale states
                35711 failed state lookup/inserts
        23876 packets sent (IPv4)
        0 packets sent (IPv6)
                0 send failed due to mbuf memory error
                5676 send error

--Bill
_______________________________________________
freebsd-pffreebsd.org mailing list

http://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to
"freebsd-pf-unsubscribefreebsd.org"