pf reply-to tcp connections stall

Hello,

I am trying to set up split routing on two Internet links,
each with
one IP address:

em0 = wan1, $em0_gw gateway
em1 = lan, NATed on em0 and em2
em2 = wan2, default gateway

pass in on em0 reply-to (em0 $em0_gw) inet proto tcp from
any to em0
flags S/SA keep state
pass in on em0 reply-to (em0 $em0_gw) inet proto udp from
any to em0 keep state
pass in on em0 reply-to (em0 $em0_gw) inet proto icmp from
any to em0 keep state

wan2 connections are working correct, no pf rules for policy
routing

wan1 tcp connections to IP of em0 (e.g. ssh) stall when a
large amount
of data is sent (e.g. running dmesg or cat file). States are
created
correctly. When ssh stalls there are some icmp packets out
on lo0 with
source and destination ip address of em0, which I believe is
not
correct (set skip on lo0 does not help). Also tried with tcp
...
modulate state but same result.

If I change default gateway to $em0_gw and disable pf all
connections
on wan1 are ok.

I also tried to use route-to instead of reply-to with:

pass out on em2 route-to (em0 $em0_gw) from em0 to any

both with keep state and no state options - same ssh
connection stall.

System is FreeBSD 7.0-STABLE amd64.

Kind regards,
Cristian
_______________________________________________
freebsd-pffreebsd.org mailing list

http://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to
"freebsd-pf-unsubscribefreebsd.org"

On Tue, May 20, 2008 at 06:30:58PM +0300, Cristian
Bradiceanu wrote:
> I am trying to set up split routing on two Internet
links, each with
> one IP address:
> 
> em0 = wan1, $em0_gw gateway
> em1 = lan, NATed on em0 and em2
> em2 = wan2, default gateway
> 
> pass in on em0 reply-to (em0 $em0_gw) inet proto tcp
from any to em0 flags S/SA keep state
> pass in on em0 reply-to (em0 $em0_gw) inet proto udp
from any to em0 keep state
> pass in on em0 reply-to (em0 $em0_gw) inet proto icmp
from any to em0 keep state
> 
> wan2 connections are working correct, no pf rules for
policy routing
> 
> wan1 tcp connections to IP of em0 (e.g. ssh) stall when
a large amount
> of data is sent (e.g. running dmesg or cat file).
States are created
> correctly. When ssh stalls there are some icmp packets
out on lo0 with
> source and destination ip address of em0, which I
believe is not
> correct (set skip on lo0 does not help). Also tried
with tcp ...
> modulate state but same result.

modulate state is known to be broken:

http://wiki.freebsd.org/JeremyChadwick/Commonly_r
eported_issues

Regarding the "when large amounts of data is sent, the
connection
breaks" issue:

I've reproduced this a few times on our systems (using the
exact same
method you do: dmesg, cat'ing large files, or scp'ing --
anything using
large TCP packets), and it's always been caused by improper
pf(4) rules
where state was broken.  In every case, the "state
mismatch" counter
shown in pfctl -s info would increase.

-- 
| Jeremy Chadwick                                jdc at
parodius.com |
| Parodius Networking                       http://www.parodius.com/
|
| UNIX Systems Administrator                  Mountain View,
CA, USA |
| Making life hard for others since 1977.              PGP:
4BD6C0CB |

_______________________________________________
freebsd-pffreebsd.org mailing list

http://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to
"freebsd-pf-unsubscribefreebsd.org"

On Tue, May 20, 2008 at 7:20 PM, Jeremy Chadwick
<koitsufreebsd.org> wrote:
> On Tue, May 20, 2008 at 06:30:58PM +0300, Cristian
Bradiceanu wrote:
>> I am trying to set up split routing on two Internet
links, each with
>> one IP address:
>>
>> em0 = wan1, $em0_gw gateway
>> em1 = lan, NATed on em0 and em2
>> em2 = wan2, default gateway
>>
>> pass in on em0 reply-to (em0 $em0_gw) inet proto
tcp from any to em0 flags S/SA keep state
>> pass in on em0 reply-to (em0 $em0_gw) inet proto
udp from any to em0 keep state
>> pass in on em0 reply-to (em0 $em0_gw) inet proto
icmp from any to em0 keep state
>>
>> wan2 connections are working correct, no pf rules
for policy routing
>>
>> wan1 tcp connections to IP of em0 (e.g. ssh) stall
when a large amount
>> of data is sent (e.g. running dmesg or cat file).
States are created
>> correctly. When ssh stalls there are some icmp
packets out on lo0 with
>> source and destination ip address of em0, which I
believe is not
>> correct (set skip on lo0 does not help). Also tried
with tcp ...
>> modulate state but same result.
>
> modulate state is known to be broken:
>
> http://wiki.freebsd.org/JeremyChadwick/Commonly_r
eported_issues
>
> Regarding the "when large amounts of data is sent,
the connection
> breaks" issue:
>
> I've reproduced this a few times on our systems (using
the exact same
> method you do: dmesg, cat'ing large files, or scp'ing
-- anything using
> large TCP packets), and it's always been caused by
improper pf(4) rules
> where state was broken.  In every case, the "state
mismatch" counter
> shown in pfctl -s info would increase.

state-mismatch counter does not increase, all
"Counters" are 0 except
match (pfctl -si).  When large amounts of data is sent the
connection
stalls and continues from time to time very slow; when it
continues
there are logged icmp packets out on lo0 from (em0) to (em0)
which
looks pretty weird to me.

Cristian
_______________________________________________
freebsd-pffreebsd.org mailing list

http://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to
"freebsd-pf-unsubscribefreebsd.org"

On 5/20/08, Cristian Bradiceanu <cbredibofhserver.net> wrote:
> On Tue, May 20, 2008 at 7:20 PM, Jeremy Chadwick
<koitsufreebsd.org> wrote:
>  > On Tue, May 20, 2008 at 06:30:58PM +0300,
Cristian Bradiceanu wrote:
>  >> I am trying to set up split routing on two
Internet links, each with
>  >> one IP address:
>  >>
>  >> em0 = wan1, $em0_gw gateway
>  >> em1 = lan, NATed on em0 and em2
>  >> em2 = wan2, default gateway
>  >>
>  >> pass in on em0 reply-to (em0 $em0_gw) inet
proto tcp from any to em0 flags S/SA keep state
>  >> pass in on em0 reply-to (em0 $em0_gw) inet
proto udp from any to em0 keep state
>  >> pass in on em0 reply-to (em0 $em0_gw) inet
proto icmp from any to em0 keep state
>  >>
>  >> wan2 connections are working correct, no pf
rules for policy routing
>  >>
>  >> wan1 tcp connections to IP of em0 (e.g. ssh)
stall when a large amount
>  >> of data is sent (e.g. running dmesg or cat
file). States are created
>  >> correctly. When ssh stalls there are some
icmp packets out on lo0 with
>  >> source and destination ip address of em0,
which I believe is not
>  >> correct (set skip on lo0 does not help). Also
tried with tcp ...
>  >> modulate state but same result.
>  >
>  > modulate state is known to be broken:
>  >
>  > http://wiki.freebsd.org/JeremyChadwick/Commonly_r
eported_issues
>  >
>  > Regarding the "when large amounts of data is
sent, the connection
>  > breaks" issue:
>  >
>  > I've reproduced this a few times on our systems
(using the exact same
>  > method you do: dmesg, cat'ing large files, or
scp'ing -- anything using
>  > large TCP packets), and it's always been caused
by improper pf(4) rules
>  > where state was broken.  In every case, the
"state mismatch" counter
>  > shown in pfctl -s info would increase.
>
>
> state-mismatch counter does not increase, all
"Counters" are 0 except
>  match (pfctl -si).  When large amounts of data is sent
the connection
>  stalls and continues from time to time very slow; when
it continues
>  there are logged icmp packets out on lo0 from (em0) to
(em0) which
>  looks pretty weird to me.
>
>
>  Cristian

    This may be a PMTUD issue. Make sure your ICMP packets
can travel
back and forth unhindered and that there are no scrub rules
that may
clear out the DF flag on them.

>
> _______________________________________________
>  freebsd-pffreebsd.org mailing list
>  
http://lists.freebsd.org/mailman/listinfo/freebsd-pf
>  To unsubscribe, send any mail to
"freebsd-pf-unsubscribefreebsd.org"
>


-- 
~/.signature: no such file or directory
_______________________________________________
freebsd-pffreebsd.org mailing list

http://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to
"freebsd-pf-unsubscribefreebsd.org"

On Wed, May 21, 2008 at 1:27 AM, Vlad GALU <dudududu.ro> wrote:
> On 5/20/08, Cristian Bradiceanu <cbredibofhserver.net> wrote:
>> On Tue, May 20, 2008 at 7:20 PM, Jeremy Chadwick
<koitsufreebsd.org> wrote:
>>  > On Tue, May 20, 2008 at 06:30:58PM +0300,
Cristian Bradiceanu wrote:
>>  >> I am trying to set up split routing on
two Internet links, each with
>>  >> one IP address:
>>  >>
>>  >> em0 = wan1, $em0_gw gateway
>>  >> em1 = lan, NATed on em0 and em2
>>  >> em2 = wan2, default gateway
>>  >>
>>  >> pass in on em0 reply-to (em0 $em0_gw)
inet proto tcp from any to em0 flags S/SA keep state
>>  >> pass in on em0 reply-to (em0 $em0_gw)
inet proto udp from any to em0 keep state
>>  >> pass in on em0 reply-to (em0 $em0_gw)
inet proto icmp from any to em0 keep state
>>  >>
>>  >> wan2 connections are working correct, no
pf rules for policy routing
>>  >>
>>  >> wan1 tcp connections to IP of em0 (e.g.
ssh) stall when a large amount
>>  >> of data is sent (e.g. running dmesg or
cat file). States are created
>>  >> correctly. When ssh stalls there are some
icmp packets out on lo0 with
>>  >> source and destination ip address of em0,
which I believe is not
>>  >> correct (set skip on lo0 does not help).
Also tried with tcp ...
>>  >> modulate state but same result.
>>  >
>>  > modulate state is known to be broken:
>>  >
>>  > http://wiki.freebsd.org/JeremyChadwick/Commonly_r
eported_issues
>>  >
>>  > Regarding the "when large amounts of
data is sent, the connection
>>  > breaks" issue:
>>  >
>>  > I've reproduced this a few times on our
systems (using the exact same
>>  > method you do: dmesg, cat'ing large files, or
scp'ing -- anything using
>>  > large TCP packets), and it's always been
caused by improper pf(4) rules
>>  > where state was broken.  In every case, the
"state mismatch" counter
>>  > shown in pfctl -s info would increase.
>>
>>
>> state-mismatch counter does not increase, all
"Counters" are 0 except
>>  match (pfctl -si).  When large amounts of data is
sent the connection
>>  stalls and continues from time to time very slow;
when it continues
>>  there are logged icmp packets out on lo0 from
(em0) to (em0) which
>>  looks pretty weird to me.
>>
>>
>>  Cristian
>
>    This may be a PMTUD issue. Make sure your ICMP
packets can travel
> back and forth unhindered and that there are no scrub
rules that may
> clear out the DF flag on them.

There's no no-df scrub flag, also no icmp filters.

Cristian
_______________________________________________
freebsd-pffreebsd.org mailing list

http://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to
"freebsd-pf-unsubscribefreebsd.org"