List Info

Thread: Re: Problems with netgraph
Re: Problems with netgraph
country flaguser name
Finland		 2008-05-07 16:42:14 
Julian Elischer wrote:
> Oleksandr Samoylyk wrote:
>> Julian Elischer wrote:
>>> Oleksandr Samoylyk wrote:
>>>> Julian Elischer wrote:
>>>>> Oleksandr Samoylyk wrote:
>>>>>> David DeSimone wrote:
>>>>>>> Julian Elischer <julianelischer.org> wrote:
>>>>>>>> unfortunatly I've been
totally  ignoring this thread because it 
>>>>>>>> said
>>>>>>>> "trouble with em"
in the topic..
>>>>>>>> If you'd said "trouble
with mpd" then maybe I'd have looked 
>>>>>>>> earlier..
>>>>>>>
>>>>>>> In the poster's defense, the
only symptom that started this was this
>>>>>>> info from ps:
>>>>>>>
>>>>>>>   PID USERNAME  THR PRI NICE  
SIZE    RES STATE  C   TIME   WCPU 
>>>>>>> COMMAND
>>>>>>>    29 root        1 -68    -   
 0K    16K CPU5   5 196:41 
>>>>>>> 100.00% em0 taskq
>>>>>>>
>>>>>>> So tracking it down to mpd has
been a process of elimination in 
>>>>>>> figuring
>>>>>>> out why packets absorb so much
CPU.
>>>>>>>
>>>>>>
>>>>>> Here is a result of profiling:
>>>>>>
>>>>>> http://lists.freebsd.org/pipermail/freebsd-ne
t/2008-May/017901.html
>>>>>>
>>>>>
>>>>>
>>>>>                  0.00        0.00     
16/1643247     igmp_input [833]
>>>>>                  0.03        0.01    
614/1643247     icmp_input [272]
>>>>>                 93.07       17.27
1642617/1643247     encap4_input [9]
>>>>> [10]    49.8   93.10       17.27
1643247         rip_input [10]
>>>>>                 14.26        0.88 
600796/749987      
>>>>> _mtx_lock_sleep [21]
>>>>>                  0.16        1.70
1643863/1643863     raw_append [93]
>>>>>                  0.00        0.24  
36345/176995      
>>>>> _mtx_unlock_sleep
>>>>> [114]
>>>>>                  0.01        0.00
1643863/5117962     jailed [278]
>>>>>                  0.00        0.00   
1292/1843        m_copym [666]
>>>>>                  0.00        0.00    
676/8214484     m_freem [34]
>>>>>
>>>>>
>>>>>
>>>>> 50% of time in rip_input???
>>>>>
>>>>> that's unexpected.. what is the
traffic?
>>>>
>>>>
>>>> more than 20k pps
>>>
>>> I mean, what KIND of traffic?
>>> I'm surprised it 's calling rip_input().. why
is it calling
>>> encap4_input()? (which calls rip_input).. what
is the IP protocol
>>> of all these packets?
>>>
>>> what does a small snippet of traffic  show?
>>>
>>> (tcpdump -i em0 -s0 -e -n -c 64)
>>
>>
>> GRE (pptp-tunnels)
>>
>> tcpdump -i em0 -s0 -e -n -c 64
>>
>> 22:43:28.818558 00:19:d1:03:ce:7e >
00:12:cf:53:61:30, ethertype IPv4 
>> (0x0800), length 250: 10.0.0.8.22 >
10.0.14.191.3513: P 
>> 3482978131:3482978327(196) ack 537019550 win 128
>> 22:43:28.818892 00:19:d1:03:ce:7e >
00:12:cf:53:61:30, ethertype IPv4 
>> (0x0800), length 1447: 10.0.0.8 > 10.0.113.178:
GREv1, call 256, seq 
>> 949074, proto PPP (0x880b), length 1413: IP
(0x0021), length 1401: 
>> 89.252.42.44.40002 > 77.120.207.229.3038: .
588169704:588171064(1360) 
>> ack 239105401 win 58593
>> 22:43:28.819014 00:19:d1:03:ce:7e >
00:12:cf:53:61:30, ethertype IPv4 
>> (0x0800), length 1447: 10.0.0.8 > 10.0.34.188:
GREv1, call 0, seq 
>> 28109460, proto PPP (0x880b), length 1413: IP
(0x0021), length 1401: 
>> 195.91.147.100.62073 > 77.120.205.18.43463: . 
>> 2979675127:2979676487(1360) ack 3166103641 win
65212
>> 22:43:28.819016 00:11:92:c5:a7:40 >
00:19:d1:03:ce:7e, ethertype IPv4 
>> (0x0800), length 111: 10.0.197.49 > 10.0.0.8:
GREv1, call 39317, seq 
>> 1841830, ack 2346966, proto PPP (0x880b), length
77: IP (0x0021), 
>> length 61: 77.120.205.65.1109 >
89.254.129.137.63441: . ack 2160932834 
>> win 17680 <nop,nop,sack 2
{6801:8161}{4081:5441}>
>> 22:43:28.819025 00:19:d1:03:ce:7e >
00:12:cf:53:61:30, ethertype IPv4 
>> (0x0800), length 1447: 10.0.0.8 > 10.0.113.178:
GREv1, call 256, seq 
>> 949075, proto PPP (0x880b), length 1413: IP
(0x0021), length 1401: 
>> 89.252.42.44.40002 > 77.120.207.229.3038: .
1360:2720(1360) ack 1 win 
>> 58593
>> 22:43:28.819109 00:19:d1:03:ce:7e >
00:12:cf:53:61:30, ethertype IPv4 
>> (0x0800), length 46: 10.0.0.8 > 10.0.197.49:
GREv1, call 16384, ack 
>> 1841830, no-payload, proto PPP (0x880b), length 12
>> 22:43:28.819152 00:19:d1:03:ce:7e >
00:12:cf:53:61:30, ethertype IPv4 
>> (0x0800), length 1447: 10.0.0.8 > 10.0.7.198:
GREv1, call 32768, seq 
>> 2854449, proto PPP (0x880b), length 1413: IP
(0x0021), length 1401: 
>> 88.81.240.172.80 > 77.120.206.140.3573: .
2985100912:2985102272(1360) 
>> ack 3056545326 win 32254
>> 22:43:28.819259 00:19:d1:03:ce:7e >
00:12:cf:53:61:30, ethertype IPv4 
>> (0x0800), length 1447: 10.0.0.8 > 10.0.34.188:
GREv1, call 0, seq 
>> 28109461, proto PPP (0x880b), length 1413: IP
(0x0021), length 1401: 
>> 195.91.147.100.62073 > 77.120.205.18.43463: .
1360:2720(1360) ack 1 
>> win 65212
>> 22:43:28.819632 00:19:d1:03:ce:7e >
00:12:cf:53:61:30, ethertype IPv4 
>> (0x0800), length 87: 10.0.0.8 > 10.0.3.36:
GREv1, call 33358, seq 
>> 809410, proto PPP (0x880b), length 53: IP (0x0021),
length 41: 
>> 89.189.135.94.29934 > 77.120.205.175.64039: .
ack 980726719 win 65535
>> 22:43:28.820102 00:11:92:c5:a7:40 >
00:19:d1:03:ce:7e, ethertype IPv4 
>> (0x0800), length 91: 10.0.7.198 > 10.0.0.8:
GREv1, call 42872, seq 
>> 1687270, ack 2854449, proto PPP (0x880b), length
57: IP (0x0021), 
>> length 41: 77.120.206.140.3573 >
88.81.240.172.80: . ack 1360 win 65535
>> 22:43:28.820172 00:19:d1:03:ce:7e >
00:12:cf:53:61:30, ethertype IPv4 
>> (0x0800), length 46: 10.0.0.8 > 10.0.7.198:
GREv1, call 32768, ack 
>> 1687270, no-payload, proto PPP (0x880b), length 12
>> 22:43:28.820255 00:19:d1:03:ce:7e >
00:12:cf:53:61:30, ethertype IPv4 
>> (0x0800), length 461: 10.0.0.8 > 10.0.7.198:
GREv1, call 32768, seq 
>> 2854450, proto PPP (0x880b), length 427: IP
(0x0021), length 415: 
>> 88.81.240.172.80 > 77.120.206.140.3573: .
1360:1734(374) ack 1 win 32254
>> 22:43:28.820382 00:19:d1:03:ce:7e >
00:12:cf:53:61:30, ethertype IPv4 
>> (0x0800), length 1447: 10.0.0.8 > 10.0.42.36:
GREv1, call 16384, seq 
>> 3749540, proto PPP (0x880b), length 1413: MLPPP
(0x003d), length 1401: 
>> seq 0x039, Flags [begin], length 1400
>> 22:43:28.820386 00:19:d1:03:ce:7e >
00:12:cf:53:61:30, ethertype IPv4 
>> (0x0800), length 56: 10.0.0.8 > 10.0.42.36:
GREv1, call 16384, seq 
>> 3749541, proto PPP (0x880b), length 22: MLPPP
(0x003d), length 10: seq 
>> 0x039, Flags [end], length 9
>> 22:43:28.820510 00:19:d1:03:ce:7e >
00:12:cf:53:61:30, ethertype IPv4 
>> (0x0800), length 259: 10.0.0.8 > 10.0.166.29:
GREv1, call 256, seq 
>> 18113, proto PPP (0x880b), length 225: IP (0x0021),
length 213: 
>> 205.188.248.197.80 > 77.120.205.253.1192: P
298024346:298024518(172) 
>> ack 1774495680 win 16384
>> 22:43:28.820514 00:19:d1:03:ce:7e >
00:12:cf:53:61:30, ethertype IPv4 
>> (0x0800), length 87: 10.0.0.8 > 10.0.141.78:
GREv1, call 32768, seq 
>> 51008, proto PPP (0x880b), length 53: IP (0x0021),
length 41: 
>> 67.228.189.192.80 > 77.120.207.6.3857: F
98423782:98423782(0) ack 
>> 337838744 win 6970
>> 22:43:28.820894 00:11:92:c5:a7:40 >
00:19:d1:03:ce:7e, ethertype IPv4 
>> (0x0800), length 1451: 10.0.162.204 > 10.0.0.8:
GREv1, call 41673, seq 
>> 145663, ack 88392, proto PPP (0x880b), length 1417:
IP (0x0021), 
>> length 1401: 77.120.206.128.3591 >
89.178.5.14.22979: . 
>> 2176487112:2176488472(1360) ack 921656808 win
16446
>> 22:43:28.820972 00:11:92:c5:a7:40 >
00:19:d1:03:ce:7e, ethertype IPv4 
>> (0x0800), length 1447: 10.0.162.204 > 10.0.0.8:
GREv1, call 41673, seq 
>> 145664, proto PPP (0x880b), length 1413: IP
(0x0021), length 1401: 
>> 77.120.206.128.3591 > 89.178.5.14.22979: .
1360:2720(1360) ack 1 win 
>> 16446
>> 22:43:28.821014 00:19:d1:03:ce:7e >
00:12:cf:53:61:30, ethertype IPv4 
>> (0x0800), length 46: 10.0.0.8 > 10.0.162.204:
GREv1, call 256, ack 
>> 145663, no-payload, proto PPP (0x880b), length 12
>> 22:43:28.821020 00:19:d1:03:ce:7e >
00:12:cf:53:61:30, ethertype IPv4 
>> (0x0800), length 1447: 10.0.0.8 > 10.0.4.108:
GREv1, call 32768, seq 
>> 39341, proto PPP (0x880b), length 1413: IP
(0x0021), length 1401: 
>> 80.93.56.145.80 > 77.120.206.211.3811: .
1861143175:1861144535(1360) 
>> ack 1545362008 win 65535
>> 22:43:28.821068 00:19:d1:03:ce:7e >
00:12:cf:53:61:30, ethertype IPv4 
>> (0x0800), length 46: 10.0.0.8 > 10.0.162.204:
GREv1, call 256, ack 
>> 145664, no-payload, proto PPP (0x880b), length 12
>> 22:43:28.821134 00:19:d1:03:ce:7e >
00:12:cf:53:61:30, ethertype IPv4 
>> (0x0800), length 1447: 10.0.0.8 > 10.0.4.108:
GREv1, call 32768, seq 
>> 39342, proto PPP (0x880b), length 1413: IP
(0x0021), length 1401: 
>> 80.93.56.145.80 > 77.120.206.211.3811: .
1360:2720(1360) ack 1 win 65535
>> 22:43:28.821264 00:19:d1:03:ce:7e >
00:12:cf:53:61:30, ethertype IPv4 
>> (0x0800), length 99: 10.0.0.8 > 10.0.75.234:
GREv1, call 16384, seq 
>> 414573, proto PPP (0x880b), length 65: IP (0x0021),
length 53: 
>> 201.246.146.143.55045 > 77.120.205.245.53009: .
ack 1074482170 win 
>> 32502 <nop,nop,timestamp 103973525 28967>
>> 22:43:28.821593 00:11:92:c5:a7:40 >
00:19:d1:03:ce:7e, ethertype IPv4 
>> (0x0800), length 106: 10.0.163.116 > 10.0.0.8:
GREv1, call 46363, seq 
>> 165235, ack 106705, proto PPP (0x880b), length 72:
IP (0x0021), length 
>> 56: 77.120.206.91.27005 > 194.50.85.251.27017:
UDP, length 27
>> 22:43:28.821670 00:19:55:d0:df:c0 >
00:19:d1:03:ce:7e, ethertype IPv4 
>> (0x0800), length 60: 10.0.14.191.3513 >
10.0.0.8.22: . ack 196 win 64240
>>
>>>>
>>>>> I see no netgraph  in the profile at
all.
>>>>> did you statically compile it all in at
compile time? (you should 
>>>>> if you want to see it)
>>>>>
>>>>
>>>> Tried both variants. Now not statically.
>>>
>>> make sure it is static and that the netgraph
nodes are all compiled 
>>> with -pg
>>>
>>
>> I'll try, but a bit later.
> 
> ok so we get somewhere.. GRE..
> 
> looks like UDP in PPP in GRE
> 
> is this a pptp session?

Yes.

>  this may be the issue:
>   http://
www.nabble.com/GRE-Mux-td16201899.html
> 

I think so. Should we hope for some progress in this
direction in future?

-- 
  Oleksandr Samoylyk
  OVS-RIPE
_______________________________________________
freebsd-netfreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to
"freebsd-net-unsubscribefreebsd.org"
Re: Problems with netgraph
country flaguser name
United Kingdom		 2008-05-07 21:52:12 
Oleksandr Samoylyk wrote:
>>
>> looks like UDP in PPP in GRE
>
> I think so. Should we hope for some progress in this
direction in future?

Probably not, unless someone is willing to come up to the
table and 
commit to writing and maintaining a Netgraph node to demux
GRE, although 
this is only shuffling the fanout elsewhere.

If MPD is relying on raw sockets to demultiplex GRE, then
this is what 
it's up against in terms of performance -- repeated
acquisitions of the 
INP sleep lock, and context switches when the socket buffer
low water 
mark is passed. It might have improved slightly in HEAD
since the move 
to rwlocks.

Like udp_input(), rip_input() suffers from the fact that the
stack has 
to deal with delivering datagrams to potentially more than
one socket, 
and there is no intermediate data structure to handle the
fan-out -- it 
walks the entire inp list every time. If you look at the
comments in 
udp_input() it's pretty clear this is a historical weakness
in the BSD 
implementation.

Windows, by the way, forces socket clients to explicitly
request 
reception of broadcast datagrams as of Windows Server 2003,
and 
multicasts are strictly delivered to group members only,
which 
eliminates that problematic loop -- you can always maintain
a tree of 
receivers that way.

I'm happy to review patches if someone else commits to
fixing it.

cheers
BMS
_______________________________________________
freebsd-netfreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to
"freebsd-net-unsubscribefreebsd.org"
[1-2]

about | contact  Other archives ( Real Estate discussion Medical topics )