|
List Info
Thread: #32287 Processes hang in "mclpl" - feedback
|
|
| #32287 Processes hang in
"mclpl" - feedback |
|
2006-08-28 01:54:03 |
>
> So we see from the data that you have the same problem
like I do. We are
> leaking mbufs in the tcp recv. I've been reading the
code, but I have not
> found a problem...
>
They finally managed to make it happen:
Aug 26 14:28:16 mirror /netbsd: WARNING: mclpool limit
reached; increase NMBCLUS
TERS
Aug 26 14:30:11 mirror /netbsd: WARNING: mclpool limit
reached; increase NMBCLUS
TERS
Aug 26 14:31:14 mirror /netbsd: WARNING: mclpool limit
reached; increase NMBCLUS
TERS
Aug 26 14:37:55 mirror last message repeated 5 times
Aug 26 15:03:28 mirror /netbsd: WARNING: mclpool limit
reached; increase NMBCLUS
TERS
Aug 26 15:07:53 mirror last message repeated 3 times
Aug 26 15:23:00 mirror last message repeated 8 times
Aug 26 15:31:48 mirror last message repeated 5 times
Aug 26 15:47:12 mirror /netbsd: WARNING: mclpool limit
reached; increase NMBCLUS
TERS
Crontab runs netstat every 5 minutes, which show just
before:
Sat Aug 26 14:20:00 CEST 2006
33001 mbufs in use:
32967 mbufs allocated to data
34 mbufs allocated to packet headers
14666 calls to protocol drain routines
small
ext cluster
route inuse 0
0 0
claims 19
0 0
releases 19
0 0
arp inuse 0
0 0
claims 604
443 443
releases 604
443 443
unix inuse 0
0 0
claims 151316
14061 5504
releases 151316
14061 5504
tcp inuse 34
0 0
claims 541276
0 0
releases 541242
0 0
tcp rx inuse 0
0 0
claims 454399042
454302211 454302211
releases 454399042
454302211 454302211
tcp tx inuse 32838
17142 17142
claims 3235280646
1585116978 1585116792
releases 3235247808
1585099836 1585099650
udp inuse 0
0 0
claims 11362
0 0
releases 11362
0 0
udp rx inuse 0
0 0
claims 25640
16844 16844
releases 25640
16844 16844
small
ext cluster
udp tx inuse 0
0 0
claims 10356
0 0
releases 10356
0 0
internet rx inuse 0
0 0
claims 454403644
454310455 454310455
releases 454403644
454310455 454310455
internet tx inuse 0
0 0
claims 693325148
3 3
releases 693325148
3 3
lo0 inuse 0
0 0
claims 93207
0 0
releases 93207
0 0
ex0 rx inuse 0
0 0
claims 454310898
454310898 454310898
releases 454310898
454310898 454310898
ex0 tx inuse 0
0 0
claims 2188204597
1076832568 1076832397
releases 2188204597
1076832568 1076832397
unknown data inuse 128
128 128
claims 2996472228
454311282 454311282
releases 2996472100
454311154 454311154
unknown header inuse 0
0 0
claims 693331839
0 0
releases 693331839
0 0
small
ext cluster
unknown soname inuse 0
0 0
claims 497540
0 0
releases 497540
0 0
unknown soopts inuse 0
0 0
claims 99586
0 0
releases 99586
0 0
unknown control inuse 0
0 0
claims 4
0 0
releases 4
0 0
And this mornings reading is:
Mon Aug 28 03:25:01 CEST 2006
53478 mbufs in use:
53429 mbufs allocated to data
49 mbufs allocated to packet headers
286534 calls to protocol drain routines
small
ext cluster
route inuse 0
0 0
claims 19
0 0
releases 19
0 0
arp inuse 0
0 0
claims 884
647 647
releases 884
647 647
unix inuse 0
0 0
claims 228266
20680 8191
releases 228266
20680 8191
tcp inuse 49
0 0
claims 736300
0 0
releases 736251
0 0
tcp rx inuse 0
0 0
claims 617399674
617271075 617271075
releases 617399674
617271075 617271075
tcp tx inuse 53301
27885 27885
claims 154404677
2183697636 2183141342
releases 154351376
2183669751 2183113457
udp inuse 0
0 0
claims 14544
0 0
releases 14544
0 0
udp rx inuse 0
0 0
claims 32344
21205 21205
releases 32344
21205 21205
small
ext cluster
udp tx inuse 0
0 0
claims 13095
0 0
releases 13095
0 0
internet rx inuse 0
0 0
claims 617440932
617317548 617317548
releases 617440932
617317548 617317548
internet tx inuse 0
0 0
claims 943883790
36582 36582
releases 943883790
36582 36582
internet inuse 0
0 0
claims 148
35 35
releases 148
35 35
lo0 inuse 0
0 0
claims 123414
0 0
releases 123414
0 0
ex0 rx inuse 0
0 0
claims 617318195
617318195 617318195
releases 617318195
617318195 617318195
ex0 tx inuse 0
0 0
claims 3002895211
1481989909 1481482587
releases 3002895211
1481989909 1481482587
unknown data inuse 128
128 128
claims 4123143040
617318579 617318579
releases 4123142912
617318451 617318451
small
ext cluster
unknown header inuse 0
0 0
claims 943861192
0 0
releases 943861192
0 0
unknown soname inuse 0
0 0
claims 674196
0 0
releases 674196
0 0
unknown soopts inuse 0
0 0
claims 140343
0 0
releases 140343
0 0
unknown control inuse 0
0 0
claims 12
0 0
releases 12
0 0
tcp tx inuse 53301
27885 27885
claims 154404677
2183697636 2183141342
releases 154351376
2183669751 2183113457
tcp rx:
154351376 + 53301 = 154404677
2183669751 + 27885 = 2183697636
2183113457 + 27885 = 2183141342
Is it really leaking?
Lund
--
Jorgen Lundman | <lundman lundman.net>
Unix Administrator | +81 (0)3 -5456-2687 ext 1017 (work)
Shibuya-ku, Tokyo | +81 (0)90-5578-8500 (cell)
Japan | +81 (0)3 -3375-1767 (home)
|
|
| #32287 Processes hang in
"mclpl" - feedback |
|
2006-08-28 02:00:54 |
On Aug 28, 10:54am, lundmanlundman.net (Jorgen
Lundman) wrote:
-- Subject: Re: #32287 Processes hang in "mclpl"
- feedback
|
| >
| > So we see from the data that you have the same
problem like I do. We are
| > leaking mbufs in the tcp recv. I've been reading the
code, but I have not
| > found a problem...
| >
|
| They finally managed to make it happen:
|
| Aug 26 14:28:16 mirror /netbsd: WARNING: mclpool limit
reached; increase NMBCLUS
| TERS
| Aug 26 14:30:11 mirror /netbsd: WARNING: mclpool limit
reached; increase NMBCLUS
| TERS
| Aug 26 14:31:14 mirror /netbsd: WARNING: mclpool limit
reached; increase NMBCLUS
| TERS
| Aug 26 14:37:55 mirror last message repeated 5 times
| Aug 26 15:03:28 mirror /netbsd: WARNING: mclpool limit
reached; increase NMBCLUS
| TERS
| Aug 26 15:07:53 mirror last message repeated 3 times
| Aug 26 15:23:00 mirror last message repeated 8 times
| Aug 26 15:31:48 mirror last message repeated 5 times
| Aug 26 15:47:12 mirror /netbsd: WARNING: mclpool limit
reached; increase NMBCLUS
| TERS
|
So in your case it is tcp tx, not rx as mine... Must be
driver dependent...
christos
|
|
| #32287 Processes hang in
"mclpl" - feedback |
|
2006-08-28 02:09:10 |
>
> So in your case it is tcp tx, not rx as mine... Must be
driver dependent...
>
> christos
>
The numbers seems to add up for me, but I perhaps just
don't know how to read
the output 
33001 mbufs in use:
tcp tx inuse 32838
17142 17142
claims 3235280646
1585116978 1585116792
releases 3235247808
1585099836 1585099650
3235280646 - 3235247808 = 32838
32838 < 33001
What am I looking for here to know that it's leaking?
Lund
--
Jorgen Lundman | <lundmanlundman.net>
Unix Administrator | +81 (0)3 -5456-2687 ext 1017 (work)
Shibuya-ku, Tokyo | +81 (0)90-5578-8500 (cell)
Japan | +81 (0)3 -3375-1767 (home)
|
|
| #32287 Processes hang in
"mclpl" - feedback |
|
2006-08-28 02:12:22 |
On Aug 28, 11:09am, lundmanlundman.net (Jorgen
Lundman) wrote:
-- Subject: Re: #32287 Processes hang in "mclpl"
- feedback
|
|
| >
| > So in your case it is tcp tx, not rx as mine... Must
be driver dependent...
| >
| > christos
| >
|
| The numbers seems to add up for me, but I perhaps just
don't know how to read
| the output
|
| 33001 mbufs in use:
|
| tcp tx inuse 32838
17142 17142
| claims 3235280646
1585116978 1585116792
| releases 3235247808
1585099836 1585099650
|
| 3235280646 - 3235247808 = 32838
| 32838 < 33001
|
| What am I looking for here to know that it's leaking?
The number of mbufs in use should not be so large. And it
keeps growing.
The fact that they are in use means that they cannot be
freed. What driver
are you using again? Anything particular with the traffic
the machine
generates?
christos
|
|
| #32287 Processes hang in
"mclpl" - feedback |
|
2006-08-28 02:18:34 |
>
> The number of mbufs in use should not be so large. And
it keeps growing.
> The fact that they are in use means that they cannot be
freed. What driver
> are you using again? Anything particular with the
traffic the machine
> generates?
Ah, I see.
ex0 is probed as:
ex0 at pci2 dev 1 function 0: 3Com 3c905CX-TX 10/100
Ethernet with mngmt (rev. 0
x40)
ex0: interrupting at irq 5
ex0: MAC address 00:26:54:08:ca:f8
The machine is an FTP mirror, which includes NetBSD. 99.9%
traffic is FTP,
although a lot of that is TLS/SSL FTP.
The only unusual settings from a vanilla NetBSD-3.0.1 is the
rediculously large
network settings:
NMBCLUSTERS=65536 (was 32768)
kern.sbmax=8388608
net.inet.tcp.sendspace=321796
net.inet.tcp.recvspace=321796
net.inet.tcp.init_win=8
FTPD is a single process setup (relying on select()..) as
opposed to the usual
process-per-login.
I can try without LOAN pages as suggested as well.
Lund
--
Jorgen Lundman | <lundmanlundman.net>
Unix Administrator | +81 (0)3 -5456-2687 ext 1017 (work)
Shibuya-ku, Tokyo | +81 (0)90-5578-8500 (cell)
Japan | +81 (0)3 -3375-1767 (home)
|
|
[1-5]
|
|
|
about | contact Other archives ( Real Estate discussion Medical topics )
|