We are using FreeBSD to run the Dante SOCKS proxy server to
accelerate a
high-latency (approximately 1-second round-trip) network
link. We need to
support many concurrent transfers of large files. To do
this, we have set
the machine up with the following parameters.
Compiled Dante with the following setting in
include/config.h
SOCKD_BUFSIZETCP = (1024*1000)
/etc/sysctl.conf :
kern.ipc.maxsockbuf=4194304
net.inet.tcp.sendspace=2097152
net.inet.tcp.recvspace=2097152
/boot/loader.conf :
kern.ipc.maxsockets="0" (also tried 25600, 51200,
102400, and 409600)
kern.ipc.nmbclusters="0" (also tried 102400 and
409600)
(Looking at the code, it seems that 0 means not to set a max
for the above
two controls.)
If kern.ipc.nmbclusters is set to 25600, the system will
hard freeze when
"vmstat -z" shows the number of clusters reaches
25600. If
kern.ipc.nmbclusters is set to 0 (or 102400), the system
will hard freeze
when "vmstat -z" shows the number of clusters is
around 66000. When it
freezes, the number of Kbytes allocated to network (as shown
by
"netstat -m") is roughly 160,000 (160MB).
For a while, we thought that there may be a limit of 65536
mbuf clusters, so
we tested building the kernel with MCLSHIFT=12, which makes
each mbcluster
4096-bytes. With this configuration, nmbclusters only
reached about 33000
before the system froze. The number of Kbytes allocated to
network (as
shown by "netstat -m") still maxed out at around
160,000.
Now, it seems that we are running into some other memory
limitation that
occurs when our network allocation gets close to 160MB. We
have tried
tuning paramaters such as KVA_PAGES, vm.kmem_size,
vm.kmem_size_max, etc.
Though, we are unsure if the mods we made there helped in
any way.
This is all being done on Celeron 2.8GHz machines with 3+ GB
of RAM running
FreeBSD 5.3. We are very much tied to this platform at the
moment, and
upgrading is not a realistic option for us. We would like
to tune the
systems to not lockup. We can currently work around the
problem (by using
smaller buffers and such), but it is at the expense of
network throughput,
which is less than ideal.
Are there any other parameters that would help us to
allocate more memory to
the kernel networking? What other options should we look
into?
Thanks,
Ed Mandy
_______________________________________________
freebsd-net freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to
"freebsd-net-unsubscribefreebsd.org"
|
On Sat, 10 Nov 2007, Ed Mandy wrote:
> If kern.ipc.nmbclusters is set to 25600, the system
will hard freeze when
> "vmstat -z" shows the number of clusters
reaches 25600. If
> kern.ipc.nmbclusters is set to 0 (or 102400), the
system will hard freeze
> when "vmstat -z" shows the number of clusters
is around 66000. When it
> freezes, the number of Kbytes allocated to network (as
shown by "netstat
> -m") is roughly 160,000 (160MB).
>
> For a while, we thought that there may be a limit of
65536 mbuf clusters, so
> we tested building the kernel with MCLSHIFT=12, which
makes each mbcluster
> 4096-bytes. With this configuration, nmbclusters only
reached about 33000
> before the system froze. The number of Kbytes
allocated to network (as
> shown by "netstat -m") still maxed out at
around 160,000.
>
> Now, it seems that we are running into some other
memory limitation that
> occurs when our network allocation gets close to 160MB.
We have tried
> tuning paramaters such as KVA_PAGES, vm.kmem_size,
vm.kmem_size_max, etc.
> Though, we are unsure if the mods we made there helped
in any way.
>
> This is all being done on Celeron 2.8GHz machines with
3+ GB of RAM running
> FreeBSD 5.3. We are very much tied to this platform at
the moment, and
> upgrading is not a realistic option for us. We would
like to tune the
> systems to not lockup. We can currently work around
the problem (by using
> smaller buffers and such), but it is at the expense of
network throughput,
> which is less than ideal.
>
> Are there any other parameters that would help us to
allocate more memory to
> the kernel networking? What other options should we
look into?
I'd like to diagnose "freeze hard" a little more
to understand what's going
on. Hopefully this won't be too disruptive for your
environment while you're
doing it.
First off, can you tell me how you're accessing the system
to run diagnostic
tools, monitor it, etc? Remember that if you run out of
clusters, you may
experience network deadlocks that prevent SSH sessions from
operating (since
there may be no memory for them to operate), so direct
console access may be
required to effectively monitor the system when in an
extreme state of low
memory in the network stack. Could you tell me if you are
using a serial
console or the video console? (Or firewire, I suppose?)
FreeBSD 5.3 was the first release to include an MPSAFE
network stack, and
there were a number of optionally compiled features that
could disable MPSAFE
networking, resulting in the Giant lock being held over
network operations.
Could you tell me what the value of the sysctl
debug.mpsafenet is?
When the system appears to hard hang, does it recover if,
say, left five
minutes? What if you unplug the network cable and leave it
five minutes?
Does the numlock key on the console work? If you leave the
console logged in
and running an application (such as "sleep
100000") and the system hangs, what
do you see if you hit Ctrl-T?
If you compile options BREAK_TO_DEBUGGER into the kernel and
generate a serial
break / hit ctrl-alt-esc, are you able to get into the
debugger? If you type
in "trace", what do you get? (There is a chapter
of the developer's handbook
that talks about using the kernel debugger, FYI). With 5.3,
we found that
usig a serial console to get to the debugger was a lot more
reliable than the
video console -- this is in part because a significant
amount of the kernel
(especially file systems and the video console) still run
under Giant, so a
thread hanging while holding Giant can prevent a console
break from getting to
the debugger. My advice would be to use a serial console
anyway, if possible,
when debugging, as it means you can use a second machine to
copy and paste DDB
output into a file to e-mail out later. After about the
third line of a
kernel stack trace, copying addresses out by hand becomes
pretty painful  .
Unfortunately, I have to say that my first advice would be
to upgrade -- not
just because a lot of work has been done relating to network
stack performance
and stability since 5.3, but also because the debugging
tools have gotten a
lot better since then. For example, in more recent versions
the kernel
debugging includes memory monitoring tools, commands to more
readily extract
debugging information, etc. 5.3 is a solid and functional
release, but when
it comes to debugging problems of this sort, being on a more
recent release
means you're more likely to see the problem already fixed,
and even if not, it
will be easier for us to fix it. I understand that may
simply not be
possible, but if you have that flexibility, it's good
advice.
A general comment on configuration: increasing the maximum
memory allocated to
the network stack can indeed increase your KVA usage
significantly. You might
well find that tuning KVA up is required to run with very
high memory
configurations for the network stack, so your intuitions
about tuning that up
aren't bad. However, when you run out of KVA, the result is
usually a panic
(since the kernel basically has to halt), so if you're not
seeing a panic then
you're probably not yet hitting the limit.
Robert N M Watson
Computer Laboratory
University of Cambridge
_______________________________________________
freebsd-netfreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to
"freebsd-net-unsubscribefreebsd.org"
|
