IFQ_MAXLEN: How large can it be?

On 15.11-10:48, Steven M. Bellovin wrote:
> > 
> > So I bumped this number on our quite busy firewall
up from 256 
> > to 1024 and later to 4096, but I still get
1'026'678 dropped 
> > packets during 8 days uptime.
> > 
> It's far from clear to me that this is a big help. 
There's a fair amount
> of literature that says that too-large router queues
are bad, since they
> end up having many retransmissions of the same data.  I
suggest that you
> look at other resources -- CPU and output line rate
come to mind -- and
> start playing with some of the fancier queueing options
on your output
> link.  (I wonder -- it would be nice to be able to do
RED on things like
> the IP input queue.  Is that possible?)
> 

What is "RED"? What do you mean bei "output
line rate"?
I wasn't aware I had queueing options on my output links.
Did you mean ALTQ? Does that work?

I'll gladly do any measuring and reporting you request if
you 
help me to interpret the values 

I have to say though, that we're somewhat limited playing
around
with this system as it is in production. My timeslot for
reboots
is some hours every two weeks.

Chris

-- 
------------------------------------------------------------
----------
Christoph Kaegi                                          
kgczhwin.ch
------------------------------------------------------------
----------

On Thu, 16 Nov 2006 08:44:32 +0100, Christoph Kaegi
<kgczhwin.ch> wrote:

> On 15.11-10:48, Steven M. Bellovin wrote:
> > > 
> > > So I bumped this number on our quite busy
firewall up from 256 
> > > to 1024 and later to 4096, but I still get
1'026'678 dropped 
> > > packets during 8 days uptime.
> > > 
> > It's far from clear to me that this is a big help.
 There's a fair amount
> > of literature that says that too-large router
queues are bad, since they
> > end up having many retransmissions of the same
data.  I suggest that you
> > look at other resources -- CPU and output line
rate come to mind -- and
> > start playing with some of the fancier queueing
options on your output
> > link.  (I wonder -- it would be nice to be able to
do RED on things like
> > the IP input queue.  Is that possible?)
> > 
> 
> What is "RED"? What do you mean bei
"output line rate"?
> I wasn't aware I had queueing options on my output
links.
> Did you mean ALTQ? Does that work?

"RED" is an output interface queueing discipline
(it stands for, as I
recall, "Random Early Drop", though the altq.conf
file says "Random Early
Detection"). Essentially, packets in an output queue
are dropped with a
probability once the queue reaches a certain length.  Yes,
that's right;
it doesnot simply drop newly-arriving packets when the queue
has reached
its maximum size.  It seems counter-intuitive, but it works
better than
the default tail-drop strategy, because it causes sending
TCPs to back off
their retransmission timers more quickly and hence avoids
duplicate copies
of packets in the queue.  (I was talking about ALTQ, but I
don't see a way
to apply that to the IP input queue, which is too bad.)

By "output line rate", I was asking how fast the
output size runs.  From
your later message about 'netstat -q', that's not the
relevant question.
I think your CPU is too slow or too overloaded compared to
your input line
speed.  What's happening is that packets are arriving faster
than your IP
stack can process them.  The only issue is how long such
bursts last;
ultimately, you're going to get packet drops if your CPU
can't keep up
with the arrival rate.

Let me repeat what I said earlier: I think that having a 4K
queue length
is not only not useful, it's quite possibly
counter-productive.

You showed four active wm interfaces in a message.  How fast
are they
running?  I know that some wm interfaces can run at gigabit
speeds -- are
you trying to do that?  How fast is your CPU?  What do 'top'
or
'sysstat' show for CPU idle percentage?

		--Steven M. Bellovin, http://www.cs.columbi
a.edu/~smb

On 16.11-09:35, Steven M. Bellovin wrote:

> > What is "RED"? What do you mean bei
"output line rate"?
> > I wasn't aware I had queueing options on my output
links.
> > Did you mean ALTQ? Does that work?
> 
> "RED" is an output interface queueing
discipline (it stands for, as I
> recall, "Random Early Drop", though the
altq.conf file says "Random Early
> Detection"). Essentially, packets in an output
queue are dropped with a
> probability once the queue reaches a certain length. 
Yes, that's right;
> it doesnot simply drop newly-arriving packets when the
queue has reached
> its maximum size.  It seems counter-intuitive, but it
works better than
> the default tail-drop strategy, because it causes
sending TCPs to back off
> their retransmission timers more quickly and hence
avoids duplicate copies
> of packets in the queue.  (I was talking about ALTQ,
but I don't see a way
> to apply that to the IP input queue, which is too bad.)

Thanks for the explanation.

> You showed four active wm interfaces in a message.  How
fast are they
> running?  I know that some wm interfaces can run at
gigabit speeds -- are
> you trying to do that?  

wm0:
flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu
1500
        media: Ethernet autoselect (1000baseT full-duplex)

wm1:
flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu
1500
        media: Ethernet autoselect (100baseTX full-duplex)

wm2:
flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu
1500
        media: Ethernet autoselect (100baseTX full-duplex)

wm3:
flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu
1500
        media: Ethernet autoselect (100baseTX full-duplex)

wm0 runs gigabit, this is the inwards interface which should
be able to keep up with traffic from internet an dmz, which
both are 100MBit/s

> How fast is your CPU?  

Inter Xeon 2.7GHz

> What do 'top' or
> 'sysstat' show for CPU idle percentage?

Usually the cpu shows between 50% and 90% idle.
Though there may be bursts which load the CPU more.i
I never really watched it maxing out.

I'll try switching on the ip4csum,tcp4csum,udp4csum,tso4 
options on the interfaces tonight and then I try to 
graph the CPU load somehow.

Thanks
Chris

-- 
------------------------------------------------------------
----------
Christoph Kaegi                                          
kgczhwin.ch
------------------------------------------------------------
----------

On Nov 16, 2006, at 6:35 AM, Steven M. Bellovin wrote:

> "RED" is an output interface queueing
discipline (it stands for, as I
> recall, "Random Early Drop", though the
altq.conf file says "Random  
> Early
> Detection").

I've always viewed this terminology discrepancy this way:

- From a router's point of view, it's Random Early Drop.

- From an endpoint's point of view, it's Random Early
Detection (of  
congestion).

-- thorpej

On Nov 16, 2006, at 8:05 AM, Christoph Kaegi wrote:

> I'll try switching on the
ip4csum,tcp4csum,udp4csum,tso4
> options on the interfaces tonight and then I try to
> graph the CPU load somehow.

ip4csum should certainly help a router.  The others may not
have so  
much of an impact, but they certainly won't hurt.

-- thorpej

Jason Thorpe <thorpejshagadelic.org> writes:

> On Nov 16, 2006, at 6:35 AM, Steven M. Bellovin wrote:
>
>> "RED" is an output interface queueing
discipline (it stands for, as I
>> recall, "Random Early Drop", though the
altq.conf file says "Random
>> Early
>> Detection").
>
> I've always viewed this terminology discrepancy this
way:
>
> - From a router's point of view, it's Random Early
Drop.
>
> - From an endpoint's point of view, it's Random Early
Detection (of
> congestion).

Sally consitently uses "detection", and that
should be definitive.

http://www.icir.or
g/floyd/red.html

-- 

    Greg Troxel <gdtir.bbn.com>

On Thu, 16 Nov 2006 17:05:17 +0100, Christoph Kaegi
<kgczhwin.ch> wrote:

> > You showed four active wm interfaces in a message.
 How fast are they
> > running?  I know that some wm interfaces can run
at gigabit speeds -- are
> > you trying to do that?  
> 
> wm0:
flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu
1500
>         media: Ethernet autoselect (1000baseT
full-duplex)
> 
> wm1:
flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu
1500
>         media: Ethernet autoselect (100baseTX
full-duplex)
> 
> wm2:
flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu
1500
>         media: Ethernet autoselect (100baseTX
full-duplex)
> 
> wm3:
flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu
1500
>         media: Ethernet autoselect (100baseTX
full-duplex)
> 
> wm0 runs gigabit, this is the inwards interface which
should
> be able to keep up with traffic from internet an dmz,
which
> both are 100MBit/s

The question, then, is which interface the dropped packets
are coming
from.  They all end up in one IP input queue.  And if the
CPU is that
idle, I'm quite perplexed.  Catching it in the act of
dropping packets
might be necessary -- watch for CPU load at the time, and
also input
packet counts on those four interfaces.


		--Steven M. Bellovin, http://www.cs.columbi
a.edu/~smb

On Thu, 16 Nov 2006 12:53:12 -0500, Greg Troxel <gdtir.bbn.com> wrote:


> Sally consitently uses "detection", and that
should be definitive.
> 
> http://www.icir.or
g/floyd/red.html
> 
Indeed.  (I was offline when I wrote my note, or I'd have
checked.  But
I've discussed the topic enough with Sally that it should
have stuck in my
brain.)

		--Steven M. Bellovin, http://www.cs.columbi
a.edu/~smb

On 16.11-12:57, Steven M. Bellovin wrote:
> On Thu, 16 Nov 2006 17:05:17 +0100, Christoph Kaegi
<kgczhwin.ch> wrote:
> 
> The question, then, is which interface the dropped
packets are coming
> from.  They all end up in one IP input queue.  And if
the CPU is that
> idle, I'm quite perplexed.  Catching it in the act of
dropping packets
> might be necessary -- watch for CPU load at the time,
and also input
> packet counts on those four interfaces.


OK, I'll try to graph CPU Load and input/output packets now.
For input/output packets I can use netstat -ni, how would I
go about getting CPU load every second? 
How should I interpret sysctl kern.cp_time?

Or can you point me to some useful C functions so I could
write and 
run a programm which outputs the relevant values every
second?

-- 
------------------------------------------------------------
----------
Christoph Kaegi                                          
kgczhwin.ch
------------------------------------------------------------
----------