Re: multiple rx/tx rings and interrupt delivery on newer nics

Thor Lancelot Simon wrote:
> On Wed, Jun 13, 2007 at 01:12:23AM -0700, Darren Reed
wrote:
>   
>> Has anyone started to think about how NetBSD can
take advantage
>> of NICs that have a larger number of rx/tx
descriptor rings,
>> and/or MSI interrupts?
>>     
>
> Think, yes -- do, no.
>
>   
>> For example, would this get tied in with ALTQ or
something else?
>>     
>
> Ooh, I hope not.  ALTQ is slow.
>
>   
>> Should I be able to dedicate a rx/tx ring pair to
http traffic
>> and another to ssh, etc?
>>     
>
> I've been thinking about this -- I know Solaris can do
this now -- and
> I have to say I'm somewhat skeptical.  Does this really
save much overhead?
>   

It's not just about speeding up delivery to an application,
but being
able to deliver a certain QOS too because you don't have to
deal
with each queue equally.

For local delivery, if you have specific rings being served
by specific
cores/cpus then it stands to reason you should see some
locality of
execution style benefits.

> I might rather have the packets all land in one ring,
tagged for what
> rule they've matched in classification.  Now, on the
other hand, what would
> be of great performance benefit for _my_ application,
at least, on a
> multiprocessor, would be the ability to use multiple
rings according to
> a slightly different set of classification parameters,
e.g. destination
> IP address.

I believe the standard matching for packets is a 5-tuple:
source address, destination address, protocol, source port,
destination port

> That way you can avoid stomping the cache of CPU A
when
> packets come in that will only ever be touched by the
network stack on
> CPU B.
>   

Right, see above.

>> ...but to do any of that will require some sort of
framework.
>>     
>
> Yes.  Do you know how what Sun recently did has been
working out?
>   

Yes, I do...the people working on the project are my
co-workers, so
this is about as involved in any NetBSD stuff as I can
realisticly be.
The project to do this is known as "crossbow" and
is hosted on
the opensolaris.org website at:
http:
//www.opensolaris.org/os/project/crossbow/

Darren

On 6/14/07, Darren Reed <darrenrfastmail.net> wrote:
> Thor Lancelot Simon wrote:
> > On Wed, Jun 13, 2007 at 01:12:23AM -0700, Darren
Reed wrote:
> >
> >> Has anyone started to think about how NetBSD
can take advantage
> >> of NICs that have a larger number of rx/tx
descriptor rings,
> >> and/or MSI interrupts?
> >>
> >
> > Think, yes -- do, no.
> >
> >
> >> For example, would this get tied in with ALTQ
or something else?
> >>
> >
> > Ooh, I hope not.  ALTQ is slow.
> >
> >
> >> Should I be able to dedicate a rx/tx ring pair
to http traffic
> >> and another to ssh, etc?
> >>
> >
> > I've been thinking about this -- I know Solaris
can do this now -- and
> > I have to say I'm somewhat skeptical.  Does this
really save much overhead?
> >
>
> It's not just about speeding up delivery to an
application, but being
> able to deliver a certain QOS too because you don't
have to deal
> with each queue equally.
>
> For local delivery, if you have specific rings being
served by specific
> cores/cpus then it stands to reason you should see some
locality of
> execution style benefits.
>
> > I might rather have the packets all land in one
ring, tagged for what
> > rule they've matched in classification.  Now, on
the other hand, what would
> > be of great performance benefit for _my_
application, at least, on a
> > multiprocessor, would be the ability to use
multiple rings according to
> > a slightly different set of classification
parameters, e.g. destination
> > IP address.
>
> I believe the standard matching for packets is a
5-tuple:
> source address, destination address, protocol, source
port, destination port
>
> > That way you can avoid stomping the cache of CPU A
when
> > packets come in that will only ever be touched by
the network stack on
> > CPU B.
> >
>
> Right, see above.
>
> >> ...but to do any of that will require some
sort of framework.
> >>
> >
> > Yes.  Do you know how what Sun recently did has
been working out?
> >
>
> Yes, I do...the people working on the project are my
co-workers, so
> this is about as involved in any NetBSD stuff as I can
realisticly be.
> The project to do this is known as "crossbow"
and is hosted on
> the opensolaris.org website at:
> http:
//www.opensolaris.org/os/project/crossbow/
>
> Darren
>
>

http://www.xensource.com/files/xensum
mit_4/xen-ny-summit-smartnic_Pratt.pdf


-- 
~Cherry