10GbE speeds

I have a project with 3 workstations all needing high speed
access to  
about 6Tb of storage. In the past I've installed technology
such as  
fibre channel connected SAN storage (using Apple's xsan) for
up to a  
dozen workstations, but with only three workstations for
this project  
I'm thinking about 10GbE. The workstations will be OSX and
the server  
FreeBSD 7 with a bunch of disks in a RAID 5 or RAID 10
configuration.

Are 10GbE NICs and drivers significantly mature enough under
FreeBSD  
to accomplish this? I'd need to achieve about 60MB/s
transfer rate  
which is theoretically quite doable, as long as the drive
array can  
keep up with three streams of that speed. I'd use netatalk,
samba or  
nfs to share files depending on which I can eek the best
speeds out of.

Alternatively I could populate the server with 1GbE NICs,
one per  
workstation and use cross over cable. That way there is
absolutely no  
contention on the network.

Any thoughts about the viability of this? Is 10GbE in
production use  
with FreeBSD and does it scale well?


Cheers
Ari Maniatis




-------------------------->
ish
http://www.ish.com.au
Level 1, 30 Wilson Street Newtown 2042 Australia
phone +61 2 9550 5001   fax +61 2 9550 4001
GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E
3E49 102A


_______________________________________________
freebsd-performancefreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-p
erformance
To unsubscribe, send any mail to
"freebsd-performance-unsubscribefreebsd.org"

At Sat, 12 Apr 2008 09:08:13 +1000,
Aristedes Maniatis wrote:
> 
> I have a project with 3 workstations all needing high
speed access to  
> about 6Tb of storage. In the past I've installed
technology such as  
> fibre channel connected SAN storage (using Apple's
xsan) for up to a  
> dozen workstations, but with only three workstations
for this project  
> I'm thinking about 10GbE. The workstations will be OSX
and the server  
> FreeBSD 7 with a bunch of disks in a RAID 5 or RAID 10
configuration.
> 
> Are 10GbE NICs and drivers significantly mature enough
under FreeBSD  
> to accomplish this? I'd need to achieve about 60MB/s
transfer rate  
> which is theoretically quite doable, as long as the
drive array can  
> keep up with three streams of that speed. I'd use
netatalk, samba or  
> nfs to share files depending on which I can eek the
best speeds out of.
> 
> Alternatively I could populate the server with 1GbE
NICs, one per  
> workstation and use cross over cable. That way there is
absolutely no  
> contention on the network.
> 
> Any thoughts about the viability of this? Is 10GbE in
production use  
> with FreeBSD and does it scale well?

I am working with the Chelsio hardware and it seems to be
well
supported on FreeBSD.  All the machines in that system are
FreeBSD
though.  How do you intend to get the OSX systems to be
10GE?

Best,
George
_______________________________________________
freebsd-performancefreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-p
erformance
To unsubscribe, send any mail to
"freebsd-performance-unsubscribefreebsd.org"

On 15/04/2008, at 10:54 PM, gnnfreebsd.org wrote:
> I am working with the Chelsio hardware and it seems to
be well
> supported on FreeBSD.  All the machines in that system
are FreeBSD
> though.  How do you intend to get the OSX systems to be
10GE?


What sort of throughput are you getting with that setup? Are
you using  
NFS between the systems?

Good question about OSX, and I hadn't got to that part yet
  But
I  
was hoping that some OSX drivers existed. My fail back plan
is to put  
3 x 1GbE NICs into the server and just use crossover cable
between the  
3 workstations and the server to avoid any contention within
the  
ethernet network.

Ari Maniatis


-------------------------->
ish
http://www.ish.com.au
Level 1, 30 Wilson Street Newtown 2042 Australia
phone +61 2 9550 5001   fax +61 2 9550 4001
GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E
3E49 102A


_______________________________________________
freebsd-performancefreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-p
erformance
To unsubscribe, send any mail to
"freebsd-performance-unsubscribefreebsd.org"

Myricom has OS X support, Chelsio has support in the works.
I don't
know about Neterion or Intel.

    -Kip



On 4/15/08, Aristedes Maniatis <ariish.com.au> wrote:
>
> On 15/04/2008, at 10:54 PM, gnnfreebsd.org wrote:
> > I am working with the Chelsio hardware and it
seems to be well
> > supported on FreeBSD.  All the machines in that
system are FreeBSD
> > though.  How do you intend to get the OSX systems
to be 10GE?
>
>
> What sort of throughput are you getting with that
setup? Are you using
> NFS between the systems?
>
> Good question about OSX, and I hadn't got to that part
yet  
But I
> was hoping that some OSX drivers existed. My fail back
plan is to put
> 3 x 1GbE NICs into the server and just use crossover
cable between the
> 3 workstations and the server to avoid any contention
within the
> ethernet network.
>
> Ari Maniatis
>
>
> -------------------------->
> ish
> http://www.ish.com.au
> Level 1, 30 Wilson Street Newtown 2042 Australia
> phone +61 2 9550 5001   fax +61 2 9550 4001
> GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A
7D2E 3E49 102A
>
>
> _______________________________________________
> freebsd-performancefreebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-p
erformance
> To unsubscribe, send any mail to
> "freebsd-performance-unsubscribefreebsd.org"
>
_______________________________________________
freebsd-performancefreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-p
erformance
To unsubscribe, send any mail to
"freebsd-performance-unsubscribefreebsd.org"

Hi Aristedes,

We are/were testing FreeBSD on a Dell PE2950 with a Myricom
10GB 
PCI-Express copper CX card.  The driver seems mature.  In
tests out of 
the box, I only saw about 3Gbps from iperf (testing against
a linux 
system...and maybe there are other issues with our
environment/that 
system to tune up yet...i wouldn't take that number too
seriously).

I think some tuning could get it up to the maximum, anyways.
 It 
certainly took a little work to get our Linux systems up to
the max, so 
I would say that in general the defaults with 10G drivers on
any system 
need some work to hit the maximum bandwidth.

-Ben

Aristedes Maniatis wrote:
> I have a project with 3 workstations all needing high
speed access to 
> about 6Tb of storage. In the past I've installed
technology such as 
> fibre channel connected SAN storage (using Apple's
xsan) for up to a 
> dozen workstations, but with only three workstations
for this project 
> I'm thinking about 10GbE. The workstations will be OSX
and the server 
> FreeBSD 7 with a bunch of disks in a RAID 5 or RAID 10
configuration.
> 
> Are 10GbE NICs and drivers significantly mature enough
under FreeBSD to 
> accomplish this? I'd need to achieve about 60MB/s
transfer rate which is 
> theoretically quite doable, as long as the drive array
can keep up with 
> three streams of that speed. I'd use netatalk, samba or
nfs to share 
> files depending on which I can eek the best speeds out
of.
> 
> Alternatively I could populate the server with 1GbE
NICs, one per 
> workstation and use cross over cable. That way there is
absolutely no 
> contention on the network.
> 
> Any thoughts about the viability of this? Is 10GbE in
production use 
> with FreeBSD and does it scale well?
> 
> 
> Cheers
> Ari Maniatis
> 
> 
> 
> 
> -------------------------->
> ish
> http://www.ish.com.au
> Level 1, 30 Wilson Street Newtown 2042 Australia
> phone +61 2 9550 5001   fax +61 2 9550 4001
> GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A
7D2E 3E49 102A
> 
> 
> _______________________________________________
> freebsd-performancefreebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-p
erformance
> To unsubscribe, send any mail to 
> "freebsd-performance-unsubscribefreebsd.org"
> 
> 

-- 
Benjeman Meekhof - UM ATLAS/AGLT2 Computing
office: 734-764-3450 cell: 734-417-6312

_______________________________________________
freebsd-performancefreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-p
erformance
To unsubscribe, send any mail to
"freebsd-performance-unsubscribefreebsd.org"

At Wed, 16 Apr 2008 10:02:33 +1000,
Aristedes Maniatis wrote:
> 
> 
> On 15/04/2008, at 10:54 PM, gnnfreebsd.org wrote:
> > I am working with the Chelsio hardware and it
seems to be well
> > supported on FreeBSD.  All the machines in that
system are FreeBSD
> > though.  How do you intend to get the OSX systems
to be 10GE?
> 
> 
> What sort of throughput are you getting with that
setup? Are you using  
> NFS between the systems?

I haven't done any performance testing of bandwidth as yet
and we are
not using these for NFS.  

> Good question about OSX, and I hadn't got to that part
yet 
But I
> was hoping that some OSX drivers existed. My fail back
plan is to
> put 3 x 1GbE NICs into the server and just use
crossover cable
> between the 3 workstations and the server to avoid any
contention
> within the ethernet network.

I think you want either a pure 10GbE network or a pure 1GbE
network
because my understanding is that 10GbE switches have issues
with mixed
links.  I have not tested that first hand yet though.

Best,
George
_______________________________________________
freebsd-performancefreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-p
erformance
To unsubscribe, send any mail to
"freebsd-performance-unsubscribefreebsd.org"

Greetings,

Benjeman J. Meekhof wrote:
> Hi Aristedes,
>
> We are/were testing FreeBSD on a Dell PE2950 with a
Myricom 10GB 
> PCI-Express copper CX card.  The driver seems mature. 
In tests out of 
> the box, I only saw about 3Gbps from iperf (testing
against a linux 
> system...and maybe there are other issues with our
environment/that 
> system to tune up yet...i wouldn't take that number too
seriously).
If I remember correctly there is a problem with iperf,
because it 
utilize too much CPU. I saw patches flying around, and part
of them are 
in FreeBSD ports collection I think.
Personally I prefer netperf for doing network tests 
>
> I think some tuning could get it up to the maximum,
anyways.  It 
> certainly took a little work to get our Linux systems
up to the max, 
> so I would say that in general the defaults with 10G
drivers on any 
> system need some work to hit the maximum bandwidth.
>
> -Ben
>
> Aristedes Maniatis wrote:
>> I have a project with 3 workstations all needing
high speed access to 
>> about 6Tb of storage. In the past I've installed
technology such as 
>> fibre channel connected SAN storage (using Apple's
xsan) for up to a 
>> dozen workstations, but with only three
workstations for this project 
>> I'm thinking about 10GbE. The workstations will be
OSX and the server 
>> FreeBSD 7 with a bunch of disks in a RAID 5 or RAID
10 configuration.
>>
>> Are 10GbE NICs and drivers significantly mature
enough under FreeBSD 
>> to accomplish this? I'd need to achieve about
60MB/s transfer rate 
>> which is theoretically quite doable, as long as the
drive array can 
>> keep up with three streams of that speed. I'd use
netatalk, samba or 
>> nfs to share files depending on which I can eek the
best speeds out of.
>>
>> Alternatively I could populate the server with 1GbE
NICs, one per 
>> workstation and use cross over cable. That way
there is absolutely no 
>> contention on the network.
>>
>> Any thoughts about the viability of this? Is 10GbE
in production use 
>> with FreeBSD and does it scale well?
>>
>>
>> Cheers
>> Ari Maniatis
>>
>>
>>
>>
>> -------------------------->
>> ish
>> http://www.ish.com.au
>> Level 1, 30 Wilson Street Newtown 2042 Australia
>> phone +61 2 9550 5001   fax +61 2 9550 4001
>> GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A
7D2E 3E49 102A
>>
>>
>> _______________________________________________
>> freebsd-performancefreebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-p
erformance
>> To unsubscribe, send any mail to 
>> "freebsd-performance-unsubscribefreebsd.org"
>>
>>
>

-- 

Best Wishes,
Stefan Lambrev
ICQ# 24134177

_______________________________________________
freebsd-performancefreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-p
erformance
To unsubscribe, send any mail to
"freebsd-performance-unsubscribefreebsd.org"

On Wed, Apr 16, 2008 at 10:02:33AM +1000, Aristedes Maniatis
wrote:
> 
> On 15/04/2008, at 10:54 PM, gnnfreebsd.org wrote:
> >I am working with the Chelsio hardware and it seems
to be well
> >supported on FreeBSD.  All the machines in that
system are FreeBSD
> >though.  How do you intend to get the OSX systems
to be 10GE?
> 
> 
> What sort of throughput are you getting with that
setup? Are you using  
> NFS between the systems?

I get about 150 MB/sec NFS random write throughput between
chelsio
NICs.  We are still in the process of optimizing NFS at the
high end.
For bulk packet throughput it is not difficult to saturate
it.

Kris

--
In God we Trust -- all others must submit an X.509
certificate.
    -- Charles Forsythe <forsythealum.mit.edu>
_______________________________________________
freebsd-performancefreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-p
erformance
To unsubscribe, send any mail to
"freebsd-performance-unsubscribefreebsd.org"

Neterion supports OS X on 10GbE NICs, I think Intel does as
well (via
third party).
Leonid

> -----Original Message-----
> From: owner-freebsd-performancefreebsd.org
[mailto:owner-freebsd-
> performancefreebsd.org] On Behalf Of Kip Macy
> Sent: Tuesday, April 15, 2008 5:09 PM
> To: Aristedes Maniatis; gnnfreebsd.org;
freebsd-performancefreebsd.org
> Subject: Re: 10GbE speeds
> 
> Myricom has OS X support, Chelsio has support in the
works. I don't
> know about Neterion or Intel.
> 
>     -Kip
> 
> 
> 
> On 4/15/08, Aristedes Maniatis <ariish.com.au> wrote:
> >
> > On 15/04/2008, at 10:54 PM, gnnfreebsd.org wrote:
> > > I am working with the Chelsio hardware and it
seems to be well
> > > supported on FreeBSD.  All the machines in
that system are FreeBSD
> > > though.  How do you intend to get the OSX
systems to be 10GE?
> >
> >
> > What sort of throughput are you getting with that
setup? Are you
using
> > NFS between the systems?
> >
> > Good question about OSX, and I hadn't got to that
part yet   But
I
> > was hoping that some OSX drivers existed. My fail
back plan is to
put
> > 3 x 1GbE NICs into the server and just use
crossover cable between
the
> > 3 workstations and the server to avoid any
contention within the
> > ethernet network.
> >
> > Ari Maniatis
> >
> >
> > -------------------------->
> > ish
> > http://www.ish.com.au
> > Level 1, 30 Wilson Street Newtown 2042 Australia
> > phone +61 2 9550 5001   fax +61 2 9550 4001
> > GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA
EF6A 7D2E 3E49 102A
> >
> >
> > _______________________________________________
> > freebsd-performancefreebsd.org mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-p
erformance
> > To unsubscribe, send any mail to
> > "freebsd-performance-unsubscribefreebsd.org"
> >
> _______________________________________________
> freebsd-performancefreebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-p
erformance
> To unsubscribe, send any mail to
"freebsd-performance-
> unsubscribefreebsd.org"
_______________________________________________
freebsd-performancefreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-p
erformance
To unsubscribe, send any mail to
"freebsd-performance-unsubscribefreebsd.org"

On Thu, Apr 17, 2008 at 10:47:15AM +0200, Markus Klaschka
wrote:
> That's interesting cause heavy reading from NFS brought
me a loadavg of 
> 70 and more if there were a lot of small files to
read.
> I thought this is a normal issue about NFS...
> By the way, are all Realtek Cards for the bin or only
the 8139...the 
> server has a 'RTL8169 Gigabit Ethernet Adapter'
> I mean, you all know this, if not read the comments in
that file ;)
> 
>    rootkalium:~ > grep worst /usr/src/sys/pci/if_rl.c
>    * probably the worst PCI ethernet controller ever
made, with the
>    possible
> 
> What could be configured wrong?
> What's the best way to test bandwidth if I only got one
well connected 
> server? pathchar?

NFS performance is limited by several things:

* server disk I/O.  With low end disk hardware you are not
going to
get good performance at high load.

* network bandwidth.  Ditto.

* NFS client and server implementation

There have been important relevant improvements in 8.0 that
improve
the client performance with many concurrent processes doing
NFS I/O.
Also 7.0 has much better performance than 6.x.

Kris

P.S. Load average doesn't tell you if your system is
performing badly,
it tells you that the system is running many processes.

--
In God we Trust -- all others must submit an X.509
certificate.
    -- Charles Forsythe <forsythealum.mit.edu>
_______________________________________________
freebsd-performancefreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-p
erformance
To unsubscribe, send any mail to
"freebsd-performance-unsubscribefreebsd.org"