Sun Fire X4500, FreeBSD and ZFS

Mjello.

Just thought I'd say that we've got a Sun Fire X4500 running
with
-CURRENT as of yesterday and ZFS. Works beautifully, after
we disabled
MSI and increased VM_KMEM_SIZE_MAX.

Without the increased VM_KMEM_SIZE_MAX, we got the usual
panic (kmem_map
too small). I haven't tried adjusting maxvnodes - that might
also have
helped. However, the machine has 16 GB RAM, so it might as
well be used
for something. I'm not quite sure how to tweak the box
efficiently, but
for now the bottleneck is our network, so we're going to
upgrade some
pieces and try again.

We configured the 48 drives as follows:

- ad52 and ad60 are magic - the BIOS is hardcoded to boot
from them, so
  we put them in a gmirror
- 5 RAIDZ2's, each with 9 disks, for a usable total of 7 per
array
- one global hotspare

# zpool list
NAME                    SIZE    USED   AVAIL    CAP  HEALTH 
   ALTROOT
void                   20.3T   62.1G   20.3T     0%  ONLINE 
   -

# zfs list
NAME          USED  AVAIL  REFER  MOUNTPOINT
void         48.2G  15.5T  41.9K  /void

All in all, a fun little toy 

-- 
Best Regards
Kenneth Schmidt
_______________________________________________
freebsd-fsfreebsd.org mailing list

http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to
"freebsd-fs-unsubscribefreebsd.org"

On Tue, May 01, 2007 at 10:33:10AM +0200, Kenneth
Vestergaard Schmidt wrote:
> Mjello.
> 
> Just thought I'd say that we've got a Sun Fire X4500
running with
> -CURRENT as of yesterday and ZFS. Works beautifully,
after we disabled
> MSI and increased VM_KMEM_SIZE_MAX.
> 
> Without the increased VM_KMEM_SIZE_MAX, we got the
usual panic (kmem_map
> too small). I haven't tried adjusting maxvnodes - that
might also have
> helped. However, the machine has 16 GB RAM, so it might
as well be used
> for something. I'm not quite sure how to tweak the box
efficiently, but
> for now the bottleneck is our network, so we're going
to upgrade some
> pieces and try again.
> 
> We configured the 48 drives as follows:
> 
> - ad52 and ad60 are magic - the BIOS is hardcoded to
boot from them, so
>   we put them in a gmirror
> - 5 RAIDZ2's, each with 9 disks, for a usable total of
7 per array
> - one global hotspare
> 
> # zpool list
> NAME                    SIZE    USED   AVAIL    CAP 
HEALTH     ALTROOT
> void                   20.3T   62.1G   20.3T     0% 
ONLINE     -
> 
> # zfs list
> NAME          USED  AVAIL  REFER  MOUNTPOINT
> void         48.2G  15.5T  41.9K  /void
> 
> All in all, a fun little toy 

If it's just a little toy for your, maybe you want to
replace it with my
teddy bear?

Great to hear that this beast works with FreeBSD!! Any
chance we can
trick you into performance comparsion between Solaris/ZFS
and
FreeBSD/ZFS?

-- 
Pawel Jakub Dawidek                       http://www.wheel.pl
pjdFreeBSD.org                           http://www.FreeBSD.org
FreeBSD committer                         Am I Evil? Yes, I
Am!

Pawel Jakub Dawidek <pjdFreeBSD.org> writes:
>> All in all, a fun little toy 
>
> If it's just a little toy for your, maybe you want to
replace it with my
> teddy bear?

Is your teddy bear rack-mountable, and does it have a
dedicated
management processor? If so, we'll talk 

> Great to hear that this beast works with FreeBSD!! Any
chance we can
> trick you into performance comparsion between
Solaris/ZFS and
> FreeBSD/ZFS?

At the very least, we want to do some heavy testing with
FreeBSD. If we
get the time, it would be fun to contrast them to Solaris.
It does
require that I reinstall Solaris, though 

How would you go about tweaking the machine for maximum
efficiency? We
have around 13 GB RAM free at the moment, just sitting doing
nothing,
and that might just as well be used for caching.

-- 
Kenneth Schmidt
_______________________________________________
freebsd-fsfreebsd.org mailing list

http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to
"freebsd-fs-unsubscribefreebsd.org"

On Fri, May 04, 2007 at 06:26:14PM +0200, Kenneth
Vestergaard Schmidt wrote:
> At the very least, we want to do some heavy testing
with FreeBSD. If we
> get the time, it would be fun to contrast them to
Solaris. It does
> require that I reinstall Solaris, though 
> 
> How would you go about tweaking the machine for maximum
efficiency? We
> have around 13 GB RAM free at the moment, just sitting
doing nothing,
> and that might just as well be used for caching.

There is a lot memory and quite a few silly limits in the
kernel.
Currently we allocate most memory for ZFS via malloc(9), so
the memory
is allocate from kmem_map. This should change in the future,
but until
then, you need to adjust vm_kmem_size, etc. to make memory
usable with
ZFS. You'd also want to tune vfs.zfs.arc_max and
vfs.zfs.arc_min,
because current auto-tuning won't be of any good from that
much RAM. I'd
start from setting vfs.zfs.arc_max to kmem_map size minus
3GB. You can
tune all of them (vfs.zfs.arc_max, vfs.zfs.arc_min,
vm.kmem_size and
vm.kmem_size_max from /boot/loader.conf).
Another old limit is for maximum number of vnodes in the
system. It is
only auto-tuned up to 100000, which you may want to change.
I'm not sure
if you can do it from kernel config, so just change
MAXVNODES_MAX define
in sys/kern/vfs_subr.c.
Be aware that most debugging options like INVARIANTS,
WITNESS, etc. have
very negative impact on ZFS performance in FreeBSD.

PS. Even if you don't want to install Solaris in there,
there are plenty
benchmarks on the net for Solaris/ZFS on this very machine.
Would be
good to know what results do we have.

Good luck!

-- 
Pawel Jakub Dawidek                       http://www.wheel.pl
pjdFreeBSD.org                           http://www.FreeBSD.org
FreeBSD committer                         Am I Evil? Yes, I
Am!