Four Drive RAID-5 on RAIDFrame Considered Harmful...

I'm building up a new server to replace my Cobalt RaQ2+
running NetBSD. 
  I got a Mini-ITX board a 1U case and hung 4 drives off it
(actually 5 
while I bring it up, but the 5th will go away once the
raidset is 
properly setup, and there's not much IO to it)

The drives are all 400GB Seagate drives, and because of the
I/O on the 
Via EN-15000G, 2 are SATA and 2 are IDE (ide drives on their
own channel).

Setting it up initially, I raided the 4 drives together with
two 
partitions on each of the components: a small one for RAID-1
to load the 
kernel, and a large one for RAID-5.  Unfortunately, the
RAID-5 had 
horrible performance: 2-3MB/sec sometimes and never higher
than about 
12MB/sec.

I tracked it down to the problem that 2 and 3 are relatively
prime 
Given 4 drives in a RAID-5 setup, you get 3 data blocks per
raid stripe, 
but the filesystem block size Must be a power of 2, so you
always have 
to write at least one partial stripe.

I'm still doing testing, but at one point I saw differences
in 
performance between 3 (+1 spare) and 4 drive RAID-5 sets of
20:1.  That 
is, adding the 4th drive into the RAID set caused
performance to drop by 
a factor of 20.

So far, it looks like the best overall performance I'm
getting is with 3 
drives, 32 SectsPerSU (for 32K Stripes) with a filesystem
block size 
also of 32K.

I'm sort of disappointed at losing 25% of my storage, but
the 
performance loss just isn't worth it.  Would a hardware RAID
card have 
these issues, or do they do tricks with buffering or
something to get 
around it?

Once I finish testing, I'll post my results.

Thanks,

Robert
-- 
Robert Thille                7575 Meadowlark Dr.;
Sebastopol, CA 95472
Home: 707.824.9753    Office/VOIP: 707.780.1560     Cell:
707.217.7544
rthillemirapoint.com    YIM:rthille     http://www.rangat.org/r
thille
Cyclist, Mountain Biker, Freediver, Kayaker, Rock Climber,
Hiker, Geek
May your spirit dive deep the blue, where the fish are many
and large!

On Oct 9, 2007, at 6:53 PM, Thor Lancelot Simon wrote:

> On Tue, Oct 09, 2007 at 04:59:58PM -0700, Robert P.
Thille wrote:
>>
>> I tracked it down to the problem that 2 and 3 are
relatively  
>> prime 
>> Given 4 drives in a RAID-5 setup, you get 3 data
blocks per raid  
>> stripe,
>> but the filesystem block size Must be a power of 2,
so you always  
>> have
>> to write at least one partial stripe.
>
> No.  Examine the "maxcontig" FFS parameter.
>

maxcontig appears to be obsolete.  Looking thru the sources
(current  
from a month or so ago), I don't think it's used by the
kernel/ 
filesystem, and the options mentioned in postings I found in
the  
mailing lists don't seem to exist anymore.

Am I missing something?

Thanks,

Robert


--
Robert Thille                7575 Meadowlark Dr.;
Sebastopol, CA 95472
Home: 707.824.9753    Office/VOIP: 707.780.1560     Cell:
707.217.7544
Robert.Thillerangat.org   YIM:rthille   http://www.rangat.org/r
thille
Cyclist, Mountain Biker, Freediver, Kayaker, Rock Climber,
Hiker, Geek
May your spirit dive deep the blue, where the fish are many
and large!

2007/10/10, Robert P. Thille
<list-netbsd-tech-performrangat.org>:
>
> On Oct 9, 2007, at 6:53 PM, Thor Lancelot Simon wrote:
>
> > On Tue, Oct 09, 2007 at 04:59:58PM -0700, Robert
P. Thille wrote:
> >>
> >> I tracked it down to the problem that 2 and 3
are relatively
> >> prime 
> >> Given 4 drives in a RAID-5 setup, you get 3
data blocks per raid
> >> stripe,
> >> but the filesystem block size Must be a power
of 2, so you always
> >> have
> >> to write at least one partial stripe.
> >
> > No.  Examine the "maxcontig" FFS
parameter.
> >
>
> maxcontig appears to be obsolete.  Looking thru the
sources (current
> from a month or so ago), I don't think it's used by the
kernel/
> filesystem, and the options mentioned in postings I
found in the
> mailing lists don't seem to exist anymore.
>
> Am I missing something?
>
> Thanks,
>
> Robert
>
>
> --
> Robert Thille                7575 Meadowlark Dr.;
Sebastopol, CA 95472
> Home: 707.824.9753    Office/VOIP: 707.780.1560    
Cell: 707.217.7544
> Robert.Thillerangat.org   YIM:rthille   http://www.rangat.org/r
thille
> Cyclist, Mountain Biker, Freediver, Kayaker, Rock
Climber, Hiker, Geek
> May your spirit dive deep the blue, where the fish are
many and large!
>

I wouldn't recommend RAID 5 at all. If one of your drives
die or your
controller dies (in the case you would have hardware raid),
you can
have a lot of trouble recovering your data. But if you have
1x RAID 0
plus RAID 1 (RAID 10 ?)  you can always replace the drives
and copy
the data since the data is always available as a whole
unit.

Zafer.

  Setting it up initially, I raided the 4 drives together
with two
  partitions on each of the components: a small one for
RAID-1 to load
  the kernel, and a large one for RAID-5.  Unfortunately,
the RAID-5 had
  horrible performance: 2-3MB/sec sometimes and never higher
than about
  12MB/sec.

I am not 100% clear on this, but I have the impression that
RAID-5
requires read-modify-write and that in the event of system
crash or
power loss you can get corruption, and thus the good
hardware
controllers have a) battery backed RAM and b) code that
won't crash.  (I
don't mean to malign the raidframe code - but because it's
in-kernel if
the kernel crashes for any reason - not unheard of - then
pending raid
writes may not happen.)

Because of this I've always just bought two big disks and
done RAID-1.

Perhaps Greg Oster will chime in, and it would be a good
addition to the
Guide to discuss the wisdom of using RAID-5.
http
://www.netbsd.org/docs/guide/en/chap-rf.html

On Oct 10, 2007, at 4:48 AM, Greg Troxel wrote:

>   Setting it up initially, I raided the 4 drives
together with two
>   partitions on each of the components: a small one for
RAID-1 to load
>   the kernel, and a large one for RAID-5. 
Unfortunately, the  
> RAID-5 had
>   horrible performance: 2-3MB/sec sometimes and never
higher than  
> about
>   12MB/sec.
>
> I am not 100% clear on this, but I have the impression
that RAID-5
> requires read-modify-write

Yep, unless you can do "full-stripe" writes, which
is why you want  
the filesystem block size to match the stripe size...

> and that in the event of system crash or
> power loss you can get corruption, and thus the good
hardware
> controllers have a) battery backed RAM and b) code that
won't  
> crash.  (I
> don't mean to malign the raidframe code - but because
it's in- 
> kernel if
> the kernel crashes for any reason - not unheard of -
then pending raid
> writes may not happen.)

Well, I'm not that familiar with the 'Raid-5 hole', but
Wikipedia  
seems to indicate that the parity can get wrong on a crash,
and  
unless you detect that before a failure, you'll lose data. 
But since  
RAIDFrame always regenerates the parity when it's not
shutdown  
cleanly, that shouldn't (?) be a problem...I think 

> Because of this I've always just bought two big disks
and done RAID-1.
>
> Perhaps Greg Oster will chime in, and it would be a
good addition  
> to the
> Guide to discuss the wisdom of using RAID-5.
> http
://www.netbsd.org/docs/guide/en/chap-rf.html

I thought about cross-posting to tech-kern, since it seems
like that  
gets a fair amount of RAIDFrame traffic, but my initial
concern was  
primarily with the performance I was seeing, and I know
cross-posting  
is frowned on.

Thanks,

Robert

--
Robert Thille                7575 Meadowlark Dr.;
Sebastopol, CA 95472
Home: 707.824.9753    Office/VOIP: 707.780.1560     Cell:
707.217.7544
Robert.Thillerangat.org   YIM:rthille   http://www.rangat.org/r
thille
Cyclist, Mountain Biker, Freediver, Kayaker, Rock Climber,
Hiker, Geek
May your spirit dive deep the blue, where the fish are many
and large!

On Tue, Oct 09, 2007 at 04:59:58PM -0700, Robert P. Thille
wrote:
> [...]
> 
> I'm sort of disappointed at losing 25% of my storage,
but the 
> performance loss just isn't worth it.  Would a hardware
RAID card have 
> these issues, or do they do tricks with buffering or
something to get 
> around it?

Hardware RAID are much faster, if they have a battery-backed
cache and
can use write-back caching policy. With the write-back cache
they can defer
the partial writes until they have enough data to do a
full-stripe write.

-- 
Manuel Bouyer <bouyerantioche.eu.org>
     NetBSD: 26 ans d'experience feront toujours la
difference
--