Strange behavior of SCSI RAID 10 (FreeBSD, LSILogic MegaRAID)

Hello world, I have faced with following issue on my
dedicated server:

8x Opteron 885, 32gm RAM, 8x36 GM 15k rpm SCSI with RAID 10

FreeBSD 7.0-generic, 64 bit version



IO performance and behavior is very strange:

1)       No other process are running:

Memory stat:

Mem: 8796K Active, 9372K Inact, 80M Wired, 36K Cache, 12M
Buf, 31G Free

Copy 3GB file first time

dd if=/home/3gb_file of=/home/3gb_file2
6291456+0 records in
6291456+0 records out
3221225472 bytes transferred in 138.842926 secs (23200501
bytes/sec)

20MBS is very poor…

Memory stat now:

Mem: 8940K Active, 5951M Inact, 287M Wired, 36K Cache, 214M
Buf, 25G Free

2)       Copy the same file again:

 dd if=/home/3gb_file of=/home/3gb_file2
6291456+0 records in
6291456+0 records out
3221225472 bytes transferred in 30.433515 secs (105844674
bytes/sec)

100MBs – much better

Mem: 9048K Active, 5951M Inact, 287M Wired, 32K Cache, 214M
Buf, 25G Free

Next attempts with this file show 100MBs spped

3)Copy other file first time
dd if=/home/test2 of=/home/test2_2
6144000+0 records in
6144000+0 records out
3145728000 bytes transferred in 141.870921 secs (22173170
bytes/sec)

20MBs again

Copy the same file again:

dd if=/home/test2 of=/home/test2_2
6144000+0 records in
6144000+0 records out
3145728000 bytes transferred in 29.560267 secs (106417441
bytes/sec)

100MBs – much better. Memory stat:

 Mem: 8964K Active, 12G Inact, 287M Wired, 28K Cache, 214M
Buf, 19G Free

Copy first file again:

dd if=/home/3gb_file of=/home/3gb_file2
6291456+0 records in
6291456+0 records out
3221225472 bytes transferred in 34.310753 secs (93883847
bytes/sec)

good speed

....

So, first copying of any file is extremely slow, second and
all next
attempts  are much better. Please help me with this issue.

Dmesg  some info:


ioapic0 <Version 1.1> irqs 0-23 on motherboard
kbd1 at kbdmux0
ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112,
RF2413, RF5413)
hptrr: HPT RocketRAID controller driver v1.1 (May 22 2008
00:49:14)
acpi0: <PTLTD   RSDT> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-safe" frequency 3579545 Hz
quality 850
acpi_timer0: <24-bit timer at 3.579545MHz> port
0x8008-0x800b on acpi0


amr0: <LSILogic MegaRAID 1.53> mem
0xdc400000-0xdc40ffff,0xdc100000-0xdc13ffff irq 16 at device
14.0 on pci140
amr0: Using 64-bit DMA
amr0: [ITHREAD]
amr0: delete logical drives supported by controller
amr0: <LSILogic MegaRAID SCSI 320-2E> Firmware 514L,
BIOS H430, 128MB RAM


hptrr: no controller detected.
acd0: DVDROM <DV-28E-R/1.8A> at ata1-slave UDMA33
amr0: delete logical drives supported by controller
amrd0: <LSILogic MegaRAID logical drive> on amr0
amrd0: 137328MB (281247744 sectors) RAID 1 (optimal)


Best regards,

Alexander.
_______________________________________________
freebsd-scsifreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-scsi

To unsubscribe, send any mail to
"freebsd-scsi-unsubscribefreebsd.org"

On Mon, May 26, 2008 at 11:41:56PM +1100, Alexander
Goncharov wrote:
> Hello world, I have faced with following issue on my
dedicated server:
> 
> 8x Opteron 885, 32gm RAM, 8x36 GM 15k rpm SCSI with
RAID 10
> 
> FreeBSD 7.0-generic, 64 bit version
> 
> 
> 
> IO performance and behavior is very strange:
> 
> 1)       No other process are running:
> 
> Memory stat:
> 
> Mem: 8796K Active, 9372K Inact, 80M Wired, 36K Cache,
12M Buf, 31G Free
> 
> Copy 3GB file first time
> 
> dd if=/home/3gb_file of=/home/3gb_file2
> 6291456+0 records in
> 6291456+0 records out
> 3221225472 bytes transferred in 138.842926 secs
(23200501 bytes/sec)
> 
> 20MBS is very poor?
> 
> Memory stat now:
> 
> Mem: 8940K Active, 5951M Inact, 287M Wired, 36K Cache,
214M Buf, 25G Free
> 
> 2)       Copy the same file again:
> 
>  dd if=/home/3gb_file of=/home/3gb_file2
> 6291456+0 records in
> 6291456+0 records out
> 3221225472 bytes transferred in 30.433515 secs
(105844674 bytes/sec)
> 
> 100MBs ? much better
> 

The "Inact" (Inactive) went up - the 3GiB file is
now cached in 
memory.  So the second (and subsequent) runs are going from
cached
memory so your 100MiB/sec transfer is actually just testing
write
speed, not read/write speed.  This is the same for your
other
tests too

"Inact" is memory that has been used and is being
kept around incase
it is used again, in other words its caching file data in
the
"Inact" region in top.  It'll be reused if
something else needs
the memory, but until then it sticks around.

Remember - you are copying the file from and to the same
filesystem - this is always going to appear slow relative
to
pure read or pure write tests.  A pure write test is
effectively
what you have when you're getting your 100MiB/sec test
result since
its just writing out from cache memory.

Regards,

Gary
_______________________________________________
freebsd-scsifreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-scsi

To unsubscribe, send any mail to
"freebsd-scsi-unsubscribefreebsd.org"

On Tue, May 27, 2008 at 1:10 AM, Gary Palmer <gpalmerfreebsd.org> wrote:

> On Mon, May 26, 2008 at 11:41:56PM +1100, Alexander
Goncharov wrote:
> > Hello world, I have faced with following issue on
my dedicated server:
> >
> > 8x Opteron 885, 32gm RAM, 8x36 GM 15k rpm SCSI
with RAID 10
> >
> > FreeBSD 7.0-generic, 64 bit version
> >
> >
> >
> > IO performance and behavior is very strange:
> >
> > 1)       No other process are running:
> >
> > Memory stat:
> >
> > Mem: 8796K Active, 9372K Inact, 80M Wired, 36K
Cache, 12M Buf, 31G Free
> >
> > Copy 3GB file first time
> >
> > dd if=/home/3gb_file of=/home/3gb_file2
> > 6291456+0 records in
> > 6291456+0 records out
> > 3221225472 bytes transferred in 138.842926 secs
(23200501 bytes/sec)
> >
> > 20MBS is very poor?
> >
> > Memory stat now:
> >
> > Mem: 8940K Active, 5951M Inact, 287M Wired, 36K
Cache, 214M Buf, 25G Free
> >
> > 2)       Copy the same file again:
> >
> >  dd if=/home/3gb_file of=/home/3gb_file2
> > 6291456+0 records in
> > 6291456+0 records out
> > 3221225472 bytes transferred in 30.433515 secs
(105844674 bytes/sec)
> >
> > 100MBs ? much better
> >
>
> The "Inact" (Inactive) went up - the 3GiB
file is now cached in
> memory.  So the second (and subsequent) runs are going
from cached
> memory so your 100MiB/sec transfer is actually just
testing write
> speed, not read/write speed.  This is the same for your
other
> tests too
>
> "Inact" is memory that has been used and is
being kept around incase
> it is used again, in other words its caching file data
in the
> "Inact" region in top.  It'll be reused if
something else needs
> the memory, but until then it sticks around.
>
> Remember - you are copying the file from and to the
same
> filesystem - this is always going to appear slow
relative to
> pure read or pure write tests.  A pure write test is
effectively
> what you have when you're getting your 100MiB/sec test
result since
> its just writing out from cache memory.
>
> Regards,
>
> Gary
>

Hi Gary,

Big thanks for your quick reply. Read only and write only
speed is good
~100MBs. But I am not sure if it really good for hw RAID 10
8x 15k rpm
drives.
I am worried about read/write speed usecase which is most
used at real tasks
(data base). I was surprised 20 MBs speed, this value is
likely to one drive
speed. But I have 8 high speed drives. Something isn't right
here.

I am waiting for freebsd community help.

Thanks in advance.
Alexander
_______________________________________________
freebsd-scsifreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-scsi

To unsubscribe, send any mail to
"freebsd-scsi-unsubscribefreebsd.org"

Alexander Goncharov wrote:
> Hello world, I have faced with following issue on my
dedicated server:
> 
> 8x Opteron 885, 32gm RAM, 8x36 GM 15k rpm SCSI with
RAID 10
> 
> FreeBSD 7.0-generic, 64 bit version
> 
> 
> 
> IO performance and behavior is very strange:
> 
> 1)       No other process are running:
> 
> Memory stat:
> 
> Mem: 8796K Active, 9372K Inact, 80M Wired, 36K Cache,
12M Buf, 31G Free
> 
> Copy 3GB file first time
> 
> dd if=/home/3gb_file of=/home/3gb_file2
> 6291456+0 records in
> 6291456+0 records out
> 3221225472 bytes transferred in 138.842926 secs
(23200501 bytes/sec)
> 
> 20MBS is very poor…
> 
> Memory stat now:
> 
> Mem: 8940K Active, 5951M Inact, 287M Wired, 36K Cache,
214M Buf, 25G Free
> 
> 2)       Copy the same file again:
> 
>  dd if=/home/3gb_file of=/home/3gb_file2
> 6291456+0 records in
> 6291456+0 records out
> 3221225472 bytes transferred in 30.433515 secs
(105844674 bytes/sec)
> 
> 100MBs – much better
> 
> Mem: 9048K Active, 5951M Inact, 287M Wired, 32K Cache,
214M Buf, 25G Free
> 
> Next attempts with this file show 100MBs spped
> 
> 3)Copy other file first time
> dd if=/home/test2 of=/home/test2_2
> 6144000+0 records in
> 6144000+0 records out
> 3145728000 bytes transferred in 141.870921 secs
(22173170 bytes/sec)
> 
> 20MBs again
> 
> Copy the same file again:
> 
> dd if=/home/test2 of=/home/test2_2
> 6144000+0 records in
> 6144000+0 records out
> 3145728000 bytes transferred in 29.560267 secs
(106417441 bytes/sec)
> 
> 100MBs – much better. Memory stat:
> 
>  Mem: 8964K Active, 12G Inact, 287M Wired, 28K Cache,
214M Buf, 19G Free
> 
> Copy first file again:
> 
> dd if=/home/3gb_file of=/home/3gb_file2
> 6291456+0 records in
> 6291456+0 records out
> 3221225472 bytes transferred in 34.310753 secs
(93883847 bytes/sec)
> 
> good speed
> 
> ....
> 
> So, first copying of any file is extremely slow, second
and all next
> attempts  are much better. Please help me with this
issue.
> 
> Dmesg  some info:
> 
> 
> ioapic0 <Version 1.1> irqs 0-23 on motherboard
> kbd1 at kbdmux0
> ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111,
RF5112, RF2413, RF5413)
> hptrr: HPT RocketRAID controller driver v1.1 (May 22
2008 00:49:14)
> acpi0: <PTLTD   RSDT> on motherboard
> acpi0: [ITHREAD]
> acpi0: Power Button (fixed)
> Timecounter "ACPI-safe" frequency 3579545 Hz
quality 850
> acpi_timer0: <24-bit timer at 3.579545MHz> port
0x8008-0x800b on acpi0
> 
> 
> amr0: <LSILogic MegaRAID 1.53> mem
> 0xdc400000-0xdc40ffff,0xdc100000-0xdc13ffff irq 16 at
device 14.0 on pci140
> amr0: Using 64-bit DMA
> amr0: [ITHREAD]
> amr0: delete logical drives supported by controller
> amr0: <LSILogic MegaRAID SCSI 320-2E> Firmware
514L, BIOS H430, 128MB RAM

I'm not sure if it's relevant but the LSILogic MegaRAID
cards i have are 
not able to do true RAID 1+0. Instead it mirrors and then
concatenates 
which will leave you relatively poor read/write performance
(no 
striping). To do RAID 1+0 on my controllers i had to do
mirror in BIOS 
and then stripe the mirror's using gstripe.

Also once you do RAID you have to pay great attention to
block 
sizes/widths/etc and alignment of such. Improper alignment
will ruin any 
performance benefits your RAID levels may provide.

Without Battery backed cache your write performance will be
poor.

Also 'dd' defaults to 512 byte blocks (poor testing).

Just my $0.2

-- 
Sten Daniel Soersdal

_______________________________________________
freebsd-scsifreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-scsi

To unsubscribe, send any mail to
"freebsd-scsi-unsubscribefreebsd.org"