installing into RAID-1: HOWTO and example

I recently set up a machine with 2 x 400G SATA disks in
RAID-1 with
Xen.  Previous postings were helpful, and I tried to
document what I
did carefully.

I learned that when one is going to use Xen, and hence at
least for
now grub, that one should not fight the PC way and use
disklabel with
no fdisk partitions.  While I hear one can make grub put the
stage1.5
elsewhere, the normal place is in the slack sectors
following the MBR.

This example has overlay partitions for root (for booting
with grub)
and swap (for dumps), and /usr and /n0 (to make sure it all
lines up).
swap in RAID is small (0.75+G ) since my dom0 will have
limited
memory.  This machine is dedicated to Xen and will have a
half-dozen
domUs.

I have not tested all the boot options.


$Id: adroit-xen.txt,v 1.6 2006/03/01 21:03:41 gdt Exp $

references:

http://mail-index.netbsd.org/port-xen/2005/11/02/0006.
html
http://mail-index.netbsd.org/port-xen/2005/11/10/0000.
html



----------------------------------------
Intel 915 chipset with 4GB RAM

BIOS set to limit VGA memory reservation to smallest values
need to investigate why 768 is reserved (pci hole?) after
xen

2 400G SATA drives

stock NetBSD-current

RAID-1 on two drives

----------------------------------------
/etc/raid0.conf
--------------------
START array
1 2 0

START disks
/dev/wd0a
/dev/wd1a

START layout
128 1 1 1

START queue
fifo 100
--------------------
PHYSICAL DISK LABELING (wd0, wd1 same)

PLAN:

one DOS primary partition (grub needs dos partitioning)

a: whole disk within partition (63-)
c: NetBSD portion: same as a
d: whole disk (0-)
# boot, dump, sanity check, sanity check
e: raid0a
f: raid0b
g: raid0e
h: raid0f

ACTUAL

#> fdisk wd0
Disk: /dev/rwd0d
NetBSD disklabel disk geometry:
cylinders: 775221, heads: 16, sectors/track: 63 (1008
sectors/cylinder)
total sectors: 781422768

BIOS disk geometry:
cylinders: 1023, heads: 255, sectors/track: 63 (16065
sectors/cylinder)
total sectors: 781422768

Partition table:
0: NetBSD (sysid 169)
    start 63, size 781422705 (381554 MB, Cyls 0-48641/81/1)
1: <UNUSED>
2: <UNUSED>
3: <UNUSED>

#> disklabel wd0
# /dev/rwd0d:
type: RAID
disk: raid
label: fictitious
flags:
bytes/sector: 512
sectors/track: 128
tracks/cylinder: 8
sectors/cylinder: 1024
cylinders: 763108
total sectors: 781422592
rpm: 3600
interleave: 1
trackskew: 0
cylinderskew: 0
headswitch: 0           # microseconds
track-to-track seek: 0  # microseconds
drivedata: 0 

8 partitions:
#        size    offset     fstype [fsize bsize cpg/sgs]
 a: 781422529        63       RAID                     #
(Cyl.      0*- 763107)
 c: 781422705        63     unused      0     0        #
(Cyl.      0*- 763108+)
 d: 781422592         0     unused      0     0        #
(Cyl.      0 - 763107)
 e:    491520       127     4.2BSD   2048 16384 25648  #
(Cyl.      0*-    480*)
 f:   1585152    491647       swap                     #
(Cyl.    480*-   2028*)
 g:   8388608   2076799     4.2BSD      0     0     0  #
(Cyl.   2028*-  10220*)
 h: 770957184  10465407     4.2BSD      0     0     0  #
(Cyl.  10220*- 763107*)
disklabel: partitions a and e overlap
disklabel: partitions a and f overlap
disklabel: partitions a and g overlap
disklabel: partitions a and h overlap
disklabel: partition c: partition extends past end of unit

----------------------------------------
RAID0 DISKLABEL

* PLAN
/    240M # starting at 0
swap 774M
/usr 4G
/n0  rest (kernel build, xen images)

* ACTUAL

#> disklabel rai0
# /dev/rraid0d:
type: RAID
disk: raid
label: fictitious
flags:
bytes/sector: 512
sectors/track: 128
tracks/cylinder: 8
sectors/cylinder: 1024
cylinders: 763107
total sectors: 781422464
rpm: 3600
interleave: 1
trackskew: 0
cylinderskew: 0
headswitch: 0           # microseconds
track-to-track seek: 0  # microseconds
drivedata: 0 

6 partitions:
#        size    offset     fstype [fsize bsize cpg/sgs]
 a:    491520         0     4.2BSD   1024  8192 40960  #
(Cyl.      0 -    479)
 b:   1585152    491520       swap                     #
(Cyl.    480 -   2027)
 d: 781422464         0     unused      0     0        #
(Cyl.      0 - 763107*)
 e:   8388608   2076672     4.2BSD      0     0     0  #
(Cyl.   2028 -  10219)
 f: 770957184  10465280     4.2BSD   2048 16384 28848  #
(Cyl.  10220 - 763107*)


----------------------------------------
/grub/menu.lst:

# Grub config file for NetBSD/xen.
default=0
fallback=4
# boot after 10s the default entry if the user didn't hit
keyboard
timeout=10
# use vga
terminal console
#
# Xen/NetBSD with 256 MB
title Xen 2.0 / NetBSD - WD0
  root(hd0,e)
  kernel (hd0,e)/xen.gz dom0_mem=262144
  module (hd0,e)/netbsd-XEN0.gz root=/dev/hda1 ro console=pc
title Xen 2.0 / NetBSD.ok - WD0
  root(hd0,0)
  kernel (hd0,e)/xen.gz dom0_mem=262144
  module (hd0,e)/netbsd-XEN0.ok.gz root=/dev/hda1 ro
console=pc
title Xen 2.0 / NetBSD - WD1
  root(hd1,e)
  kernel (hd1,e)/xen.gz dom0_mem=262144
  module (hd1,e)/netbsd-XEN0.gz root=/dev/hda1 ro console=pc
title Xen 2.0 / NetBSD.ok - WD1
  root(hd1,0)
  kernel (hd1,e)/xen.gz dom0_mem=262144
  module (hd1,e)/netbsd-XEN0.ok.gz root=/dev/hda1 ro
console=pc
#
# Load the NetBSD bootloader, letting it load the
NetBSD/i386 kernel.
title NetBSD chain - WD0
  root (hd0,0)
  chainloader +1
title NetBSD chain - WD1
  root (hd1,0)
  chainloader +1
#
# Regular NetBSD - does not get boot options
title NetBSD - WD0
  root (hd0,e)
  kernel --type=netbsd /netbsd-GENERIC.gz
title NetBSD - WD1
  root (hd1,e)
  kernel --type=netbsd /netbsd-GENERIC.gz
#
# Regular NetBSD - does not get boot options
title NetBSD.ok - WD0
  root (hd0,e)
  kernel --type=netbsd /netbsd-GENERIC.ok.gz
title NetBSD.ok - WD1
  root (hd1,e)
  kernel --type=netbsd /netbsd-GENERIC.ok.gz
## end of grub config file.
----------------------------------------

GRUB setup

# mkdir /grub
# cp -p /usr/pkg/lib/grub/i386-/* /grub

setup menu.lst as above

# grub --no-floppy

grub> root (hd1,e)
 Filesystem type is ffs, partition type 0xa9

grub> setup (hd1)
 Checking if "/boot/grub/stage1" exists... no
 Checking if "/grub/stage1" exists... yes
 Checking if "/grub/stage2" exists... yes
 Checking if "/grub/ffs_stage1_5" exists... yes
 Running "embed /grub/ffs_stage1_5 (hd1)"...  14
sectors are embedded.
succeeded
 Running "install /grub/stage1 (hd1) (hd1)1+14 p
(hd1,0,e)/grub/stage2 /grub/menu.lst"...
 succeeded
Done.

grub> root (hd0,e)
 Filesystem type is ffs, partition type 0xa9

grub> setup (hd0)
 Checking if "/boot/grub/stage1" exists... no
 Checking if "/grub/stage1" exists... yes
 Checking if "/grub/stage2" exists... yes
 Checking if "/grub/ffs_stage1_5" exists... yes
 Running "embed /grub/ffs_stage1_5 (hd0)"...  14
sectors are embedded.
succeeded
 Running "install /grub/stage1 (hd0) (hd0)1+14 p
(hd0,0,e)/grub/stage2 /grub/menu.lst"...
 succeeded
Done.

----------------------------------------
In domU: fstab

# $Id: adroit-xen.txt,v 1.6 2006/03/01 21:03:41 gdt Exp $
/dev/raid0a     /       ffs     rw              1 1
/dev/raid0b     none    swap    sw              0 0
/dev/raid0b     /tmp    mfs     rw,-s=1049328
/dev/raid0e     /usr    ffs     rw              1 2
/dev/raid0f     /n0     ffs     rw              1 3
/dev/wd0f       none    none    dp      
kernfs          /kern   kernfs  rw
procfs          /proc   procfs  rw,noauto
/dev/cd0a       /cdrom  cd9660  ro,noauto

----------------------------------------
In all domU, /etc/rc.conf has powerd=YES to invoke shutdown
from
shutdown of dom0.

manuel, or someone...

could You link to this on the port-xen web page at
netbsd.org?
I think this will save many people from headaches 

florian

-- 

florian heigl			 http://deranfangvomen.de/

Grid Computing erfreut sich gerade bei Windows-Nutzern sehr
regem Zuspruch, auch
wenn die Rechnerbesitzer meist nichts von ihrem Glück
wissen. --f. weimer

On Wed, Mar 01, 2006 at 08:45:58PM -0500, Greg Troxel wrote:
> [...]
> ----------------------------------------
> Intel 915 chipset with 4GB RAM
> 
> BIOS set to limit VGA memory reservation to smallest
values
> need to investigate why 768 is reserved (pci hole?)
after xen

Yes, this is the PCI hole. I've got the same issue with a
i945.
Despite the claim that the motherboard
"supports" 4Gb and using a EM64T CPU,
768M were stolen for PCI. By playing with BIOS options,
disabling all unused
devices I trimmed this down to ~600M.

In order to really use 4GB you need a i955X chipset.

-- 
Manuel Bouyer <bouyerantioche.eu.org>
     NetBSD: 26 ans d'experience feront toujours la
difference
--

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Manuel,

On 2 Mar 2006, at 23:13, Manuel Bouyer wrote:

> On Wed, Mar 01, 2006 at 08:45:58PM -0500, Greg Troxel
wrote:
>> [...]
>> ----------------------------------------
>> Intel 915 chipset with 4GB RAM
>>
>> BIOS set to limit VGA memory reservation to
smallest values
>> need to investigate why 768 is reserved (pci hole?)
after xen
>
> Yes, this is the PCI hole. I've got the same issue with
a i945.
> Despite the claim that the motherboard
"supports" 4Gb and using a  
> EM64T CPU,
> 768M were stolen for PCI. By playing with BIOS options,
disabling  
> all unused
> devices I trimmed this down to ~600M.
>
> In order to really use 4GB you need a i955X chipset.

I have an i965 chipset and I also lose 768MB of 4GB to the
PCI hole  
(it's sort of ok, at least I was aware of it when deploying
this  
machine).

However, my question is what would happen if I add more
memory to the  
machine. I.e. if I boost it to 8GB would I still only get
3.25GB for  
Xen or would I get 7.25GB not having to bother about the
hole in the  
middle?

Johan

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iQCVAwUBRidvh/otlDfa2H4ZAQKZfgQAk2+aS8efHSH1Le8DE5nLMovjLUt2
Jrk4
YB6ydwA618JobNQu8X1pEUdFxURmNUTva4Fs5UR2/WcO4H/MzbjYGvlK/hvr
2tai
wDJFI9Ykcw25XM4Z7HL52T24u+2EaFRgHPeX+uzUA/BgIJmWaCcYL7NZvab5
6V9f
7WzLnVMaGcE=
=4Zn1
-----END PGP SIGNATURE-----

On Thu, Apr 19, 2007 at 03:32:55PM +0200, Johan Ihren
wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Hi Manuel,
> 
> On 2 Mar 2006, at 23:13, Manuel Bouyer wrote:
> 
> >On Wed, Mar 01, 2006 at 08:45:58PM -0500, Greg
Troxel wrote:
> >>[...]
> >>----------------------------------------
> >>Intel 915 chipset with 4GB RAM
> >>
> >>BIOS set to limit VGA memory reservation to
smallest values
> >>need to investigate why 768 is reserved (pci
hole?) after xen
> >
> >Yes, this is the PCI hole. I've got the same issue
with a i945.
> >Despite the claim that the motherboard
"supports" 4Gb and using a  
> >EM64T CPU,
> >768M were stolen for PCI. By playing with BIOS
options, disabling  
> >all unused
> >devices I trimmed this down to ~600M.
> >
> >In order to really use 4GB you need a i955X
chipset.
> 
> I have an i965 chipset and I also lose 768MB of 4GB to
the PCI hole  
> (it's sort of ok, at least I was aware of it when
deploying this  
> machine).

With a 32bits or 64bits OS ? with 32bits, you won't see more
than 4GB
including the PCI hole anyway, even if the chipset supports
remaming the
remaining RAM above 4GB.

> 
> However, my question is what would happen if I add more
memory to the  
> machine. I.e. if I boost it to 8GB would I still only
get 3.25GB for  
> Xen or would I get 7.25GB not having to bother about
the hole in the  
> middle?

NetBSD/Xen is still 32bits, so it's limited to 4GB.

-- 
Manuel Bouyer, LIP6, Universite Paris VI.          
Manuel.Bouyerlip6.fr
     NetBSD: 26 ans d'experience feront toujours la
difference
--

On Thu, 19 Apr 2007, Manuel Bouyer wrote:
>> However, my question is what would happen if I add
more memory to the
>> machine. I.e. if I boost it to 8GB would I still
only get 3.25GB for
>> Xen or would I get 7.25GB not having to bother
about the hole in the
>> middle?
>
> NetBSD/Xen is still 32bits, so it's limited to 4GB.

Hosting providers want to run NetBSD XenUs on either 32-bit
PAE or 64-bit 
Xen Dom0s;  quite often, it doesn't make much sense to split
up a box with 
less than 4GB ram.

Are there plans to make NetBSD-xen work on 64-bit
platforms?

alternately, I have read  the 32-PAE on 64 support for Xen
3.0.5 is 
supposed to be nice;  any word if that will support 32
non-PAE guests? 
that would be good enough for me.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

>>> Yes, this is the PCI hole. I've got the same
issue with a i945.
>>> Despite the claim that the motherboard
"supports" 4Gb and using a
>>> EM64T CPU,
>>> 768M were stolen for PCI. By playing with BIOS
options, disabling
>>> all unused
>>> devices I trimmed this down to ~600M.
>>>
>>> In order to really use 4GB you need a i955X
chipset.
>>
>> I have an i965 chipset and I also lose 768MB of 4GB
to the PCI hole
>> (it's sort of ok, at least I was aware of it when
deploying this
>> machine).
>
> With a 32bits or 64bits OS ? with 32bits, you won't see
more than 4GB
> including the PCI hole anyway, even if the chipset
supports  
> remaming the
> remaining RAM above 4GB.
>
>> However, my question is what would happen if I add
more memory to the
>> machine. I.e. if I boost it to 8GB would I still
only get 3.25GB for
>> Xen or would I get 7.25GB not having to bother
about the hole in the
>> middle?
>
> NetBSD/Xen is still 32bits, so it's limited to 4GB.

That's what I feared. Yes, I understand that a 32bit OS is
limited to  
4GB, but I thought that perhaps the Xen hypervisor had no
such  
restrictions and as long as dom0 and all domU's where small
enough it  
might work. But then I've also seen the discussions about
PAE and  
page table structures having to be the same between
hypervisor, dom0  
and domU, so I'm not surprised, really.

Ho hum. I suspect this will bite me hard around the end of
the year.  
Oh, well.

Johan

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iQCVAwUBRifOGvotlDfa2H4ZAQI7oQQAp2AQOck6g3QW7Jjj73ADqjIA8HMJ
0ss0
MzY0M9e0KrsE7pwNmW/dXK3A9+nKS/uZV+1PrFjYa0j3VUfsgVDqeMI8nfQe
ESOS
FRBlngMyk+0Q1atty+wNYM8es090rIa29SfA6FJiz0bJchJMItE5zKin21F2
QhCh
GJa/K36MleQ=
=0l90
-----END PGP SIGNATURE-----

On Thu, Apr 19, 2007 at 02:34:39PM -0700, Luke S. Crawford
wrote:
> On Thu, 19 Apr 2007, Manuel Bouyer wrote:
> >>However, my question is what would happen if I
add more memory to the
> >>machine. I.e. if I boost it to 8GB would I
still only get 3.25GB for
> >>Xen or would I get 7.25GB not having to bother
about the hole in the
> >>middle?
> >
> >NetBSD/Xen is still 32bits, so it's limited to
4GB.
> 
> Hosting providers want to run NetBSD XenUs on either
32-bit PAE or 64-bit 
> Xen Dom0s;  quite often, it doesn't make much sense to
split up a box with 
> less than 4GB ram.
> 
> Are there plans to make NetBSD-xen work on 64-bit
platforms?

This was proposed as a GSOC project, but unfortunably wasn't
accepted :(
I'll probably work on this myself this summer.

> 
> alternately, I have read  the 32-PAE on 64 support for
Xen 3.0.5 is 
> supposed to be nice;  any word if that will support 32
non-PAE guests? 

I don't think it will. But you can always run NetBSD/i386 in
an HVM guest 

-- 
Manuel Bouyer, LIP6, Universite Paris VI.          
Manuel.Bouyerlip6.fr
     NetBSD: 26 ans d'experience feront toujours la
difference
--

On Thu, Apr 19, 2007 at 04:10:31PM +0200, Manuel Bouyer
wrote:
> > However, my question is what would happen if I add
more memory to the  
> > machine. I.e. if I boost it to 8GB would I still
only get 3.25GB for  
> > Xen or would I get 7.25GB not having to bother
about the hole in the  
> > middle?
> NetBSD/Xen is still 32bits, so it's limited to 4GB.

Out of curiosity:
How much memory can you actually use in a Xen 32 bit domU
with PAE support?
3.5GB or 4GB?

	Kind regards

-- 
Matthias Scheler                                  http://zhadum.org.uk/

On Fri, Apr 20, 2007 at 09:55:38PM +0100, Matthias Scheler
wrote:
> On Thu, Apr 19, 2007 at 04:10:31PM +0200, Manuel Bouyer
wrote:
> > > However, my question is what would happen if
I add more memory to the  
> > > machine. I.e. if I boost it to 8GB would I
still only get 3.25GB for  
> > > Xen or would I get 7.25GB not having to
bother about the hole in the  
> > > middle?
> > NetBSD/Xen is still 32bits, so it's limited to
4GB.
> 
> Out of curiosity:
> How much memory can you actually use in a Xen 32 bit
domU with PAE support?
> 3.5GB or 4GB?

I think you should be able to have more than 4GB, but a
single process
can't use more than 2 or 3 GB (depending on how much VM the
kernel
keeps for itself).

-- 
Manuel Bouyer <bouyerantioche.eu.org>
     NetBSD: 26 ans d'experience feront toujours la
difference
--