NetBSD crashes on 4 GB Supermicro

I'm having a problem running a NetBSD domU on one particular
type of
system.

	SuperMicro PDSMI+ motherboard
	4 GB RAM (about 3.5 MB usable)
	Xen 3.0.3-0 (non-PAE, of course)
	dom0: Linux 2.6.16.29

Normally NetBSD will crash immediately upon startup on these
systems.
Linux domUs work just fine. NetBSD also works fine on other
4GB systems.

The only way I can get NetBSD to work on these is if I tell
Xen to
limit the RAM usage to 3G or less on the command line
(mem=3072m).

That last 512MB or so may not seem like much, but without
PAE support,
I'm trying to make use of every last available MB.

I have no idea how to debug a domU. Any help would be
appreciated.

Thanks.

----------

This is what Xen has to say about it:

(XEN) domain_crash_sync called from entry.S (ff150e49)
(XEN) Domain 1 (vcpu#0) crashed on cpu#1:
(XEN) ----[ Xen-3.0.3-0  x86_32  debug=n  Not tainted ]----
(XEN) CPU:    1
(XEN) EIP:    e019:[<c02b6ea9>]
(XEN) EFLAGS: 00000246   CONTEXT: guest
(XEN) eax: ffffffff   ebx: d097d003   ecx: d097d000   edx:
00000000
(XEN) esi: 0000005b   edi: c0455000   ebp: c0317bb4   esp:
c0317b78
(XEN) cr0: 8005003b   cr4: 000026d0   cr3: d097c000   cr2:
00000000
(XEN) ds: e021   es: e021   fs: 0000   gs: 0000   ss: e021  
cs: e019
(XEN) Guest stack trace from esp=c0317b78:
(XEN)    00000000 c02b6ea9 0001e019 00010046 c02b798c
d097a154 d0973061 00000000
(XEN)    00000000 00000000 c0454000 00000005 c0450c10
c0450000 c0459000 00000000
(XEN)    c0100179 c0317bbc 0000e021 c01044bc c0104520
00000000 00000000 00000000
(XEN)    00000000 00000000 00000000 c02ec8a0 00000000
00000000 00000000 00000000
(XEN)    00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000
[many more lines of zeros]

On Thu, May 03, 2007 at 12:18:02PM -0400, marcottepanix.com
wrote:
> I'm having a problem running a NetBSD domU on one
particular type of
> system.
> 
> 	SuperMicro PDSMI+ motherboard
> 	4 GB RAM (about 3.5 MB usable)
> 	Xen 3.0.3-0 (non-PAE, of course)
> 	dom0: Linux 2.6.16.29
> 
> Normally NetBSD will crash immediately upon startup on
these systems.
> Linux domUs work just fine. NetBSD also works fine on
other 4GB systems.
> 
> The only way I can get NetBSD to work on these is if I
tell Xen to
> limit the RAM usage to 3G or less on the command line
(mem=3072m).
> 
> That last 512MB or so may not seem like much, but
without PAE support,
> I'm trying to make use of every last available MB.

I've seen the same issue on a Dell 2950 system. From memory,
linux
domU couldn't start either, but this system isn't running
Xen now so
I can't check.

> 
> I have no idea how to debug a domU. Any help would be
appreciated.
> 
> Thanks.
> 
> ----------
> 
> This is what Xen has to say about it:
> 
> (XEN) domain_crash_sync called from entry.S (ff150e49)
> (XEN) Domain 1 (vcpu#0) crashed on cpu#1:
> (XEN) ----[ Xen-3.0.3-0  x86_32  debug=n  Not tainted
]----
> (XEN) CPU:    1
> (XEN) EIP:    e019:[<c02b6ea9>]
> (XEN) EFLAGS: 00000246   CONTEXT: guest
> (XEN) eax: ffffffff   ebx: d097d003   ecx: d097d000  
edx: 00000000
> (XEN) esi: 0000005b   edi: c0455000   ebp: c0317bb4  
esp: c0317b78
> (XEN) cr0: 8005003b   cr4: 000026d0   cr3: d097c000  
cr2: 00000000
> (XEN) ds: e021   es: e021   fs: 0000   gs: 0000   ss:
e021   cs: e019
> (XEN) Guest stack trace from esp=c0317b78:
> (XEN)    00000000 c02b6ea9 0001e019 00010046 c02b798c
d097a154 d0973061 00000000
> (XEN)    00000000 00000000 c0454000 00000005 c0450c10
c0450000 c0459000 00000000
> (XEN)    c0100179 c0317bbc 0000e021 c01044bc c0104520
00000000 00000000 00000000
> (XEN)    00000000 00000000 00000000 c02ec8a0 00000000
00000000 00000000 00000000
> (XEN)    00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000
> [many more lines of zeros]


build a XEN3_DOMU kernel with -g and boot it. Then use gdb
netbsd.gdb
to match the eip value (c02b6ea9 here) to function/line in
the source
code. It's quite possible that this will be in assembly
code;
then you can use 'disass' to find the instruction.
You can also try to walk the stack dump to reconstruct the
function
call graph.

-- 
Manuel Bouyer <bouyerantioche.eu.org>
     NetBSD: 26 ans d'experience feront toujours la
difference
--

> build a XEN3_DOMU kernel with -g and boot it. Then use
gdb netbsd.gdb

I remember why I rarely use gdb.

I think it looks like this (3.1 sources):

  locore.S:456            start
  xen_machdep.c:422       xpmap_init
  xen_machdep.c:311       xpmap_get_bootpte (crashed here)

xpmap_get_bootpte may have called these before crashing:

	xpmap_get_bootptep
	  xpmap_get_vbootpde
	    xpmap_get_bootpde
	    xpmap_mtop

I can't tell by just looking at the code if this is NetBSD's
fault or
Xen's.

Is there a way to single step a running kernel under xen, or
do I
need to do some sort of printf thing? Which would I use
(XENPRINTF,
XENPRINTK, etc. )?

Oh, I've tried every version of NetBSD from 3.1 to recent
-current. It's
all the same.

Thanks.

--
- Brian

On Thu, May 03, 2007 at 07:33:44PM -0400, Brian Marcotte
wrote:
> > build a XEN3_DOMU kernel with -g and boot it. Then
use gdb netbsd.gdb
> 
> I remember why I rarely use gdb.
> 
> I think it looks like this (3.1 sources):
> 
>   locore.S:456            start
>   xen_machdep.c:422       xpmap_init
>   xen_machdep.c:311       xpmap_get_bootpte (crashed
here)
> 
> xpmap_get_bootpte may have called these before
crashing:
> 
> 	xpmap_get_bootptep
> 	  xpmap_get_vbootpde
> 	    xpmap_get_bootpde
> 	    xpmap_mtop
> 
> I can't tell by just looking at the code if this is
NetBSD's fault or
> Xen's.
> 
> Is there a way to single step a running kernel under
xen,

I fear it's not possible at this point; not enough things
have been
set up to use ddb

> or do I
> need to do some sort of printf thing? Which would I use
(XENPRINTF,
> XENPRINTK, etc. )?

printf() won't work either; but I suspect printk (XENPRINTK)
may (this
will show up on xen's console, not the domU console).
You'll need a DEBUG xen,gz for this; but when building
you'll have
to disable NDEBUG manually (I don't have the sources handy
to tell you
where, sorry) because for debug xen uses one of the free PTE
bits that
NetBSD use too.

-- 
Manuel Bouyer <bouyerantioche.eu.org>
     NetBSD: 26 ans d'experience feront toujours la
difference
--

> > I can't tell by just looking at the code if this
is NetBSD's fault or
> > Xen's.

Good news: I don't have to try to debug NetBSD within Xen.

I upgraded to Xen-3.0.4-1 and NetBSD now works on these
machines.

Thanks for your help. If I had not seen the NetBSD code, I
may not have
considered Xen could be at fault.

--
- Brian