The "HYPERVISOR_mmu_update failed" panic

I noticed that, late July / early August, Eric Delcamp
posted a message 
to this forum about getting a "HYPERVISOR_mmu_update
failed" panic when 
booting INSTALL_XEN3_DOMU on a Linux dom0, with a recent
Xen. (see 
http://mail-index.netbsd.org/port-xen/2006/07/26/0007.
html)

I found that message, because I got the message myself when
using the 
latest development Xen sources (with a few mods), running
the 
OpenSolaris dom0, and then booting INSTALL_XEN3_DOMU in
-current.

Anyway, this is just a note that I tracked down this
problem. When 
compiled with debugging options on, Xen uses an unused PTE
bit to track 
granted table pages, in an effort to catch cleanup problems
early. 
However, the NetBSD pmap already uses all of those bits. So,
at the 
first attempt to enter a tracked (pv_list) mapping, Xen
refuses, because 
the same bit it is using for debugging purposes is set in
the PTE. The 
definition (_PAGE_GNTTAB) is even marked as "has to be
disabled for *BSD").

I guess there are Xen compiles out there that have debug on
by default, 
and with the current Xen sources, NetBSD (and *BSD in
general), won't 
boot on that.

- Frank

Frank van der Linden wrote:
> I noticed that, late July / early August, Eric Delcamp
posted a 
> message to this forum about getting a
"HYPERVISOR_mmu_update failed" 
> panic when booting INSTALL_XEN3_DOMU on a Linux dom0,
with a recent 
> Xen. (see http://mail-index.netbsd.org/port-xen/2006/07/26/0007.
html)
>
> I found that message, because I got the message myself
when using the 
> latest development Xen sources (with a few mods),
running the 
> OpenSolaris dom0, and then booting INSTALL_XEN3_DOMU in
-current.
>
> Anyway, this is just a note that I tracked down this
problem. When 
> compiled with debugging options on, Xen uses an unused
PTE bit to 
> track granted table pages, in an effort to catch
cleanup problems 
> early. However, the NetBSD pmap already uses all of
those bits. So, at 
> the first attempt to enter a tracked (pv_list) mapping,
Xen refuses, 
> because the same bit it is using for debugging purposes
is set in the 
> PTE. The definition (_PAGE_GNTTAB) is even marked as
"has to be 
> disabled for *BSD").
>
> I guess there are Xen compiles out there that have
debug on by 
> default, and with the current Xen sources, NetBSD (and
*BSD in 
> general), won't boot on that.
>
> - Frank
Yes, Xen debug builds use the bit reserved by pmap to mark
wired pages 
for its own purposes.
Maybe we could check if the hypervisor is OK to run BSD and
panic with 
the appropriate message if not. I'm not sure we can see in
the 
hypervisor start info struct that it was built with -DDEBUG,
but a least 
we can try to set the bit on some page early and check the
result (like 
in pmap_bootstrap()).

-- Mathieu

Frank van der Linden wrote:
> I noticed that, late July / early August, Eric Delcamp
posted a 
> message to this forum about getting a
"HYPERVISOR_mmu_update failed" 
> panic when booting INSTALL_XEN3_DOMU on a Linux dom0,
with a recent 
> Xen. (see http://mail-index.netbsd.org/port-xen/2006/07/26/0007.
html)
>
> I found that message, because I got the message myself
when using the 
> latest development Xen sources (with a few mods),
running the 
> OpenSolaris dom0, and then booting INSTALL_XEN3_DOMU in
-current.
>
> Anyway, this is just a note that I tracked down this
problem. When 
> compiled with debugging options on, Xen uses an unused
PTE bit to 
> track granted table pages, in an effort to catch
cleanup problems 
> early. However, the NetBSD pmap already uses all of
those bits. So, at 
> the first attempt to enter a tracked (pv_list) mapping,
Xen refuses, 
> because the same bit it is using for debugging purposes
is set in the 
> PTE. The definition (_PAGE_GNTTAB) is even marked as
"has to be 
> disabled for *BSD").
>
> I guess there are Xen compiles out there that have
debug on by 
> default, and with the current Xen sources, NetBSD (and
*BSD in 
> general), won't boot on that.
>
> - Frank
Yes, Xen debug builds use the bit reserved by pmap to mark
wired pages 
for its own purposes.
Maybe we could check if the hypervisor is OK to run BSD and
panic with 
the appropriate message if not. I'm not sure we can see in
the 
hypervisor start info struct that it was built with -DDEBUG,
but a least 
we can try to set the bit on some page early and check the
result (like 
in pmap_bootstrap()).

-- Mathieu

Frank van der Linden wrote:
> I noticed that, late July / early August, Eric Delcamp
posted a 
> message to this forum about getting a
"HYPERVISOR_mmu_update failed" 
> panic when booting INSTALL_XEN3_DOMU on a Linux dom0,
with a recent 
> Xen. (see http://mail-index.netbsd.org/port-xen/2006/07/26/0007.
html)
>
> I found that message, because I got the message myself
when using the 
> latest development Xen sources (with a few mods),
running the 
> OpenSolaris dom0, and then booting INSTALL_XEN3_DOMU in
-current.
>
> Anyway, this is just a note that I tracked down this
problem. When 
> compiled with debugging options on, Xen uses an unused
PTE bit to 
> track granted table pages, in an effort to catch
cleanup problems 
> early. However, the NetBSD pmap already uses all of
those bits. So, at 
> the first attempt to enter a tracked (pv_list) mapping,
Xen refuses, 
> because the same bit it is using for debugging purposes
is set in the 
> PTE. The definition (_PAGE_GNTTAB) is even marked as
"has to be 
> disabled for *BSD").
>
> I guess there are Xen compiles out there that have
debug on by 
> default, and with the current Xen sources, NetBSD (and
*BSD in 
> general), won't boot on that.
>
> - Frank
Yes, Xen debug builds use the bit reserved by pmap to mark
wired pages 
for its own purposes.
Maybe we could check if the hypervisor is OK to run BSD and
panic with 
the appropriate message if not. I'm not sure we can see in
the 
hypervisor start info struct that it was built with -DDEBUG,
but a least 
we can try to set the bit on some page early and check the
result (like 
in pmap_bootstrap()).

-- Mathieu