|
List Info
Thread: Strange events leading to "Hotplug scripts not working" and xen death
|
|
| Strange events leading to "Hotplug
scripts not working" and xen death |
 
United States |
2007-05-15 10:49:27 |
I apologise to anyone who is on xen-users or xen-devel. I
posted this to
both lists and was told I should talk to the netbsd xen
maintainers 
Can anyone here help me with this? I really odn't want to
have to use
Linux for my Dom0, but I might be forced to at this rate
:-(
I think it's a bug in whatever is supposed to execute the
vif-bridge
script (but I'm not sure if that is xend, or xenbackend).
System: NetBSD 4.0 BETA2 as of ~1 week ago, Xen 3 compiled
from pkgsrc.
I have Dom0 working just fine. It boots without a problem.
I also have a file-backed Linux DomU which starts. The
config for this
is at the end.
I've found a rather.. strange series of events that causes
Xen to stop
creating new DomUs. Even my working Linux DomUs fail to
create.
The thing that makes this strange, is that there doesn't
seem to be one
single thing thing in particular that causes this. For
instance,
following the instructions here on creating a FreeBSD DomU
work the
first time, and after that everything stops working:
http://www.freebsd.org/doc/en
_US.ISO8859-1/books/handbook/virtualization-guest.html
Also, downloading CentOS 5 from jailtime.org causes
everything to stop
working. The configs I used for FreeBSD and CentOS5 are at
the end also.
When things "stop working", I see messages like
this when trying to
create a domain:
[root xen1] /xen/configs # xm create -c freebsd1
Using config file "./freebsd1".
Error: Device 0 (vif) could not be connected. Hotplug
scripts not working.
Some digging indicated that the scripts (eg,
/usr/pkg/etc/xen/scripts/vif-bridge) are not actually being
accessed any
more - their access time does not change again until I
reboot to fix
this problem. Restarting xend or any other xen daemons
doesn't work.
*Finally* commands
like 'xenstore-ls' just hang too.
The xenstore commands seem to be hanging trying to read from
the store.
/var/run/xenstored/tdb is getting updated though, before 'xm
create'
hangs.
I believe the problem may have something to do with the
storage, but I'm
not sure.
The end of the xenstore-trace.log looks like this:
OUT 0x805b0b0 20070513 19:53:35 TRANSACTION_END (OK )
IN 0x805b0b0 20070513 19:53:35 TRANSACTION_START ( )
OUT 0x805b0b0 20070513 19:53:35 TRANSACTION_START (447 )
IN 0x805b0b0 20070513 19:53:35 READ
(/local/domain/244/device/vif/0/backend )
OUT 0x805b0b0 20070513 19:53:35 READ
(/local/domain/0/backend/vif/244/0)
IN 0x805b0b0 20070513 19:53:35 TRANSACTION_END (T )
DESTROY transaction 0x8063230
OUT 0x805b0b0 20070513 19:53:35 TRANSACTION_END (OK )
IN 0x805b0b0 20070513 19:53:35 WATCH
(/local/domain/0/backend/vif/244/0/hotplug-status 138635916
)
CREATE watch 0x8062030
OUT 0x805b0b0 20070513 19:53:35 WATCH (OK )
OUT 0x805b0b0 20070513 19:53:35 WATCH_EVENT
(/local/domain/0/backend/vif/244/0/hotplug-status 138635916
)
IN 0x805b0b0 20070513 19:53:35 TRANSACTION_START ( )
OUT 0x805b0b0 20070513 19:53:35 TRANSACTION_START (448 )
IN 0x805b0b0 20070513 19:53:35 READ
(/local/domain/0/backend/vif/244/0/hotplug-status )
OUT 0x805b0b0 20070513 19:53:35 ERROR (ENOENT )
IN 0x805b0b0 20070513 19:53:35 TRANSACTION_END (T )
DESTROY transaction 0x8063230
OUT 0x805b0b0 20070513 19:53:35 TRANSACTION_END (OK )
Can anyone help? The only suggestion that the many Google
results on
"Hotplug scripts not working" errors say to make
sure udev is working,
but I don't think that applies to NetBSD just Linux.
Working Linux DomU config:
kernel = "/xen/kernels/vmlinuz-2.6.16.33-xen"
memory = 128
name = "Debian_Sarge_Image"
vif = [ 'mac=00:11:22:33:44:55, bridge=bridge0' ]
disk = [ 'file:/xen/nfs_root/sarge1/sarge1.img,hda1,w' ]
root = "/dev/hda1 ro"
extra = "init=/bin/sh"
FreeBSD 7 which causes breakage:
kernel = "/xen/kernels/freebsd7-kernel"
memory = 256
name = "freebsd_xen_INSTALL"
vif = [ 'mac=00:11:22:33:44:66, bridge=bridge0' ]
disk = [ 'file:/xen/nfs_root/freebsd1/mdroot-7.0,hda1,w'
]
extra = "boot_verbose"
extra += ",boot_single"
extra += ",kern.hz=100"
extra += ",vfs.root.mountfrom=ufs:/dev/xbd769a"
CentOS 5 which causes breakage:
kernel = "/xen/kernels/vmlinuz-2.6.16.33-xen"
memory = 128
name = "centos.5-0"
vif = [ '' ]
dhcp = "dhcp"
disk =
['file:/xen/nfs_root/centos50/centos.5-0.img,sda1,w',
'file:/xen/nfs_root/centos50/centos.swap,sda2,w']
root = "/dev/sda1 ro"
--
Avleen Vig
Systems Administrator
Personal: www.silverwraith.com
"It's not me that's crazy, it's you."
"Monkey?"
|
|
| Re: Strange events leading to
"Hotplug scripts not working"
and xen death |
|
2007-05-15 12:34:19 |
On Tue, May 15, 2007 at 08:49:27AM -0700, Avleen Vig wrote:
> I apologise to anyone who is on xen-users or xen-devel.
I posted this to
> both lists and was told I should talk to the netbsd xen
maintainers
>
> Can anyone here help me with this? I really odn't want
to have to use
> Linux for my Dom0, but I might be forced to at this
rate :-(
> I think it's a bug in whatever is supposed to execute
the vif-bridge
> script (but I'm not sure if that is xend, or
xenbackend).
It's xenbackend
>
> System: NetBSD 4.0 BETA2 as of ~1 week ago, Xen 3
compiled from pkgsrc.
Is it xen-3.0.3 or xen-3.0.4 ?
>
> I have Dom0 working just fine. It boots without a
problem.
> I also have a file-backed Linux DomU which starts. The
config for this
> is at the end.
>
> I've found a rather.. strange series of events that
causes Xen to stop
> creating new DomUs. Even my working Linux DomUs fail to
create.
> The thing that makes this strange, is that there
doesn't seem to be one
> single thing thing in particular that causes this. For
instance,
> following the instructions here on creating a FreeBSD
DomU work the
> first time, and after that everything stops working:
>
> http://www.freebsd.org/doc/en
_US.ISO8859-1/books/handbook/virtualization-guest.html
>
> Also, downloading CentOS 5 from jailtime.org causes
everything to stop
> working. The configs I used for FreeBSD and CentOS5 are
at the end also.
>
> When things "stop working", I see messages
like this when trying to
> create a domain:
> [rootxen1] /xen/configs # xm create -c freebsd1
> Using config file "./freebsd1".
> Error: Device 0 (vif) could not be connected. Hotplug
scripts not working.
>
> Some digging indicated that the scripts (eg,
> /usr/pkg/etc/xen/scripts/vif-bridge) are not actually
being accessed any
> more - their access time does not change again until I
reboot to fix
> this problem. Restarting xend or any other xen daemons
doesn't work.
>
> *Finally* commands
like 'xenstore-ls' just hang too.
> The xenstore commands seem to be hanging trying to read
from the store.
So it's most probably an issue with the store. Could you try
ktrace to
see on what xenstore-ls hangs ? Is xenstored still running
?
You could try to ktrace it too.
--
Manuel Bouyer <bouyerantioche.eu.org>
NetBSD: 26 ans d'experience feront toujours la
difference
--
|
|
| Re: Strange events leading to
"Hotplug scripts not working"
and xen death |
United States |
2007-05-15 13:17:24 |
On Tue, May 15, 2007 at 07:34:19PM +0200, Manuel Bouyer
wrote:
> It's xenbackend
Cool good to know
> > System: NetBSD 4.0 BETA2 as of ~1 week ago, Xen 3
compiled from pkgsrc.
>
> Is it xen-3.0.3 or xen-3.0.4 ?
xen-3.0.4
> > *Finally* commands
like 'xenstore-ls' just hang too.
> > The xenstore commands seem to be hanging trying to
read from the store.
>
> So it's most probably an issue with the store. Could
you try ktrace to
> see on what xenstore-ls hangs ? Is xenstored still
running ?
> You could try to ktrace it too.
xenstored is still running.
I rebooted, ktraced xenstored, and then tried to create the
centos
domain, which reliably breaks xen every time:
http://silverwraith.com/xen/xenstore_during_x
m_create.ktrace.out.gz
~1mb (~12mb uncompressed)
xenstore-ls current works though with no errors...
I tried to create the FreeBSD7 domain, and it failed with:
[XEN] Initialising virtual ethernet driver.
panic: not implemented
Uptime: 1s
That looks like a problem with the FreeBSD kernel, but as a
result,
xenstore-ls now gives this error at the end:
vbd = ""
1 = ""
2 = ""
769 = ""
domain = "freebsd_xen_INSTALL"
frontend =
"/local/domain/2/device/vbd/769"
uuid =
"807bc57d-ff85-27b7-a7fc-1dd311d02d73"
dev = "hda1"
state = "5"
params =
"/xen/nfs_root/freebsd1/mdroot-7.0"
mode = "w"
online = ""
xenstore-ls: xs_directory
(/local/domain/0/backend/vbd/2/769/online): Bad file
descriptor
Running xenstore for a second time now makes it hang.
Ktrace for it is at: http://silverwraith.com/xen/xenstore_ls.ktrace.out.gz
Thank you!
|
|
| Re: Strange events leading to
"Hotplug scripts not working"
and xen death |
|
2007-05-15 18:22:11 |
On Wed, 16 May 2007 03:34:19 am Manuel Bouyer wrote:
> It's xenbackend
> > Also, downloading CentOS 5 from jailtime.org
causes everything to stop
> > working. The configs I used for FreeBSD and
CentOS5 are at the end also.
> >
> > When things "stop working", I see
messages like this when trying to
> > create a domain:
> > [rootxen1] /xen/configs # xm create -c
freebsd1
> > Using config file "./freebsd1".
> > Error: Device 0 (vif) could not be connected.
Hotplug scripts not
> > working.
This isn't possibly a sparse file issue? I've seen some
weird errors where
some domus are ok but others fail with no usable error ...
Sarton
|
|
| Re: Strange events leading to
"Hotplug scripts not working"
and xen death |
United States |
2007-05-17 11:17:33 |
On Tue, May 15, 2007 at 07:34:19PM +0200, Manuel Bouyer
wrote:
> > *Finally* commands
like 'xenstore-ls' just hang too.
> > The xenstore commands seem to be hanging trying to
read from the store.
>
> So it's most probably an issue with the store. Could
you try ktrace to
> see on what xenstore-ls hangs ? Is xenstored still
running ?
> You could try to ktrace it too.
I ran xenstore-ls while trying to create one of these broken
domains.
The /local/3/device/vif/0/backend line is.. interesting:
"backend = "/locaû""
Hopefully this problem is simply that the store is being
writen to
incorrectly.
I then deleted all of the xen related files in /var/run,
rebooted and
ran the `xm create` and `xenstore-ls`. This time it spat
back a "cannot
allocate memory" error :-( The outputs from both runs
are below.
============================================================
============
First run:
[rootxen1] ~ # xenstore-ls
tool = ""
xenstored = ""
vm = ""
00000000-0000-0000-0000-000000000000 = ""
on_xend_stop = "ignore"
shadow_memory = "0"
uuid = "00000000-0000-0000-0000-000000000000"
on_reboot = "restart"
on_poweroff = "destroy"
on_xend_start = "ignore"
on_crash = "restart"
xend = ""
restart_count = "0"
vcpus = "1"
vcpu_avail = "1"
memory = "64"
maxmem = "64"
name = "Domain-0"
00ada6e1-ff9c-facb-80b5-2acad2e97eac = ""
image = "(linux (kernel
/xen/kernels/vmlinuz-2.6.16.33-xen) (args
'ip=10.0.0.13:10.0.0.12:10.0.0..."
ostype = "linux"
kernel = "/xen/kernels/vmlinuz-2.6.16.33-xen"
cmdline =
"ip=10.0.0.13:10.0.0.12:10.0.0.1:255.255.255.0::eth0:of
f root=/dev/nfs nfsroot=10.0.0..."
ramdisk = ""
on_xend_stop = "ignore"
shadow_memory = "0"
uuid = "00ada6e1-ff9c-facb-80b5-2acad2e97eac"
on_reboot = "restart"
start_time = "1179406408.47"
on_poweroff = "destroy"
on_xend_start = "ignore"
on_crash = "restart"
xend = ""
restart_count = "0"
vcpus = "1"
vcpu_avail = "1"
memory = "64"
maxmem = "64"
name = "test_os"
local = ""
domain = ""
0 = ""
vm =
"/vm/00000000-0000-0000-0000-000000000000"
cpu = ""
0 = ""
availability = "online"
memory = ""
target = "65536"
name = "Domain-0"
console = ""
limit = "1048576"
domid = "0"
backend = ""
vif = ""
1 = ""
2 = ""
3 = ""
0 = ""
domain = "test_os"
handle = "0"
uuid =
"d51098d5-d658-0e05-e520-756e2f8c6a36"
script =
"/usr/pkg/etc/xen/scripts/vif-bridge"
ip = "10.0.0.13"
state = "2"
frontend = "/local/domain/3/device/vif/0"
mac = "00:16:3e:29:67:16"
online = "1"
frontend-id = "3"
3 = ""
vm =
"/vm/00ada6e1-ff9c-facb-80b5-2acad2e97eac"
device-misc = ""
vif = ""
nextDeviceID = "1"
device = ""
vif = ""
0 = ""
backend-id = "0"
mac = "00:16:3e:29:67:16"
handle = "0"
state = "1"
backend = "/locaû"
console = ""
ring-ref = "509853"
port = "2"
limit = "1048576"
tty = "/dev/ttyp2"
name = "test_os"
domid = "3"
cpu = ""
0 = ""
availability = "online"
memory = ""
target = "65536"
store = ""
ring-ref = "509854"
port = "1"
============================================================
============
Second run:
[rootxen1] ~ # xenstore-ls
tool = ""
xenstored = ""
vm = ""
00000000-0000-0000-0000-000000000000 = ""
on_xend_stop = "ignore"
shadow_memory = "0"
uuid = "00000000-0000-0000-0000-000000000000"
on_reboot = "restart"
on_poweroff = "destroy"
on_xend_start = "ignore"
on_crash = "restart"
xend = ""
restart_count = "0"
vcpus = "1"
vcpu_avail = "1"
memory = "64"
maxmem = "64"
name = "Domain-0"
ef25b12c-a2ed-cb7f-6bae-3d7a7f6b5965 = ""
image = "(linux (kernel
/xen/kernels/vmlinuz-2.6.16.33-xen) (args
'ip=10.0.0.13:10.0.0.12:10.0.0.1:255.255...."
ostype = "linux"
kernel = "/xen/kernels/vmlinuz-2.6.16.33-xen"
cmdline =
"ip=10.0.0.13:10.0.0.12:10.0.0.1:255.255.255.0::eth0:of
f root=/dev/nfs nfsroot=10.0.0.12:/tmp/xe..."
ramdisk = ""
on_xend_stop = "ignore"
shadow_memory = "0"
uuid = "ef25b12c-a2ed-cb7f-6bae-3d7a7f6b5965"
on_reboot = "restart"
start_time = "1179406927.98"
on_poweroff = "destroy"
on_xend_start = "ignore"
on_crash = "restart"
xend = ""
restart_count = "0"
vcpus = "1"
vcpu_avail = "1"
memory = "64"
maxmem = "64"
name = "test_os"
local = ""
domain = ""
0 = ""
vm =
"/vm/00000000-0000-0000-0000-000000000000"
cpu = ""
0 = ""
availability = "online"
memory = ""
target = "65536"
name = "Domain-0"
console = ""
limit = "1048576"
domid = "0"
backend = ""
vif = ""
1 = ""
0 = ""
domain = "test_os"
handle = "0"
uuid =
"ac21d8c9-9aba-07ef-6cb3-d2481673ce62"
script =
"/usr/pkg/etc/xen/scripts/vif-bridge"
ip = "10.0.0.13"
state = "2"
frontend = "/local/domain/1/device/vif/0"
mac = "00:16:3e:6a:69:2e"
online = "1"
frontend-id = "1"
1 = ""
vm =
"/vm/ef25b12c-a2ed-cb7f-6bae-3d7a7f6b5965"
device-misc = ""
vif = ""
nextDeviceID = "1"
device = ""
vif = ""
0 = ""
backend-id = "0"
mac = "00:16:3e:6a:69:2e"
handle = "0"
state = "1"
backend = "/local/domain/0/backend/vif/1/0"
console = ""
xenstore-ls: xs_directory (/local/domain/1/console): Cannot
allocate memory
|
|
[1-5]
|
|