New kernel does not boot up on remote host

Hi ML,

I need your help to find the right way.....

I am running a

Linux sd-4421 2.6.20-gentoo-r8 #1 SMP Sat Jun 16 19:27:48
CEST 2007
i686 VIA Esther processor 2000MHz CentaurHauls GNU/Linux

remote box. This kernel was compiled on my own. It is good.
But I need
a xen box now. So I emerged the linux-2.6.16.49-xen sources
and
compiled them successfully (I made an oldconfig on the
/proc/config.gz
of the running 2.6.20). I followed the
http://ge
ntoo-wiki.com/HOWTO_Xen_and_Gentoo doc for dom0-Xen
specific
kernel config.

Well, now it is time to boot up the new dom0_2.6.18...
grub.conf got
the changes, fall-backing to previous kernel if
required....

... the matter is Grub falls back to previous 2.6.20 kernel.
So I have
no log at all about what went wrong during the dom0_2.6.18
boot.

Any idea to know what went wrong ?

Many thanks for your help,

Gal'
-- 
gentoo-usergentoo.org mailing list

On Wed, 2007-07-04 at 21:54 +0200, Galevsky wrote:
> ... the matter is Grub falls back to previous 2.6.20
kernel. So I have
> no log at all about what went wrong during the
dom0_2.6.18 boot.
> 
> Any idea to know what went wrong ?
> 

Fall back is usually "useless" in the sense that
if grub finds your
kernel and can boot it, but it happens to be a bad kernel,
then there is
no benefit.  If/Since you *are* falling back leads me to
believe that
grub either can't find your kernel or otherwise can't load
it (typo in
grub.conf).  Nevertheless, if you want to actually know what
the error
is then you should disable the fallback.


--
Albert W. Hopkins

-- 
gentoo-usergentoo.org mailing list

Many thanks for you suggestion. I tried a boot with no fall
back, but
nothing added neither to boot.log nor to dmesg. I am sure to
target
the right kernel image with no typo into grub.conf . This
new kernel
is a =xen-sources-2.6.16.49, and I configured it making an
oldconfig
based on current =gentoo-sources-2.6.20 running kernel. So,
errors
cannot come from drivers or such things, but specific xen
options.
Does anyone know any xen option able to prevent the kernel
to boot up
?

Many thanks for your support,

Gal'

2007/7/4, Albert Hopkins <mardukgentoo.org>:
> On Wed, 2007-07-04 at 21:54 +0200, Galevsky wrote:
> > ... the matter is Grub falls back to previous
2.6.20 kernel. So I have
> > no log at all about what went wrong during the
dom0_2.6.18 boot.
> >
> > Any idea to know what went wrong ?
> >
>
> Nevertheless, if you want to actually know what the
error
> is then you should disable the fallback.
>
>
> --
> Albert W. Hopkins
>
> --
> gentoo-usergentoo.org mailing list
>
>
-- 
gentoo-usergentoo.org mailing list

On Thu, 2007-07-05 at 11:36 +0200, Galevsky wrote:
> Many thanks for you suggestion. I tried a boot with no
fall back, but
> nothing added neither to boot.log nor to dmesg. I am
sure to target
> the right kernel image with no typo into grub.conf .
This new kernel
> is a =xen-sources-2.6.16.49, and I configured it making
an oldconfig
> based on current =gentoo-sources-2.6.20 running kernel.
So, errors
> cannot come from drivers or such things, but specific
xen options.
> Does anyone know any xen option able to prevent the
kernel to boot up
> ?

I guess I (still) don't understand what your issue is.  So I
went back
and read your original post.  Maybe I misunderstood it.

      * What do you mean by "remote host"?
      * If your box "does not boot" how would you
expect to see a
        boot.log or dmesg?  They don't exist if the system
has not
        booted.  In fact "does not boot" implies
that you can't even log
        in to check for dmesg or boot logs.  Perhaps you can
explain
        what it is exactly you mean by "does not
boot".  
      * Usually there is an error message on the console if
there is a
        boot loader/kernel issue with booting.
      * Also you never posted your grub.conf.

So hopefully you can come up with a specific explanation of
"does not
boot" as well as your config and any console messages
you get.  Else
this becomes a "blind leading the blind" issue.



--
Albert W. Hopkins

-- 
gentoo-usergentoo.org mailing list

2007/7/5, Albert Hopkins <mardukgentoo.org>:
> On Thu, 2007-07-05 at 11:36 +0200, Galevsky wrote:
> > Many thanks for you suggestion. I tried a boot
with no fall back, but
> > nothing added neither to boot.log nor to dmesg. I
am sure to target
> > the right kernel image with no typo into grub.conf
. This new kernel
> > is a =xen-sources-2.6.16.49, and I configured it
making an oldconfig
> > based on current =gentoo-sources-2.6.20 running
kernel. So, errors
> > cannot come from drivers or such things, but
specific xen options.
> > Does anyone know any xen option able to prevent
the kernel to boot up
> > ?
>
> I guess I (still) don't understand what your issue is. 
So I went back
> and read your original post.  Maybe I misunderstood
it.
>
>       * What do you mean by "remote host"?

I rent a dedicated host. Thus, I have no physical access to
the
machine. And the reason why I used the fall-back feature
into grub is
to avoid the use of a boring rescue system (via a web
interface) to
take back the control of a not-responding box.

>       * If your box "does not boot" how would
you expect to see a
>         boot.log or dmesg ? They don't exist if the
system has not
>         booted.  In fact "does not boot"
implies that you can't even log
>         in to check for dmesg or boot logs.  Perhaps
you can explain
>         what it is exactly you mean by "does not
boot".

When I try to boot on the new kernel, I set it as the
"default" one in
grub.conf and reboot the box. And I wait for a few minutes.
Then I try
to log on, praying to find the box responding.

First, with grub fall-back activated, the boot had fallen
back and I
was able to read the logs that contained the running kernel
logs, but
no info about the kernel boot that failed. That was the
matter of my
first mail: how to read  the reasons of the first
kernel-boot-attempt
failure.

Secondly, you advised me to turn fall back off. I do. The
box is no
more responding after reboot, and I have to launch the
"rescue system"
to log on my box, and read the /var/log/*. I thought
previously that
some pb happened -I didn't know when exactly-... and was
waiting for
more info in log files. But files were empty. So, I conclude
that the
boot procedure failed (and you too ;o)).

Finally, I have to find out why my kernel is not booting. I
tuned up
well my previous kernel, a gentoo-sources-2.6.20 -this
kernel runs
currently my box very well- and the kernel I want to boot
now is a
xen-sources-2.6.16. I made an oldconfig on /proc/config.gz
of the
2.6.20 running kernel, and fill in xen-dom0 specific options
as
indicated on the wiki tutorial [1]. So, I guess I did
something wrong
with the dom0 xen config, and asked for advice on xen
specific options
with my second post.

>       * Usually there is an error message on the
console if there is a
>         boot loader/kernel issue with booting.
>       * Also you never posted your grub.conf.
>
> So hopefully you can come up with a specific
explanation of "does not
> boot" as well as your config and any console
messages you get.  Else
> this becomes a "blind leading the blind"
issue.

I have no console message to provide you, I just know that
my remote
box is not responding when I try to boot another kernel (If
you know
the way to get logs, I'll be grateful ). And I
checked the
/var/log/* to be sure that the kernel was not running but
just missing
any network feature so that I could not log on the box
despite the
kernel was actually running.

As for my grub.conf + 'ls -l /boot' , I will provide you in
a while (I
need to go home before), but I am not sure it is relevant.
(just to
check for typo I guess).

> --
> Albert W. Hopkins
>
> --
> gentoo-usergentoo.org mailing list

Many thanks to take care of my problem 

Gal'

[1]: http://ge
ntoo-wiki.com/HOWTO_Xen_and_Gentoo
-- 
gentoo-usergentoo.org mailing list

On Thu, 5 Jul 2007 18:40:20 +0200
Galevsky <galevskygmail.com> wrote:

> 2007/7/5, Albert Hopkins <mardukgentoo.org>:
> > On Thu, 2007-07-05 at 11:36 +0200, Galevsky
wrote:
> > > Many thanks for you suggestion. I tried a
boot with no fall back,
> > > but nothing added neither to boot.log nor to
dmesg. I am sure to
> > > target the right kernel image with no typo
into grub.conf . This
> > > new kernel is a =xen-sources-2.6.16.49, and I
configured it
> > > making an oldconfig based on current
=gentoo-sources-2.6.20
> > > running kernel. So, errors cannot come from
drivers or such
> > > things, but specific xen options. Does anyone
know any xen option
> > > able to prevent the kernel to boot up ?
> >
> > I guess I (still) don't understand what your issue
is.  So I went
> > back and read your original post.  Maybe I
misunderstood it.
> >
> >       * What do you mean by "remote
host"?
> 
> I rent a dedicated host. Thus, I have no physical
access to the
> machine. And the reason why I used the fall-back
feature into grub is
> to avoid the use of a boring rescue system (via a web
interface) to
> take back the control of a not-responding box.
> 
> >       * If your box "does not boot" how
would you expect to see a
> >         boot.log or dmesg ? They don't exist if
the system has not
> >         booted.  In fact "does not boot"
implies that you can't
> > even log in to check for dmesg or boot logs. 
Perhaps you can
> > explain what it is exactly you mean by "does
not boot".
> 
> When I try to boot on the new kernel, I set it as the
"default" one in
> grub.conf and reboot the box. And I wait for a few
minutes. Then I try
> to log on, praying to find the box responding.
> 
> First, with grub fall-back activated, the boot had
fallen back and I
> was able to read the logs that contained the running
kernel logs, but
> no info about the kernel boot that failed. That was the
matter of my
> first mail: how to read  the reasons of the first
kernel-boot-attempt
> failure.
> 
> Secondly, you advised me to turn fall back off. I do.
The box is no
> more responding after reboot, and I have to launch the
"rescue system"
> to log on my box, and read the /var/log/*. I thought
previously that
> some pb happened -I didn't know when exactly-... and
was waiting for
> more info in log files. But files were empty. So, I
conclude that the
> boot procedure failed (and you too ;o)).
> 
> Finally, I have to find out why my kernel is not
booting. I tuned up
> well my previous kernel, a gentoo-sources-2.6.20 -this
kernel runs
> currently my box very well- and the kernel I want to
boot now is a
> xen-sources-2.6.16. I made an oldconfig on
/proc/config.gz of the
> 2.6.20 running kernel, and fill in xen-dom0 specific
options as
> indicated on the wiki tutorial [1]. So, I guess I did
something wrong
> with the dom0 xen config, and asked for advice on xen
specific options
> with my second post.
> 
> >       * Usually there is an error message on the
console if there
> > is a boot loader/kernel issue with booting.
> >       * Also you never posted your grub.conf.
> >
> > So hopefully you can come up with a specific
explanation of "does
> > not boot" as well as your config and any
console messages you get.
> > Else this becomes a "blind leading the
blind" issue.
> 
> I have no console message to provide you, I just know
that my remote
> box is not responding when I try to boot another kernel
(If you know
> the way to get logs, I'll be grateful ). And I
checked the
> /var/log/* to be sure that the kernel was not running
but just missing
> any network feature so that I could not log on the box
despite the
> kernel was actually running.
> 
> As for my grub.conf + 'ls -l /boot' , I will provide
you in a while (I
> need to go home before), but I am not sure it is
relevant. (just to
> check for typo I guess).
> 
> > --
> > Albert W. Hopkins
> >
> > --
> > gentoo-usergentoo.org mailing list
> 
> Many thanks to take care of my problem 
> 
> Gal'
> 
> [1]: http://ge
ntoo-wiki.com/HOWTO_Xen_and_Gentoo

from /etc/conf.d/rc:
# RC_BOOTLOG will generate a log of the boot messages shown
on the
console. # Useful for headless machines or debugging.  You
need to
emerge the # app-admin/showconsole package for this to work.
 Note that
this probably # won't work correctly with boot splash.

RC_BOOTLOG="no"

I recommend you install showconsole and set RC_BOOTLOG to
yes, that
might help you.  It is possible that maybe something as
mundane as
networking is failing for a stupid reason, and therefore you
cant get
to the computer because it cant finish booting.
-- 
gentoo-usergentoo.org mailing list

I recently discovered 'savefallback' in grub which I did not
know about.
I'll assume you're using that but again, I haven't seen your
grub.conf.

Dan Farrell made an intresting point about boot log and
showconsole, but
I assumed you were already using that.  But I still don't
that will help
you because your problem appears to be one of the
following:

      * Grub is not loading your kernel and is falling back
(same as my
        original theory)
      * Your kernel/hypervisor is loading but is crashing
immediately.
        If the Xen hypervisor is crashing you really need
the console.
        AFAIK there's no option to log.  The only option you
really have
        is whether or not to immediately reboot when it
crashes.  If Xen
        is loading successfully then it loads your dom0. If
that's
        crashing it's probably crashing immediately (i.e.
not even
        mounting root). If it is the dom0 then that seems to
be the case
        since you can't find any record of it having booted.
 If you
        crash before you mount root read/write then
showconsole and
        bootlog are useless.

All things said, I'm still guessing that it's either a grub
problem or
Xen doesn't like your hardware.  Xen is picky about hardware
and
sometimes you have to turn on/off things in the BIOS or as a
parameter
to Xen (like the ACPI controller) but it's going to be hard
to guess
without an error message and I'm betting that error message
appears
before bootlog/showconsole take effect.


--
Albert W. Hopkins

-- 
gentoo-usergentoo.org mailing list

2007/7/5, Dan Farrell <danspore.ath.cx>:
> On Thu, 5 Jul 2007 18:40:20 +0200
> Galevsky <galevskygmail.com> wrote:
>
> > 2007/7/5, Albert Hopkins <mardukgentoo.org>:
> > > On Thu, 2007-07-05 at 11:36 +0200, Galevsky
wrote:
> > > > Many thanks for you suggestion. I tried
a boot with no fall back,
> > > > but nothing added neither to boot.log
nor to dmesg. I am sure to
> > > > target the right kernel image with no
typo into grub.conf . This
> > > > new kernel is a =xen-sources-2.6.16.49,
and I configured it
> > > > making an oldconfig based on current
=gentoo-sources-2.6.20
> > > > running kernel. So, errors cannot come
from drivers or such
> > > > things, but specific xen options. Does
anyone know any xen option
> > > > able to prevent the kernel to boot up ?
> > >
> > > I guess I (still) don't understand what your
issue is.  So I went
> > > back and read your original post.  Maybe I
misunderstood it.
> > >
> > >       * What do you mean by "remote
host"?
> >
> > I rent a dedicated host. Thus, I have no physical
access to the
> > machine. And the reason why I used the fall-back
feature into grub is
> > to avoid the use of a boring rescue system (via a
web interface) to
> > take back the control of a not-responding box.
> >
> > >       * If your box "does not boot"
how would you expect to see a
> > >         boot.log or dmesg ? They don't exist
if the system has not
> > >         booted.  In fact "does not
boot" implies that you can't
> > > even log in to check for dmesg or boot logs. 
Perhaps you can
> > > explain what it is exactly you mean by
"does not boot".
> >
> > When I try to boot on the new kernel, I set it as
the "default" one in
> > grub.conf and reboot the box. And I wait for a few
minutes. Then I try
> > to log on, praying to find the box responding.
> >
> > First, with grub fall-back activated, the boot had
fallen back and I
> > was able to read the logs that contained the
running kernel logs, but
> > no info about the kernel boot that failed. That
was the matter of my
> > first mail: how to read  the reasons of the first
kernel-boot-attempt
> > failure.
> >
> > Secondly, you advised me to turn fall back off. I
do. The box is no
> > more responding after reboot, and I have to launch
the "rescue system"
> > to log on my box, and read the /var/log/*. I
thought previously that
> > some pb happened -I didn't know when exactly-...
and was waiting for
> > more info in log files. But files were empty. So,
I conclude that the
> > boot procedure failed (and you too ;o)).
> >
> > Finally, I have to find out why my kernel is not
booting. I tuned up
> > well my previous kernel, a gentoo-sources-2.6.20
-this kernel runs
> > currently my box very well- and the kernel I want
to boot now is a
> > xen-sources-2.6.16. I made an oldconfig on
/proc/config.gz of the
> > 2.6.20 running kernel, and fill in xen-dom0
specific options as
> > indicated on the wiki tutorial [1]. So, I guess I
did something wrong
> > with the dom0 xen config, and asked for advice on
xen specific options
> > with my second post.
> >
> > >       * Usually there is an error message on
the console if there
> > > is a boot loader/kernel issue with booting.
> > >       * Also you never posted your
grub.conf.
> > >
> > > So hopefully you can come up with a specific
explanation of "does
> > > not boot" as well as your config and any
console messages you get.
> > > Else this becomes a "blind leading the
blind" issue.
> >
> > I have no console message to provide you, I just
know that my remote
> > box is not responding when I try to boot another
kernel (If you know
> > the way to get logs, I'll be grateful ). And I
checked the
> > /var/log/* to be sure that the kernel was not
running but just missing
> > any network feature so that I could not log on the
box despite the
> > kernel was actually running.
> >
> > As for my grub.conf + 'ls -l /boot' , I will
provide you in a while (I
> > need to go home before), but I am not sure it is
relevant. (just to
> > check for typo I guess).
> >
> > > --
> > > Albert W. Hopkins
> > >
> > > --
> > > gentoo-usergentoo.org mailing list
> >
> > Many thanks to take care of my problem 
> >
> > Gal'
> >
> > [1]: http://ge
ntoo-wiki.com/HOWTO_Xen_and_Gentoo
>
> from /etc/conf.d/rc:
> # RC_BOOTLOG will generate a log of the boot messages
shown on the
> console. # Useful for headless machines or debugging. 
You need to
> emerge the # app-admin/showconsole package for this to
work.  Note that
> this probably # won't work correctly with boot splash.
>
> RC_BOOTLOG="no"
>
> I recommend you install showconsole and set RC_BOOTLOG
to yes, that
> might help you.  It is possible that maybe something as
mundane as
> networking is failing for a stupid reason, and
therefore you cant get
> to the computer because it cant finish booting.
> --
> gentoo-usergentoo.org mailing list
>
>

Thank you Dan, but I did it before, and boot.log remains
empty. In
fact, the new kernel boot turns on like grub couldn't find
the kernel
image....

Hereafter my /boot content:

sd-4421 boot # ll /boot
total 13M
-rw-rw-r-- 1 root root 767k Jul  6 00:02
System.map-2.6.16.49-xendedibox_r6_final
-rw-r--r-- 1 root root 929k Jun 16 19:29
System.map-2.6.20-gentoo-r8
lrwxrwxrwx 1 root root    1 Apr 30 19:40 boot -> ./
-rw-rw-r-- 1 root root  31k Jul  6 00:02
config-2.6.16.49-xendedibox_r6_final
-rw-r--r-- 1 root root  34k Jun 16 19:29
config-2.6.20-gentoo-r8
-rw-r--r-- 1 root root  753 Jul  4 23:30
gentoo-sources-2.6.18
drwxr-xr-x 2 root root 1.1k Jun 16 16:58 grub/
drwx------ 2 root root  13k Apr 30 19:40 lost+found/
drwxr-xr-x 2 root root 1.1k Jun 16 23:21 ref/
-rw-r--r-- 1 root root 5.9M Jul  6 00:02
vmlinux-syms-2.6.16.49-xendedibox_r6_final
lrwxrwxrwx 1 root root   37 Jul  6 00:02 vmlinuz-2.6-xen
->
vmlinuz-2.6.16.49-xendedibox_r6_final
lrwxrwxrwx 1 root root   37 Jul  6 00:02 vmlinuz-2.6.16-xen
->
vmlinuz-2.6.16.49-xendedibox_r6_final
-rw-r--r-- 1 root root 2.4M Jul  6 00:02
vmlinuz-2.6.16.49-xendedibox_r6_final
-rw-r--r-- 1 root root 2.4M Jun 16 19:29
vmlinuz-2.6.20-gentoo-r8

and my grub.conf:

### START (grub.conf)
sd-4421 boot # cat /boot/grub/grub.conf
# Customized boot procedure

default 0
timeout 1
#fallback 1 2

title Gentoo Linux 2.6.16-gentoo_xen_dom0
  root (hd0,0)
  kernel /boot/vmlinuz-2.6-xen ro root=/dev/sda2


title Gentoo Linux 2.6.20-r8
  root (hd0,0)
  kernel /boot/vmlinuz-2.6.20-gentoo-r8 ro root=/dev/sda2


title Gentoo Linux 2.6.18-r4-dedibox_r6_final
  root (hd0,0)
  kernel /boot/ref/2.6.18-gentoo-r4dedibox_r6_final ro
root=/dev/sda2
### END (grub.conf)

Well, let's try a boot on kernel /boot/vmlinuz-2.6-xen

=> box not responding..... and via the rescue system:

# ls /mnt/sda2/var/log/
portage   user.log  xen
# more  /mnt/sda2/var/log/user.log
Jul  6 00:12:30 sd-4421 shutdown[4571]: shutting down for
system reboot

thus no log at all (xen log also empty).


Gal'
-- 
gentoo-usergentoo.org mailing list

Sorry, I was building again my kernel image to confirm that
It was not
a stupid mistake.

2007/7/6, Albert Hopkins <mardukgentoo.org>:
> I recently discovered 'savefallback' in grub which I
did not know about.
> I'll assume you're using that but again, I haven't seen
your grub.conf.
>
> Dan Farrell made an intresting point about boot log and
showconsole, but
> I assumed you were already using that.

Ya, I did it. No boot.log created when booting the
xen-kernel, but
reporting good status when I boot up the good 2.6.20 one.
So, feature
is on.

>  But I still don't that will help
> you because your problem appears to be one of the
following:
>
>       * Grub is not loading your kernel and is falling
back (same as my
>         original theory)

Ya, you should be right...

>       * Your kernel/hypervisor is loading but is
crashing immediately.
>         If the Xen hypervisor is crashing you really
need the console.
>         AFAIK there's no option to log.  The only
option you really have
>         is whether or not to immediately reboot when it
crashes.  If Xen
>         is loading successfully then it loads your
dom0. If that's
>         crashing it's probably crashing immediately
(i.e. not even
>         mounting root). If it is the dom0 then that
seems to be the case
>         since you can't find any record of it having
booted.  If you
>         crash before you mount root read/write then
showconsole and
>         bootlog are useless.
> All things said, I'm still guessing that it's either a
grub problem or
> Xen doesn't like your hardware.  Xen is picky about
hardware and
> sometimes you have to turn on/off things in the BIOS or
as a parameter
> to Xen (like the ACPI controller) but it's going to be
hard to guess
> without an error message and I'm betting that error
message appears
> before bootlog/showconsole take effect.
>

Okay, thanks a lot, I am looking for such information since
I am new
to xen. I will google for people running xen solutions on
the same
kind of boxes.

>
> --
> Albert W. Hopkins
>
> --
> gentoo-usergentoo.org mailing list
>
>

Many thanks for your support guys ;o)

Gal'
-- 
gentoo-usergentoo.org mailing list

On Fri, 2007-07-06 at 00:44 +0200, Galevsky wrote:
> and my grub.conf:
> 
> ### START (grub.conf)
> sd-4421 boot # cat /boot/grub/grub.conf
> # Customized boot procedure
> 
> default 0
> timeout 1
> #fallback 1 2
> 
> title Gentoo Linux 2.6.16-gentoo_xen_dom0
>   root (hd0,0)
>   kernel /boot/vmlinuz-2.6-xen ro root=/dev/sda2
> 
> 
> title Gentoo Linux 2.6.20-r8
>   root (hd0,0)
>   kernel /boot/vmlinuz-2.6.20-gentoo-r8 ro
root=/dev/sda2
> 
> 
> title Gentoo Linux 2.6.18-r4-dedibox_r6_final
>   root (hd0,0)
>   kernel /boot/ref/2.6.18-gentoo-r4dedibox_r6_final ro
root=/dev/sda2
> ### END (grub.conf) 

Wow, it really does make a difference when we can see the
configuration!

Actually you are not using savedefault like I was assuming. 
Which
basically means fallback only works when grub fails to load
the kernel. 

You said you followed the "HOWTO Xen and Gentoo"
but looks like you
ignored section 6 on configuring the boot loader.  It should
look more
like this:

        title Gentoo Linux 2.6.16-gentoo_xen_dom0
            root (hd0,0)
            kernel /boot/xen.gz dom0_mem=98M
            module /boot/vmlinuz-2.6.20-gentoo-r8 ro
root=/dev/sda2

Likely the dom0 kernel is failing because it expects to be
run within
the hypervisor.  You need to load that first as in the
above.

Hope this helps.

--
Albert W. Hopkins

-- 
gentoo-usergentoo.org mailing list