LKM 'pf': environment compile options mismatch - LKM '', kernel 'DEBUG,DIAGNOSTIC'

> seems more like a snapshot kernel to me, still only the
provider can 
> put insight to this.

They told me that my kernel was same as the /netbsd kernel.
And they told 
me they made no changes.

I see that the default XENU config does have DEBUG and
DIAGNOSTIC. I am 
just not sure why the module does not.

How can I find out if the pf.o is actually from the
NetBSD/xen?

I can try to download a differnt pf.o to try. I haven't
looked yet. But 
is there an download site that have individual files like
that to 
download? Or what tarball can I extract it from for 3.0?

> [1] how come YOU are using pf? 

You probably got me confused with the other Reed. Depending
on the system, 
I use IPF, PF, IPFW, IP Tables...  By the way, I am still
working on 
getting a PF book published -- it is almost ready 

On Wed, Jul 19, 2006 at 10:06:56AM -0500, reedreedmedia.net wrote:
> > seems more like a snapshot kernel to me, still
only the provider can 
> > put insight to this.
> 
> They told me that my kernel was same as the /netbsd
kernel. And they told 
> me they made no changes.
> 
> I see that the default XENU config does have DEBUG and
DIAGNOSTIC. I am 
> just not sure why the module does not.
> 
> How can I find out if the pf.o is actually from the
NetBSD/xen?

Well, modules are always compiled without options in the
distrib.
As Xen kernels are compiled with debug options, these are
incompatible.
You'll have to rebuild  a LKM with these options, but I
don't know the
details.

-- 
Manuel Bouyer <bouyerantioche.eu.org>
     NetBSD: 26 ans d'experience feront toujours la
difference
--

On Wed, Jul 19, 2006 at 10:06:56AM -0500, reedreedmedia.net wrote:
> > seems more like a snapshot kernel to me, still
only the provider can 
> > put insight to this.
> 
> They told me that my kernel was same as the /netbsd
kernel. And they told 
> me they made no changes.
> 
> I see that the default XENU config does have DEBUG and
DIAGNOSTIC. I am 
> just not sure why the module does not.
> 
> How can I find out if the pf.o is actually from the
NetBSD/xen?

Well, modules are always compiled without options in the
distrib.
As Xen kernels are compiled with debug options, these are
incompatible.
You'll have to rebuild  a LKM with these options, but I
don't know the
details.

-- 
Manuel Bouyer <bouyerantioche.eu.org>
     NetBSD: 26 ans d'experience feront toujours la
difference
--

> Well, modules are always compiled without options in
the distrib.
> As Xen kernels are compiled with debug options, these
are incompatible.
> You'll have to rebuild  a LKM with these options, but
I don't know the
> details.

So I retrieved netbsd-3 via cvs. On my NetBSD/i386 3.99.21
build system, I 
did:

cd /usr/src/netbsd-3/src/sys/lkm/net/pf

modified the Makefile to create Makefile.reed with:

-CPPFLAGS+=     -I$S/dist/pf -I$S -DINET6 -DINET
+CPPFLAGS+=     -I$S/dist/pf -I$S -DINET6 -DINET
-DDIAGNOSTIC -DDEBUG

and the ran:

 make USETOOLS=NEVER -f Makefile.reed

I copied the resulting pf.o to the NetBSD 3.0 (XENU) system
and ran:

# /sbin/modload pf.o
Module loaded as ID 0
# modstat
Type    Id   Offset Loadaddr Size Info     Rev Module Name
DEV       0  -1/161 cb820000 0098 cb8413a0   2 pf

Moments later it panicked. (Not at modload time.)

Went to console and saw:

kernel: protection fault trap, code=0
Stopped at      0xcb8235bf:     cli
emul_freebsd_object(cb844610,c049cd90,1,0,64) at 0xcb8235bf
softclock(0,c049cdd0,c0335a03,3b9aca00,0) at
netbsd:softclock+0x2e3
softintr_dispatch(0,fffffffe,800,3,1) at
netbsd:softintr_dispatch+0xb3
DDB lost frame for netbsd:Xsoftclock+0x2e, trying 0xc049cdbc
Xsoftclock() at netbsd:Xsoftclock+0x2e
--- interrupt ---
?(31,11,11,f,2468f) at 0
ds          0x11
es          0x11
fs          0x31
gs          0x11
edi         0xcb844610
esi         0x1
ebp         0xc049cd54  emul_freebsd_object+0x7db40
?(31,11,11,f,2468f) at 0
ds          0x11
es          0x11
fs          0x31
gs          0x11
edi         0xcb844610
esi         0x1
ebp         0xc049cd54  emul_freebsd_object+0x7db40
ebx         0xffffc000
edx         0x246
ecx         0xffffffff
eax         0x786c2578
eip         0xcb8235bf
cs          0x9
eflags      0x10246
esp         0xc049cd4c  emul_freebsd_object+0x7db38
ss          0x11
0xcb8235bf:     cli
Stopped at      0xcb8235bf:     cli
db> 
db> reboot
syncing disks... done
unmounting file systems...
unmounting /kern (kernfs)...
unmounting / (/dev/xbd0a)...kernel: page fault trap, code=0
Stopped at      netbsd:buf_lotsfree+0xa:        testb  
$0x40,0x23(%eax)
db> 
rebooting...


So what is the correct way to build a pf.o?

Does anyone use kernel modules with Xen?

(I still can't figure out how it worked for me a couple
weeks ago.)

Thanks,

Jeremy C. Reed

> Well, modules are always compiled without options in
the distrib.
> As Xen kernels are compiled with debug options, these
are incompatible.
> You'll have to rebuild  a LKM with these options, but
I don't know the
> details.

So I retrieved netbsd-3 via cvs. On my NetBSD/i386 3.99.21
build system, I 
did:

cd /usr/src/netbsd-3/src/sys/lkm/net/pf

modified the Makefile to create Makefile.reed with:

-CPPFLAGS+=     -I$S/dist/pf -I$S -DINET6 -DINET
+CPPFLAGS+=     -I$S/dist/pf -I$S -DINET6 -DINET
-DDIAGNOSTIC -DDEBUG

and the ran:

 make USETOOLS=NEVER -f Makefile.reed

I copied the resulting pf.o to the NetBSD 3.0 (XENU) system
and ran:

# /sbin/modload pf.o
Module loaded as ID 0
# modstat
Type    Id   Offset Loadaddr Size Info     Rev Module Name
DEV       0  -1/161 cb820000 0098 cb8413a0   2 pf

Moments later it panicked. (Not at modload time.)

Went to console and saw:

kernel: protection fault trap, code=0
Stopped at      0xcb8235bf:     cli
emul_freebsd_object(cb844610,c049cd90,1,0,64) at 0xcb8235bf
softclock(0,c049cdd0,c0335a03,3b9aca00,0) at
netbsd:softclock+0x2e3
softintr_dispatch(0,fffffffe,800,3,1) at
netbsd:softintr_dispatch+0xb3
DDB lost frame for netbsd:Xsoftclock+0x2e, trying 0xc049cdbc
Xsoftclock() at netbsd:Xsoftclock+0x2e
--- interrupt ---
?(31,11,11,f,2468f) at 0
ds          0x11
es          0x11
fs          0x31
gs          0x11
edi         0xcb844610
esi         0x1
ebp         0xc049cd54  emul_freebsd_object+0x7db40
?(31,11,11,f,2468f) at 0
ds          0x11
es          0x11
fs          0x31
gs          0x11
edi         0xcb844610
esi         0x1
ebp         0xc049cd54  emul_freebsd_object+0x7db40
ebx         0xffffc000
edx         0x246
ecx         0xffffffff
eax         0x786c2578
eip         0xcb8235bf
cs          0x9
eflags      0x10246
esp         0xc049cd4c  emul_freebsd_object+0x7db38
ss          0x11
0xcb8235bf:     cli
Stopped at      0xcb8235bf:     cli
db> 
db> reboot
syncing disks... done
unmounting file systems...
unmounting /kern (kernfs)...
unmounting / (/dev/xbd0a)...kernel: page fault trap, code=0
Stopped at      netbsd:buf_lotsfree+0xa:        testb  
$0x40,0x23(%eax)
db> 
rebooting...


So what is the correct way to build a pf.o?

Does anyone use kernel modules with Xen?

(I still can't figure out how it worked for me a couple
weeks ago.)

Thanks,

Jeremy C. Reed

On Wed, Jul 19, 2006 at 02:59:37PM -0500, reedreedmedia.net wrote:
> > Well, modules are always compiled without options
in the distrib.
> > As Xen kernels are compiled with debug options,
these are incompatible.
> > You'll have to rebuild  a LKM with these options,
but I don't know the
> > details.
> 
> So I retrieved netbsd-3 via cvs. On my NetBSD/i386
3.99.21 build system, I 
> did:
> 
> cd /usr/src/netbsd-3/src/sys/lkm/net/pf
> 
> modified the Makefile to create Makefile.reed with:
> 
> -CPPFLAGS+=     -I$S/dist/pf -I$S -DINET6 -DINET
> +CPPFLAGS+=     -I$S/dist/pf -I$S -DINET6 -DINET
-DDIAGNOSTIC -DDEBUG
> 
> and the ran:
> 
>  make USETOOLS=NEVER -f Makefile.reed
> 
> I copied the resulting pf.o to the NetBSD 3.0 (XENU)
system and ran:
> 
> # /sbin/modload pf.o
> Module loaded as ID 0
> # modstat
> Type    Id   Offset Loadaddr Size Info     Rev Module
Name
> DEV       0  -1/161 cb820000 0098 cb8413a0   2 pf
> 
> Moments later it panicked. (Not at modload time.)
> 
> Went to console and saw:
> 
> kernel: protection fault trap, code=0
> Stopped at      0xcb8235bf:     cli
> emul_freebsd_object(cb844610,c049cd90,1,0,64) at
0xcb8235bf
> softclock(0,c049cdd0,c0335a03,3b9aca00,0) at
netbsd:softclock+0x2e3
> softintr_dispatch(0,fffffffe,800,3,1) at
netbsd:softintr_dispatch+0xb3
> DDB lost frame for netbsd:Xsoftclock+0x2e, trying
0xc049cdbc
> Xsoftclock() at netbsd:Xsoftclock+0x2e
> --- interrupt ---
> ?(31,11,11,f,2468f) at 0
> ds          0x11
> es          0x11
> fs          0x31
> gs          0x11
> edi         0xcb844610
> esi         0x1
> ebp         0xc049cd54  emul_freebsd_object+0x7db40
> ?(31,11,11,f,2468f) at 0
> ds          0x11
> es          0x11
> fs          0x31
> gs          0x11
> edi         0xcb844610
> esi         0x1
> ebp         0xc049cd54  emul_freebsd_object+0x7db40
> ebx         0xffffc000
> edx         0x246
> ecx         0xffffffff
> eax         0x786c2578
> eip         0xcb8235bf
> cs          0x9
> eflags      0x10246
> esp         0xc049cd4c  emul_freebsd_object+0x7db38
> ss          0x11
> 0xcb8235bf:     cli
> Stopped at      0xcb8235bf:     cli
> db> 
> db> reboot
> syncing disks... done
> unmounting file systems...
> unmounting /kern (kernfs)...
> unmounting / (/dev/xbd0a)...kernel: page fault trap,
code=0
> Stopped at      netbsd:buf_lotsfree+0xa:        testb  
$0x40,0x23(%eax)
> db> 
> rebooting...
> 
> 
> So what is the correct way to build a pf.o?
> 
> Does anyone use kernel modules with Xen?

I don't think so. In fact, I think there's too much
differences between
i386 and xen kernel headers to be able to share LKMs between
i386 and Xen.

The lkms need to be built with includes from arch/xen. But I
don't know
what needs to be changed in the Makefile for that. Maybe you
need to
change MACHINE and/or MACHINE_ARCH make variables

-- 
Manuel Bouyer <bouyerantioche.eu.org>
     NetBSD: 26 ans d'experience feront toujours la
difference
--

On Wed, Jul 19, 2006 at 02:59:37PM -0500, reedreedmedia.net wrote:
> > Well, modules are always compiled without options
in the distrib.
> > As Xen kernels are compiled with debug options,
these are incompatible.
> > You'll have to rebuild  a LKM with these options,
but I don't know the
> > details.
> 
> So I retrieved netbsd-3 via cvs. On my NetBSD/i386
3.99.21 build system, I 
> did:
> 
> cd /usr/src/netbsd-3/src/sys/lkm/net/pf
> 
> modified the Makefile to create Makefile.reed with:
> 
> -CPPFLAGS+=     -I$S/dist/pf -I$S -DINET6 -DINET
> +CPPFLAGS+=     -I$S/dist/pf -I$S -DINET6 -DINET
-DDIAGNOSTIC -DDEBUG
> 
> and the ran:
> 
>  make USETOOLS=NEVER -f Makefile.reed
> 
> I copied the resulting pf.o to the NetBSD 3.0 (XENU)
system and ran:
> 
> # /sbin/modload pf.o
> Module loaded as ID 0
> # modstat
> Type    Id   Offset Loadaddr Size Info     Rev Module
Name
> DEV       0  -1/161 cb820000 0098 cb8413a0   2 pf
> 
> Moments later it panicked. (Not at modload time.)
> 
> Went to console and saw:
> 
> kernel: protection fault trap, code=0
> Stopped at      0xcb8235bf:     cli
> emul_freebsd_object(cb844610,c049cd90,1,0,64) at
0xcb8235bf
> softclock(0,c049cdd0,c0335a03,3b9aca00,0) at
netbsd:softclock+0x2e3
> softintr_dispatch(0,fffffffe,800,3,1) at
netbsd:softintr_dispatch+0xb3
> DDB lost frame for netbsd:Xsoftclock+0x2e, trying
0xc049cdbc
> Xsoftclock() at netbsd:Xsoftclock+0x2e
> --- interrupt ---
> ?(31,11,11,f,2468f) at 0
> ds          0x11
> es          0x11
> fs          0x31
> gs          0x11
> edi         0xcb844610
> esi         0x1
> ebp         0xc049cd54  emul_freebsd_object+0x7db40
> ?(31,11,11,f,2468f) at 0
> ds          0x11
> es          0x11
> fs          0x31
> gs          0x11
> edi         0xcb844610
> esi         0x1
> ebp         0xc049cd54  emul_freebsd_object+0x7db40
> ebx         0xffffc000
> edx         0x246
> ecx         0xffffffff
> eax         0x786c2578
> eip         0xcb8235bf
> cs          0x9
> eflags      0x10246
> esp         0xc049cd4c  emul_freebsd_object+0x7db38
> ss          0x11
> 0xcb8235bf:     cli
> Stopped at      0xcb8235bf:     cli
> db> 
> db> reboot
> syncing disks... done
> unmounting file systems...
> unmounting /kern (kernfs)...
> unmounting / (/dev/xbd0a)...kernel: page fault trap,
code=0
> Stopped at      netbsd:buf_lotsfree+0xa:        testb  
$0x40,0x23(%eax)
> db> 
> rebooting...
> 
> 
> So what is the correct way to build a pf.o?
> 
> Does anyone use kernel modules with Xen?

I don't think so. In fact, I think there's too much
differences between
i386 and xen kernel headers to be able to share LKMs between
i386 and Xen.

The lkms need to be built with includes from arch/xen. But I
don't know
what needs to be changed in the Makefile for that. Maybe you
need to
change MACHINE and/or MACHINE_ARCH make variables

-- 
Manuel Bouyer <bouyerantioche.eu.org>
     NetBSD: 26 ans d'experience feront toujours la
difference
--

On Wed, Jul 19, 2006 at 11:00:38PM +0200, Manuel Bouyer
wrote:
> On Wed, Jul 19, 2006 at 02:59:37PM -0500, reedreedmedia.net wrote:
> > > Well, modules are always compiled without
options in the distrib.
> > > As Xen kernels are compiled with debug
options, these are incompatible.
> > > You'll have to rebuild  a LKM with these
options, but I don't know the
> > > details.
> > 
> > So I retrieved netbsd-3 via cvs. On my NetBSD/i386
3.99.21 build system, I 
~~~ snip ~~~
> > 
> > So what is the correct way to build a pf.o?

pf.o is somehow inherently incompatible with a XENU kernel,
with or
without DEBUG,DIAGNOSTIC.  I have a very similar panic when
I load
a standard pf.o into the kernel I use (see below).

> > 
> > Does anyone use kernel modules with Xen?

Yes. NetBSD 3.0/Xen 2.0. Both in Dom0 and DomU.  In this
case the module
is nnpfs_mod.o from the Arla (v. 0.41 and 0.43) AFS client. 
I usually
will use a custom xen kernel without DEBUG,DIAGNOSTIC so
that the module
will work with both a i386 and xen kernel.

The only trouble I've had was when I upgraded to a netbsd-3
kernel and
arla 0.43, after rebuilding nnpfs_mod.o against the new
sources
my AFS access will go out to lunch after a few hours, but no
panics.
Finding the actual problem has not made it's way to the top
of the
to do list yet.

> 
> I don't think so. In fact, I think there's too much
differences between
> i386 and xen kernel headers to be able to share LKMs
between i386 and Xen.
 
I really hope that this isn't actually the case.  Having
AFS access is
essential.

> The lkms need to be built with includes from arch/xen.
But I don't know
> what needs to be changed in the Makefile for that.
Maybe you need to
> change MACHINE and/or MACHINE_ARCH make variables

I built nnpfs_mod.o no differently than I would on any other
architecture.
Of course, this module is basically a character device and a
file system,
not a module that becomes part of the networking subsystem.

	Jonathan Kollasch

On Wed, Jul 19, 2006 at 11:00:38PM +0200, Manuel Bouyer
wrote:
> On Wed, Jul 19, 2006 at 02:59:37PM -0500, reedreedmedia.net wrote:
> > > Well, modules are always compiled without
options in the distrib.
> > > As Xen kernels are compiled with debug
options, these are incompatible.
> > > You'll have to rebuild  a LKM with these
options, but I don't know the
> > > details.
> > 
> > So I retrieved netbsd-3 via cvs. On my NetBSD/i386
3.99.21 build system, I 
~~~ snip ~~~
> > 
> > So what is the correct way to build a pf.o?

pf.o is somehow inherently incompatible with a XENU kernel,
with or
without DEBUG,DIAGNOSTIC.  I have a very similar panic when
I load
a standard pf.o into the kernel I use (see below).

> > 
> > Does anyone use kernel modules with Xen?

Yes. NetBSD 3.0/Xen 2.0. Both in Dom0 and DomU.  In this
case the module
is nnpfs_mod.o from the Arla (v. 0.41 and 0.43) AFS client. 
I usually
will use a custom xen kernel without DEBUG,DIAGNOSTIC so
that the module
will work with both a i386 and xen kernel.

The only trouble I've had was when I upgraded to a netbsd-3
kernel and
arla 0.43, after rebuilding nnpfs_mod.o against the new
sources
my AFS access will go out to lunch after a few hours, but no
panics.
Finding the actual problem has not made it's way to the top
of the
to do list yet.

> 
> I don't think so. In fact, I think there's too much
differences between
> i386 and xen kernel headers to be able to share LKMs
between i386 and Xen.
 
I really hope that this isn't actually the case.  Having
AFS access is
essential.

> The lkms need to be built with includes from arch/xen.
But I don't know
> what needs to be changed in the Makefile for that.
Maybe you need to
> change MACHINE and/or MACHINE_ARCH make variables

I built nnpfs_mod.o no differently than I would on any other
architecture.
Of course, this module is basically a character device and a
file system,
not a module that becomes part of the networking subsystem.

	Jonathan Kollasch

> pf.o is somehow inherently incompatible with a XENU
kernel, with or
> without DEBUG,DIAGNOSTIC.  I have a very similar panic
when I load
> a standard pf.o into the kernel I use (see below).
> > > 
> > > Does anyone use kernel modules with Xen?
> 
> Yes. NetBSD 3.0/Xen 2.0. Both in Dom0 and DomU.  In
this case the module
> is nnpfs_mod.o from the Arla (v. 0.41 and 0.43) AFS
client.  I usually
> will use a custom xen kernel without DEBUG,DIAGNOSTIC
so that the module
> will work with both a i386 and xen kernel.

The strange thing for me is that I was using pf.o with Xen
for two weeks. 
It was definitely working as I was using it with spamd (and
my logs showed 
spamd's usage).

I am using Xen through a hosting provider and they made some
changes last 
week that made me lose service for approximately 15 hours.
(They moved my 
service to a new server I think and I received new console
IP.)

I never recorded what kernel I used before (and my logs with
the details 
has already rotated). The Xen hosting provider told me that
they made no 
changes to my kernel or modules. But the two times I
attempted to use pf.o 
(modload -f of pf.o without the options and then using a
pf.o with DEBUG 
and DIAGNOSTIC) I had kernel panics.