Re: source line numbers and modules (on x86_64)

Hi,

I searched the archives and found that you've discussed an
issue I'm
seeing with x86_64 kernels where crash doesn't have line
numbers for
modules' symbols:
https://www.redhat.com/archives/crash-ut
ility/2008-January/msg00021.html

I'm using crash-4.0-6.3 on a RHEL5U1 x86_64 system with a
custom
2.6.22.19 kernel.  Given that the RHEL5U1 x86_64 kernels
clearly do
provide accurate line numbers for modules, has anyone
identified how
that is?  I have to believe the redhat kernel is patched to
fix this
issue.

I looked over the various redhat patches that are applied to
RHEL5's
2.6.18 sources but can't see a patch that stands out as
specifically
addressing this x86_64 issue.  But I could easily be
overlooking some
patch.

please advise, thanks.
Mike

ps. please cc me as I've not yet been able to join the list

--
Crash-utility mailing list
Crash-utilityredhat.com
https://www.redhat.com/mailman/listinfo/crash-utility

Mike Snitzer wrote:
> Hi,
>
> I searched the archives and found that you've discussed
an issue I'm
> seeing with x86_64 kernels where crash doesn't have
line numbers for
> modules' symbols:
> https://www.redhat.com/archives/crash-ut
ility/2008-January/msg00021.html
>
> I'm using crash-4.0-6.3 on a RHEL5U1 x86_64 system with
a custom
> 2.6.22.19 kernel.  Given that the RHEL5U1 x86_64
kernels clearly do
> provide accurate line numbers for modules, has anyone
identified how
> that is?  I have to believe the redhat kernel is
patched to fix this
> issue.
>
> I looked over the various redhat patches that are
applied to RHEL5's
> 2.6.18 sources but can't see a patch that stands out as
specifically
> addressing this x86_64 issue.  But I could easily be
overlooking some
> patch.
>
> please advise, thanks.
> Mike
>
> ps. please cc me as I've not yet been able to join the
list
>   
You're on the list now...

Anyway, there are no kernel patches applied to RHEL5
kernels
to make it work that I'm aware of.  That being said, I
don't
have an answer as to why they don't work for modules on
your
custom kernel (or why they don't work in RHEL4 kernels...).

Dave
 

--
Crash-utility mailing list
Crash-utilityredhat.com
https://www.redhat.com/mailman/listinfo/crash-utility

On Wed, Apr 30, 2008 at 6:43 PM, Dave Anderson
<andersonredhat.com> wrote:
>
> Mike Snitzer wrote:
>
> > Hi,
> >
> > I searched the archives and found that you've
discussed an issue I'm
> > seeing with x86_64 kernels where crash doesn't
have line numbers for
> > modules' symbols:
> > https://www.redhat.com/archives/crash-ut
ility/2008-January/msg00021.html
> >
> > I'm using crash-4.0-6.3 on a RHEL5U1 x86_64 system
with a custom
> > 2.6.22.19 kernel.  Given that the RHEL5U1 x86_64
kernels clearly do
> > provide accurate line numbers for modules, has
anyone identified how
> > that is?  I have to believe the redhat kernel is
patched to fix this
> > issue.
> >
> > I looked over the various redhat patches that are
applied to RHEL5's
> > 2.6.18 sources but can't see a patch that stands
out as specifically
> > addressing this x86_64 issue.  But I could easily
be overlooking some
> > patch.
> >
> > please advise, thanks.
> > Mike
> >
> > ps. please cc me as I've not yet been able to join
the list
> >
> >
>  You're on the list now...
>
>  Anyway, there are no kernel patches applied to RHEL5
kernels
>  to make it work that I'm aware of.  That being said, I
don't
>  have an answer as to why they don't work for modules
on your
>  custom kernel (or why they don't work in RHEL4
kernels...).

I configured my x86_64 kernel to be relocatable.  Could this
be the
reason for modules' line number debug data being off? 
From:
http://people.redhat.com/anderson/crash.changelog.
html#4_0_4_5

"Using /proc/kallsyms or a --reloc=[size] as a command
line argument
 is similar to using a System.map file, in that it results
in the loss
 of the use of line number debug data."

Still doesn't explain why RHEL4 isn't happy in this regard
either...

I'm trying a non-relocatable variant of my kernel now and
will report
back with the results.

Mike

--
Crash-utility mailing list
Crash-utilityredhat.com
https://www.redhat.com/mailman/listinfo/crash-utility

Mike Snitzer wrote:
> On Wed, Apr 30, 2008 at 6:43 PM, Dave Anderson
<andersonredhat.com> wrote:
>   
>> Mike Snitzer wrote:
>>
>>     
>>> Hi,
>>>
>>> I searched the archives and found that you've
discussed an issue I'm
>>> seeing with x86_64 kernels where crash doesn't
have line numbers for
>>> modules' symbols:
>>> https://www.redhat.com/archives/crash-ut
ility/2008-January/msg00021.html
>>>
>>> I'm using crash-4.0-6.3 on a RHEL5U1 x86_64
system with a custom
>>> 2.6.22.19 kernel.  Given that the RHEL5U1
x86_64 kernels clearly do
>>> provide accurate line numbers for modules, has
anyone identified how
>>> that is?  I have to believe the redhat kernel
is patched to fix this
>>> issue.
>>>
>>> I looked over the various redhat patches that
are applied to RHEL5's
>>> 2.6.18 sources but can't see a patch that
stands out as specifically
>>> addressing this x86_64 issue.  But I could
easily be overlooking some
>>> patch.
>>>
>>> please advise, thanks.
>>> Mike
>>>
>>> ps. please cc me as I've not yet been able to
join the list
>>>
>>>
>>>       
>>  You're on the list now...
>>
>>  Anyway, there are no kernel patches applied to
RHEL5 kernels
>>  to make it work that I'm aware of.  That being
said, I don't
>>  have an answer as to why they don't work for
modules on your
>>  custom kernel (or why they don't work in RHEL4
kernels...).
>>     
>
> I configured my x86_64 kernel to be relocatable.  Could
this be the
> reason for modules' line number debug data being off? 
From:
> http://people.redhat.com/anderson/crash.changelog.
html#4_0_4_5
>
> "Using /proc/kallsyms or a --reloc=[size] as a
command line argument
>  is similar to using a System.map file, in that it
results in the loss
>  of the use of line number debug data."
>   

No, that particular issue only applies to the base kernel,
and is 
irrelevant with respect to modules.
(Note that the RHEL5 kernel is relocatable as well.)
> Still doesn't explain why RHEL4 isn't happy in this
regard either...
>
> I'm trying a non-relocatable variant of my kernel now
and will report
> back with the results.
>   

What you might confirm is whether you're getting any
debuginfo data when 
you load
the module?  Are you sure that your modules are being built
with -g?

Dave

--
Crash-utility mailing list
Crash-utilityredhat.com
https://www.redhat.com/mailman/listinfo/crash-utility

On Wed, Apr 30, 2008 at 7:41 PM, Dave Anderson
<andersonredhat.com> wrote:
>
> Mike Snitzer wrote:
>
> > On Wed, Apr 30, 2008 at 6:43 PM, Dave Anderson
<andersonredhat.com>

> > >  Anyway, there are no kernel patches applied
to RHEL5 kernels
> > >  to make it work that I'm aware of.  That
being said, I don't
> > >  have an answer as to why they don't work for
modules on your
> > >  custom kernel (or why they don't work in
RHEL4 kernels...).
> > >
> > >
> >
> > I configured my x86_64 kernel to be relocatable. 
Could this be the
> > reason for modules' line number debug data being
off?  From:
> > http://people.redhat.com/anderson/crash.changelog.
html#4_0_4_5
> >
> > "Using /proc/kallsyms or a --reloc=[size] as
a command line argument
> >  is similar to using a System.map file, in that it
results in the loss
> >  of the use of line number debug data."
> >
> >
>
>  No, that particular issue only applies to the base
kernel, and is
> irrelevant with respect to modules.
>  (Note that the RHEL5 kernel is relocatable as well.)

OK, yeah making the base kernel non-relocatable didn't
help.

>  What you might confirm is whether you're getting any
debuginfo data when
> you load the module?  Are you sure that your modules
are being built with -g?

How would I confirm that I'm getting debuginfo data when I
'mod -s
ext3'?  I'm using an spec file that is based heavily (at
least
relative to debuginfo) on the redhat kernel-2.6.spec but
I'll have to
verify that I'm building with -g...

Mike

--
Crash-utility mailing list
Crash-utilityredhat.com
https://www.redhat.com/mailman/listinfo/crash-utility

Mike Snitzer wrote:
> On Wed, Apr 30, 2008 at 7:41 PM, Dave Anderson
<andersonredhat.com> wrote:
>   
>> Mike Snitzer wrote:
>>
>>     
>>> On Wed, Apr 30, 2008 at 6:43 PM, Dave Anderson
<andersonredhat.com>
>>>       
>
>   
>>>>  Anyway, there are no kernel patches
applied to RHEL5 kernels
>>>>  to make it work that I'm aware of.  That
being said, I don't
>>>>  have an answer as to why they don't work
for modules on your
>>>>  custom kernel (or why they don't work in
RHEL4 kernels...).
>>>>
>>>>
>>>>         
>>> I configured my x86_64 kernel to be
relocatable.  Could this be the
>>> reason for modules' line number debug data
being off?  From:
>>> http://people.redhat.com/anderson/crash.changelog.
html#4_0_4_5
>>>
>>> "Using /proc/kallsyms or a --reloc=[size]
as a command line argument
>>>  is similar to using a System.map file, in that
it results in the loss
>>>  of the use of line number debug data."
>>>
>>>
>>>       
>>  No, that particular issue only applies to the base
kernel, and is
>> irrelevant with respect to modules.
>>  (Note that the RHEL5 kernel is relocatable as
well.)
>>     
>
> OK, yeah making the base kernel non-relocatable didn't
help.
>
>   
>>  What you might confirm is whether you're getting
any debuginfo data when
>> you load the module?  Are you sure that your
modules are being built with -g?
>>     
>
> How would I confirm that I'm getting debuginfo data
when I 'mod -s
> ext3'?  I'm using an spec file that is based heavily
(at least
> relative to debuginfo) on the redhat kernel-2.6.spec
but I'll have to
> verify that I'm building with -g...
>
> Mike
>   

So your build procedure ends up with a kernel-debuginfo
package that 
contains
a set of "split" <module-name>.ko.debug
files?  If you take the module 
filename
displayed after the "mod -s" is done, and run it
through "readelf -a", 
you'll see
a .gnu_debuglink section.  That
/lib/modules/<path-to>/module.ko module 
will have
a link to its partner in 
/usr/lib/debug/lib/modules/<path-to>/module.ko.debug.

Dave

 

--
Crash-utility mailing list
Crash-utilityredhat.com
https://www.redhat.com/mailman/listinfo/crash-utility

On Wed, Apr 30, 2008 at 8:21 PM, Dave Anderson
<andersonredhat.com> wrote:
>
> Mike Snitzer wrote:
>
> > On Wed, Apr 30, 2008 at 7:41 PM, Dave Anderson
<andersonredhat.com>
> wrote:
> >
> >
> > > Mike Snitzer wrote:
> > >
> > >
> > >
> > > > On Wed, Apr 30, 2008 at 6:43 PM, Dave
Anderson <andersonredhat.com>
> > > >
> > > >
> > >
> >
> >
> >
> > >
> > > >
> > > > >  Anyway, there are no kernel
patches applied to RHEL5 kernels
> > > > >  to make it work that I'm aware of.
 That being said, I don't
> > > > >  have an answer as to why they
don't work for modules on your
> > > > >  custom kernel (or why they don't
work in RHEL4 kernels...).
> > > > >
> > > > >
> > > > >
> > > > >
> > > > I configured my x86_64 kernel to be
relocatable.  Could this be the
> > > > reason for modules' line number debug
data being off?  From:
> > > > http://people.redhat.com/anderson/crash.changelog.
html#4_0_4_5
> > > >
> > > > "Using /proc/kallsyms or a
--reloc=[size] as a command line argument
> > > >  is similar to using a System.map file,
in that it results in the loss
> > > >  of the use of line number debug
data."
> > > >
> > > >
> > > >
> > > >
> > >  No, that particular issue only applies to
the base kernel, and is
> > > irrelevant with respect to modules.
> > >  (Note that the RHEL5 kernel is relocatable
as well.)
> > >
> > >
> >
> > OK, yeah making the base kernel non-relocatable
didn't help.
> >
> >
> >
> > >  What you might confirm is whether you're
getting any debuginfo data
> when
> > > you load the module?  Are you sure that your
modules are being built
> with -g?
> > >
> > >
> >
> > How would I confirm that I'm getting debuginfo
data when I 'mod -s
> > ext3'?  I'm using an spec file that is based
heavily (at least
> > relative to debuginfo) on the redhat
kernel-2.6.spec but I'll have to
> > verify that I'm building with -g...
> >
> > Mike
> >
> >
>
>  So your build procedure ends up with a
kernel-debuginfo package that
> contains a set of "split"
<module-name>.ko.debug files?

Yes.

> If you take the module filename displayed after the
"mod -s" is done,
> and run it through "readelf -a", you'll see a
.gnu_debuglink section.

My kernel's ext3.ko has:
  [23] .gnu_debuglink    PROGBITS         0000000000000000 
00036558
       0000000000000014  0000000000000000           0     0 
   4

which would appear fairly comparable to what the RHEL5
kernel's ext3.ko has:
  [22] .gnu_debuglink    PROGBITS         0000000000000000 
00037418
       0000000000000014  0000000000000000           0     0 
   4

> That /lib/modules/<path-to>/module.ko module will
have
>  a link to its partner in
>
/usr/lib/debug/lib/modules/<path-to>/module.ko.debug.

readelf -x $SECTION /lib/modules/<path-to>/ext3.ko
shows that
.gnu_debuglink points to ext3.ko.debug as expected.

Given the "so far so good" nature of what you're
helping me verify
here it still begs the question: so why doesn't it work on
my
2.6.22.19 x86_64 kernel? ;)

Where does the line number debug data get stored in the
module.ko.debug?  In my case the source comes up as
"include/linux/clocksource.h: 121" for all kernel
modules' symbols.

Any further pointers would be greatly appreciated, thanks a
lot for
your time already!
Mike

--
Crash-utility mailing list
Crash-utilityredhat.com
https://www.redhat.com/mailman/listinfo/crash-utility

Mike Snitzer wrote:
> On Wed, Apr 30, 2008 at 8:21 PM, Dave Anderson
<andersonredhat.com> wrote:
>   
>> Mike Snitzer wrote:
>>
>>     
>>> On Wed, Apr 30, 2008 at 7:41 PM, Dave Anderson
<andersonredhat.com>
>>>       
>> wrote:
>>     
>>>       
>>>> Mike Snitzer wrote:
>>>>
>>>>
>>>>
>>>>         
>>>>> On Wed, Apr 30, 2008 at 6:43 PM, Dave
Anderson <andersonredhat.com>
>>>>>
>>>>>
>>>>>           
>>>
>>>       
>>>>>>  Anyway, there are no kernel
patches applied to RHEL5 kernels
>>>>>>  to make it work that I'm aware of.
 That being said, I don't
>>>>>>  have an answer as to why they
don't work for modules on your
>>>>>>  custom kernel (or why they don't
work in RHEL4 kernels...).
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>             
>>>>> I configured my x86_64 kernel to be
relocatable.  Could this be the
>>>>> reason for modules' line number debug
data being off?  From:
>>>>> http://people.redhat.com/anderson/crash.changelog.
html#4_0_4_5
>>>>>
>>>>> "Using /proc/kallsyms or a
--reloc=[size] as a command line argument
>>>>>  is similar to using a System.map file,
in that it results in the loss
>>>>>  of the use of line number debug
data."
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>           
>>>>  No, that particular issue only applies to
the base kernel, and is
>>>> irrelevant with respect to modules.
>>>>  (Note that the RHEL5 kernel is relocatable
as well.)
>>>>
>>>>
>>>>         
>>> OK, yeah making the base kernel non-relocatable
didn't help.
>>>
>>>
>>>
>>>       
>>>>  What you might confirm is whether you're
getting any debuginfo data
>>>>         
>> when
>>     
>>>> you load the module?  Are you sure that
your modules are being built
>>>>         
>> with -g?
>>     
>>>>         
>>> How would I confirm that I'm getting debuginfo
data when I 'mod -s
>>> ext3'?  I'm using an spec file that is based
heavily (at least
>>> relative to debuginfo) on the redhat
kernel-2.6.spec but I'll have to
>>> verify that I'm building with -g...
>>>
>>> Mike
>>>
>>>
>>>       
>>  So your build procedure ends up with a
kernel-debuginfo package that
>> contains a set of "split"
<module-name>.ko.debug files?
>>     
>
> Yes.
>
>   
>> If you take the module filename displayed after the
"mod -s" is done,
>> and run it through "readelf -a", you'll
see a .gnu_debuglink section.
>>     
>
> My kernel's ext3.ko has:
>   [23] .gnu_debuglink    PROGBITS        
0000000000000000  00036558
>        0000000000000014  0000000000000000           0  
  0     4
>
> which would appear fairly comparable to what the RHEL5
kernel's ext3.ko has:
>   [22] .gnu_debuglink    PROGBITS        
0000000000000000  00037418
>        0000000000000014  0000000000000000           0  
  0     4
>
>   
>> That /lib/modules/<path-to>/module.ko module
will have
>>  a link to its partner in
>>
/usr/lib/debug/lib/modules/<path-to>/module.ko.debug.
>>     
>
> readelf -x $SECTION
/lib/modules/<path-to>/ext3.ko shows that
> .gnu_debuglink points to ext3.ko.debug as expected.
>
> Given the "so far so good" nature of what
you're helping me verify
> here it still begs the question: so why doesn't it work
on my
> 2.6.22.19 x86_64 kernel? ;)
>
> Where does the line number debug data get stored in
the
> module.ko.debug?  In my case the source comes up as
> "include/linux/clocksource.h: 121" for all
kernel modules' symbols.
>
> Any further pointers would be greatly appreciated,
thanks a lot for
> your time already!
> Mike
>   

In the crash sources, gdb-6.1/gdb/symtab.c contains several
crash 
utility "hooks" into
the native gdb code.  The function called
gdb_get_line_number()  calls  
the pre-existing
gdb find_pc_line()  function to get the line number
information.  As to 
why it works
with some kernels and not with others is the question at
hand, and to which
I have no answer.  The behavior you see is similar to what
happens with
x86_64 in RHEL4 kernels, although on my RHEL4 x86_64
machine, it always
seems to return a line in include/linux/seqlock.h (?).  In
any case, the 
module
debuginfo data contains a linkage between text offset values
to the 
source code line
numbers.  In other words, do an "nm -Bn
<module>.ko" and you'll see that 
module
text locations are offsets, as opposed to compiled-in
addresses that you 
see if you
do an "nm -Bn vmlinux".  So my guess is that the
problem is related to 
the fact that
the module objects are relocatable, where the module object
has offset 
values applied
to each text offset, whereas the the PC values that get
passed to the 
gdb code
are the kernel vmalloc()'d addresses where the module got
loaded.  When the
"mod" command is done, it does a gdb
"add-symbol-file" command, which
essentially transforms of the PC offset values in the module
object into the
loaded vmalloc address.  But why the resultant vmalloc()
addresses work for
some kernels and not others is a mystery to me.

Dave





--
Crash-utility mailing list
Crash-utilityredhat.com
https://www.redhat.com/mailman/listinfo/crash-utility