|
List Info
Thread: enhance bt command
|
|
| enhance bt command |
 
United States |
2008-03-01 16:38:37 |
Hi All
When use bt -f to show stack data, I need a quick way to
find out what
are these stack data. For example, does any of these data
are a inode
pointer, or ... So here is always what i do.
bt -f > stack
kmem -S inode_cache > inode
then use sort and comm utility to find value that appear in
both files.
Is there a better way to do this?
I wish we can have a bt -f slab1 slab2...
and try to match the stack data with content from these slab
cache
automatically.
Thanks!
--
Ming Zhang
 #$%^
purging memory... (*!%
http://blackmag
ic02881.wordpress.com/
http://www
.linkedin.com/in/blackmagic02881
--------------------------------------------
--
Crash-utility mailing list
Crash-utilityredhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
|
|
| Re: JWW SPAM-bad-from enhance bt
command |
United States |
2008-03-01 19:15:05 |
I think you'll have better luck consulting the source code,
and
disassembly to figure out what is on the stack.
For the example you've cited, if a given function uses an
inode pointer,
it shouldn't take but a minute or so to determine where in
the stack
frame this inode is located. (unless of course it turns out
to be in a
register only, in which case you have to look for it to be
spilled in a
subsequent frame)
- jim
On Sat, 2008-03-01 at 17:38 -0500, Ming Zhang wrote:
> Hi All
>
> When use bt -f to show stack data, I need a quick way
to find out what
> are these stack data. For example, does any of these
data are a inode
> pointer, or ... So here is always what i do.
>
> bt -f > stack
> kmem -S inode_cache > inode
>
> then use sort and comm utility to find value that
appear in both files.
>
> Is there a better way to do this?
>
> I wish we can have a bt -f slab1 slab2...
>
> and try to match the stack data with content from these
slab cache
> automatically.
>
> Thanks!
>
>
--
Crash-utility mailing list
Crash-utilityredhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
|
|
| Re: JWW SPAM-bad-from enhance bt
command |
United States |
2008-03-01 19:26:08 |
On Sat, 2008-03-01 at 17:15 -0800, James Washer wrote:
> I think you'll have better luck consulting the source
code, and
> disassembly to figure out what is on the stack.
checking asm code and finding out what is on the stack is
sure the most
accurate way. but a hint could speed up the process.
>
> For the example you've cited, if a given function uses
an inode pointer,
> it shouldn't take but a minute or so to determine where
in the stack
> frame this inode is located. (unless of course it turns
out to be in a
> register only, in which case you have to look for it to
be spilled in a
> subsequent frame)
quite some time you can see from the asm code that a
important parameter
is stored in register and you have to go several level
deeper before
finding it. so a hint which quickly locate a potential 'hot'
data and
later revalidate with code reading is still very appealing.
>
> - jim
>
> On Sat, 2008-03-01 at 17:38 -0500, Ming Zhang wrote:
> > Hi All
> >
> > When use bt -f to show stack data, I need a quick
way to find out what
> > are these stack data. For example, does any of
these data are a inode
> > pointer, or ... So here is always what i do.
> >
> > bt -f > stack
> > kmem -S inode_cache > inode
> >
> > then use sort and comm utility to find value that
appear in both files.
> >
> > Is there a better way to do this?
> >
> > I wish we can have a bt -f slab1 slab2...
> >
> > and try to match the stack data with content from
these slab cache
> > automatically.
> >
> > Thanks!
> >
> >
>
--
Ming Zhang
#$%^
purging memory... (*!%
http://blackmag
ic02881.wordpress.com/
http://www
.linkedin.com/in/blackmagic02881
--------------------------------------------
--
Crash-utility mailing list
Crash-utilityredhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
|
|
| Re: JWW SPAM-bad-from enhance bt
command |
United States |
2008-03-03 08:37:25 |
On Mon, 2008-03-03 at 09:19 -0500, Dave Anderson wrote:
> Ming Zhang wrote:
> > On Sat, 2008-03-01 at 17:15 -0800, James Washer
wrote:
> >> I think you'll have better luck consulting the
source code, and
> >> disassembly to figure out what is on the
stack.
> >
> > checking asm code and finding out what is on the
stack is sure the most
> > accurate way. but a hint could speed up the
process.
> >
> >
> >> For the example you've cited, if a given
function uses an inode pointer,
> >> it shouldn't take but a minute or so to
determine where in the stack
> >> frame this inode is located. (unless of course
it turns out to be in a
> >> register only, in which case you have to look
for it to be spilled in a
> >> subsequent frame)
> >
> > quite some time you can see from the asm code that
a important parameter
> > is stored in register and you have to go several
level deeper before
> > finding it. so a hint which quickly locate a
potential 'hot' data and
> > later revalidate with code reading is still very
appealing.
>
> I'm not sure whether cluttering the bt -f command
output would
> be all that worthwhile, but I'm thinking that this
request may have
> some merit with respect to the "rd" command.
Currently the
> "rd -s" option recognizes and translates
kernel symbols that it
> finds in the raw memory output. Maybe something like a
"-S"
> option could do that -- plus also recognize kernel
virtual memory
> addresses that come from the slab cache, and
alternatively display
> the slab cache namestring in some recognizable manner,
i.e. bracketed,
> or something like to that effect.
>
> Then since "bt -f" displays the block of
memory associated with
> each stack frame, you could easily transfer the stack
address
> to a "rd -S [stack-address] [count]"
command.
yes. it will be great if we can have a rd -S!
>
> Dave
>
> >
> >
> >
> >> - jim
> >>
> >> On Sat, 2008-03-01 at 17:38 -0500, Ming Zhang
wrote:
> >>> Hi All
> >>>
> >>> When use bt -f to show stack data, I need
a quick way to find out what
> >>> are these stack data. For example, does
any of these data are a inode
> >>> pointer, or ... So here is always what i
do.
> >>>
> >>> bt -f > stack
> >>> kmem -S inode_cache > inode
> >>>
> >>> then use sort and comm utility to find
value that appear in both files.
> >>>
> >>> Is there a better way to do this?
> >>>
> >>> I wish we can have a bt -f slab1 slab2...
> >>>
> >>> and try to match the stack data with
content from these slab cache
> >>> automatically.
> >>>
> >>> Thanks!
> >>>
> >>>
>
--
Ming Zhang
#$%^
purging memory... (*!%
http://blackmag
ic02881.wordpress.com/
http://www
.linkedin.com/in/blackmagic02881
--------------------------------------------
--
Crash-utility mailing list
Crash-utilityredhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
|
|
| Re: JWW SPAM-bad-from enhance bt
command |
|
2008-03-03 10:19:40 |
Ming Zhang wrote:
> On Mon, 2008-03-03 at 09:19 -0500, Dave Anderson
wrote:
>> I'm not sure whether cluttering the bt -f command
output would
>> be all that worthwhile, but I'm thinking that this
request may have
>> some merit with respect to the "rd"
command. Currently the
>> "rd -s" option recognizes and translates
kernel symbols that it
>> finds in the raw memory output. Maybe something
like a "-S"
>> option could do that -- plus also recognize kernel
virtual memory
>> addresses that come from the slab cache, and
alternatively display
>> the slab cache namestring in some recognizable
manner, i.e. bracketed,
>> or something like to that effect.
>>
>> Then since "bt -f" displays the block of
memory associated with
>> each stack frame, you could easily transfer the
stack address
>> to a "rd -S [stack-address] [count]"
command.
>
> yes. it will be great if we can have a rd -S!
>
Apply the attached patch and see what you think. The
"rd -S"
option supplements "-s" by also recognizing memory
from slab
caches -- and alternatively displaying the slab cache name
in brackets.
Thanks,
Dave
--
Crash-utility mailing list
Crash-utilityredhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
|
|
 |
| Re: JWW SPAM-bad-from enhance bt
command |
United States |
2008-03-03 11:30:14 |
On Mon, 2008-03-03 at 11:19 -0500, Dave Anderson wrote:
> Ming Zhang wrote:
> > On Mon, 2008-03-03 at 09:19 -0500, Dave Anderson
wrote:
> >> I'm not sure whether cluttering the bt -f
command output would
> >> be all that worthwhile, but I'm thinking that
this request may have
> >> some merit with respect to the "rd"
command. Currently the
> >> "rd -s" option recognizes and
translates kernel symbols that it
> >> finds in the raw memory output. Maybe
something like a "-S"
> >> option could do that -- plus also recognize
kernel virtual memory
> >> addresses that come from the slab cache, and
alternatively display
> >> the slab cache namestring in some recognizable
manner, i.e. bracketed,
> >> or something like to that effect.
> >>
> >> Then since "bt -f" displays the
block of memory associated with
> >> each stack frame, you could easily transfer
the stack address
> >> to a "rd -S [stack-address] [count]"
command.
> >
> > yes. it will be great if we can have a rd -S!
> >
>
> Apply the attached patch and see what you think. The
"rd -S"
> option supplements "-s" by also recognizing
memory from slab
> caches -- and alternatively displaying the slab cache
name
> in brackets.
>
this is what it looks like
10078b2ba30: vprintk+498 .LC391+195657
10078b2ba40: 0000000000000471 [size-32]
10078b2ba50: 000000000000016e 0000000000000001
10078b2ba60: 0000000000000005 [kmem_cache]
10078b2ba70: 000000000000003c [size-1024]
10078b2ba80: [kmem_cache] 0000000000000000
10078b2ba90: 0000000000000001 00000000801108a1
[raw value: cache name] will be better
strangely, not all the value is correct. i am looking at
it.
> Thanks,
> Dave
>
--
Ming Zhang
#$%^
purging memory... (*!%
http://blackmag
ic02881.wordpress.com/
http://www
.linkedin.com/in/blackmagic02881
--------------------------------------------
--
Crash-utility mailing list
Crash-utilityredhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
|
|
| Re: enhance bt command |
|
2008-03-03 11:51:04 |
Ming Zhang wrote:
> On Mon, 2008-03-03 at 11:19 -0500, Dave Anderson
wrote:
>> Ming Zhang wrote:
>>> On Mon, 2008-03-03 at 09:19 -0500, Dave
Anderson wrote:
>>>> I'm not sure whether cluttering the bt -f
command output would
>>>> be all that worthwhile, but I'm thinking
that this request may have
>>>> some merit with respect to the
"rd" command. Currently the
>>>> "rd -s" option recognizes and
translates kernel symbols that it
>>>> finds in the raw memory output. Maybe
something like a "-S"
>>>> option could do that -- plus also recognize
kernel virtual memory
>>>> addresses that come from the slab cache,
and alternatively display
>>>> the slab cache namestring in some
recognizable manner, i.e. bracketed,
>>>> or something like to that effect.
>>>>
>>>> Then since "bt -f" displays the
block of memory associated with
>>>> each stack frame, you could easily transfer
the stack address
>>>> to a "rd -S [stack-address]
[count]" command.
>>> yes. it will be great if we can have a rd -S!
>>>
>> Apply the attached patch and see what you think.
The "rd -S"
>> option supplements "-s" by also
recognizing memory from slab
>> caches -- and alternatively displaying the slab
cache name
>> in brackets.
>>
>
> this is what it looks like
>
> 10078b2ba30: vprintk+498 .LC391+195657
> 10078b2ba40: 0000000000000471 [size-32]
> 10078b2ba50: 000000000000016e 0000000000000001
> 10078b2ba60: 0000000000000005 [kmem_cache]
> 10078b2ba70: 000000000000003c [size-1024]
> 10078b2ba80: [kmem_cache] 0000000000000000
> 10078b2ba90: 0000000000000001 00000000801108a1
>
>
> [raw value: cache name] will be better
That would take up too much space, especially on a 32-bit
machine's display, which shows 4 addresses per line.
>
> strangely, not all the value is correct. i am looking
at it.
I'm not sure what you mean by them not being correct, but
if you run "kmem -s [address]", it should show the
samec
slab cache container (since it uses the same facility).
For example:
crash> rd d799df80 4
d799df80: 00000004 c0470395 d799dfa4 ca9ec8c0
......G.........
crash> rd -S d799df80 4
d799df80: 00000004 vfs_read+159 [size-4096] [filp]
crash> kmem -s d799dfa4
CACHE NAME OBJSIZE ALLOCATED TOTAL
SLABS SSIZE
dff74e80 size-4096 4096 215 233
233 4k
SLAB MEMORY TOTAL ALLOCATED FREE
c64f9e20 d799d000 1 1 0
FREE / [ALLOCATED]
[d799dfa4]
crash> kmem -s ca9ec8c0
CACHE NAME OBJSIZE ALLOCATED TOTAL
SLABS SSIZE
dfc7a280 filp 192 4271 4580
229 4k
SLAB MEMORY TOTAL ALLOCATED FREE
ca9ec000 ca9ec080 20 9 11
FREE / [ALLOCATED]
[ca9ec8c0]
crash>
Dave
--
Crash-utility mailing list
Crash-utilityredhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
|
|
| Re: enhance bt command |
United States |
2008-03-03 12:15:32 |
On Mon, 2008-03-03 at 12:51 -0500, Dave Anderson wrote:
> Ming Zhang wrote:
> > On Mon, 2008-03-03 at 11:19 -0500, Dave Anderson
wrote:
> >> Ming Zhang wrote:
> >>> On Mon, 2008-03-03 at 09:19 -0500, Dave
Anderson wrote:
> >>>> I'm not sure whether cluttering the bt
-f command output would
> >>>> be all that worthwhile, but I'm
thinking that this request may have
> >>>> some merit with respect to the
"rd" command. Currently the
> >>>> "rd -s" option recognizes
and translates kernel symbols that it
> >>>> finds in the raw memory output. Maybe
something like a "-S"
> >>>> option could do that -- plus also
recognize kernel virtual memory
> >>>> addresses that come from the slab
cache, and alternatively display
> >>>> the slab cache namestring in some
recognizable manner, i.e. bracketed,
> >>>> or something like to that effect.
> >>>>
> >>>> Then since "bt -f" displays
the block of memory associated with
> >>>> each stack frame, you could easily
transfer the stack address
> >>>> to a "rd -S [stack-address]
[count]" command.
> >>> yes. it will be great if we can have a rd
-S!
> >>>
> >> Apply the attached patch and see what you
think. The "rd -S"
> >> option supplements "-s" by also
recognizing memory from slab
> >> caches -- and alternatively displaying the
slab cache name
> >> in brackets.
> >>
> >
> > this is what it looks like
> >
> > 10078b2ba30: vprintk+498 .LC391+195657
> > 10078b2ba40: 0000000000000471 [size-32]
> > 10078b2ba50: 000000000000016e
0000000000000001
> > 10078b2ba60: 0000000000000005 [kmem_cache]
> > 10078b2ba70: 000000000000003c [size-1024]
> > 10078b2ba80: [kmem_cache]
0000000000000000
> > 10078b2ba90: 0000000000000001
00000000801108a1
> >
> >
> > [raw value: cache name] will be better
>
> That would take up too much space, especially on a
32-bit
> machine's display, which shows 4 addresses per line.
>
> >
> > strangely, not all the value is correct. i am
looking at it.
>
> I'm not sure what you mean by them not being correct,
but
> if you run "kmem -s [address]", it should
show the samec
> slab cache container (since it uses the same
facility).
> For example:
my English sucks...
this is what i got. my side size-512 or size-1024 also
works. just stuff
like dentry cache or inode cache does not work.
crash> rd 10078b2bca0 2
10078b2bca0: 000001007877c8d4 00000100710fd5c8
..wx.......q....
crash> rd -S 10078b2bca0 2
10078b2bca0: [dentry_cache] 00000100710fd5c8
crash> kmem -s 10078b2bca0
kmem: address is not allocated in slab subsystem:
10078b2bca0
>
> crash> rd d799df80 4
> d799df80: 00000004 c0470395 d799dfa4 ca9ec8c0
......G.........
> crash> rd -S d799df80 4
> d799df80: 00000004 vfs_read+159 [size-4096] [filp]
> crash> kmem -s d799dfa4
> CACHE NAME OBJSIZE ALLOCATED
TOTAL SLABS SSIZE
> dff74e80 size-4096 4096 215
233 233 4k
> SLAB MEMORY TOTAL ALLOCATED FREE
> c64f9e20 d799d000 1 1 0
> FREE / [ALLOCATED]
> [d799dfa4]
> crash> kmem -s ca9ec8c0
> CACHE NAME OBJSIZE ALLOCATED
TOTAL SLABS SSIZE
> dfc7a280 filp 192 4271
4580 229 4k
> SLAB MEMORY TOTAL ALLOCATED FREE
> ca9ec000 ca9ec080 20 9 11
> FREE / [ALLOCATED]
> [ca9ec8c0]
> crash>
>
> Dave
>
>
--
Ming Zhang
#$%^
purging memory... (*!%
http://blackmag
ic02881.wordpress.com/
http://www
.linkedin.com/in/blackmagic02881
--------------------------------------------
--
Crash-utility mailing list
Crash-utilityredhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
|
|
| Re: enhance bt command |
|
2008-03-03 13:04:59 |
Ming Zhang wrote:
> my English sucks...
>
> this is what i got. my side size-512 or size-1024 also
works. just stuff
> like dentry cache or inode cache does not work.
>
>
> crash> rd 10078b2bca0 2
> 10078b2bca0: 000001007877c8d4 00000100710fd5c8
..wx.......q....
> crash> rd -S 10078b2bca0 2
> 10078b2bca0: [dentry_cache] 00000100710fd5c8
> crash> kmem -s 10078b2bca0
> kmem: address is not allocated in slab subsystem:
10078b2bca0
No, your English is fine -- it's your command-entering that
sucks!
Above, when you entered "kmem -s 10078b2bca0",
you're incorrectly entering
the *address* where the dentry_cache reference (1007877c8d4)
is located.
And so it's telling you that 10078b2bca0 is not allocated in
the slab
subsystem, which it isn't...
But if you entered "kmem -s 1007877c8d4", you'd
see the dentry_cache
information.
Dave
--
Crash-utility mailing list
Crash-utilityredhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
|
|
| Re: enhance bt command |
United States |
2008-03-03 13:35:03 |
On Mon, 2008-03-03 at 14:04 -0500, Dave Anderson wrote:
> Ming Zhang wrote:
>
> > my English sucks...
> >
> > this is what i got. my side size-512 or size-1024
also works. just stuff
> > like dentry cache or inode cache does not work.
> >
> >
> > crash> rd 10078b2bca0 2
> > 10078b2bca0: 000001007877c8d4
00000100710fd5c8 ..wx.......q....
> > crash> rd -S 10078b2bca0 2
> > 10078b2bca0: [dentry_cache]
00000100710fd5c8
> > crash> kmem -s 10078b2bca0
> > kmem: address is not allocated in slab subsystem:
10078b2bca0
>
> No, your English is fine -- it's your command-entering
that sucks!
>
> Above, when you entered "kmem -s
10078b2bca0", you're incorrectly entering
> the *address* where the dentry_cache reference
(1007877c8d4) is located.
> And so it's telling you that 10078b2bca0 is not
allocated in the slab
> subsystem, which it isn't...
>
> But if you entered "kmem -s 1007877c8d4",
you'd see the dentry_cache
> information.
yes. i noticed that. need some tea when handling 3 bugs at
the same
time...
so here is my understanding. it actually shows the slab
object that
contain that address. it might not be the address of that
object. so
here is what i need to do
crash> rd -x 10078b2bca0
10078b2bca0: 000001007877c8d4
crash> rd -S 10078b2bca0
10078b2bca0: [dentry_cache]
crash> kmem -s 000001007877c8d4
CACHE NAME OBJSIZE ALLOCATED
TOTAL
SLABS SSIZE
10037ffc080 dentry_cache 240 9429
10560
660 4k
SLAB MEMORY TOTAL ALLOCATED FREE
1007877c040 1007877c088 16 9 7
FREE / [ALLOCATED]
[1007877c8d4]
this only tell me that it belongs to one dentry object, but
no idea
which one
then i have to use kmem -S dentry_cache, find out this
piece
SLAB MEMORY TOTAL ALLOCATED FREE
1007877c040 1007877c088 16 9 7
FREE / [ALLOCATED]
[1007877c088]
[1007877c178]
1007877c268 (cpu 1 cache)
1007877c358 (cpu 1 cache)
[1007877c448]
1007877c538 (cpu 1 cache)
[1007877c628]
[1007877c718]
[1007877c808]
~~~~~~~~~~~~~~~~~~~~ then find out it is here.
[1007877c8f8]
1007877c9e8 (cpu 1 cache)
1007877cad8 (cpu 1 cache)
[1007877cbc8]
[1007877ccb8]
1007877cda8 (cpu 1 cache)
1007877ce98 (cpu 1 cache)
then i know the object contain this address is 1007877c808.
then
crash> dentry.d_iname
struct dentry {
[0xcc] unsigned char d_iname[36];
}
crash> eval c808+cc
hexadecimal: c8d4
decimal: 51412
octal: 144324
show me that variable in stack actually is d_iname.
then can we have the output format as
10078b2bca0: [000001007877c808+cc: dentry_cache]
so we know the object address, which slab it is in, and the
offset,
(thus can derive the raw value), all in one shot?
Thanks!
>
> Dave
>
>
>
--
Ming Zhang
#$%^
purging memory... (*!%
http://blackmag
ic02881.wordpress.com/
http://www
.linkedin.com/in/blackmagic02881
--------------------------------------------
--
Crash-utility mailing list
Crash-utilityredhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
|
|
|
|