Which thread caused dump?

Hi!

I have a multithreaded program which coredumps after ~20
hours of
running.
Is it possible to see which thread caused the crash?
Usually I use thr x and bt for seeing if some exception was
thrown, but
I have a case where no thread is throwing any exceptions.

I've enclosed output from the actual core dump.. any other
cmd's I
should know of?
Any help on discovering why it is crashing would be greatly
appreciated..

/Steffen


testnxq2# uname -a
FreeBSD testnxq2.xxx.dk 6.1-RELEASE FreeBSD 6.1-RELEASE #0:
Sun May  7
04:32:43 UTC 2006
rootopus.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC 
i386

testnxq2# gdb measuring measuring.core
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public
License, and you
are
welcome to change it and/or distribute copies of it under
certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show
warranty" for
details.
This GDB was configured as
"i386-marcel-freebsd"...
Core was generated by `measuring'.
Program terminated with signal 11, Segmentation fault.
Reading symbols from /usr/lib/libpcap.so.4...done.
Loaded symbols for /usr/lib/libpcap.so.4
Reading symbols from /usr/local/lib/libqosmysql.so.1...done.
Loaded symbols for /usr/local/lib/libqosmysql.so.1
Reading symbols from
/usr/local/lib/libqosthreads.so.1...done.
Loaded symbols for /usr/local/lib/libqosthreads.so.1
Reading symbols from /usr/local/lib/libqoslog.so.1...done.
Loaded symbols for /usr/local/lib/libqoslog.so.1
Reading symbols from
/usr/local/lib/libqoslegacynetstuff.so.1...done.
Loaded symbols for /usr/local/lib/libqoslegacynetstuff.so.1
Reading symbols from
/usr/local/lib/libqoscommon.so.1...done.
Loaded symbols for /usr/local/lib/libqoscommon.so.1
Reading symbols from
/usr/local/lib/libqosmeasuring.so.1...done.
Loaded symbols for /usr/local/lib/libqosmeasuring.so.1
Reading symbols from /usr/local/lib/libqosdb.so.1...done.
Loaded symbols for /usr/local/lib/libqosdb.so.1
Reading symbols from
/usr/local/lib/libqospacket.so.1...done.
Loaded symbols for /usr/local/lib/libqospacket.so.1
Reading symbols from
/usr/local/lib/libqosaggregation.so.1...done.
Loaded symbols for /usr/local/lib/libqosaggregation.so.1
Reading symbols from /usr/lib/libstdc++.so.5...done.
Loaded symbols for /usr/lib/libstdc++.so.5
Reading symbols from /lib/libm.so.4...done.
Loaded symbols for /lib/libm.so.4
Reading symbols from /usr/lib/libpthread.so.2...done.
Loaded symbols for /usr/lib/libpthread.so.2
Reading symbols from /lib/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /usr/local/lib/libmysqlpp.so.2...done.
Loaded symbols for /usr/local/lib/libmysqlpp.so.2
Reading symbols from /usr/local/lib/libqoscmd.so.1...done.
Loaded symbols for /usr/local/lib/libqoscmd.so.1
Reading symbols from /lib/libz.so.3...done.
Loaded symbols for /lib/libz.so.3
Reading symbols from
/usr/local/lib/mysql/libmysqlclient.so.15...done.
Loaded symbols for /usr/local/lib/mysql/libmysqlclient.so.15
Reading symbols from /lib/libcrypt.so.3...done.
Loaded symbols for /lib/libcrypt.so.3
Reading symbols from /libexec/ld-elf.so.1...done.
Loaded symbols for /libexec/ld-elf.so.1
#0  0x2834f4ab in pthread_testcancel () from
/usr/lib/libpthread.so.2
[New Thread 0x806bc00 (sleeping)]
[New Thread 0x806ba00 (runnable)]
[New Thread 0x806b800 (sleeping)]
[New Thread 0x806b600 (runnable)]
[New Thread 0x8057800 (runnable)]
[New Thread 0x8057600 (runnable)]
[New Thread 0x8057400 (runnable)]
[New Thread 0x8057200 (LWP 100041)]
[New Thread 0x8057000 (sleeping)]
[New LWP 100080]
(gdb) info threads
* 10 LWP 100080  0x2834f4ab in pthread_testcancel () from
/usr/lib/libpthread.so.2
  9 Thread 0x8057000 (sleeping)  0x28347f0f in
pthread_mutexattr_init ()
from /usr/lib/libpthread.so.2
  8 Thread 0x8057200 (LWP 100041)  0x2834f46b in
pthread_testcancel ()
from /usr/lib/libpthread.so.2
  7 Thread 0x8057400 (runnable)  0x28347f0f in
pthread_mutexattr_init ()
from /usr/lib/libpthread.so.2
  6 Thread 0x8057600 (runnable)  0x28347f0f in
pthread_mutexattr_init ()
from /usr/lib/libpthread.so.2
  5 Thread 0x8057800 (runnable)  0x282960de in
std::_Rb_tree_increment
() from /usr/lib/libstdc++.so.5
  4 Thread 0x806b600 (runnable)  0x2840c7df in read () from
/lib/libc.so.6
  3 Thread 0x806b800 (sleeping)  0x28347f0f in
pthread_mutexattr_init ()
from /usr/lib/libpthread.so.2
  2 Thread 0x806ba00 (runnable)  0x283ffacb in gettimeofday
() from
/lib/libc.so.6
  1 Thread 0x806bc00 (sleeping)  0x28347f0f in
pthread_mutexattr_init ()
from /usr/lib/libpthread.so.2
(gdb) thr 1
[Switching to thread 1 (Thread 0x806bc00 (sleeping))]#0 
0x28347f0f in
pthread_mutexattr_init () from /usr/lib/libpthread.so.2
(gdb) bt
#0  0x28347f0f in pthread_mutexattr_init () from
/usr/lib/libpthread.so.2
#1  0x283419ae in _nanosleep () from
/usr/lib/libpthread.so.2
#2  0x283d7669 in sleep () from /lib/libc.so.6
#3  0x283373aa in sleep () from /usr/lib/libpthread.so.2
#4  0x280c4b26 in WatchDog::execute (this=0x805a2c0) at
WatchDog.cpp:138
#5  0x2813d015 in IThread<WatchDog>::entrypoint
(pthis=0x805a2c0) at
IThread.h:175
#6  0x28340319 in pthread_create () from
/usr/lib/libpthread.so.2
#7  0x283f4637 in _ctx_start () from /lib/libc.so.6
(gdb) thr 2
[Switching to thread 2 (Thread 0x806ba00 (runnable))]#0 
0x283ffacb in
gettimeofday () from /lib/libc.so.6
(gdb) bt
#0  0x283ffacb in gettimeofday () from /lib/libc.so.6
#1  0x283f31de in time () from /lib/libc.so.6
#2  0x280d4161 in LogSender::lsWrite (this=0x8051d00,
message=
        {static npos = 4294967295, _M_dataplus =
{<std::allocator<char>>
= {<__gnu_cxx::new_allocator<char>> = {<No
data fields>}, <No data
fields>}, _M_p = 0x8071a2c "sending 3
results"}}, level=7 '\a') at
LogSender.cpp:25
#3  0x280d40a5 in LogSender::lsWrite (this=0x8051d00,
message=
        {static npos = 4294967295, _M_dataplus =
{<std::allocator<char>>
= {<__gnu_cxx::new_allocator<char>> = {<No
data fields>}, <No data
fields>}, _M_p = 0x7 <Address 0x7 out of bounds>}})
at LogSender.cpp:17
#4  0x280d43aa in LogSender::bufferChar (this=0x8051d00,
c=10 '\n') at
LogSender.cpp:63
#5  0x280d4e74 in LogSender::LogStreambuf::overflow
(this=0x8051d18,
c=10) at LogSender.h:119
#6  0x282c13ff in std::basic_streambuf<char,
std::char_traits<char>
>:sputn
() from /usr/lib/libstdc++.so.5
#7  0x282b78eb in std::operator<<
<std::char_traits<char> > () from
/usr/lib/libstdc++.so.5
#8  0x2820a415 in ResultHandler::execute (this=0x806c400) at
ResultHandler.cpp:85
#9  0x2813cf65 in
IMonitoredThread<ResultHandler>::entrypoint
(pthis=0x806c400) at IMonitoredThread.h:217
#10 0x28340319 in pthread_create () from
/usr/lib/libpthread.so.2
#11 0x283f4637 in _ctx_start () from /lib/libc.so.6
(gdb) thr 3
[Switching to thread 3 (Thread 0x806b800 (sleeping))]#0 
0x28347f0f in
pthread_mutexattr_init () from /usr/lib/libpthread.so.2
(gdb) bt
#0  0x28347f0f in pthread_mutexattr_init () from
/usr/lib/libpthread.so.2
#1  0x283419ae in _nanosleep () from
/usr/lib/libpthread.so.2
#2  0x283f31b7 in usleep () from /lib/libc.so.6
#3  0x283373f6 in usleep () from /usr/lib/libpthread.so.2
#4  0x2812fb04 in PacketScheduler::execute (this=0x806c100)
at
PacketScheduler.cpp:104
#5  0x2813ce81 in
IMonitoredThread<PacketScheduler>::entrypoint
(pthis=0x806c100) at IMonitoredThread.h:217
#6  0x28340319 in pthread_create () from
/usr/lib/libpthread.so.2
#7  0x283f4637 in _ctx_start () from /lib/libc.so.6
(gdb) thr 4
[Switching to thread 4 (Thread 0x806b600 (runnable))]#0 
0x2840c7df in
read () from /lib/libc.so.6
(gdb) bt
#0  0x2840c7df in read () from /lib/libc.so.6
#1  0x2833819a in read () from /usr/lib/libpthread.so.2
#2  0x2808a5ac in pcap_lookupnet () from
/usr/lib/libpcap.so.4
#3  0x2808b57e in pcap_dispatch () from
/usr/lib/libpcap.so.4
#4  0x2808b666 in pcap_next () from /usr/lib/libpcap.so.4
#5  0x2814cc60 in Receiver::getPacket (this=0x8088000,
pac=0xbf5faf2c,
t=0xbf5faf10) at Receiver.cpp:94
#6  0x2814cdd2 in Receiver::serviceQueue (this=0x8088000) at
Receiver.cpp:144
#7  0x2814cb2c in Receiver::execute (this=0x8088000) at
Receiver.cpp:72
#8  0x2813cd9d in
IMonitoredThread<Receiver>::entrypoint
(pthis=0x8088000) at IMonitoredThread.h:217
#9  0x28340319 in pthread_create () from
/usr/lib/libpthread.so.2
#10 0x283f4637 in _ctx_start () from /lib/libc.so.6
(gdb) thr 5
[Switching to thread 5 (Thread 0x8057800 (runnable))]#0 
0x282960de in
std::_Rb_tree_increment () from /usr/lib/libstdc++.so.5
(gdb) bt
#0  0x282960de in std::_Rb_tree_increment () from
/usr/lib/libstdc++.so.5
#1  0x2813c0e6 in
std::_Rb_tree_iterator<std::pair<unsigned int const,
std::list<FinishedPacket,
SAllocator<FinishedPacket> > > >::operator++
(
    this=0xbf6fbf30) at stl_tree.h:189
#2  0x28135ba8 in Measuring::execute (this=0x805d480) at
Measuring.cpp:114
#3  0x0804c13a in
IMonitoredThread<Measuring>::entrypoint
(pthis=0x805d480) at IMonitoredThread.h:217
#4  0x28340319 in pthread_create () from
/usr/lib/libpthread.so.2
#5  0x283f4637 in _ctx_start () from /lib/libc.so.6
(gdb) thr 6
[Switching to thread 6 (Thread 0x8057600 (runnable))]#0 
0x28347f0f in
pthread_mutexattr_init () from /usr/lib/libpthread.so.2
(gdb) bt
#0  0x28347f0f in pthread_mutexattr_init () from
/usr/lib/libpthread.so.2
#1  0x283419ae in _nanosleep () from
/usr/lib/libpthread.so.2
#2  0x283d7669 in sleep () from /lib/libc.so.6
#3  0x283373aa in sleep () from /usr/lib/libpthread.so.2
#4  0x280d5fa2 in LogPostOffice::execute (this=0x805a040) at
LogPostOffice.cpp:75
#5  0x280d9bb9 in
IMonitoredThread<LogPostOffice>::entrypoint
(pthis=0x805a040) at IMonitoredThread.h:217
#6  0x28340319 in pthread_create () from
/usr/lib/libpthread.so.2
#7  0x283f4637 in _ctx_start () from /lib/libc.so.6
(gdb) tht 7
Undefined command: "tht".  Try
"help".
(gdb) thr 7
[Switching to thread 7 (Thread 0x8057400 (runnable))]#0 
0x28347f0f in
pthread_mutexattr_init () from /usr/lib/libpthread.so.2
(gdb) bt
#0  0x28347f0f in pthread_mutexattr_init () from
/usr/lib/libpthread.so.2
#1  0x283419ae in _nanosleep () from
/usr/lib/libpthread.so.2
#2  0x28337fce in select () from /usr/lib/libpthread.so.2
#3  0x00000000 in ?? ()
(gdb) thr 8
[Switching to thread 8 (Thread 0x8057200 (LWP 100041))]#0 
0x2834f46b in
pthread_testcancel () from /usr/lib/libpthread.so.2
(gdb) bt
#0  0x2834f46b in pthread_testcancel () from
/usr/lib/libpthread.so.2
#1  0x28347e3c in pthread_mutexattr_init () from
/usr/lib/libpthread.so.2
#2  0x284fa450 in ?? ()
(gdb) thr 9
[Switching to thread 9 (Thread 0x8057000 (sleeping))]#0 
0x28347f0f in
pthread_mutexattr_init () from /usr/lib/libpthread.so.2
(gdb) bt
#0  0x28347f0f in pthread_mutexattr_init () from
/usr/lib/libpthread.so.2
#1  0x283419ae in _nanosleep () from
/usr/lib/libpthread.so.2
#2  0x283d7669 in sleep () from /lib/libc.so.6
#3  0x283373aa in sleep () from /usr/lib/libpthread.so.2
#4  0x0804a7ae in debug () at main.cpp:92
#5  0x0804a365 in main () at main.cpp:54
(gdb) thr 10
[Switching to thread 10 (LWP 100080)]#0  0x2834f46b in
pthread_testcancel () from /usr/lib/libpthread.so.2
(gdb) bt
#0  0x2834f46b in pthread_testcancel () from
/usr/lib/libpthread.so.2
#1  0x28347e3c in pthread_mutexattr_init () from
/usr/lib/libpthread.so.2
#2  0x284fa450 in ?? ()
(gdb)

On Tue, Aug 29, 2006 at 12:42:40PM +0200, Steffen Schumacher
wrote:
> Hi!
> 
> I have a multithreaded program which coredumps after
~20 hours of
> running.
> Is it possible to see which thread caused the crash?
> Usually I use thr x and bt for seeing if some exception
was thrown, but
> I have a case where no thread is throwing any
exceptions.
> 
> I've enclosed output from the actual core dump.. any
other cmd's I
> should know of?
> Any help on discovering why it is crashing would be
greatly
> appreciated..

Generally the first thread that comes up is the one that
crashed.

> Program terminated with signal 11, Segmentation fault.

> * 10 LWP 100080  0x2834f4ab in pthread_testcancel ()
from
> /usr/lib/libpthread.so.2

> (gdb) thr 10
> [Switching to thread 10 (LWP 100080)]#0  0x2834f46b in
> pthread_testcancel () from /usr/lib/libpthread.so.2
> (gdb) bt
> #0  0x2834f46b in pthread_testcancel () from
/usr/lib/libpthread.so.2
> #1  0x28347e3c in pthread_mutexattr_init () from
> /usr/lib/libpthread.so.2
> #2  0x284fa450 in ?? ()
> (gdb)
>

So probably right there.

-- 
Daniel Jacobowitz
CodeSourcery

Ok but it doesn't show me much. Hmm.. 

My stack must have been overwritten:
> (gdb) bt
> #0  0x2834f46b in pthread_testcancel () from
/usr/lib/libpthread.so.2
> #1  0x28347e3c in pthread_mutexattr_init () from
> /usr/lib/libpthread.so.2
> #2  0x284fa450 in ?? ()
> (gdb)

#2 seems to start out of the blue..
How do you go about chasing a bug like this?

/Steffen

-----Oprindelig meddelelse-----
Fra: Daniel Jacobowitz [mailto:drowfalse.org] 
Sendt: 29. august 2006 14:39
Til: Steffen Schumacher
Cc: gdbsourceware.org
Emne: Re: Which thread caused dump?

On Tue, Aug 29, 2006 at 12:42:40PM +0200, Steffen Schumacher
wrote:
> Hi!
> 
> I have a multithreaded program which coredumps after
~20 hours of
> running.
> Is it possible to see which thread caused the crash?
> Usually I use thr x and bt for seeing if some exception
was thrown,
but
> I have a case where no thread is throwing any
exceptions.
> 
> I've enclosed output from the actual core dump.. any
other cmd's I
> should know of?
> Any help on discovering why it is crashing would be
greatly
> appreciated..

Generally the first thread that comes up is the one that
crashed.

> Program terminated with signal 11, Segmentation fault.

> * 10 LWP 100080  0x2834f4ab in pthread_testcancel ()
from
> /usr/lib/libpthread.so.2

> (gdb) thr 10
> [Switching to thread 10 (LWP 100080)]#0  0x2834f46b in
> pthread_testcancel () from /usr/lib/libpthread.so.2
> (gdb) bt
> #0  0x2834f46b in pthread_testcancel () from
/usr/lib/libpthread.so.2
> #1  0x28347e3c in pthread_mutexattr_init () from
> /usr/lib/libpthread.so.2
> #2  0x284fa450 in ?? ()
> (gdb)
>

So probably right there.

-- 
Daniel Jacobowitz
CodeSourcery

On Tue, Aug 29, 2006 at 03:23:11PM +0200, Steffen Schumacher
wrote:
> Ok but it doesn't show me much. Hmm.. 
> 
> My stack must have been overwritten:
> > (gdb) bt
> > #0  0x2834f46b in pthread_testcancel () from
/usr/lib/libpthread.so.2
> > #1  0x28347e3c in pthread_mutexattr_init () from
> > /usr/lib/libpthread.so.2
> > #2  0x284fa450 in ?? ()
> > (gdb)
> 
> #2 seems to start out of the blue..
> How do you go about chasing a bug like this?

Maybe try getting system libraries with debugging; GDB has
apparently
not got enough information to backtrace out of
libpthread.so.2.

-- 
Daniel Jacobowitz
CodeSourcery