single-stepping and internal breakpoints on a multi-threaded program

Hello members, 

My colleague has found a problem about handling
single-stepping on a
multi-threaded target running on Linux box.  

The attached program makes two sub threads, and we do
single-stepping
on one of them.  
When the target get stopped by a internal breakpoint hit
(ex. thread
creation, thread death, shared library loading) during
singe-stepping,
GDB unconditionally resume all the thread and it result in
running
over the range of single-stepping.  

The session log below shows one of the phenomena reproduced
on X86
FC6; the problem occured when the second thread was created.
 
I can see them 10-20% of the time on our Cell hardware
running FC5
(note that the program is compiled just for PPC64) and X86
FC5/FC6.  

----
GNU gdb 6.6.50.20070330-cvs
Copyright (C) 2007 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public
License, and you are
welcome to change it and/or distribute copies of it under
certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show
warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...
Using host libthread_db library
"/lib/libthread_db.so.1".
(gdb) b 14
Breakpoint 1 at 0x80484fa: file mt.c, line 14.
(gdb) l 14
9       {
10        int i;
11        int cnt = *((int *) arg);
12        pid_t pid = getpid();
13
14        for (i=0; i<cnt; i++)
15          printf ("[%d]counter: %dn", pid, i);
16
17        return NULL;
18      }
(gdb) r
Starting program: /project/07/user/suzuki/test/mt/mt
[Thread debugging using libthread_db enabled]
[New Thread -1208011072 (LWP 28472)]
[28472]start
[New Thread -1208013936 (LWP 28475)]
[Switching to Thread -1208013936 (LWP 28475)]

Breakpoint 1, counter (arg=0xbfd491b0) at mt.c:14
14        for (i=0; i<cnt; i++)
(gdb) s
[New Thread -1218503792 (LWP 28476)]
[28472]no_counter
[Thread -1218503792 (LWP 28476) exited]
[28472]counter: 0
[28472]counter: 1
[28472]counter: 2
[28472]counter: 3
[28472]counter: 4
[28472]counter: 5
[28472]counter: 6
[28472]counter: 7
[28472]counter: 8
[28472]counter: 9
[Thread -1208013936 (LWP 28475) exited]
Thread Event Breakpoint: gdb should not stop!
(gdb) 
---- 

I saw that the context of single-stepping thread is
preserved by
context_switch called in handle_inferior_event, but GDB
doesn't go
back to that context till the thread is trapped in another
reason
(in the above case, the death of the thread).  

I think it should either stop immediately and prompt the
user that the
single-stepping has been cancelled, or goes back to the
preserved
context somehow and continue single-stepping.  
How do you think of it? 

Thanks in advance, 
-- 
Emi SUZUKI

#include <stdio.h>
#include <sys/types.h>
#include <pthread.h>
#include <unistd.h>
#include <string.h>

void *
counter(void *arg)
{
  int i;
  int cnt = *((int *) arg);
  pid_t pid = getpid();

  for (i=0; i<cnt; i++)
    printf ("[%d]counter: %dn", pid, i);

  return NULL;
}

void *
no_counter (void *arg)
{
  pid_t pid = getpid();

  printf ("[%d]no_countern", pid);

  return NULL;
}

int
main (void)
{
  pid_t p_pid;
  pthread_t thread_id1, thread_id2;
  int status;
  int count = 10;

  p_pid = getpid ();
  printf ("[%d]startn", p_pid);

  status = pthread_create (&thread_id1, NULL, counter,
(void *)&count);
  if (status != 0)
    {
      fprintf (stderr, "pthread_create : %s",
strerror(status));
      return (1);
    }

  status = pthread_create (&thread_id2, NULL,
no_counter, (void *)NULL);
  if (status != 0)
    {
      fprintf (stderr, "pthread_create : %s",
strerror(status));
      pthread_join (thread_id1, NULL);
      return (1);
    }

  pthread_join (thread_id2, NULL);
  pthread_join (thread_id1, NULL);

  printf ("[%d]endn",p_pid);
  return 0;
}

Emi SUZUKI <emi-suzukitjsys.co.jp> writes:
> The attached program makes two sub threads, and we do
single-stepping
> on one of them.  
> When the target get stopped by a internal breakpoint
hit (ex. thread
> creation, thread death, shared library loading) during
singe-stepping,
> GDB unconditionally resume all the thread and it result
in running
> over the range of single-stepping.  

I'm able to reproduce this problem on my FC6 x86 dual core
system.
Thanks very much for the clear reproduction instructions.

> I saw that the context of single-stepping thread is
preserved by
> context_switch called in handle_inferior_event, but GDB
doesn't go
> back to that context till the thread is trapped in
another reason
> (in the above case, the death of the thread).  
>
> I think it should either stop immediately and prompt
the user that the
> single-stepping has been cancelled, or goes back to the
preserved
> context somehow and continue single-stepping.  
> How do you think of it? 

So, if I'm understanding correctly, we single-step the
'counter'
thread, which proceeds normally until the 'no_counter'
thread hits its
TD_CREATE event, to tell GDB about its existence.  Then, GDB
fails to
resume single-stepping the 'counter' thread, letting it run
until it
exits and reports its TD_DEATH event.

The fact that the inferior uses breakpoints to report thread
creation
and thread death is supposed to be an internal detail of the
thread
library --- or, more properly, of the thread debugging
interface.  One
thread stepping shouldn't be affected by other threads being
created
(although GDB should still print the 'new thread'
messages).

So I think the right behavior would be for GDB to continue
single-stepping after processing creation and death events
for other
threads.

I have forgetten exactly how GDB handles multi-threaded
single
stepping; I'm afraid I can't suggest how to do this.  If
you're unable
to fix it yourself, please file a bug report, and include
your test
program.

On Fri, 2007-04-06 at 15:12 -0700, Jim Blandy wrote:
> Emi SUZUKI <emi-suzukitjsys.co.jp> writes:
> > The attached program makes two sub threads, and we
do single-stepping
> > on one of them.  
> > When the target get stopped by a internal
breakpoint hit (ex. thread
> > creation, thread death, shared library loading)
during singe-stepping,
> > GDB unconditionally resume all the thread and it
result in running
> > over the range of single-stepping.  
> 
> I'm able to reproduce this problem on my FC6 x86 dual
core system.
> Thanks very much for the clear reproduction
instructions.
> 
> > I saw that the context of single-stepping thread
is preserved by
> > context_switch called in handle_inferior_event,
but GDB doesn't go
> > back to that context till the thread is trapped in
another reason
> > (in the above case, the death of the thread).  
> >
> > I think it should either stop immediately and
prompt the user that the
> > single-stepping has been cancelled, or goes back
to the preserved
> > context somehow and continue single-stepping.  
> > How do you think of it? 
> 
> So, if I'm understanding correctly, we single-step the
'counter'
> thread, which proceeds normally until the 'no_counter'
thread hits its
> TD_CREATE event, to tell GDB about its existence. 
Then, GDB fails to
> resume single-stepping the 'counter' thread, letting it
run until it
> exits and reports its TD_DEATH event.
> 
> The fact that the inferior uses breakpoints to report
thread creation
> and thread death is supposed to be an internal detail
of the thread
> library --- or, more properly, of the thread debugging
interface.  One
> thread stepping shouldn't be affected by other threads
being created
> (although GDB should still print the 'new thread'
messages).
> 
> So I think the right behavior would be for GDB to
continue
> single-stepping after processing creation and death
events for other
> threads.
> 
> I have forgetten exactly how GDB handles multi-threaded
single
> stepping; I'm afraid I can't suggest how to do this. 
If you're unable
> to fix it yourself, please file a bug report, and
include your test
> program.

I'm thinking that gdb just loses the "I'm stepping this
thread" 
bit of state info, which is normally handled in
"proceed" and 
stored in the infrun-state struct - whereupon it just
continues
both threads without stepping.


But that's just a hunch.

Jim and Michael,

Thank you for your comments.  

From: Michael Snyder <Michael dot Snyder at
access-company dot com>
Subject: Re: single-stepping and internal breakpoints on a
multi-threaded program
Date: Fri, 06 Apr 2007 15:25:09 -0700

> > I have forgetten exactly how GDB handles
multi-threaded single
> > stepping; I'm afraid I can't suggest how to do
this.  If you're unable
> > to fix it yourself, please file a bug report, and
include your test
> > program.
> 
> I'm thinking that gdb just loses the "I'm stepping
this thread" 
> bit of state info, which is normally handled in
"proceed" and 
> stored in the infrun-state struct - whereupon it just
continues
> both threads without stepping.

I think it's true in some ways: it seemed that the clue for
GDB to
decide that "I'm stepping this thread" is
eventually only the value of
step_range_end.  

When the target get stopped on a thread different from which
GDB
expected to stop, it was preserved in the thread_info
structure by
context_switch, but never had an oppotunity to be looked
back till the
target get stopped on that thread again.  It is not just
lost, but no
better than lost.  

I think I should work more on the problem, but
unfortunately, my
copyright assignment to FSF is still not ready.  So I will
file it to
the bug tracking system, following Jim's advice.  
I will be back on it when I find that anything is not done
for it
after my assignment is ready.  I hope not, though 

Thanks anyway, 
-- 
Emi SUZUKI