Keeping breakpoints inserted

One of the infrastructure bits necessary for the non-stop
threads
debugging is always-inserted-breakpoints mode. If GDB has
stopped
one thread, and other threads are running, we want those
other threads
to still hit breakpoints and watchpoints. However, current
GDB removes
all breakpoints from the target before giving user a prompt,
and this
has to change.

I've spend quite time examining breakpoint.c and infrun.c
and 
cleaning/localizing the decisions as to when breakpoints are

inserted/removed, and I believe I now have a fully workable
plan 
to make breakpoints always inserted. We need to:

1. Generally, whenever a breakpoint is added, changed, or 
removed, immediately communicate this change to the target.

2. In order to do that, we need to have a single function
called on each
breakpoint change. That function will be responsible to
update 
bp_location_chain, using locations of existing breakpoint,
removing
breakpoint locations that are no longer necessary, and
inserting newly 
added locations. Since we can have several breakpoints at
the same address, 
the function should take care not to remove location
belonging to a 
deleted breakpoint, if there's location at the same address
belonging 
to a still-present breakpoint.

Specifically, the function will walk over current
bp_location_chain. 
For each location:

        - if no breakpoint has a location with the same
address, 
	we remove the location from target
        - if no breakpoint has this bp_location* object, we
free it.

All locations present in some breakpoint but not yet present
in 
bp_location_chain are added to bp_location_chain and
inserted to target.

3. The functions that read memory should be modified to
restore memory 
content overwritten by breakpoints. I believe there's a 
single function that is in all memory-reading code paths, so
this change is
easy.

4. infrun.c should be modified to avoid removing breakpoints
when 
we stop -- most probably under control of a new variable.

5. Sometimes, infrun.c still need to remove breakpoints --
in particular 
when stepping over them. We'll implement smart code that
allows to 
step over breakpoint without removing it -- Jim will
separately post details
on this. For watchpoints, we still have to remove them when
stepping 
over watchpoint (for the case of non-continuable
non-steppable watchpoints).

6. It's possible that we get an error when inserting
breakpoint. 
In present GDB, such an error will be printed when we try to
resume
target, and the target won't be resumed. With
always-inserted mode, 
the error will be printed when we add or change the
breakpoint. However, 
resuming target with not-inserted breakpoint still seems 
wrong behaviour, 
so we'll try to insert breakpoints again when resuming, and
if that fails,
stop.

Anybody has comments on this approach?

- Volodya

On Thu, 2007-11-29 at 22:24 +0300, Vladimir Prus wrote:
> One of the infrastructure bits necessary for the
non-stop threads
> debugging is always-inserted-breakpoints mode. If GDB
has stopped
> one thread, and other threads are running, we want
those other threads
> to still hit breakpoints and watchpoints. However,
current GDB removes
> all breakpoints from the target before giving user a
prompt, and this
> has to change.
> 
> I've spend quite time examining breakpoint.c and
infrun.c and 
> cleaning/localizing the decisions as to when
breakpoints are 
> inserted/removed, and I believe I now have a fully
workable plan 
> to make breakpoints always inserted. 

> [...]
> 
> Anybody has comments on this approach?

Might there be a user preference, under some circumstances,

to NOT have them inserted while some threads run and some
are stopped?

On Friday 30 November 2007 04:12:14 you wrote:
> On Thu, 2007-11-29 at 22:24 +0300, Vladimir Prus
wrote:
> > One of the infrastructure bits necessary for the
non-stop threads
> > debugging is always-inserted-breakpoints mode. If
GDB has stopped
> > one thread, and other threads are running, we want
those other threads
> > to still hit breakpoints and watchpoints. However,
current GDB removes
> > all breakpoints from the target before giving user
a prompt, and this
> > has to change.
> >
> > I've spend quite time examining breakpoint.c and
infrun.c and
> > cleaning/localizing the decisions as to when
breakpoints are
> > inserted/removed, and I believe I now have a fully
workable plan
> > to make breakpoints always inserted.
> >
> > [...]
> >
> > Anybody has comments on this approach?
>
> Might there be a user preference, under some
circumstances,
> to NOT have them inserted while some threads run and
some
> are stopped?

Honestly, originally I planned to have breakpoints always
inserted,
unconditionally. Unfortunately, that does not work with
remote targets --
since the Z0 packet may use memory breakpoints, and remote
protocol does
not requires the memory read packet to return original
memory content,
without breakpoints, we'd need to explicitly read the memory
ourself --
which is not necessary without non-stop mode.

At the same time, I don't yet know any scenario when 
always-inserted-breakpoints mode should not be used in
non-stop mode -- 
as non-stop mode needs some support from remote side anyway,
we might
as well demand "right" behaviour from Z0/m.

- Volodya

Hi,

On Thu, 2007-11-29 at 22:24 +0300, Vladimir Prus wrote:
> Anybody has comments on this approach?

When adding and removing breakpoints, will GDB stop all
threads or just
the one under inspection?

On first thought, I think that if adding a breakpoint can be
done
atomically (i.e., trap instruction is 1 word wide), then it
wouldn't be
necessary to stop other threads. Am I being too naive here?

-- 
[]'s
Thiago Jung Bauermann
Software Engineer
IBM Linux Technology Center

On Fri, 2007-11-30 at 19:03 -0200, Thiago Jung Bauermann
wrote:
> Hi,
> 
> On Thu, 2007-11-29 at 22:24 +0300, Vladimir Prus
wrote:
> > Anybody has comments on this approach?
> 
> When adding and removing breakpoints, will GDB stop all
threads or just
> the one under inspection?
> 
> On first thought, I think that if adding a breakpoint
can be done
> atomically (i.e., trap instruction is 1 word wide),
then it wouldn't be
> necessary to stop other threads. Am I being too naive
here? 

There will be some as-it-were non-deterministic behavior, 
it seems to me.  If threads are running while we are
inserting
breakpoints, then there will (at some point) be breakpoint
events while we are inserting breakpoints, and the order of
these events (and of running-threads stopping) will depend
on the order in which we insert them, as well as on what 
the running threads happen to be doing at the time.

I should think that this would be more intrusive (in the 
sense of changing the behavior of the target program) than
we already are.


We could see a deadlock develop during the time it takes us
to insert all the breakpoints.

Thiago Jung Bauermann <bauerman at br.ibm.com>
writes:
> On Thu, 2007-11-29 at 22:24 +0300, Vladimir Prus
wrote:
>> Anybody has comments on this approach?
>
> When adding and removing breakpoints, will GDB stop all
threads or just
> the one under inspection?
>
> On first thought, I think that if adding a breakpoint
can be done
> atomically (i.e., trap instruction is 1 word wide),
then it wouldn't be
> necessary to stop other threads. Am I being too naive
here? 

The Linux kernel has a facility called 'kprobes' (which is
extremely
cool, BTW) that can insert breakpoint instructions into
kernel code
while other processors are executing it.  This interface has
been
implemented on many different architectures.  We're using it
as our
model, since it does essentially what we need to do, in an
even more
challenging context.  I haven't seen one where it needs to
halt all
other processors while inserting the breakpoint.

On the i386, you pretty much just save the underlying
instruction and
drop the trap in.  There are processor errata (AH 33 in
ftp://download.intel.com/design/mobile/SPECUPDT/31407915.pdf
) that
warn against the dangers of one processor making changes in
another's
code stream, but it turns out that the 'int3' instruction
--- the one
used for breakpoints --- is specifically excluded from those
errata;
so we're off the hook.

Michael Snyder <msnyder at specifix.com> writes:
> There will be some as-it-were non-deterministic
behavior, 
> it seems to me.  If threads are running while we are
inserting
> breakpoints, then there will (at some point) be
breakpoint
> events while we are inserting breakpoints, and the
order of
> these events (and of running-threads stopping) will
depend
> on the order in which we insert them, as well as on
what 
> the running threads happen to be doing at the time.
>
> I should think that this would be more intrusive (in
the 
> sense of changing the behavior of the target program)
than
> we already are.
>
> We could see a deadlock develop during the time it
takes us
> to insert all the breakpoints.

Since breakpoints only make the threads that hit them
execute "very
slowly", I don't see how they could introduce a
deadlock in code that
didn't already have the potential for deadlock in the
absence of a
debugger.

It's true that the breakpoints won't all appear in the
address space
simultaneously.  It might be useful to be able to insert
breakpoints
disabled, and then enable them as a group.  Even in all-stop
mode, GDB
isn't able to stop all threads simulataneously; it has to
obtain the
list of threads and send each one a signal.  So it seems to
me that
some degree of 'play' in breakpoint insertion isn't so
different from
what we have now.

Inserting breakpoints into code that's currently being
executed
inevitably entails race conditions.  But I expect one would
only do
such a thing in circumstances where it's meaningful and
controllable.

For example, you might know that you'll reach the breakpoint
only in
response to a certain input, which you'll send to the
program after
you've set the breakpoint.

Or, it might not matter which particular execution of the
code you
catch first, you just want to interrupt it.

Non-stop debugging is more invasive than all-stop debugging,
if you
assess invasiveness by looking at how the threads progress
relative to
each other.  Non-stop debugging can make one thread run
'extremely
slowly' while other threads progress at a normal rate. 
All-stop
debugging stops all threads roughly equally (but even then
not
perfectly so).

However, non-stop debugging can be less invasive than
all-stop
debugging, if you assess invasiveness by looking at how
responsive the
system remains to outside events.  With non-stop debugging,
you can
debug one particular thread out of the entire system while
the others
continue to work normally.

Deciding which mode to use depends on the developer's
knowledge of the
system they're debugging.

On Fri, 2007-11-30 at 16:08 -0800, Jim Blandy wrote:
> Michael Snyder <msnyder at specifix.com> writes:
> > There will be some as-it-were non-deterministic
behavior, 
> > it seems to me.  If threads are running while we
are inserting
> > breakpoints, then there will (at some point) be
breakpoint
> > events while we are inserting breakpoints, and the
order of
> > these events (and of running-threads stopping)
will depend
> > on the order in which we insert them, as well as
on what 
> > the running threads happen to be doing at the
time.
> >
> > I should think that this would be more intrusive
(in the 
> > sense of changing the behavior of the target
program) than
> > we already are.
> >
> > We could see a deadlock develop during the time it
takes us
> > to insert all the breakpoints.
> 
> Since breakpoints only make the threads that hit them
execute "very
> slowly", I don't see how they could introduce a
deadlock in code that
> didn't already have the potential for deadlock in the
absence of a
> debugger.

Well, just 'cause you've changed the timing.  Thread A
holds
a resource that thread B needs; thread A hits a breakpoint 
but thread B gets to keep running (for a while) because you
haven't finished inserting breakpoints.

> It's true that the breakpoints won't all appear in the
address space
> simultaneously.  It might be useful to be able to
insert breakpoints
> disabled, and then enable them as a group.  Even in
all-stop mode, GDB
> isn't able to stop all threads simulataneously; it has
to obtain the
> list of threads and send each one a signal.  So it
seems to me that
> some degree of 'play' in breakpoint insertion isn't so
different from
> what we have now.

No, it's just more of what is already an intrusive thing.
It's worse in degree, not in kind.

On Nov 30, 2007 5:30 PM, Michael Snyder <msnyderspecifix.com> wrote:
> > Since breakpoints only make the threads that hit
them execute "very
> > slowly", I don't see how they could introduce
a deadlock in code that
> > didn't already have the potential for deadlock in
the absence of a
> > debugger.
>
> Well, just 'cause you've changed the timing.  Thread A
holds
> a resource that thread B needs; thread A hits a
breakpoint
> but thread B gets to keep running (for a while) because
you
> haven't finished inserting breakpoints.

Non-stop debugging can affect the relative timing of threads
more than
all-stop debugging.  And even if the developer only stops
one thread,
others may stop too, because they're waiting for the
developer's
thread to do something.  But I don't think what you've
described is
deadlock: B is blocked waiting for A, which may take a long
time
(until the developer lets go of it, to be exact) to get
through what
it's doing.  When A resumes, both will resume.

The original concern you raised was that non-stop debugging
is "more
intrusive than we already are".  But clearly all-stop
debugging on a
live system is maximally intrusive to the system's users;
non-stop
debugging has the potential to be much less intrusive, when
used with
knowledge of the interactions between the system's threads.

I think what's bothering you is that the degree of
effective
interference depends on the developer knowing stuff about
the system
under debug; it's not just a flat-out guarantee made by the
debugger:
"no matter what you do, we promise X, or at least
roughly X."  But
there are reasonable debugger use cases that are simply not
possible
at all with all-stop debugging.  We want to support those,
in addition
to the ones we support now.

On Sat, 2007-12-01 at 09:52 -0800, Jim Blandy wrote:
> On Nov 30, 2007 5:30 PM, Michael Snyder <msnyderspecifix.com> wrote:
> The original concern you raised was that non-stop
debugging is "more
> intrusive than we already are".  But clearly
all-stop debugging on a
> live system is maximally intrusive to the system's
users; non-stop
> debugging has the potential to be much less intrusive,
when used with
> knowledge of the interactions between the system's
threads.

There are cases when a developer will want to use non-stop
debugging but
minimize change of relative timing of threads. Suppose that
a developer
is trying to debug a deadlock situation in a program with 3
threads. A
and B are deadlocking, and C is a "supporting"
thread without which the
other two can't run. He can't use all-stop debugging because
while
inspecting A and B, C needs to be running. In this case,
relative timing
of threads is important in order to have better chance at
reproducing
the deadlock.
-- 
[]'s
Thiago Jung Bauermann
Software Engineer
IBM Linux Technology Center