plugin interface for GDB

Hi folks,

I am interested in developing a plugin interface for GDB,
and would
like to know what are your thoughts about it. Would you find
such
feature useful?  Would it be integrated into GDB?

This is useful to me, so I'll probably do it even if there's
no plan
to integrate the plugin interface upstream (in fact, I have
a very
basic working prototype already). But if there is community
interest in
such thing, I'd like to do it in a way that will increase
its chance of
integration and shape it so that more people could benefit
from it.

I saw some discussion on this subject in the following
thread:

ht
tp://sourceware.org/ml/gdb/2002-04/msg00371.html

But it didn't come to a conclusion, so I'd like to revisit
the idea.

Motivation for having a plugin interface:

- enable seamless debugging of programs written in multiple
languages
  (Java + C, Python + C, etc) (in fact, this is the reason I
need the
  plugin interface.)

- useful for specialized debugging (specific problem
domain)

- lessens the need to maintain custom GDBs. You just use a
standard GDB
  and write a plugin (or use an existing one). If you decide
to upgrade
  GDB, you won't need to port your code and not even
recompile the
  plugin (ideally, if we manage to design a resilient
interface).

- allow people outside of the core GDB team to implement
distribute and
  test their plugins without modifying GDB.
  
- lowers the bar to creating modifications to GDB (exposes a
well
  defined interface, so there's no need to learn the ins and
outs of
  GDB to write the plugin you need/want)

Scott Moser's idea was to have very basic plugin support
(loading and
unloading), and leave to the plugin writer the work of
searching which
internal GDB functions he'd need to use and directly call
them from the
plugin code. I am more inclined to go into a different
direction, which
is to provide an abstraction layer which would expose
functionality
which is potentially useful for a plugin.

Each approach has advantages and disadvantages, but some of
the
advantages I pointed out above only hold if you have such
abstraction
layer.
-- 
[]'s
Thiago Jung Bauermann
Software Engineer
IBM Linux Technology Center

Thiago Jung Bauermann wrote:

> Scott Moser's idea was to have very basic plugin
support (loading and
> unloading), and leave to the plugin writer the work of
searching which
> internal GDB functions he'd need to use and directly
call them from the
> plugin code. I am more inclined to go into a different
direction, which
> is to provide an abstraction layer which would expose
functionality
> which is potentially useful for a plugin.

You should read the recent threads about adding scripting
support to
gdb.  The one from several months ago
<http://sourceware.org/ml/gdb/2007-01/threads.html#001
26> seemed to
mostly converge on Python, with tcl and guile as runners
up.

To me it seems like it would be a lot easier to go in that
direction
than to actually allow for plugins in the sense of
dynamically loading a
shared module of compiled code.  In either case, you have to
implement
some kind of API that exposes the internals of gdb to the
plugin or the
script, and in either case you can extend the functionality
of gdb in
pretty much arbitrary ways.  But in the case of scripting
you don't have
ABI headaches to worry about, and  it increases
accessilbility since you
can express relatively high level concepts easily in just a
short amount
of python script, compared to having to code the same amount
of
functionality with a C plugin.

Brian

On Fri, Apr 20, 2007 at 04:43:48PM -0300, Thiago Jung
Bauermann wrote:
> I am interested in developing a plugin interface for
GDB, and would
> like to know what are your thoughts about it. Would you
find such
> feature useful?  Would it be integrated into GDB?

I don't think it very likely that we would add a C plugin
interface.
We've discussed the disadvantages many times before.  But at
the same
time, I'm very interested in a "plugin"-like
system using a scripting
language.  You can find a discussion about using Python
earlier in
this year's archives.

In my opinion the main difference is the character of the
interface;
using a scripting language has plenty of inherent
advantages, plus it
makes it clear that you do not get to grub around in any
other bit of
GDB, period, no matter how clever you think you are.

> - enable seamless debugging of programs written in
multiple languages
>   (Java + C, Python + C, etc) (in fact, this is the
reason I need the
>   plugin interface.)

What does this have to do with a plugin?  That is, why is a
plugin
useful for it, rather than additional support in GDB proper?
 It's a
very vague problem statement - an interesting one, though.

> - lessens the need to maintain custom GDBs. You just
use a standard GDB
>   and write a plugin (or use an existing one). If you
decide to upgrade
>   GDB, you won't need to port your code and not even
recompile the
>   plugin (ideally, if we manage to design a resilient
interface).
> 
> - allow people outside of the core GDB team to
implement distribute and
>   test their plugins without modifying GDB.
>   
> - lowers the bar to creating modifications to GDB
(exposes a well
>   defined interface, so there's no need to learn the
ins and outs of
>   GDB to write the plugin you need/want)

Remember, we want GDB contributors!  That's the whole point
- getting
people to contribute.  So for things that would be generally
useful,
which all of the above are, helping people do them without
contributing to GDB is entirely the wrong way round.

I do not believe that you can come up with an abstraction
layer big
enough to be useful that is not a significant maintenance
burden for
the GDB developers.  I've been surprised before, though!

-- 
Daniel Jacobowitz
CodeSourcery

On Fri, 2007-04-20 at 16:00 -0400, Daniel Jacobowitz wrote:
> On Fri, Apr 20, 2007 at 04:43:48PM -0300, Thiago Jung
Bauermann wrote:
> > I am interested in developing a plugin interface
for GDB, and would
> > like to know what are your thoughts about it.
Would you find such
> > feature useful?  Would it be integrated into GDB?
> 
> I don't think it very likely that we would add a C
plugin interface.
> We've discussed the disadvantages many times before. 
But at the same
> time, I'm very interested in a "plugin"-like
system using a scripting
> language.  You can find a discussion about using Python
earlier in
> this year's archives.

Yes, I saw those. Thanks for pointing them out, though. I
think it's
very interesting and useful as well.

> In my opinion the main difference is the character of
the interface;
> using a scripting language has plenty of inherent
advantages, plus it
> makes it clear that you do not get to grub around in
any other bit of
> GDB, period, no matter how clever you think you are.

I agree that a plugin interface and a scripting language
overlap a lot
in terms of functionality. I'm not sure it would cut it in
my specific
case... I'm looking at something which will potentially need
to examine
10's of GB of inferior memory in some cases and I'm a bit
worried about
the performance. If there was already scripting language
support in GDB,
I could do some benchmarks. It's quite possible that the
overhead of the
scripting language would be low or acceptable. But right now
I don't
want to work (probably a lot) on developing such support
only to find
out it won't meet my performance needs... 

> > - enable seamless debugging of programs written in
multiple languages
> >   (Java + C, Python + C, etc) (in fact, this is
the reason I need the
> >   plugin interface.)
> 
> What does this have to do with a plugin?  That is, why
is a plmaybe ugin
> useful for it, rather than additional support in GDB
proper?

We want to add support in GDB to debug both IBM's JVM and
programs
running on top of it at the same time. It could go into GDB
proper,
except that we need to keep our modifications internal and I
want to
minimize the burden of porting the code to new GDB releases,
so I don't
want to mess too much with the guts of GDB. (objectionable
motivation, I
know. But these are the constraints I need to work with).

As I said, a plugin interface is a good answer to that need,
and I'll
probably do such interface even if only to keep it
internal.

I believe there are other people in the same situation as
mine. I've
seen messages here stating something like "I'm using an
old version of
GDB because I customized it and still didn't port the code
to a newer
version". A plugin interface would help those people.

Granted, a scripting language would possibly solve their
needs, but
maybe not for all of them. Knowing more reasons which lead
to folks
getting into the trouble of maintaining an internal GDB
version would
probably help this discussion. I can only speculate...

>   It's a vague problem statement - an interesting one,
though.

Think Java code which calls native methods via JNI (very
common), or the
other way around: a C or C++ program which fires up a JVM.
It's
cumbersome to debug the combination today. You can't have a
backtrace
which shows both the interpreted language and the native
language
frames, you can't step into a function which is written in
the other
language.. You can't define some breakpoints in the
interpreted language
part and others in the C part. And so on.

I agree, it's a very interesting problem, and it mostly
doesn't need a
plugin interface to be implemented. But there might be a few
little
known or unpopular interpreted languages for which someone
might have an
interest in writing such support, but then wouldn't be
accepted upstream
for one reason or another (e.g., because GDB maintainers
could think
it's not worth the effort to keep it in mainline
(hypotetical situation,
of course!  )).

> > - lessens the need to maintain custom GDBs. You
just use a standard GDB
> >   and write a plugin (or use an existing one). If
you decide to upgrade
> >   GDB, you won't need to port your code and not
even recompile the
> >   plugin (ideally, if we manage to design a
resilient interface).
> > 
> > - allow people outside of the core GDB team to
implement distribute and
> >   test their plugins without modifying GDB.
> >   
> > - lowers the bar to creating modifications to GDB
(exposes a well
> >   defined interface, so there's no need to learn
the ins and outs of
> >   GDB to write the plugin you need/want)
> 
> Remember, we want GDB contributors!  That's the whole
point - getting
> people to contribute.  So for things that would be
generally useful,
> which all of the above are, helping people do them
without
> contributing to GDB is entirely the wrong way round.

Agreed. But like in the example I gave above, there might be
some
features which don't interest the GDB community for one
reason or
another, but other people need or want.

I don't think this would turn people away from contributing
to GDB,
because things which make sense to be in GDB proper would
still go
there. It's less work to have a patch accepted into GDB
mainline than
have to maintain one's own plugin. Besides, the integrated
patch will
reach more people (all GDB users, as opposed to users of the
plugin).
But for the things which can't go into GDB for one reason or
another,
it's a good option.

In fact, this could even attract developers because of the
"lower the
bar" point. Developing a plugin would help people get
familiar with GDB
and also instill courage to start hacking on it.

> I do not believe that you can come up with an
abstraction layer big
> enough to be useful that is not a significant
maintenance burden for
> the GDB developers.  I've been surprised before,
though!

I think you'd be surprised. 

I was. In a very short time we were able to come up with a
working
prototype which provides very little functionality
(basically, just
interaction with the user and peek inferior memory) but at
the same time
is enough to meet some of our needs.

Of course there's a long way to go from there, and code will
start
getting more complex. But if it's useful enough, the
maintenance burden
is bearable.

BTW, Apple did write a plugin interface for GDB. It's
available from
their Darwin project website, if you're curious:

http://www.opens
ource.apple.com/darwinsource/DevToolsAug2006/gdb-563/macsbug
/gdb_plugin_support/

-- 
[]'s
Thiago Jung Bauermann
Software Engineer
IBM Linux Technology Center

On Fri, 2007-04-20 at 12:57 -0700, Brian Dessent wrote:
> > Scott Moser's idea was to have very basic plugin
support (loading and
> > unloading), and leave to the plugin writer the
work of searching which
> > internal GDB functions he'd need to use and
directly call them from the
> > plugin code. I am more inclined to go into a
different direction, which
> > is to provide an abstraction layer which would
expose functionality
> > which is potentially useful for a plugin.
> 
> You should read the recent threads about adding
scripting support to
> gdb.  The one from several months ago
> <http://sourceware.org/ml/gdb/2007-01/threads.html#001
26> seemed to
> mostly converge on Python, with tcl and guile as
runners up.
> 
> To me it seems like it would be a lot easier to go in
that direction
> than to actually allow for plugins in the sense of
dynamically loading a
> shared module of compiled code.

I saw that thread. It's certainly very desirable
functionality. My
problem is that maybe it would meet my needs, maybe not.
Like I just
mentioned in a message to Daniel, we'll be looking at lots
of data. I'm
not sure what would be the performance of that...

Also, we have a lot of C code which we'll integrate with
GDB, so a layer
of interpreted language is unnecessary to us and just adds
complexity.

>   In either case, you have to implement
> some kind of API that exposes the internals of gdb to
the plugin or the
> script, and in either case you can extend the
functionality of gdb in
> pretty much arbitrary ways.

Agreed. Both features overlap a lot in terms of
functionality provided
to users, but I'm afraid they are not completely equivalent.
I think
this discussion is a good opportunity to try to determine
this.

>   But in the case of scripting you don't have
> ABI headaches to worry about,

You mean keeping binary compatibility with plugins accross
different GDB
versions? That's a nice goal, but not worth worrying about
too much, I
think.

>  and  it increases accessilbility since you
> can express relatively high level concepts easily in
just a short amount
> of python script, compared to having to code the same
amount of
> functionality with a C plugin.

Agreed.
-- 
[]'s
Thiago Jung Bauermann
Software Engineer
IBM Linux Technology Center

On Fri, Apr 20, 2007 at 06:21:11PM -0300, Thiago Jung
Bauermann wrote:
> I agree that a plugin interface and a scripting
language overlap a lot
> in terms of functionality. I'm not sure it would cut it
in my specific
> case... I'm looking at something which will potentially
need to examine
> 10's of GB of inferior memory in some cases and I'm a
bit worried about
> the performance. If there was already scripting
language support in GDB,
> I could do some benchmarks. It's quite possible that
the overhead of the
> scripting language would be low or acceptable. But
right now I don't
> want to work (probably a lot) on developing such
support only to find
> out it won't meet my performance needs... 

Well, it's going to get developed eventually in either case.
 If you
can't write things fast enough in Python, Python itself lets
you load
C modules to do the real work; the question is how fast GDB
can
provide memory access to the Python layer.  And I think the
answer is
roughly as fast as GDB could get at it directly.

> We want to add support in GDB to debug both IBM's JVM
and programs
> running on top of it at the same time. It could go into
GDB proper,
> except that we need to keep our modifications internal
and I want to
> minimize the burden of porting the code to new GDB
releases, so I don't
> want to mess too much with the guts of GDB.
(objectionable motivation, I
> know. But these are the constraints I need to work
with).

Not surprising constraints.  The trick, for you and for us,
is to
minimize the amount of it is local to IBM and specific to
the JVM.

> As I said, a plugin interface is a good answer to that
need, and I'll
> probably do such interface even if only to keep it
internal.
> 
> I believe there are other people in the same situation
as mine. I've
> seen messages here stating something like "I'm
using an old version of
> GDB because I customized it and still didn't port the
code to a newer
> version". A plugin interface would help those
people.

This, you see, is the part I don't believe.  I think that a
_perfect_
plugin interface would help.  You get to draw your own
conclusions on
how likely that is 

> > I do not believe that you can come up with an
abstraction layer big
> > enough to be useful that is not a significant
maintenance burden for
> > the GDB developers.  I've been surprised before,
though!
> 
> I think you'd be surprised. 
> 
> I was. In a very short time we were able to come up
with a working
> prototype which provides very little functionality
(basically, just
> interaction with the user and peek inferior memory) but
at the same time
> is enough to meet some of our needs.
> 
> Of course there's a long way to go from there, and code
will start
> getting more complex. But if it's useful enough, the
maintenance burden
> is bearable.

The question is what you need, and how easy it is to expose
in another
way.  I did a prototype of Guile integration in just a
couple of days
that did basically the same amount you've got here; and if
I'd done it
in Python (being a language I can actually write things in,
unlike the
staggering around I do in Guile), it could even have been
useful.

-- 
Daniel Jacobowitz
CodeSourcery

>>>>> "Daniel" == Daniel Jacobowitz
<drowfalse.org> writes:

 Daniel> On Fri, Apr 20, 2007 at 06:21:11PM -0300, Thiago
Jung
 Daniel> Bauermann wrote:
 >> I agree that a plugin interface and a scripting
language overlap a
 >> lot in terms of functionality. I'm not sure it
would cut it in my
 >> specific case... I'm looking at something which
will potentially
 >> need to examine 10's of GB of inferior memory in
some cases and
 >> I'm a bit worried about the performance. If there
was already
 >> scripting language support in GDB, I could do
some
 >> benchmarks. It's quite possible that the overhead
of the scripting
 >> language would be low or acceptable. But right now
I don't want to
 >> work (probably a lot) on developing such support
only to find out
 >> it won't meet my performance needs... 

 Daniel> Well, it's going to get developed eventually in
either case.
 Daniel> If you can't write things fast enough in Python,
Python
 Daniel> itself lets you load C modules to do the real
work; the
 Daniel> question is how fast GDB can provide memory
access to the
 Daniel> Python layer.  And I think the answer is roughly
as fast as
 Daniel> GDB could get at it directly.

Very likely.  "Scripting language" often makes
people expect low
performance, and for some that may be true (shell scripts? 
Tcl
perhaps).

Python is very fast.  It compiles to bytecode, and the
execution
machinery seems to work quite well.  I have an application
that runs
through very large files, doing some simple string matching.
 It runs
at 50 MB/s or better, on a PC.

So I don't think speed will be a concern.

   paul

Paul Koning wrote:

> Python is very fast.  It compiles to bytecode, and the
execution
> machinery seems to work quite well.  I have an
application that runs
> through very large files, doing some simple string
matching.  It runs
> at 50 MB/s or better, on a PC.

And in fact Python could be a lot faster. In a class project
we did
a proof of concept project to build a Python using the same
approach
as was used for SPITBOL, compiling to real executable code.
Python
programs ran 5-50 times faster than current interpretors.
Maybe one
day