repo to work on python scripting support

Hi folks,

Since Tromey announced this in another thread, it might have
been
missed by some people: Tromey set up a git repo to host our
work of
integrating python into GDB. To play with it:

$ git clone git://git.gitorious.org/gdb-python/mainline.git
$ cd mainline
$ git remote add origin gitgitorious.org:gdb-python/mainline.git
$ git checkout --track -b python-revisited
origin/python-revisited

The current code has some rudimentary support for exposing
value
structs as python objects, some initial frame querying from
python,
evaluating python expressions on the GDB command line (or in
a script),
evaluating GDB commands from python, and some varobj
trickery which is
explained here:

ht
tp://sourceware.org/ml/gdb/2008-02/msg00140.html

There's also some reference to all this on the wiki:

http://sou
rceware.org/gdb/wiki/OngoingWork

And quoting Tromey:

"If you are interested in helping out, and you have an
FSF assignment
on file, I can set you up with write
access."
-- 
[]'s
Thiago Jung Bauermann
Software Engineer
IBM Linux Technology Center

I did some hacking on the Python integration over the last
couple of
days, enough to warrant a status update.

You can evaluate python, either as a block, as a one-liner,
or as an
expression.  I cleaned up the code so that it works properly
in other
block structures, like commands or loops.  The expression
evaluation
code makes an attempt to convert python values to gdb values
as
appropriate (strings, ints, and floating-point types).  Some
contrived
examples...

Block form:

    python
    print 23
    end

One liner:

    python print 23

Expression:

    break foo if $(23 > 5)


There is minimal hook-up to frames.  You can get the name of
the
function at any frame depth and the number of frames.  This
area needs
an OO rewrite.

I haven't tried out the value support at all yet (sorry
Thiago and
Volodya).


Breakpoints are now visible as objects and some of their
properties
are inspectable.  Some properties can be set from Python as
well.
Enough works that I was able to write a simple "save my
breakpoints"
script -- it is appended.

Breakpoints need some more plumbing.  I want to make it so
you can
write: new gdb.Breakpoint('location') and have it do the
right thing.
(This looks pretty easy.)


You can set Python hooks on a few events but this is not
documented
and not tested yet.  There's also a Python command to pass
any string
to the gdb CLI, and another one to get the value of any
"set/show"
variable.


As you can see it is pretty rough still, but I think it is
coming
along rather well.  It is not very hard to add new
functionality.
There are a few things I don't understand about hooking this
code to
gdb, I'll ask in a separate thread.

Comments, constructive criticism, etc, welcome.

Tom

# Set up a python function to do the work.
python
from __future__ import with_statement
import os
def gdb_save_breakpoints(filename):
    with open(filename, 'w') as file:
        for bp in gdb.breakpoints():
    	    if not bp:
	       continue
	    if not bp.is_valid():
	       continue
	    print >> file, "break",
bp.get_location(),
    	    cond = bp.get_condition()
	    if cond:
	       print >> file, " if", cond,
	    print >> file
	    commands = bp.get_commands()
	    if commands:
	       print >> file, "commands"
	       print >> file, commands,
	       print >> file, "end"
	    if not bp.is_enabled():
	       print >> file, "disable $bpnum"
end

# And now the gdb user command.
# Try:  save_breakpoints /tmp/out
define save_breakpoints
dont-repeat
# Silly quoting games...
python gdb_save_breakpoints('$arg0')
end

On Wed, Mar 26, 2008 at 11:16:37AM -0600, Tom Tromey wrote:
> Are you concerned about the parsing problem for
C-defined functions?
> We could push the parsing-style declarations down to
the C level to
> solve that.

No, I'm concerned about what the C-defined functions would
include
or call to interpret the details.  I don't want them
including any
of GDB's normal headers.

> Daniel> print $(py "this is a python
"program"")
> 
> I was thinking some functions could ask for an
uninterpreted string.
> 
>   print $(py do something in python)

But what if there's quotes in there?  Or parentheses, or
both?

-- 
Daniel Jacobowitz
CodeSourcery

On Wed, Mar 26, 2008 at 10:28:53AM -0700, Doug Evans wrote:
> On Mon, Mar 24, 2008 at 10:32 PM, Tom Tromey
<tromeyredhat.com> wrote:
> >  I've been thinking... maybe we do want $(...) to
allow things other
> >  than Python commands.  Or, more precisely, maybe
we want to let Python
> >  code register a function by name for use in
$(...).  The Python
> >  function would get the uninterpreted string
argument for processing.
> 
> OOC, Would there be an intent to not disallow
dynamically loaded C/C++
> instead of Python for $(foo)? [Emphasis on OOC.]

My plan on this, to date, has been to insist on use of the
Python
interface for marshalling and unmarshalling.  This keeps the
interface
explicit.  Of course, Python can load C modules itself and
there'd be
nothing to stop that if you needed to do e.g. some more
intensive
computation in the C side; and with a little forethought we
could
probably keep likely bulky operations (like memory reading)
as
zero-copy.

> Parsing two separate languages in one line is
problematic too (I'm not
> sure how far the Pythonic version was intended to be
interpreted).

Yes, I think we'll not want to allow raw Python in most GDB
expressions; if it's useful in a convenience expression the
outer layer of quoting should probably belong to GDB.

print $(py "this is a python
"program"")

-- 
Daniel Jacobowitz
CodeSourcery

>>>>> "Doug" == Doug Evans
<djegoogle.com> writes:

>> I've been thinking... maybe we do want $(...) to
allow things other
>> than Python commands.  Or, more precisely, maybe we
want to let Python
>> code register a function by name for use in $(...).
 The Python
>> function would get the uninterpreted string
argument for processing.

Doug> OOC, Would there be an intent to not disallow
dynamically loaded C/C++
Doug> instead of Python for $(foo)? [Emphasis on OOC.]

I think once we've decided to do some extra parsing of
$(...), instead
of just passing it verbatim to Python for interpretation, we
can do
whatever we like.  I wouldn't be opposed to being able to
register new
functions implemented in C.

>> break inner if $(gdb.caller_is ('outer'))

Doug> Parsing two separate languages in one line is
problematic too (I'm not
Doug> sure how far the Pythonic version was intended to
be interpreted).

In the current implementation the `...' in $(...) is just
passed to
Python for evaluation.  So, it has to follow Python quoting
rules, for
instance.

This is simple to implement , but I
think yields a not-very-nice
user experience.

Tom

On Tue, Mar 25, 2008 at 12:31 PM, Paul Koning
<Paul_Koningdell.com> wrote:
>  The notion of asking for a particular type is a bit
foreign to Python;
>  arguments have no fixed type.

It's not really about type.  For example, Emacs has about a
dozen
interactive spec code letters that compute string arguments
for the
function.  Some are filenames, some are buffer names, some
are bits of
text...  It's about how to prompt for the arguments.  Or, in
the case
of $() functions in GDB, how to parse the arguments.

>  Another possibility is to pass strings but then have
standard
>  conversion routines (things callable by Python and
supplied by gdb).
>  For example parse_and_eval_address.  And the target
functions.  And so
>  on.
>
>  def walklist (head):
>     addr = parse_and_eval_address (head)
>     while addr:
>           print "list item at", addr
>           addr = target_read_memory (addr, 4)

Using "None" this way means that the function's
arguments are (say)
"address, or None if I should prompt
interactively."

The nice thing about interactive specs is that the function
itself has
a single contract to follow: each of its arguments always
means the
same thing, whether the function is invoked from code or
interactively
by the user.

You may also get interactive prompting by accident, if you
pass the
wrong arguments to the function.

On Mon, Mar 24, 2008 at 11:32:16PM -0600, Tom Tromey wrote:
> I've been thinking... maybe we do want $(...) to allow
things other
> than Python commands.  Or, more precisely, maybe we
want to let Python
> code register a function by name for use in $(...). 
The Python
> function would get the uninterpreted string argument
for processing.

I was thinking about this unrelatedly last night and I
agree, we
should register the arguments independently.  I'm not so
sure about
quoting decisions for the arguments yet, though.  What I
don't want to
do is end up with the GDB CLI again, where everyone who adds
a command
picks a different way to quote the arguments.

Maybe we can handle this the same way that I have been
speculating
about handling CLI commands: register a
"prototype" of the function?
For example, you could register:
  $(strcmp expression, "foo")
as "EXPRESSION, EXPRESSION".  Other functions
might not have the comma
if the preceeding argument was not an EXPRESSION.

-- 
Daniel Jacobowitz
CodeSourcery

On Wed, Mar 26, 2008 at 7:15 AM, Paul Koning
<Paul_Koningdell.com> wrote:
>  It seems to me that there's an attempt here to treat
Python as if it
>  were Elisp.  But it isn't, and it isn't similar.

You're right, Python and Emacs Lisp are not the same, and we
shouldn't
thoughtlessly adapt ideas from Emacs Lisp to GDB Python.

However, Emacs does have a similar situation to the one we
have
here, in that Emacs commands are simultaneously:
- ordinary Emacs Lisp functions callable from Lisp, and
- interactive commands that can be bound to keys or invoked
by name,
  and when this happens need to prompt or otherwise gather
their
  arguments appropriately.

This is directly analogous to what we're doing with GDB.  We
want user
commands and functions to be simultaneously:
- Python functions (or at least objects) that can be used in
the
  ordinary fashion in Python code, and
- either commands that can be typed at the (gdb) prompt, or
functions
  that can be used in GDB expressions using the $(FUNC ARGS)
syntax.

So I think it's a sound analogy that rests on the novel uses
to which
we're putting functions, not on anything specific to Emacs
Lisp.

> In Python, the caller picks the type of the arguments
(implicitly, by
> picking the variables and expressions in the call). 
The called
> function then either examines what it received and
converts as needed,
> or assumes that what it got was what it wanted and uses
exceptions to
> convert as needed, etc.

I'm proposing we continue to use Python in exactly this
way.

> If you add decorations of some sort, or magical methods
in classes, to
> do Elisp-style argument mapping, a Python programmer is
going to look
> at that and say "what language is this?"

Have you written code in Emacs Lisp?  It sounds like you've
gotten the
impression that what's going on is much more magic and
involved than
it really is.

It's possible to write Emacs Lisp functions that work like
the Python
code you posted that checks arguments against None and
prompts for
them if needed.  They would work fine, and interactive users
wouldn't
be able to distinguish them from ordinary Emacs commands.

But Emacs Lisp coders don't write their functions this way:

- They would lose error checking when calling their
functions from
  Lisp: if they forget an argument to the function, the
function will
  prompt interactively instead of raising an error.

- They would lose the ability to use optional arguments in
the way one
  usually does --- when there's an obvious or useful
default, or for
  backwards compatibility.  Instead of using default values,
the
  function would have to prompt for their values
interactively.

- Interactive specs are often terser, and because they're an
idiom
  they're a little easier to read.

>>>>> "Daniel" == Daniel Jacobowitz
<drowfalse.org> writes:

>> OOC, Would there be an intent to not disallow
dynamically loaded C/C++
>> instead of Python for $(foo)? [Emphasis on OOC.]

Daniel> My plan on this, to date, has been to insist on
use of the Python
Daniel> interface for marshalling and unmarshalling. 
This keeps the interface
Daniel> explicit.

Interesting -- since I just said the opposite 

Are you concerned about the parsing problem for C-defined
functions?
We could push the parsing-style declarations down to the C
level to
solve that.

I did want to say to Doug, but forgot -- is this something
you're
actually interested in?  AFAIK you can't load gdb commands
written in
C or C++, so it would be a bit odd to add that feature just
for these
functions.

Not that this matters hugely.  And dynamically loading
commands might
also be useful.

Daniel> print $(py "this is a python
"program"")

I was thinking some functions could ask for an uninterpreted
string.

  print $(py do something in python)

I'm not wedded to it.

Tom

What kind of functions for parsing Python does the Python
API export?
Is there anything that would, say, parse an expression as
far as it
can, and let us know where the expression ends?  If so, we
could use
that for $(py <python code here>), and things like
embedded quotes and
parens would just work.

Python decorators seem to be in the latest python, at
least:
http://docs.
python.org/ref/function.html

The discussion Tom linked to was back in 2004.

Decorators would be perfect for this, I think:

Gdb.interactive (Gdb.Expr, Gdb.Expr)
def strcmp (str1, str2):
    // compare strings

Easy squeezy.

Hi everybody,

My turn at the status update. 

I did the OO rewrite for the frames support, it has a few
useful methods
already (equals, is_inner_than, get_name, get_pc, get_prev,
get_next,
find_sal). You can also call gdb.frames(), which will return
a tuple
containing all frames in the stack (or throw an exception if
there's a
problem with the unwinding).

I also exposed a little bit of struct symtab and struct
symtab_and_line
(which is returned by Frame.find_sal), just enough to be
able to
implement a mostly complete backtrace command in Python
(it's only
missing ability to print function arguments, which I plan to
work on
later).

Regarding the values work that Tromey mentioned, I didn't
add
functionality to the code that Volodya implemented, just
converted it to
OO form and worked on lifetime of struct value vs lifetime
of Python
Value object (I believe I is correct now).

Here's the Python implementation of an "rbt"
command which prints the
backtrace in reverse (or right :-P) order:

python
import sys

def gdb_rbacktrace ():
  frames = gdb.frames ()
  for i in reversed (range (len (frames))):
    sal = frames[i].find_sal ()
    pc = frames[i].get_pc ()

    sys.stdout.write ("#%-2d" % i)
    if pc != sal.get_pc () or not sal.symtab:
      sys.stdout.write (" 0x%08x in" % pc)
    sys.stdout.write (" " + frames[i].get_name
())
    if sal.symtab and sal.symtab.get_filename ():
      sys.stdout.write (" at " +
sal.symtab.get_filename ())
      sys.stdout.write (":" + str (sal.get_line
()))
    if not frames[i].get_name () or (not sal.symtab or not
sal.symtab.get_filename ()):
       lib = gdb.solib_address (pc)
       if lib:
         sys.stdout.write (" from " + lib)
    sys.stdout.write ("n")
end

define rbt
dont-repeat
python gdb_rbacktrace ()
end

Here's sample output:

(gdb) rbt
#2  0x080483bb in main at ../../src/examples/funcs.c:15
#1  0x08048391 in f1 at ../../src/examples/funcs.c:10
#0  f2 at ../../src/examples/funcs.c:5
(gdb)
-- 
[]'s
Thiago Jung Bauermann
Software Engineer
IBM Linux Technology Center

On Wed, 2008-03-26 at 11:44 -0600, Tom Tromey wrote:
> Long-term compatibility in the Python API is something
we haven't
> discussed.  I'm really not sure what to do about this
-- we really are
> exposing a decent amount of gdb internals to the world
this way.

We are, but at least up until now we've been exposing things
which are
there in GDB for a long time now (struct breakpoint, struct
frame_info,
struct value), and attributes which are fairly intrinsic to
them, like
frame.get_pc(). I expect this stuff to be stable between
releases.

I also expect that most (if not all) of what is exposed is
along these
lines, that is, providing a "view" of the general
GDB internal
components, as oposed to exposing exoteric internal GDB code
which is
likely to change often.

Since the C <-> Python glue code provide some
decoupling, it's possible
to accomodate for some changes in the GDB internals.

All in all, my hope is that compatibility of python scripts
accross GDB
versions won't be a frequent issue. We could provide a
method which
returns the GDB versions so that scripts can accomodate for
changes, if
the need arises?
-- 
[]'s
Thiago Jung Bauermann
Software Engineer
IBM Linux Technology Center

>>>>> "Jim" == Jim Blandy
<jimbred-bean.com> writes:

 Jim> It's worth putting some investment into the way
arguments get
 Jim> handled, to make writing commands as easy as
possible. 

 Jim> I don't know the most Pythonish way of doing
things, but it
 Jim> seems to me that Python could register not
functions, but
 Jim> objects, to be callable with $(). 

You can make objects callable, but I don't know if that's
the answer.
Calling a class object is how you construct instances, by
the way...

An obvious Pythonic way of doing this is with default
arguments, which
allows a call to omit some of the arguments defined for the
function.

For example:

    def foo (arg1, arg2=None):
	if arg2 is None:
	   arg2 = raw_input ("please supply arg2: ")

   paul

>>>>> "Tom" == Tom Tromey
<tromeyredhat.com> writes:

 Tom> The basic problem is that we have a syntax for
embedding a
 Tom> python call in an expression that looks like
$(stuff).

 Tom> Now, internally to gdb, "stuff" is just a
string.  But, most of
 Tom> the time, the implementation of this function,
whatever it is,
 Tom> won't want just a string -- it will want an
expression, or a
 Tom> file name, or something.

 Tom> So, what Jim and Daniel want, I think, is a
declarative way for
 Tom> the Python code (which implements the given
function) to tell
 Tom> gdb's core how to parse this string.

Something akin to the way that C extension modules inside
Python tell
the Python execution machinery what data type it wants might
serve.

The notion of asking for a particular type is a bit foreign
to Python;
arguments have no fixed type.

Another possibility is to pass strings but then have
standard
conversion routines (things callable by Python and supplied
by gdb).
For example parse_and_eval_address.  And the target
functions.  And so
on.

def walklist (head):
    addr = parse_and_eval_address (head)
    while addr:
	  print "list item at", addr
	  addr = target_read_memory (addr, 4)

   paul

On Tue, Mar 25, 2008 at 11:18:12AM -0700, Jim Blandy wrote:
> I don't know the most Pythonish way of doing things,
but it seems to
> me that Python could register not functions, but
objects, to be
> callable with $().  The object could have an
'interactive' member to
> specify how to call it (say, "e" might mean,
evaluate an expression
> and pass its value to me, and "s" might mean,
just hand it to me as a
> string), and then some other method to actually do the
call.

Hmm, I've worked with some packages that did similar things
using
mandatory docstrings.

def strcmp (expr1, expr2):
  """strcmp: EXPR, EXPR

  Compare expr1 and expr2 as strings."""
  doit

Dunno if that's wise.

-- 
Daniel Jacobowitz
CodeSourcery

On Wed, Mar 26, 2008 at 11:44:39AM -0600, Tom Tromey wrote:
> Maybe we could do the same kind of parsing for
commands.

Yep.

The only part of this I'm concerned about is separators. 
Basically,
an expression can be followed only by a comma or the end of
arguments.
If you want just a number or address, though, you can use
spaces.

Maybe we should always use commas to simplify.

-- 
Daniel Jacobowitz
CodeSourcery

>>>>> "Paul" == Paul Koning
<Paul_KoningDell.com> writes:

Paul> An obvious Pythonic way of doing this is with
default arguments, which
Paul> allows a call to omit some of the arguments defined
for the function.
Paul> For example:
Paul>     def foo (arg1, arg2=None):
Paul> 	if arg2 is None:
Paul> 	   arg2 = raw_input ("please supply arg2:
")

I think default arguments are part of the solution, but not
the whole
solution.

The basic problem is that we have a syntax for embedding a
python call
in an expression that looks like $(stuff).

Now, internally to gdb, "stuff" is just a string. 
But, most of the
time, the implementation of this function, whatever it is,
won't want
just a string -- it will want an expression, or a file name,
or
something.

So, what Jim and Daniel want, I think, is a declarative way
for the
Python code (which implements the given function) to tell
gdb's core
how to parse this string.

I think the hope here is that we avoid the mess of gdb
commands, which
really do get just a plain string and then proceed to be
inconsistent
with each other and just generally hard to modify.


It would be interesting to see if we could enumerate the
core
possibilities for argument types.

Tom

On Mon, Mar 24, 2008 at 10:32 PM, Tom Tromey <tromeyredhat.com> wrote:
>  I've been thinking... maybe we do want $(...) to allow
things other
>  than Python commands.  Or, more precisely, maybe we
want to let Python
>  code register a function by name for use in $(...). 
The Python
>  function would get the uninterpreted string argument
for processing.

OOC, Would there be an intent to not disallow dynamically
loaded C/C++
instead of Python for $(foo)? [Emphasis on OOC.]

>  My reason is that after playing a bit, I'm not
over-fond of the
>  verboseness and extra quoting required by the Python
bits.  It is more
>  gdb-ish, I think, not to do this.
>
>  Any opinions?
>
>  The canonical example is a function to match the
calling function at a
>  breakpoint.  gdb-ish:
>
>     break inner if $(caller-is outer)
>
>  Pythonic:
>
>     break inner if $(gdb.caller_is ('outer'))

Parsing two separate languages in one line is problematic
too (I'm not
sure how far the Pythonic version was intended to be
interpreted).

>>>>> "Daniel" == Daniel Jacobowitz
<drowfalse.org> writes:

Daniel> No, I'm concerned about what the C-defined
functions would include
Daniel> or call to interpret the details.  I don't want
them including any
Daniel> of GDB's normal headers.

Gotcha, that makes sense to me.

Long-term compatibility in the Python API is something we
haven't
discussed.  I'm really not sure what to do about this -- we
really are
exposing a decent amount of gdb internals to the world this
way.

>> print $(py do something in python)

Daniel> But what if there's quotes in there?  Or
parentheses, or both?

Yeah.  They have to match and if gdb does any processing (I
didn't
look into this), you have to know how to trick it.  Talk
about
unsatisfactory...  OTOH double-quoting the python code is
also a bit
yucky.  But, I suppose this will be a relatively unusual
thing to want
to do.

I'll try to implement this soon.

So far my list of declarative argument "kinds"
is:

* expression
* linespec (or whatever the thing that "break"
takes is called)
* filename (do we need it?)

I was thinking that the declaration could be a tuple using
named
constants; string elements in the tuple would be looked for
explicitly, like: (EXPRESSION "," EXPRESSION)
would parse an
expression, a comma (ignoring whitespace around it), and
another
expression.

I guess we need a syntax for optional arguments as well,
perhaps just
a named 'OPTIONAL' constant in the tuple.

Maybe we could do the same kind of parsing for commands.

Tom