libSegFault and just in time debugging

Got an idea to pass in front of you...

If you don't know how libSegFault works, it is a dynamic
library that
comes packaged with glibc.  To use it, you get the runtime
loader to
preload it and link it to your programs, using either the
environment
variable LD_PRELOAD or the file /etc/ld.so.preload.  Upon
loading,
it takes over the signal handling vectors for half a dozen
fatal signals
(SEGV, BUS, ILL, FPU...), but it does so *before* the main
user program
is invoked -- so there's no interferance with programs that
want to use
those vectors themselves.  If the program *doesn't* handle
the signal,
then libSegFault catches it and prints some useful debugging
info to
the system console or syslog (before passing the signal back
to the
kernel so that a core file can be generated).

What if, instead of doing that, libSegFault (or another
similar library)
were to open a socket to a daemon and say "I caught a
crash -- what
do you want me to do?".  And then wait for a reply. 
All that can be
done with async-signal-safe function calls.

The daemon, because it is NOT in a signal handler, can do
anything
it wants, eg. open a GUI window and say "Program XYZ
has crashed:
would you like to debug it?"  User clicks a button,
daemon fires up
gdb, attaches to crashing program, then responds to the
signal handler
library with a message saying "get out of the way,
please...".  The
library removes itself from the signal vector, re-raises the
signal,
and returns.  And gdb finds itself at the point where the
signal was
originally raised.

If the user doesn't want to debug it, the daemon can tell
the library
to just let it crash, or to log the information a la
libSegFault, or
any number of other things.

Interesting?

On Fri, Jun 29, 2007 at 12:10:32PM -0700, Michael Snyder
wrote:
> What if, instead of doing that, libSegFault (or another
similar library)
> were to open a socket to a daemon and say "I
caught a crash -- what
> do you want me to do?".  And then wait for a
reply.  All that can be
> done with async-signal-safe function calls.

It's a brilliant idea.  Ubuntu did it   It uses
the Linux kernel's
core handling support, and is called apport.

  https://wiki.ubuntu.co
m/Apport

You'll notice a bit of a way down that there's GDB output;
in this
case it comes from a core dump, but I'm pretty sure the same
kernel
hooks can be used to take control before the core is dumped.
 I'd have
to check.

-- 
Daniel Jacobowitz
CodeSourcery

> On Fri, Jun 29, 2007 at 12:10:32PM -0700, Michael
Snyder wrote:
> > What if, instead of doing that, libSegFault (or
another similar library)
> > were to open a socket to a daemon and say "I
caught a crash -- what
> > do you want me to do?".  And then wait for a
reply.  All that can be
> > done with async-signal-safe function calls.
>
> It's a brilliant idea.  Ubuntu did it   It uses
the Linux kernel's
> core handling support, and is called apport.

Yes, apport is slick, but it relies on kernel mods.
This doesn't.  In fact, it isn't even peculiar to Linux, it
would
work on any glibc system, and in principle even on systems
that don't use glibc.  Probably any unix, and even cygwin.

On Fri, Jun 29, 2007 at 12:52:42PM -0700, Michael Snyder
wrote:
> Yes, apport is slick, but it relies on kernel mods.
> This doesn't.  In fact, it isn't even peculiar to
Linux, it would
> work on any glibc system, and in principle even on
systems
> that don't use glibc.  Probably any unix, and even
cygwin.

FYI, cygwin already does this, too; it can invoke dumper (to
generate core
dumps) or GDB.  I imagine you could invoke apport from a
preloaded
library easily; after all, you can hook into it from Python
exceptions.

-- 
Daniel Jacobowitz
CodeSourcery

We did this with Mac OS X for a while (the way Mach
Exceptions work  
it's a little easier to implement, you don't need a special
library or  
anything like that).  It's pretty neat.  We were doing it  
automatically with all processes on the system, which is
even handier,  
but you do have to be careful - if you are attaching with
gdb rather  
than dumping core
On Jun 29, 2007, at 1:00 PM, Daniel Jacobowitz wrote:

> On Fri, Jun 29, 2007 at 12:52:42PM -0700, Michael
Snyder wrote:
>> Yes, apport is slick, but it relies on kernel
mods.
>> This doesn't.  In fact, it isn't even peculiar to
Linux, it would
>> work on any glibc system, and in principle even on
systems
>> that don't use glibc.  Probably any unix, and even
cygwin.
>
> FYI, cygwin already does this, too; it can invoke
dumper (to  
> generate core
> dumps) or GDB.  I imagine you could invoke apport from
a preloaded
> library easily; after all, you can hook into it from
Python  
> exceptions.
>
> -- 
> Daniel Jacobowitz
> CodeSourcery

Oops, hit send too soon...

Meant to say if you are automatically attaching to
everything on the  
system, you have to be careful about auto-launched daemons,
because  
they won't die all the way while waiting for the connection
to the  
debugger, and that can cause them not to get restarted.  Not
a problem  
in general with released systems, where hopefully the
daemons aren't  
crashing all that often.  But with development systems it
could cause  
a problem...

Jim

On Jun 29, 2007, at 1:12 PM, Jim Ingham wrote:

> We did this with Mac OS X for a while (the way Mach
Exceptions work  
> it's a little easier to implement, you don't need a
special library  
> or anything like that).  It's pretty neat.  We were
doing it  
> automatically with all processes on the system, which
is even  
> handier, but you do have to be careful - if you are
attaching with  
> gdb rather than dumping core
> On Jun 29, 2007, at 1:00 PM, Daniel Jacobowitz wrote:
>
>> On Fri, Jun 29, 2007 at 12:52:42PM -0700, Michael
Snyder wrote:
>>> Yes, apport is slick, but it relies on kernel
mods.
>>> This doesn't.  In fact, it isn't even peculiar
to Linux, it would
>>> work on any glibc system, and in principle even
on systems
>>> that don't use glibc.  Probably any unix, and
even cygwin.
>>
>> FYI, cygwin already does this, too; it can invoke
dumper (to  
>> generate core
>> dumps) or GDB.  I imagine you could invoke apport
from a preloaded
>> library easily; after all, you can hook into it
from Python  
>> exceptions.
>>
>> -- 
>> Daniel Jacobowitz
>> CodeSourcery
>

[Daniel Jacobowitz]
> FYI, cygwin already does this, too; it can invoke
dumper (to
> generate core dumps) or GDB.

That's right -- I'd forgotten about that.

[Jim Ingham:]

> We did this with Mac OS X for a while (the way Mach
Exceptions work
> it's a little easier to implement, you don't need a
special library or
> anything like that).  It's pretty neat.  We were doing
it
> automatically with all processes on the system, which
is even handier,
> but you do have to be careful - if you are attaching
with gdb rather
> than dumping core.

Of course the old Mac model ("macsbug"?) is the
classic instance
of just in time debugging.  And then (surprise), Windows had
it too!

I'm just wondering if linux/unix users might not like to
have the same
thing.

Michael

> Date: Fri, 29 Jun 2007 16:00:00 -0400
> From: Daniel Jacobowitz <drowfalse.org>
> Cc: gdbsourceware.org
> 
> FYI, cygwin already does this, too; it can invoke
dumper (to generate core
> dumps) or GDB.

By Cygwin does this using a Windows-specific technique,
doesn't it?

> Date: Sat, 30 Jun 2007 13:52:35 +0300
> From: Eli Zaretskii <elizgnu.org>
> CC: msnydersonic.net, gdbsourceware.org
> 
> By Cygwin does this using a Windows-specific technique,
doesn't it?
  ^^

Meant to say "But", of course.