(linux/threads) Interesting side-effect of "auto-solib-add 0"

Hello,

One of our customers noticed that when you set
auto-solib-add to off
and then start your program, GDB usually breaks down like
so:

    (gdb) c
    Continuing.
    
    Program terminated with signal SIGTRAP, Trace/breakpoint
trap.
    The program no longer exists.

This is on x86-linux.

The important clue here is that the program is using
threads, and
that turning off auto-solib-add causes GDB to not push the
thread
support module.

I did my analysis with a debugger derived from gdb-6.4. With
the
current GDB from CVS, the errors are not so obvious - at
least
I can't seem to reproduce them. But I did double-check that
GDB
indeed doesn't start the thread module (by checking with
another
debugger the value of "current_target") either.

 From what I can tell, the thread support is pushed by
check_for_thread_db
which is called (through several calls) by
solib_read_symbols, which
itself is called by solib_add:

    for (gdb = so_list_head; gdb; gdb = gdb->next)
      if (! pattern || re_exec (gdb->so_name))
        {
          any_matches = 1;
          if (readsyms && solib_read_symbols (gdb,
from_tty))
            loaded_any_symbols = 1;
        }

The call to check_for_thread_db occurs through a hook:

        /* Call predecessor on chain, if any.  */
        if (remote_new_objfile_chain != 0 &&
            remote_desc == 0)
          remote_new_objfile_chain (objfile);

Which is set as follow in linux-thread-db.c:

        /* Add ourselves to objfile event chain.  */
        target_new_objfile_chain =
deprecated_target_new_objfile_hook;
        deprecated_target_new_objfile_hook =
thread_db_new_objfile;

and remote.c:

        /* Hook into new objfile notification.  */
        remote_new_objfile_chain =
deprecated_target_new_objfile_hook;
        deprecated_target_new_objfile_hook  =
remote_new_objfile;

(ugh! another candidate for a new observer???)

So basically, the linux thread layer hooks itself up to the
objfile
creation callback. But unfortunately for us, the objfile for
libpthread.so does not get created since we turned
auto-solib-add off,
the hook doesn't get called, and as a result, thread
support stays off.

It seems to me that we should be able to fix this by adding
an observer
of the `solib_loaded' event to replace the hook that
linux-thread-db
installed. I actually gave it a try, using the sources from
CVS, and
found out something surprising: Although the solib_loaded
observer
I just created gets called as expected, the linux thread
layers does
not detect that threads are in use:

        /* Now attempt to open a connection to the thread
library.  */
        err = td_ta_new_p (&proc_handle,
&thread_agent);
        switch (err)
          {
          case TD_NOLIBTHREAD:
            /* No thread library was detected.  */
            break;

For some reason that I cannot understand,
"td_ta_new" doesn't detect
thread usage if this function is called from the observer.

I made another experiment: Still with the old hook
deactivated, I added
the following loop at the end of "solib_add",
which basically causes
a second round of notifications after solib_add had time to
do its
thing:

        so = so_list_head;
        while (so != NULL)
          {
            observer_notify_solib_loaded (so);
            so = so->next;
          }

I first did a run with "auto-solib-add"
activated (unchanged), and
linux-thread-db.c DID detect thread usage. I then turned off
auto-solib-add, and this time thread usages was NOT
detected.

The only difference I can see between the two runs is that
symbols
are read in one case, and not read in the other case. Is
libthread_db.so
dependent on having the symbol table being loaded or
something of that
sort? Any idea as to using the solib_loaded observer didn't
work?

Thanks,
-- 
Joel

On Tue, Sep 12, 2006 at 05:27:13PM -0700, Joel Brobecker
wrote:
> It seems to me that we should be able to fix this by
adding an observer
> of the `solib_loaded' event to replace the hook that
linux-thread-db
> installed. I actually gave it a try, using the sources
from CVS, and
> found out something surprising: Although the
solib_loaded observer
> I just created gets called as expected, the linux
thread layers does
> not detect that threads are in use:

You've missed the pertinent fact here.  We're not waiting
for
libpthread.so to be loaded just for kicks!  The only way
libthread_db
can operate is by querying back through GDB for the
locations of
symbols in the thread library.  If we don't load that
objfile, then
it can't initialize.

> The only difference I can see between the two runs is
that symbols
> are read in one case, and not read in the other case.
Is libthread_db.so
> dependent on having the symbol table being loaded or
something of that
> sort?

Yes.  Don't load libpthread, won't get threads.

-- 
Daniel Jacobowitz
CodeSourcery

> You've missed the pertinent fact here.  We're not
waiting for
> libpthread.so to be loaded just for kicks!  The only
way libthread_db
> can operate is by querying back through GDB for the
locations of
> symbols in the thread library.  If we don't load that
objfile, then
> it can't initialize.

It's really strange. I put traces everywhere inside
linux-thread-db
so that if there was any callbacks called during the
detection,
I would see them. As you can see, I'm not very familiar
with how
this library works. Do you know where the callbacks are in
the
debugger code? I've been trying to find a document that
explains
how the pthread_db library works, is there one?

> > The only difference I can see between the two runs
is that symbols
> > are read in one case, and not read in the other
case. Is libthread_db.so
> > dependent on having the symbol table being loaded
or something of that
> > sort?
> 
> Yes.  Don't load libpthread, won't get threads.

Sounds fair enough to me.

Eli, everyone,

Would it be something worth documenting somewhere in the
manual?
I could probably write a small paragraph somewhere after the
description
of the auto-solib-add setting.

Thanks!
-- 
Joel

On Tue, Sep 12, 2006 at 05:47:20PM -0700, Joel Brobecker
wrote:
> > You've missed the pertinent fact here.  We're
not waiting for
> > libpthread.so to be loaded just for kicks!  The
only way libthread_db
> > can operate is by querying back through GDB for
the locations of
> > symbols in the thread library.  If we don't load
that objfile, then
> > it can't initialize.
> 
> It's really strange. I put traces everywhere inside
linux-thread-db
> so that if there was any callbacks called during the
detection,
> I would see them. As you can see, I'm not very
familiar with how
> this library works. Do you know where the callbacks are
in the
> debugger code?

Ah, simple: they're not there.  Take a look at the ps_*
routines in
proc-service.c.  In particular, ps_pglobal_lookup.  It's
not at all
surprising that you couldn't find them; this is a very rare
library
interface.  The shared library actually has undefined
symbols which
must be satisfied by GDB.

Do you want to stick a comment in linux-thread-db.c
somewhere
suggesting a look over there too?

> I've been trying to find a document that explains
> how the pthread_db library works, is there one?

Well, the way we use glibc's isn't documented anywhere,
but this
library is historically Sun's - their documentation may be
enlightening.

> > > The only difference I can see between the two
runs is that symbols
> > > are read in one case, and not read in the
other case. Is libthread_db.so
> > > dependent on having the symbol table being
loaded or something of that
> > > sort?
> > 
> > Yes.  Don't load libpthread, won't get threads.
> 
> Sounds fair enough to me.

The technical alternative, if you want to implement it,
would be to
always load symbols for libpthread.  What do you think?  We
don't even
really need debugging symbols - just the minimal (ELF)
symbol table.

> Eli, everyone,
> 
> Would it be something worth documenting somewhere in
the manual?
> I could probably write a small paragraph somewhere
after the description
> of the auto-solib-add setting.

If we leave it the way it is, I think this is a great idea. 
Please be
sure to mark it as a GNU/Linux specific issue.

-- 
Daniel Jacobowitz
CodeSourcery

> Date: Tue, 12 Sep 2006 17:47:20 -0700
> From: Joel Brobecker <brobeckeradacore.com>
> 
> Eli, everyone,
> 
> Would it be something worth documenting somewhere in
the manual?
> I could probably write a small paragraph somewhere
after the description
> of the auto-solib-add setting.

Yes, please do.

> Ah, simple: they're not there.  Take a look at the
ps_* routines in
> proc-service.c.  In particular, ps_pglobal_lookup. 
It's not at all
> surprising that you couldn't find them; this is a very
rare library
> interface.  The shared library actually has undefined
symbols which
> must be satisfied by GDB.

Ah ha, I see! I didn't realize it was using undefined
references...
I've been used to seeing callbacks...

> Do you want to stick a comment in linux-thread-db.c
somewhere
> suggesting a look over there too?

Absolutely, might be useful to someone else. I'll do that
tomorrow.

> The technical alternative, if you want to implement it,
would be to
> always load symbols for libpthread.  What do you think?
 We don't even
> really need debugging symbols - just the minimal (ELF)
symbol table.

I have been thinking about that, but there were several
little factors
that made me reconsider. The one that I didn't like the
most was the
fact that the only way I could see to detect the pthread
library being
loaded is by looking at its name. It doesn't feel very
elegant to have
to search for "/libpthread.so" or somesuch in
GDB. I don't know very
well how GNU/Linux distributions are made, but couldn't
that library
be named libpthread-1.8.0.so, for instance, making its
identification
harder?

At one point, I considered the idea of detecting
libpthread.so, and
printing a warning saying that, because auto-solib-add has
been
deactivated, thread support hasn't been enabled. But
that's almost
no less work than the above, so pointless.

In the end, because I wasn't able to produce any unexpected
behavior
except maybe lack of thread support, and given the probable
little
number of times this capability is used (I'm guessing,
though), it
didn't seem worth the effort.

I do realize, however, that it shouldn't be too much work,
though.
I'm open to both alternatives:

  1. Enhance GDB
  2. Document the limitation (Eli also agreed with the idea)

-- 
Joel

On Tue, Sep 12, 2006 at 08:45:18PM -0700, Joel Brobecker
wrote:
> I have been thinking about that, but there were several
little factors
> that made me reconsider. The one that I didn't like
the most was the
> fact that the only way I could see to detect the
pthread library being
> loaded is by looking at its name. It doesn't feel very
elegant to have
> to search for "/libpthread.so" or somesuch
in GDB. I don't know very
> well how GNU/Linux distributions are made, but
couldn't that library
> be named libpthread-1.8.0.so, for instance, making its
identification
> harder?

Well, if you wanted to be that precise about it, you could
do it by
SONAME.  But I think matching libpthread wouldn't be that
bad of a
hack.

>   2. Document the limitation (Eli also agreed with the
idea)

Let's go with this for now then.

-- 
Daniel Jacobowitz
CodeSourcery