gdb and shared libraries

gdb and shared libraries

[wim delvaux]

Hi all,

I am running the 5.3.90-2003_06_29-cvs-debian version of gdb and STILL i have 
problems with breakpoints in shared libraries...

The problem is that you can set a breakpoint in a library but if you restart 
the program all breakpoints in that library are lost (i.e. gdb knows about 
them but seems to have disabled them).

Now I know that you are going to say that the library gets unloaded and hence 
the breakpoint cannot be set ... but it seems that gdb tries to set them 
because I get messages that It cannot set a shlib breakpoint.

Now I wonder if it is really that difficult to remember over gdb restarts that 
breakpoints were set for a certain library and once that library get reloaded 
set the breakpoints again.

1. without the messages that it cannot set them
2. without appearing to forget that the breakpoints were active ..

I say STILL because the problem has existed in all 5-versions I know of but 
did not exist in the last 4-version.

W

Re: gdb and shared libraries

[Daniel Jacobowitz]

On Wed, Feb 08, 2006 at 09:54:35PM -0800, Jim Blandy wrote:
> On 2/8/06, Patrick Alken <patrick.alken@colorado.edu> wrote:
> > So it is step #4 that I am hoping someone on the list can give me
> > details for. How do you know when all the libraries have been loaded
> > so that you can stop execution of the program while you load
> > the libs symbol tables? How do you get the pathnames of the libraries
> > after they have been loaded so you can mmap them?
> 
> Well, each time it loads a shared library, the dynamic linker calls a
> specific function (I think it's _dl_debug_state on Linux), and there
> is a designated data structure for you to walk to find the current
> list of shlibs.  See solib.c and solib-svr4.c.
> 
> But I can't remember, nor can I Google, where this stuff is
> documented.  I guess we're just depending on undocumented behavior (on
> Linux, at least).  But the whole reason it's there is to help out the
> debugger, so I don't know why it'd be undocumented.

Well, that's how _we_ do it.  How you're really _supposed_ to do it is
unclear, but involves the dynamic DT_DEBUG tag.

-- 
Daniel Jacobowitz
CodeSourcery

Re: gdb and shared libraries

[Jim Blandy]

On 2/8/06, Patrick Alken <patrick.alken@colorado.edu> wrote:
> So it is step #4 that I am hoping someone on the list can give me
> details for. How do you know when all the libraries have been loaded
> so that you can stop execution of the program while you load
> the libs symbol tables? How do you get the pathnames of the libraries
> after they have been loaded so you can mmap them?

Well, each time it loads a shared library, the dynamic linker calls a
specific function (I think it's _dl_debug_state on Linux), and there
is a designated data structure for you to walk to find the current
list of shlibs.  See solib.c and solib-svr4.c.

But I can't remember, nor can I Google, where this stuff is
documented.  I guess we're just depending on undocumented behavior (on
Linux, at least).  But the whole reason it's there is to help out the
debugger, so I don't know why it'd be undocumented.

gdb and shared libraries

[Patrick Alken]

Hello,

  I am the developer of ald (http://ald.sourceforge.net), which
is a GNU licensed debugger designed specifically for assembly
level debugging.  Right now I am trying to solve the problem of
loading in the symbol tables of shared libraries for a target
program so that breakpoints can be set at addresses in these
libraries.

  Unfortunately it has not been easy to find information on
how to best do this, and after studying the gdb source code
for a while I decided it was best to ask this list for help.

  I am looking for a big picture with perhaps some useful
details for the more complicated steps. As an example,

1. fork + ptrace(TRACE_ME) + exec (child)
2. catch SIGTRAP (parent)
3. user types "run"
4. (start singlestepping/continuing/whatever until shared libraries
    have been loaded and ?somehow? get names of all libraries
    loaded by dynamic linker)
5. mmap() that list of libraries and then load their symbol tables
   into memory so breakpoints can be set there

So it is step #4 that I am hoping someone on the list can give me
details for. How do you know when all the libraries have been loaded
so that you can stop execution of the program while you load
the libs symbol tables? How do you get the pathnames of the libraries
after they have been loaded so you can mmap them?

Thank you in advance,
Patrick Alken

Re: GDB and Shared Libraries

[Daniel Jacobowitz]

On Tue, Jan 06, 2004 at 02:06:57PM -0800, Drummonds, Scott B wrote:
> Hi, everyone,
> 
> I'm noticing a discrepancy between the way that GDB handles shared
> libraries and my Linux system does during execution.  Specifically, my
> program works on the command line (well, except for the actual run-time
> but that I'm trying to find) but aborts in GDB for lack of shared
> libraries.  Specifically:
> 
> <quote>
> (gdb) run <my_args>
> Starting program: <my_binary> <my_args>
> <my_binary>: error while loading shared libraries: libtcl8.0.so: cannot
> open shared object file: No such file or directory
> 
> Program exited with code 0177.
> </quote>
> 
> Can anyone tell me why this is happening?  This is an old version of GDB
> (5.2-2) but I can't imagine that the problem here is anything other than
> my incorrect or incomplete understanding of GDB.

This usually means that you have a shell startup script (.bashrc,
.profile, et cetera) which is clobbering the library path.

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

GDB and Shared Libraries

[Drummonds, Scott B]

Hi, everyone,

I'm noticing a discrepancy between the way that GDB handles shared
libraries and my Linux system does during execution.  Specifically, my
program works on the command line (well, except for the actual run-time
but that I'm trying to find) but aborts in GDB for lack of shared
libraries.  Specifically:

<quote>
(gdb) run <my_args>
Starting program: <my_binary> <my_args>
<my_binary>: error while loading shared libraries: libtcl8.0.so: cannot
open shared object file: No such file or directory

Program exited with code 0177.
</quote>

Can anyone tell me why this is happening?  This is an old version of GDB
(5.2-2) but I can't imagine that the problem here is anything other than
my incorrect or incomplete understanding of GDB.

Thanks,
Scott

Re: GDB and shared libraries

[Stephen Smith]

Ok, my code shipment went out the other day, so I am back on this topic.  Since this is the first time I have ever
modified GNU code, you may have to bear with me.

For the record, I have read Section 13 of the gdb manual (especially the section on the q packet).  I also read my gdb
server
code, and kevin's solib-svr4.h and solib-svr4.c files. from the 20010306 snapshot.

First a couple of questions:
    1)  Andrew:  you state that "Most shared library implementations can, in theory, be debugged remotely
         without any modifications to the GDB remote protocol."  But, if kevin is correct, the q packet will have to
         have some additional features added.  Correct?
    2)  I am planning on adding a solib-remote-elf.h and solib-remote-elf.c files.
         where should I add the hooks in the rest of the files - I don't see where the svr4 stuff is hooked in and I haven't

         figured out where to hook into remote communications code for modifying the q packet (if I do have to modify the
protocol).

Thanks for helping the newbie.
sps

Below is the summary of the current discussion, just so we don't have to go over it.

>  Andrew Cagney
>  Just FYI.
>
>  Most shared library implementations can, in theory, be debugged remotely
>  without any modifications to the GDB remote protocol.  This is because
>  the SHLIBs are handled using only memory read/writes and breakpoints.
>
>  The ``exercise left to the reader'' is getting what was native-only
>  SHLIB code re-written in a way that is host independant.
>
>  good luck,
>          Andrew

>  Kevin Buettner wrote:
> > Stephen Smith wrote:
> > On the target I have the application testapp.exe and the library
> > libtestlib.so.  When the OS loads testapp.exe and libtestlib.so it
> > does the fixups and (since we wrote the OS and the gdbserver) we can
> > see the elf headers for both files.  We also currently can debug
> > testapp.exe itself.
> >
> > Now, the function main() has a call to lib1Func() which is defined
> > inside of libtestlib.so.  I would like to trace into that function.
> >
> > Questions:
> >      1)  In the protocol
> >            http://sources.redhat.com/gdb/onlinedocs/gdb_14.html#SEC120
> >          I don't see a way to pass back the fixup results that the loader
> >          performed.  Don't I need to do that?
>
> Maybe, maybe not.  As Andrew indicated in his message, you might
> be able to do it all by reading memory, writing memory, and setting
> breakpoints.  In the event that you can't, I suggest using the "q"
> packet so that GDB can fetch the shared library information from
> gdbserver.  You'll need to modify gdb to generate (send) the "q"
> packet at the appropriate point in the code (which will likely be in
> your solib backend) and you'll need to modify gdbserver to recognize
> your special "q" packet and return the shared library information in a
> mutually agreed upon format.
>
> >      2) When I step into the function, lib1Func(), how does the
> >         gdbserver tell gdb that it needs to load the symbol table
> >         from the file libtestlib.so?
>
> The idea is for the symbol table to be loaded (by GDB) prior to ending
> up in lib1Func().  For this to happen, GDB needs to be told when a
> shared library has been loaded by the dynamic linker.  Again, one of
> the ways this is commonly done is for gdb to set an internal
> breakpoint in a special do-nothing function that the shared library
> machinery agrees to call after it's loaded a shared library.  But it
> could be done in other ways too.
>
> Anyway, once GDB has been told that there are (potentially) new
> libraries loaded, GDB then attempts to find the load information
> that you refer to.  It then loads the symbol table and applies
> the necessary relocations.
>
> The point is that all of this occurs before you've ever hit lib1Func().
>
> Kevin

Re: GDB and shared libraries

[Stephen Smith]

Now, I am getting somewhere!

Thanks.

Stephen

Re: GDB and shared libraries

[Kevin Buettner]

On Feb 27,  2:15pm, Stephen Smith wrote:

> On the target I have the application testapp.exe and the library
> libtestlib.so.  When the OS loads testapp.exe and libtestlib.so it
> does the fixups and (since we wrote the OS and the gdbserver) we can
> see the elf headers for both files.  We also currently can debug
> testapp.exe itself.
> 
> Now, the function main() has a call to lib1Func() which is defined
> inside of libtestlib.so.  I would like to trace into that function.
> 
> Questions:
>      1)  In the protocol
>            http://sources.redhat.com/gdb/onlinedocs/gdb_14.html#SEC120
>          I don't see a way to pass back the fixup results that the loader
>          performed.  Don't I need to do that?

Maybe, maybe not.  As Andrew indicated in his message, you might
be able to do it all by reading memory, writing memory, and setting
breakpoints.  In the event that you can't, I suggest using the "q"
packet so that GDB can fetch the shared library information from
gdbserver.  You'll need to modify gdb to generate (send) the "q"
packet at the appropriate point in the code (which will likely be in
your solib backend) and you'll need to modify gdbserver to recognize
your special "q" packet and return the shared library information in a
mutually agreed upon format.

>      2) When I step into the function, lib1Func(), how does the
>         gdbserver tell gdb that it needs to load the symbol table
>         from the file libtestlib.so?

The idea is for the symbol table to be loaded (by GDB) prior to ending
up in lib1Func().  For this to happen, GDB needs to be told when a
shared library has been loaded by the dynamic linker.  Again, one of
the ways this is commonly done is for gdb to set an internal
breakpoint in a special do-nothing function that the shared library
machinery agrees to call after it's loaded a shared library.  But it
could be done in other ways too.

Anyway, once GDB has been told that there are (potentially) new
libraries loaded, GDB then attempts to find the load information
that you refer to.  It then loads the symbol table and applies
the necessary relocations.

The point is that all of this occurs before you've ever hit lib1Func().

Kevin

Re: GDB and shared libraries

[Stephen Smith]

Andrew Cagney wrote:

> > Ok,  what do I need to add (commands, data, etc.) to my gdb server to trace into these libraries?  Is the spec written
> > down?
>
> Just FYI.
>
> Most shared library implementations can, in theory, be debugged remotely
> without any modifications to the GDB remote protocol.  This is because
> the SHLIBs are handled using only memory read/writes and breakpoints.
>
> The ``exercise left to the reader'' is getting what was native-only
> SHLIB code re-written in a way that is host independant.

From what I have been told and from looking at the spec it appears that I may have what I need already implemented as far as
the remote protocol is concerned (unless I missed something).  BTW, thanks kevin for your help.

Now here is an example that will go along with my question that hopefully will explain why I am so confused.  Remember I am
running
on target=powerpc-motorola-elf   and host = i686-cygwin-pe (this means that gdb is the insight version 5.0).

On the target I have the application testapp.exe and the library libtestlib.so.  When the OS loads testapp.exe and
libtestlib.so it does the fixups and (since we wrote the OS and the gdbserver) we can see the elf headers for both files.
We also currently can debug testapp.exe itself.

Now, the function main() has a call to lib1Func()  which is defined inside of libtestlib.so.  I would like to trace into
that function.

Questions:
     1)  In the protocol   http://sources.redhat.com/gdb/onlinedocs/gdb_14.html#SEC120  I don't see a way to pass back the
fixup results that the loader performed.
          Don't I need to do that?

     2) When I step into the function, lib1Func(), how does the gdbserver tell gdb that it needs to load the symbol table
from the file libtestlib.so?

sps

Re: GDB and shared libraries

[Andrew Cagney]

> Ok,  what do I need to add (commands, data, etc.) to my gdb server to trace into these libraries?  Is the spec written
> down?

Just FYI.

Most shared library implementations can, in theory, be debugged remotely
without any modifications to the GDB remote protocol.  This is because
the SHLIBs are handled using only memory read/writes and breakpoints.

The ``exercise left to the reader'' is getting what was native-only
SHLIB code re-written in a way that is host independant.

good luck,
	Andrew

Re: GDB and shared libraries

[Stephen Smith]

>
> Or (more likely) it's possible that the OS provides some other
> interface for getting at this information.  Again, the two things you
> need are 1) some way to be notified that the dynamic linker has loaded
> (or unloaded) one of the shared libraries and 2) some way to get the
> names of the libraries, their load addresses, and perhaps other
> assorted information.  You'll need to consult the docs for your
> proprietary OS and figure out what these mechanisms are.  It's quite
> possible that they've provided you with a debugging API for getting at
> this information.
>

Well, since we wrote the OS and the gdbserver, I think that is doable.

>
> Anyway, once you've figured out these details, you'll still need to
> put a shared library backend on your target.  Fortunately, if you make
> the target stub (gdb server) do most of the work, it ought to be
> pretty simple.  I think the generally accepted mechanism for gdb to
> request this kind of target/stub-specific information is the "q"
> packet.  GDB's remote protocol is documented at
>
>     http://sources.redhat.com/gdb/onlinedocs/gdb_14.html#SEC120
>
> At this point, I'll let others jump in because my experience with the
> remote protocol is actually rather limited...
>
> Kevin

Thanks for pointing me to the doc

Stephen

Re: GDB and shared libraries

[Kevin Buettner]

Stephen,

Without knowing more about the shared library mechanism of the
proprietary embeddded OS, it is difficult to say.  For an SVR4-like
shared library system, gdb does not need any help from the target's
gdb server beyond that which it already provides (reading memory,
writing memory, etc) for normal debugging.  The reason for this is
that GDB is able to obtain all the information it needs regarding
the dynamic linker's activities from the memory of the target
process.  It is certainly possible that the OS you are dealing with
provides the information in a similar, albeit different way.

Or (more likely) it's possible that the OS provides some other
interface for getting at this information.  Again, the two things you
need are 1) some way to be notified that the dynamic linker has loaded
(or unloaded) one of the shared libraries and 2) some way to get the
names of the libraries, their load addresses, and perhaps other
assorted information.  You'll need to consult the docs for your
proprietary OS and figure out what these mechanisms are.  It's quite
possible that they've provided you with a debugging API for getting at
this information.

Anyway, once you've figured out these details, you'll still need to
put a shared library backend on your target.  Fortunately, if you make
the target stub (gdb server) do most of the work, it ought to be
pretty simple.  I think the generally accepted mechanism for gdb to
request this kind of target/stub-specific information is the "q"
packet.  GDB's remote protocol is documented at

    http://sources.redhat.com/gdb/onlinedocs/gdb_14.html#SEC120

At this point, I'll let others jump in because my experience with the
remote protocol is actually rather limited...

Kevin

On Feb 26, 10:06pm, Stephen Smith wrote:
> Subject: Re: GDB and shared libraries
> Ah, now I know why we are not communicating.
> 
> I have an embedded system with a proprietary OS.  This system has a minimal gdb server running on it.  Let's call this
> the target
> system.
> 
> My host is a Windows NT/Cygwin or Linux Box running gdb which is talking to the gdb server on the target machine.
> The problem is that I have a process running on the target (no video, X, etc.) and am debugging on the host.  My process
> uses a shared library and I need to add the capability of tracing into it.
> 
> Ok,  what do I need to add (commands, data, etc.) to my gdb server to trace into these libraries?  Is the spec written
> down?
> 
> Kevin Buettner wrote:
> 
> > On Feb 26,  6:17pm, Stephen Smith wrote:
> >
> > > I am told that the "minimal" (and I don't know how minimal)
> > > gdbserver doesn't follow the SVR4 spec.  Well, I could have fun.
> >
> > Okay, you've lost me.  Where does gdbserver enter into the picture?
> >
> > Kevin
>-- End of excerpt from Stephen Smith

Re: GDB and shared libraries

[Stephen Smith]

Ah, now I know why we are not communicating.

I have an embedded system with a proprietary OS.  This system has a minimal gdb server running on it.  Let's call this
the target
system.

My host is a Windows NT/Cygwin or Linux Box running gdb which is talking to the gdb server on the target machine.
The problem is that I have a process running on the target (no video, X, etc.) and am debugging on the host.  My process
uses a shared library and I need to add the capability of tracing into it.

Ok,  what do I need to add (commands, data, etc.) to my gdb server to trace into these libraries?  Is the spec written
down?

Kevin Buettner wrote:

> On Feb 26,  6:17pm, Stephen Smith wrote:
>
> > I am told that the "minimal" (and I don't know how minimal)
> > gdbserver doesn't follow the SVR4 spec.  Well, I could have fun.
>
> Okay, you've lost me.  Where does gdbserver enter into the picture?
>
> Kevin

Re: GDB and shared libraries

[Stephen Smith]

Let's pretend that my system is not SVR4 in nature and that I have source so a very minimal gdb server for an embedded
system that currently communicates with gdb for debugging of normal gcc applications.

Let's not assume that there are any special hooks in gdb for debugging shared libraries.  What commands or data would
the gdb server have to send to gdb to enable the debugging of shared libraries.

Stephen

Re: GDB and shared libraries

[Kevin Buettner]

On Feb 26,  5:40pm, Stephen Smith wrote:

> Kevin Buettner wrote:
> 
> >
> >     1) Modify your shared library implementation so that solib-svr4.c
> >        (in the gdb sources) will work for you.  Recall that in private
> >        email I suggested defining SVR4_FETCH_LINK_MAP_OFFSETS for your
> >        target.  This is easy to do; just take a look at struct
> >        r_debug and struct link_map and figure out the offsets from
> >        the beginning of the struct to each member n the struct.
> >        Then construct a small function to fill in the fields of
> >        GDB's struct link_map_offsets.  You may even be able to
> >        use the example at
> >
> >            http://sources.redhat.com/ml/gdb-patches/2000-10/msg00147.html
> >
> >        verbatim.
>
> I take it you meant, Modify your shared library implementation of gdb...
> 
> Is that correct?

No.

From your other email (including some of the private ones) it sounded
to me like you had the source to the OS and you could modify the
dynamic loader.  So I meant for the above to be split up into two
parts:

    a) OS / Library side work in making the dynamic loader behave
       like other SVR4 shared library implementations.

    b) In GDB, define SVR4_FETCH_LINK_MAP_OFFSETS for your target.

If you can do (a), then doing (b) will be trivial.  If you look over
some of my past email, you'll see that I've sketched out what needs
to be done for (a).

If you end up going the other route of adding a new solib backend
for GDB, you may still end up doing some work in the dynamic loader
anyway.  Basically, your dynamic loader and GDB need to agree on
an interface for 1) notifying GDB of when a shared object has been
loaded/unloaded, and 2) obtaining the names of the currently loaded
shared objects and their load addresses.

Kevin

Re: GDB and shared libraries

[Stephen Smith]

I take it you meant, Modify your shared library implementation of gdb...

Is that correct?

Kevin Buettner wrote:

>
>     1) Modify your shared library implementation so that solib-svr4.c
>        (in the gdb sources) will work for you.  Recall that in private
>        email I suggested defining SVR4_FETCH_LINK_MAP_OFFSETS for your
>        target.  This is easy to do; just take a look at struct
>        r_debug and struct link_map and figure out the offsets from
>        the beginning of the struct to each member n the struct.
>        Then construct a small function to fill in the fields of
>        GDB's struct link_map_offsets.  You may even be able to
>        use the example at
>
>            http://sources.redhat.com/ml/gdb-patches/2000-10/msg00147.html
>
>        verbatim.

Re: GDB and shared libraries

[Stephen Smith]

>
> However, you ought to first verify that your shared library
> implementation is SVR4-like enough for it to help.  (See above.)
>
> If it doesn't, you can take one of two approaches:
>
>
>     2) Add a solib-yourtarget.c backend.  There are now two examples
>        in the sources: solib-svr4.c and solib-aix5.c.  The eventual
>        goal is to migrate the other shared library support to use this
>        framework too.  (So don't look at the other shared library support
>        that hasn't been migrated yet.  Or if you do, don't do it that
>        way ;-)
>
> Kevin

Since the other platform (that I have to get to work later) is x86 pe format I think I will go this route if necessary
(and then add the code back into the gdb baseline).

sps

Re: GDB and shared libraries

[Kevin Buettner]

On Feb 26,  3:51pm, Stephen Smith wrote:

> > Last October, I made some changes to GDB's infrastructure to better
> > support debugging of shared libraries in an embedded environment.
> > But this infrastructure work will only really work for an SVR4-like
> > shared library implementation (which, BTW, is what Linux uses).
> >
> > See http://sources.redhat.com/ml/gdb-patches/2000-10/msg00142.html
> 
> Ok,  when I run "$target-gcc -E -dM -x c /dev/null" I get
> 
> #define __PPC__ 1
> #define _BIG_ENDIAN 1
> #define PPC 1
> #define __svr4__ 1
> #define __GNUC_MINOR__ 95
> #define __CHAR_UNSIGNED__ 1
> #define _ARCH_PPC 1
> #define __unix 1
> #define __unix__ 1
> #define __GNUC__ 2
> #define _CALL_SYSV 1
> #define __PPC 1
> #define __BIG_ENDIAN__ 1
> #define unix 1
> 
> so it appears that would be a canidate.

Maybe.  You'll need to dig deeper though to know if it'll really work. 
For starters, you'll need to determine if your dynamic loader calls
one of the magic symbols listed in solib_break_names[] in
solib-svr4.c.  Also, you'll need to verify that it exports the r_debug
struct via the mechanism used to find it by locate_base() in
solib-svr4.c.  In a nutshell, this means that the contents of the
.dynamic section as found on the target needs to have the value
corresponding to the DT_DEBUG tag be the address of the r_debug
struct.

> Which would you recommend:
>     a) Patching my current code (the 5.0 release)
>     b) Getting the latest snapshot

Get the latest snapshot.

However, you ought to first verify that your shared library
implementation is SVR4-like enough for it to help.  (See above.)

If it doesn't, you can take one of two approaches:

    1) Modify your shared library implementation so that solib-svr4.c
       (in the gdb sources) will work for you.  Recall that in private
       email I suggested defining SVR4_FETCH_LINK_MAP_OFFSETS for your
       target.  This is easy to do; just take a look at struct
       r_debug and struct link_map and figure out the offsets from
       the beginning of the struct to each member n the struct.
       Then construct a small function to fill in the fields of
       GDB's struct link_map_offsets.  You may even be able to
       use the example at

	   http://sources.redhat.com/ml/gdb-patches/2000-10/msg00147.html

       verbatim.

    2) Add a solib-yourtarget.c backend.  There are now two examples
       in the sources: solib-svr4.c and solib-aix5.c.  The eventual
       goal is to migrate the other shared library support to use this
       framework too.  (So don't look at the other shared library support
       that hasn't been migrated yet.  Or if you do, don't do it that
       way ;-)

Kevin