debugging core files from other machines with archived symbol files

debugging core files from other machines with archived symbol files

[Erik Gustafsson]

Hi,

I have a problem using gdb in a core file senario:

A machine with my stripped shared library crashes somewere in the world.
The core file is sent to me together with information on what version it was 
that crashed. I have archived non-stripped versions of my shared library, 
and now I want to get gdb to read the non-stripped version of my shared lib.

1: How do I find the base address where the text segment of my lib was 
loaded?
2: How can I tell gdb to load my library from my archive instead of the path 
where my lib was installed on the customer machine that crashed?

Is there any way of doing this accept from creating a file system structure 
matching the one on the machine where the crash occured, and copy the 
correct version of the non-stripped lib file to that directory?

Thx,

/Staffan

_________________________________________________________________
Add photos to your messages with MSN 8. Get 2 months FREE*. 
http://join.msn.com/?page=features/featuredemail

Re: debugging core files from other machines with archived symbol files

[Daniel Jacobowitz]

On Tue, Aug 19, 2003 at 01:25:52PM +0200, Erik Gustafsson wrote:
> Hi,
> 
> I have a problem using gdb in a core file senario:
> 
> A machine with my stripped shared library crashes somewere in the world.
> The core file is sent to me together with information on what version it 
> was that crashed. I have archived non-stripped versions of my shared 
> library, and now I want to get gdb to read the non-stripped version of my 
> shared lib.
> 
> 1: How do I find the base address where the text segment of my lib was 
> loaded?
> 2: How can I tell gdb to load my library from my archive instead of the 
> path where my lib was installed on the customer machine that crashed?
> 
> Is there any way of doing this accept from creating a file system structure 
> matching the one on the machine where the crash occured, and copy the 
> correct version of the non-stripped lib file to that directory?

Not really.  You can use solib-absolute-prefix to move the file
structure around, but it has to look the same.

You can work out the load address by poking around with objdump, and
then load the shared library with add-symbol-file.  But this is not for
the faint of heart.

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

Re: debugging core files from other machines with archived symbol files

[Erik Gustafsson]

Hmm.. Isn't this something you want to be able to do?

Seems to me that everybody that is shipping software to customers would like 
to be able to debug crash dumps from customer sites in a simple way.

I'm not sure I understand on what level the problem is. Is this a missing 
feature in GDB or missing information in the core file?

Thx,

/Staffan

>From: Daniel Jacobowitz <drow@mvista.com>
>To: Erik Gustafsson <loa_gus@hotmail.com>
>CC: gdb@sources.redhat.com
>Subject: Re: debugging core files from other machines with archived symbol 
>files
>Date: Tue, 19 Aug 2003 08:43:42 -0400
>
>On Tue, Aug 19, 2003 at 01:25:52PM +0200, Erik Gustafsson wrote:
> > Hi,
> >
> > I have a problem using gdb in a core file senario:
> >
> > A machine with my stripped shared library crashes somewere in the world.
> > The core file is sent to me together with information on what version it
> > was that crashed. I have archived non-stripped versions of my shared
> > library, and now I want to get gdb to read the non-stripped version of 
>my
> > shared lib.
> >
> > 1: How do I find the base address where the text segment of my lib was
> > loaded?
> > 2: How can I tell gdb to load my library from my archive instead of the
> > path where my lib was installed on the customer machine that crashed?
> >
> > Is there any way of doing this accept from creating a file system 
>structure
> > matching the one on the machine where the crash occured, and copy the
> > correct version of the non-stripped lib file to that directory?
>
>Not really.  You can use solib-absolute-prefix to move the file
>structure around, but it has to look the same.
>
>You can work out the load address by poking around with objdump, and
>then load the shared library with add-symbol-file.  But this is not for
>the faint of heart.
>
>--
>Daniel Jacobowitz
>MontaVista Software                         Debian GNU/Linux Developer

_________________________________________________________________
MSN 8 helps eliminate e-mail viruses. Get 2 months FREE*. 
http://join.msn.com/?page=features/virus

Re: debugging core files from other machines with archived symbol files

[Daniel Jacobowitz]

On Tue, Aug 19, 2003 at 03:21:04PM +0200, Erik Gustafsson wrote:
> Hmm.. Isn't this something you want to be able to do?
> 
> Seems to me that everybody that is shipping software to customers would 
> like to be able to debug crash dumps from customer sites in a simple way.
> 
> I'm not sure I understand on what level the problem is. Is this a missing 
> feature in GDB or missing information in the core file?

But I don't find it any particular problem to do it by recreating the
filesystem image, if I don't have the debugging libraries installed on
my system anyway.  That's what solib-absolute-prefix is for anyway.

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

Re: debugging core files from other machines with archived symbol files

[Paul Koning]

>>>>> "Erik" == Erik Gustafsson <loa_gus@hotmail.com> writes:

 Erik> Hmm.. Isn't this something you want to be able to do?  Seems to
 Erik> me that everybody that is shipping software to customers would
 Erik> like to be able to debug crash dumps from customer sites in a
 Erik> simple way.

 Erik> I'm not sure I understand on what level the problem is. Is this
 Erik> a missing feature in GDB or missing information in the core
 Erik> file?

I don't think there is a real issue in practice.  Daniel briefly
stated the requirements -- you need to have a file system subtree
somewhere that has the same shape as the target system tree.  That in
general isn't a big deal.

Given that, you simply point gdb to that subtree with
solib-absolute-prefix, and you're good to go.

For example, suppose your build procedure produces full (unstripped)
binaries in .../targetroot/usr/lib, .../targetroot/usr/bin, etc. Then
this gets the job done:
     gdb .../targetroot/usr/bin/brokenapp
     (gdb) set solib-absolute-prefix .../targetroot
     (gdb) core customercorefile

GDB will find the relevant file names and the load addresses from data
stored in the corefile.  Well, that depends on the OS; some don't do a
particularly good job.  For example, you may discover, as I did, that
you can find text sections of libs but not data sections.

    paul

RE: debugging core files from other machines with archived symbol files

[Staffan Gustafsson]

Problem is mostly in the automation area. On windows, we have a
simple script that starts the debugger on the core file, maps in 
the correct symbols and images from a symbol server, the correct 
version of the source and takes me to the line causing 
the crash.

When debugging a lot of dumps, that's a big saver. The story 
using redhat/gdb seems much weeker.

So while I could create a directory structure similar to what some 
client has chosen for his/her machine, copy my files to that 
location, it is a harder to automate, since I have to start gdb on 
the core file just to find out what that directory structure is.

But I guess it can be done. This is however an area where Microsoft 
has a more streamlined solution, and it wouldn't hurt to have the 
possibility to look up symbols/images from a server in gdb.

Just my $.02

/Staffan

> -----Original Message-----
> From: Paul Koning [mailto:pkoning@equallogic.com] 
> Sent: den 19 augusti 2003 15:32
> To: loa_gus@hotmail.com
> Cc: drow@mvista.com; gdb@sources.redhat.com
> Subject: Re: debugging core files from other machines with 
> archived symbol files
> 
> 
> >>>>> "Erik" == Erik Gustafsson <loa_gus@hotmail.com> writes:
> 
>  Erik> Hmm.. Isn't this something you want to be able to do?  
> Seems to  Erik> me that everybody that is shipping software 
> to customers would  Erik> like to be able to debug crash 
> dumps from customer sites in a  Erik> simple way.
> 
>  Erik> I'm not sure I understand on what level the problem 
> is. Is this  Erik> a missing feature in GDB or missing 
> information in the core  Erik> file?
> 
> I don't think there is a real issue in practice.  Daniel 
> briefly stated the requirements -- you need to have a file 
> system subtree somewhere that has the same shape as the 
> target system tree.  That in general isn't a big deal.
> 
> Given that, you simply point gdb to that subtree with 
> solib-absolute-prefix, and you're good to go.
> 
> For example, suppose your build procedure produces full 
> (unstripped) binaries in .../targetroot/usr/lib, 
> .../targetroot/usr/bin, etc. Then this gets the job done:
>      gdb .../targetroot/usr/bin/brokenapp
>      (gdb) set solib-absolute-prefix .../targetroot
>      (gdb) core customercorefile
> 
> GDB will find the relevant file names and the load addresses 
> from data stored in the corefile.  Well, that depends on the 
> OS; some don't do a particularly good job.  For example, you 
> may discover, as I did, that you can find text sections of 
> libs but not data sections.
> 
>     paul
> 
> 
> 
> 
>