Writing core files to contain buildids

Writing core files to contain buildids

[Paul Smith]

Hi all;

I'm maintaining a fork of the old Google coredumper code:

Original: https://github.com/anatol/google-coredumper
My fork: https://github.com/madscientist/google-coredumper

We use this for various reasons rather than relying on the Linux kernel
coredumper and it's worked well for us for many years now.  However, I
would like to start using build IDs in our shared libs and binaries and
to take best advantage of that I need to ensure that the build ID
values appear in the generated core files, which, at the moment, they
do not unfortunately.

I'm hoping someone can point me to some information about what extra
content I need to make sure is preserved in the generated coredump so
that I can retrieve build IDs from it.

Thanks!

Re: Writing core files to contain buildids

[Mark Wielaard]

Hi Paul,

On Mon, 2020-10-26 at 03:06 -0400, Paul Smith wrote:
> We use this for various reasons rather than relying on the Linux
> kernel
> coredumper and it's worked well for us for many years now.  However, I
> would like to start using build IDs in our shared libs and binaries and
> to take best advantage of that I need to ensure that the build ID
> values appear in the generated core files, which, at the moment, they
> do not unfortunately.
> 
> I'm hoping someone can point me to some information about what extra
> content I need to make sure is preserved in the generated coredump so
> that I can retrieve build IDs from it.

The basic idea behind getting buildids into core files is that they
(the GNU ELF notes) are at the start of the file in the first page that
is dumped (together with the phdrs) in the core file so when core file
consumers can walk the program headers and find the buildid note
(almost immediately following those headers).

A few more technical details can be found at:
https://fedoraproject.org/wiki/Releases/FeatureBuildId#Include_build_IDs_in_core_dumps

Hope that helps,

Mark

Re: Writing core files to contain buildids

[Paul Smith]

Thanks Mark,

On Tue, 2020-10-27 at 15:39 +0100, Mark Wielaard wrote:
> The basic idea behind getting buildids into core files is that they
> (the GNU ELF notes) are at the start of the file in the first page
> that is dumped (together with the phdrs) in the core file so when
> core file consumers can walk the program headers and find the buildid
> note (almost immediately following those headers).

Something is missing; if I check the binary I see a build ID in it.  I
can control whether the userspace coredumper or the kernel coredumper
is used via an environment variable: the kernel-generated core contains
a full set of build ID values including vdso.so, the binary, and all
normal .so's.

But when using the userspace coredumper I get a completely valid
coredump that GDB is happy with, for example, but there is only the
vdso.so build ID present: I don't see any build IDs from the binary or
shared libraries.

I guess another possibility is that they are there but eu-unstrip
--core can't find them?

The userspace coredumper uses the output of /proc/self/maps and
/proc/self/smaps to determine what memory to dump; the code is a bit
hairy but I will investigate what's going on.

The parsing of the maps starts here (I don't really expect anyone to
read this but if you're curious):

https://github.com/madscientist/google-coredumper/blob/1b64fbc282ba08654853f34c2bda00ffe5e26855/src/elfcore.c#L760

and the write-out starts here:

https://github.com/madscientist/google-coredumper/blob/1b64fbc282ba08654853f34c2bda00ffe5e26855/src/elfcore.c#L1019

> A few more technical details can be found at:
> https://fedoraproject.org/wiki/Releases/FeatureBuildId#Include_build_IDs_in_core_dumps

Thanks I did see that but I will read it harder.

Re: Writing core files to contain buildids

[Mark Wielaard]

Hi Paul,

On Tue, Oct 27, 2020 at 04:20:51PM -0400, Paul Smith wrote:
> On Tue, 2020-10-27 at 15:39 +0100, Mark Wielaard wrote:
> > The basic idea behind getting buildids into core files is that they
> > (the GNU ELF notes) are at the start of the file in the first page
> > that is dumped (together with the phdrs) in the core file so when
> > core file consumers can walk the program headers and find the buildid
> > note (almost immediately following those headers).
> 
> Something is missing; if I check the binary I see a build ID in it.  I
> can control whether the userspace coredumper or the kernel coredumper
> is used via an environment variable: the kernel-generated core contains
> a full set of build ID values including vdso.so, the binary, and all
> normal .so's.
> 
> But when using the userspace coredumper I get a completely valid
> coredump that GDB is happy with, for example, but there is only the
> vdso.so build ID present: I don't see any build IDs from the binary or
> shared libraries.
> 
> I guess another possibility is that they are there but eu-unstrip
> --core can't find them?

Do you have the generated core files somehwere so others can look at
them? How exactly are you testing the build-id notes are there?

Cheers,

Mark

Re: Writing core files to contain buildids

[Paul Smith]

On Tue, 2020-10-27 at 23:23 +0100, Mark Wielaard wrote:
> Do you have the generated core files somehwere so others can look at
> them? How exactly are you testing the build-id notes are there?

I don't have one to publish but I can create one.  That's a good idea
anyway as it will be simpler to debug than the actual system which is
much more complex.

I'm using eu-unstrip -n --core core.xxx to check for build IDs.

Re: Writing core files to contain buildids

[Mark Wielaard]

Hi Paul,

On Wed, 2020-10-28 at 00:40 -0400, Paul Smith wrote:
> On Tue, 2020-10-27 at 23:23 +0100, Mark Wielaard wrote:
> > Do you have the generated core files somehwere so others can look
> > at
> > them? How exactly are you testing the build-id notes are there?
> 
> I don't have one to publish but I can create one.  That's a good idea
> anyway as it will be simpler to debug than the actual system which is
> much more complex.
> 
> I'm using eu-unstrip -n --core core.xxx to check for build IDs.

OK, so say you have a core file (this one produced by the kernel
crashing a local sleep process). Then eu-unstrip -n --core would show:

0x400000+0x208000 ec28c24670b6b52fb99b43338e47fe2d4b6fb4ea@0x400284 - - /usr/bin/sleep
0x7ff037345000+0x3cd000 dd9a6ba0d81c91f5ca7dbb4a1ac58319cc26dd5a@0x7ff037345280 - - /usr/lib64/libc-2.17.so
0x7ff037712000+0x224000 a527fe72908703c5972ae384e78d1850d1881ee7@0x7ff0377121d8 - - /usr/lib64/ld-2.17.so
0x7ffed73bf000+0x1000 63023e5151c4d5bbdb7f6f85a6f95a6b95dc780e@0x7ffed73bf328 . - linux-vdso.so.1

What this means is that it found the build-id for libc-2.17.so at
address 0x7ff037345280 in the segment spanning 0x7ff037345000+0x3cd000.

So that address is very near the beginning of that segment.

eu-readelf -l on the core file will show a segment that corresponds to
that:

  LOAD           0x025000 0x00007ff037345000 0x0000000000000000 0x001000 0x1c2000 R E 0x1000

Note that the filesize is just the first page (0x001000) even though
the file itself is much bigger (as shown by the memsize of 0x1c2000)

You'll have to make sure your coredumper program produces a similar
segment in the core file covering the same area of libc.so to get the
build-id in there.

Hope that helps,

Mark