Automatically fetching Build ID from remote libraries and resole them locally?

Automatically fetching Build ID from remote libraries and resole them locally?

[Norbert Lange]

Hello,

I am mostly remote debugging, and I often have mismatches from the
remote binaries and the local build environment (typically this makes
thing worse than having no debug info)
This can include the app themself, but most commonly its slightly
different builds of libraries.

Right now I can pick between setting up a sysroot path, or DL the
(stripped) libraries from the target. Neither is optimal.
If your sysroot is compromised of several builds, its also not
feasible to have all combinations of them around.

Build-Ids already solve those problems on a local gdb session. Would
it be possibly to use the benefits on remote sessions aswell?

ie. DL only some ELF headers or remotely inspect the Build-ID,
then lookup and resolve the matching library and debuginfo in the
debug-file-directory database.
Potentially transfer the whole library from the target if that fails.

Such a scheme would make connection to a remote target rather foolproof, and
allowing other stuff like manually loaded libraries (no valid
filename) to work aswell.

Norbert

Re: Automatically fetching Build ID from remote libraries and resole them locally?

[Frank Ch. Eigler]

Norbert Lange via Gdb <gdb@sourceware.org> writes:

> [...]
> ie. DL only some ELF headers or remotely inspect the Build-ID,
> then lookup and resolve the matching library and debuginfo in the
> debug-file-directory database. [...]

By the way, this part of the problem is solved e.g. using the elfutils
debuginfod facility, which is now part of gdb.  If gdb can get the
buildids, then it can auto-download matching binaries & source code.

https://sourceware.org/elfutils/Debuginfod.html

- FChE

Re: Automatically fetching Build ID from remote libraries and resole them locally?

[Norbert Lange]

Am Sa., 28. März 2020 um 01:40 Uhr schrieb Frank Ch. Eigler <fche@redhat.com>:
>
>
> Norbert Lange via Gdb <gdb@sourceware.org> writes:
>
> > [...]
> > ie. DL only some ELF headers or remotely inspect the Build-ID,
> > then lookup and resolve the matching library and debuginfo in the
> > debug-file-directory database. [...]
>
> By the way, this part of the problem is solved e.g. using the elfutils
> debuginfod facility, which is now part of gdb.  If gdb can get the
> buildids, then it can auto-download matching binaries & source code.

Yes. Just want to make sure that I am talking about the other (left) side:

Remote target (gdbserver) <-> Local gdb session <-> debug file source.

Currently its not possible to detect the Build ID without transferring
the whole library
from Remote target to the local gdb. I suppose with coredumps its a
similar problem.
What I would like is using build IDs wherever possible. That means
retrieve them from the remote,
and maybe define some fileformat to augment coredumps with the information.

Norbert

Re: Automatically fetching Build ID from remote libraries and resole them locally?

[Jan Kratochvil]

On Mon, 30 Mar 2020 10:35:30 +0200, Norbert Lange via Gdb wrote:
> Currently its not possible to detect the Build ID without transferring the
> whole library from Remote target to the local gdb. I suppose with coredumps
> its a similar problem. What I would like is using build IDs wherever
> possible. That means retrieve them from the remote, and maybe define some
> fileformat to augment coredumps with the information.

Coredumps already contain build-ids of executables + shared libraries as long
as it is enabled (it is by default):
	https://www.kernel.org/doc/Documentation/filesystems/proc.txt
  - (bit 4) ELF header pages in file-backed private memory areas (it is
            effective only if the bit 2 is cleared)

build-id loading from remote files should be also somehow supported by recent
GDBs (done by Gary Benson).

Fedora+RHEL GDB has patchset
	https://src.fedoraproject.org/rpms/gdb/blob/master/f/gdb-6.6-buildid-locate.patch
	https://src.fedoraproject.org/rpms/gdb/blob/master/f/gdb-6.6-buildid-locate-solib-missing-ids.patch
	https://src.fedoraproject.org/rpms/gdb/blob/master/f/gdb-6.6-buildid-locate-core-as-arg.patch
to locate appropriate executable + shared libraries from such core file by
build-id - it locates then through symlinks in:
	/usr/lib/debug/.build-id/6b/6cd649300ec28b47638e766e9cbb58648cbdac{,.debug}

IIUC that patchset should be now being replaced by the elfutils debuginfod.


Jan

Re: Automatically fetching Build ID from remote libraries and resole them locally?

[Norbert Lange]

Am Mo., 30. März 2020 um 10:45 Uhr schrieb Jan Kratochvil
<jan.kratochvil@redhat.com>:
>
> On Mon, 30 Mar 2020 10:35:30 +0200, Norbert Lange via Gdb wrote:
> > Currently its not possible to detect the Build ID without transferring the
> > whole library from Remote target to the local gdb. I suppose with coredumps
> > its a similar problem. What I would like is using build IDs wherever
> > possible. That means retrieve them from the remote, and maybe define some
> > fileformat to augment coredumps with the information.
>
> Coredumps already contain build-ids of executables + shared libraries as long
> as it is enabled (it is by default):
>         https://www.kernel.org/doc/Documentation/filesystems/proc.txt
>   - (bit 4) ELF header pages in file-backed private memory areas (it is
>             effective only if the bit 2 is cleared)
>
> build-id loading from remote files should be also somehow supported by recent
> GDBs (done by Gary Benson).

Thats great then.

>
> Fedora+RHEL GDB has patchset
>         https://src.fedoraproject.org/rpms/gdb/blob/master/f/gdb-6.6-buildid-locate.patch
>         https://src.fedoraproject.org/rpms/gdb/blob/master/f/gdb-6.6-buildid-locate-solib-missing-ids.patch
>         https://src.fedoraproject.org/rpms/gdb/blob/master/f/gdb-6.6-buildid-locate-core-as-arg.patch
> to locate appropriate executable + shared libraries from such core file by
> build-id - it locates then through symlinks in:
>         /usr/lib/debug/.build-id/6b/6cd649300ec28b47638e766e9cbb58648cbdac{,.debug}
>
> IIUC that patchset should be now being replaced by the elfutils debuginfod.

Still thinking you are misunderstanding me, debuginfod aint helping here.

I am running gdbserver on 'remote' and gdb on 'localpc',
then attaching to the remote target.

I have a version of the libraries on 'localpc' (almost mirrored for
compiling), but they might
not be the exact ones used on the 'remote' (rebuilds, additional libs, etc..)
I would want to use 'set sysroot /path/to/staging/dir', but the downside is that
mismatches will not be detected and can badly mess up debugging.
GDB 9.0 certainly does not use *Build-Ids from the 'remote'*,
neither for files existing on the 'localpc' or not (used strace to
monitor the file lookups).

I could only set 'set sysroot target:/',
at which point all DSOs will be transmitted from 'remote' to
'localpc'. Not exactly what I had in mind.

So, instead I would want GDB to just inspect the Build-ID's from 'remote',
and look them up from 'localpc' (whether its using debuginfod or not,
this is irrelevant here).

Norbert

Re: Automatically fetching Build ID from remote libraries and resole them locally?

[Jan Kratochvil]

On Mon, 30 Mar 2020 11:04:30 +0200, Norbert Lange wrote:
> I would want to use 'set sysroot /path/to/staging/dir', but the downside is that
> mismatches will not be detected and can badly mess up debugging.

You should have setup symlinks in /usr/lib/debug/.build-id/
(or where you 'set debug-file-directory') for all build-ids you ever build.
That works for core-files.


> GDB 9.0 certainly does not use *Build-Ids from the 'remote'*,

According to a test I have made now I agree. Maybe Gary Benson will suggest
more.


Jan

Re: Automatically fetching Build ID from remote libraries and resole them locally?

[Gary Benson]

Jan Kratochvil wrote:
> On Mon, 30 Mar 2020 11:04:30 +0200, Norbert Lange wrote:
> > I would want to use 'set sysroot /path/to/staging/dir', but the
> > downside is that mismatches will not be detected and can badly
> > mess up debugging.
> 
> You should have setup symlinks in /usr/lib/debug/.build-id/
> (or where you 'set debug-file-directory') for all build-ids you ever
> build.  That works for core-files.
> 
> > GDB 9.0 certainly does not use *Build-Ids from the 'remote'*,
> 
> According to a test I have made now I agree. Maybe Gary Benson will
> suggest more.

Sorry to disappoint, but I don't think I've ever done anything with
build-id.

Cheers,
Gary

Re: Automatically fetching Build ID from remote libraries and resole them locally?

[Norbert Lange]

Am Mo., 30. März 2020 um 11:19 Uhr schrieb Jan Kratochvil
<jan.kratochvil@redhat.com>:
>
> On Mon, 30 Mar 2020 11:04:30 +0200, Norbert Lange wrote:
> > I would want to use 'set sysroot /path/to/staging/dir', but the downside is that
> > mismatches will not be detected and can badly mess up debugging.
>
> You should have setup symlinks in /usr/lib/debug/.build-id/
> (or where you 'set debug-file-directory') for all build-ids you ever build.
> That works for core-files.

Thats the plan.

I verified that the coredump contains all Build-Ids with
eu-unstrip -n --core /tmp/core

But Id have to first locate the binary manually, then starting
gdb PATH_TO_BINARY PATH_TO_CORE

Whats worse is, that gdb seems to lookup for the library first (filepath),
if it exists it will use the Build-Id from this potentially unrelated
version (and use the wrong debug info if one exists),
if it does not exist, then it wont even try searching in the
debug-library (neither via name or build-id).

In other words, its completely broken unless the system you debug is
the very same (and in the same state) as the executable did run on.

If I'd want to debug a coredump now, I would have to inspect the
buildids, copy the debug-infos under the name they where used on the
target in a own sysroot.
(Almost the same as for remote debug now).

I tried both with sysroot unset (default) and set to a directory with
partially missing/outdated versions.

> > GDB 9.0 certainly does not use *Build-Ids from the 'remote'*,
>
> According to a test I have made now I agree. Maybe Gary Benson will suggest
> more.

Norbert

Re: Automatically fetching Build ID from remote libraries and resole them locally?

[Jan Kratochvil]

On Mon, 06 Apr 2020 13:31:43 +0200, Norbert Lange wrote:
> But Id have to first locate the binary manually, then starting
> gdb PATH_TO_BINARY PATH_TO_CORE
> 
> Whats worse is, that gdb seems to lookup for the library first (filepath),
> if it exists it will use the Build-Id from this potentially unrelated
> version (and use the wrong debug info if one exists),
> if it does not exist, then it wont even try searching in the
> debug-library (neither via name or build-id).
> 
> In other words, its completely broken unless the system you debug is
> the very same (and in the same state) as the executable did run on.

What you describe is behaving correctly on Fedora with my off-trunk patches
(I have not tested it now but I was fixing exactly this problem in its initial
version many years ago). If you run FSF GDB maybe Red Hat should upstream more
of the Fedora patches.


Jan

Re: Automatically fetching Build ID from remote libraries and resole them locally?

[Norbert Lange]

Hmm,

How do I read the list at https://src.fedoraproject.org/rpms/gdb/tree/master.
Are all patches applied, or are the ones starting with 'gdb-6.6' only
for that version?

Yeah, if all of them are missing, upstream is lacking alot.
An a different note, I dont know if you are a GDB maintainer,
but the topic should be the upstream version?
It was not clear to me Redhat gdb and sourceware gdb are two different things.

Am Mo., 6. Apr. 2020 um 13:49 Uhr schrieb Jan Kratochvil
<jan.kratochvil@redhat.com>:
>
> On Mon, 06 Apr 2020 13:31:43 +0200, Norbert Lange wrote:
> > But Id have to first locate the binary manually, then starting
> > gdb PATH_TO_BINARY PATH_TO_CORE
> >
> > Whats worse is, that gdb seems to lookup for the library first (filepath),
> > if it exists it will use the Build-Id from this potentially unrelated
> > version (and use the wrong debug info if one exists),
> > if it does not exist, then it wont even try searching in the
> > debug-library (neither via name or build-id).
> >
> > In other words, its completely broken unless the system you debug is
> > the very same (and in the same state) as the executable did run on.
>
> What you describe is behaving correctly on Fedora with my off-trunk patches
> (I have not tested it now but I was fixing exactly this problem in its initial
> version many years ago). If you run FSF GDB maybe Red Hat should upstream more
> of the Fedora patches.
>
>
> Jan
>

Re: Automatically fetching Build ID from remote libraries and resole them locally?

[Jan Kratochvil]

On Mon, 06 Apr 2020 14:08:55 +0200, Norbert Lange wrote:
> How do I read the list at https://src.fedoraproject.org/rpms/gdb/tree/master.
> Are all patches applied, or are the ones starting with 'gdb-6.6' only
> for that version?

All of them are applied, in this order:
	https://src.fedoraproject.org/rpms/gdb/blob/master/f/_patch_order

Although I would find it most easy to just use some Fedora virtual machine
(when you do not run Fedora natively).


> An a different note, I dont know if you are a GDB maintainer,

I am no longer.


> but the topic should be the upstream version?

This mailing list is sure about the upstream version.  Just saying the problem
is solved in Fedora version so the patches could be somehow imported.


> It was not clear to me Redhat gdb and sourceware gdb are two different things.

(sourceware == FSF == upstream) != (Fedora == Red Hat; in this case)


Jan

Re: Automatically fetching Build ID from remote libraries and resole them locally?

[Norbert Lange]

Am Mo., 6. Apr. 2020 um 14:16 Uhr schrieb Jan Kratochvil
<jan.kratochvil@redhat.com>:
>
> On Mon, 06 Apr 2020 14:08:55 +0200, Norbert Lange wrote:
> > How do I read the list at https://src.fedoraproject.org/rpms/gdb/tree/master.
> > Are all patches applied, or are the ones starting with 'gdb-6.6' only
> > for that version?
>
> All of them are applied, in this order:
>         https://src.fedoraproject.org/rpms/gdb/blob/master/f/_patch_order
>
> Although I would find it most easy to just use some Fedora virtual machine
> (when you do not run Fedora natively).

I doubt that, given that I tend to use gdb via IDE, and that we
currently internally distribute dev tools over a debian repository.
Lastly, Ill have to build a gdbserver for the target also.

>
>
> > An a different note, I dont know if you are a GDB maintainer,
>
> I am no longer.

Ok, then sorry if I came across disgruntled (I am disgruntled but you
certainly arent to blame)

>
> > but the topic should be the upstream version?
>
> This mailing list is sure about the upstream version.  Just saying the problem
> is solved in Fedora version so the patches could be somehow imported.

Thanks, gonna try applying them locally and bugging debian maintainers
to add them.
Upstream might take a while, from what it seems. If you can, please
try if you can push those patches somehow.

Norbert

Re: Automatically fetching Build ID from remote libraries and resole them locally?

[Norbert Lange]

@Jan: You said

> > GDB 9.0 certainly does not use *Build-Ids from the 'remote'*,
>
> According to a test I have made now I agree. Maybe Gary Benson will suggest
> more.

did you talk about upstream or Fedora (with your patches) here?

I got GDB compiling with almost all Fedora patches included. Ready to test soon.

I encountered some headaches when cross-compiling, it seems you have an bug
with --with-rpm=auto (the default).
(thats the output following the lines 'if test "x$with_rpm" != "xno"; then')

+ test xauto = xyes
+ test xauto = xauto
+ LIBRPM=librpm.so
+ RPM_REQUIRE=false
+ DLOPEN_REQUIRE=false
+ LIBRPM_STRING='"librpm.so"'
+ printf '%s\n' 'configure:6650: checking specific librpm version'
+ printf %s 'checking specific librpm version... '
checking specific librpm version... + HAVE_DLOPEN_LIBRPM=false
+ save_LIBS='-ldl '
+ LIBS='-ldl  -ldl'
+ test yes = yes
+ :
+ printf '%s\n' 'configure:6656: error: in `/tmp/build/gdb/gdb'\'':'
+ printf '%s\n' 'configure: error: in `/tmp/build/gdb/gdb'\'':'
configure: error: in `/tmp/build/gdb/gdb':
+ as_fn_error 'cannot run test program while cross compiling
See `config.log'\'' for more details.' 6659 5
+ as_status='cannot run test program while cross compiling
See `config.log'\'' for more details.'
+ test cannot run test program while cross compiling See
'`config.log'\''' for more details. -eq 0
/tmp/build/source/gdb/gdb/configure: line 406: test: too many arguments
+ test ''
+ printf '%s\n' 'configure: error: 6659'
configure: error: 6659
+ as_fn_exit cannot run test program while cross compiling See
'`config.log'\''' for more details.
+ set +e
+ as_fn_set_status cannot
+ return cannot
/tmp/build/source/gdb/gdb/configure: line 295: return: cannot: numeric
argument required
+ exit cannot
/tmp/build/source/gdb/gdb/configure: line 305: exit: cannot: numeric
argument required
+ exit_status=2
+ echo

Am Mo., 6. Apr. 2020 um 14:53 Uhr schrieb Norbert Lange <nolange79@gmail.com>:
>
> Am Mo., 6. Apr. 2020 um 14:16 Uhr schrieb Jan Kratochvil
> <jan.kratochvil@redhat.com>:
> >
> > On Mon, 06 Apr 2020 14:08:55 +0200, Norbert Lange wrote:
> > > How do I read the list at https://src.fedoraproject.org/rpms/gdb/tree/master.
> > > Are all patches applied, or are the ones starting with 'gdb-6.6' only
> > > for that version?
> >
> > All of them are applied, in this order:
> >         https://src.fedoraproject.org/rpms/gdb/blob/master/f/_patch_order
> >
> > Although I would find it most easy to just use some Fedora virtual machine
> > (when you do not run Fedora natively).
>
> I doubt that, given that I tend to use gdb via IDE, and that we
> currently internally distribute dev tools over a debian repository.
> Lastly, Ill have to build a gdbserver for the target also.
>
> >
> >
> > > An a different note, I dont know if you are a GDB maintainer,
> >
> > I am no longer.
>
> Ok, then sorry if I came across disgruntled (I am disgruntled but you
> certainly arent to blame)
>
> >
> > > but the topic should be the upstream version?
> >
> > This mailing list is sure about the upstream version.  Just saying the problem
> > is solved in Fedora version so the patches could be somehow imported.
>
> Thanks, gonna try applying them locally and bugging debian maintainers
> to add them.
> Upstream might take a while, from what it seems. If you can, please
> try if you can push those patches somehow.
>
> Norbert