[Bug gdb/17917] New: gdbserver should lookup build-ids in the remote binaries

[Bug gdb/17917] New: gdbserver should lookup build-ids in the remote binaries

[eseidel at google dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=17917

            Bug ID: 17917
           Summary: gdbserver should lookup build-ids in the remote
                    binaries
           Product: gdb
           Version: unknown
            Status: NEW
          Severity: normal
          Priority: P2
         Component: gdb
          Assignee: unassigned at sourceware dot org
          Reporter: eseidel at google dot com

gdbserver should lookup build-ids in the remote binaries

Right now gdbserver only sends the library_name back to gdb and gdb looks for
library_name in $PATH, etc.  Only if it finds library_name on the local machine
*then* it begins the .build-id search using the build-id it found in the local
binary.  In my application I don't always have a local binary (or have an easy
way to pull the binary from the remote), but I do have a .build-id tree full of
links to the symbold versions of all binarys on the remote.  Having gdbserver
send the build-id to gdb would be useful.


Steps:

1. Build your program (and libraries) with build-ids enabled
gcc -Wlbuild-id -o my_library.so library.

2. Setup a .build-id directory tree:
https://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html
e.g.:
% readelf -n -e my_library.so
build-id: abcdef123456

mkdir -p .build-id/ab
ln -s my_libary_with_symbols.so .build-id/ab/cdef123456.debug

3. Launch under gdbserver on a remote system.
gdbserver :1234 my_program

4.  Connect to remote system:
gdb my_program
(gdb) target remote 1.2.3.4:1234

5.  Be sad that my_library has no symbols, even though you have a .build-id
directory for it.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/17917] gdbserver should lookup build-ids in the remote binaries

[daniel at mariadb dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=17917

daniel at mariadb dot org <daniel at mariadb dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |daniel at mariadb dot org

--- Comment #1 from daniel at mariadb dot org <daniel at mariadb dot org> ---
Totally agree. In trying to make self contained debug environments like
containers, to help users with developers tackle tricky problems:

Example:

$ podman run -d --rm -p 2345  --cap-add CAP_SYS_PTRACE --name mdb-debug -e
MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1  
quay.io/mariadb-foundation/mariadb-debug:10.5
1f2f0246ad656dad10be8cacedf312f94f7b98651906f29d09f07dbf048c4f70

$ podman port mdb-debug 2345
0.0.0.0:44663

$ podman exec -ti  mdb-debug gosu mysql gdbserver --attach  :2345 1
Attached; pid = 1
Listening on port 2345


$ gdb
GNU gdb (GDB) Fedora 10.2-3.fc34
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
(gdb) set debug separate-debug-file 1
(gdb) target extended-remote  :44663
Remote debugging using :44663
Reading /usr/sbin/mariadbd from remote target...
warning: File transfers from remote targets can be slow. Use "set sysroot" to
access files locally instead.
Reading /usr/sbin/mariadbd from remote target...
Reading symbols from target:/usr/sbin/mariadbd...

Looking for separate debug info (build-id) for target:/usr/sbin/mariadbd
  Trying
/usr/lib/debug/.build-id/57/c35022dc84dd28479db41cc278f625776c4adc.debug...
Looking for separate debug info (debug link) for target:/usr/sbin/mariadbd
  Trying
target:/usr/sbin/c35022dc84dd28479db41cc278f625776c4adc.debug...Reading
/usr/sbin/c35022dc84dd28479db41cc278f625776c4adc.debug from remote target...
 no, unable to open.
  Trying
target:/usr/sbin/.debug/c35022dc84dd28479db41cc278f625776c4adc.debug...Reading
/usr/sbin/.debug/c35022dc84dd28479db41cc278f625776c4adc.debug from remote
target...
 no, unable to open.
  Trying
target:/usr/lib/debug//usr/sbin/c35022dc84dd28479db41cc278f625776c4adc.debug...Reading
/usr/lib/debug//usr/sbin/c35022dc84dd28479db41cc278f625776c4adc.debug from
remote target...
 no, unable to open.
  Trying
target:/usr/lib/debug/usr/sbin//c35022dc84dd28479db41cc278f625776c4adc.debug...Reading
/usr/lib/debug/usr/sbin//c35022dc84dd28479db41cc278f625776c4adc.debug from
remote target...
 no, unable to open.
  Trying
target:target:/usr/lib/debug/usr/sbin//c35022dc84dd28479db41cc278f625776c4adc.debug...Reading
target:/usr/lib/debug/usr/sbin//c35022dc84dd28479db41cc278f625776c4adc.debug
from remote target...
 no, unable to open.
Missing separate debuginfo for target:/usr/sbin/mariadbd
Try: dnf --enablerepo='*debug*' install
/usr/lib/debug/.build-id/57/c35022dc84dd28479db41cc278f625776c4adc.debug
(No debugging symbols found in target:/usr/sbin/mariadbd)


/usr/lib/debug/.build-id/57/c35022dc84dd28479db41cc278f625776c4adc.debug
existed on the target if gdb asked.


podman exec -ti  mdb-debug ls -la 
/usr/lib/debug/.build-id/57/c35022dc84dd28479db41cc278f625776c4adc.debug

-rw-r--r--. 1 root root 120833256 Aug 27 11:26
/usr/lib/debug/.build-id/57/c35022dc84dd28479db41cc278f625776c4adc.debug

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/17917] gdbserver should lookup build-ids in the remote binaries

[daniel at mariadb dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=17917

--- Comment #2 from daniel at mariadb dot org <daniel at mariadb dot org> ---
With https://sourceware.org/pipermail/gdb-patches/2021-August/181718.html


$ ./gdb/gdb 
GNU gdb (GDB) 12.0.50.20210828-git
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
(gdb)  set debug separate-debug-file 1
(gdb)  target extended-remote  :44663
Remote debugging using :44663
warning: Can not parse XML target description; XML support was disabled at
compile time
Reading /usr/sbin/mariadbd from remote target...
warning: File transfers from remote targets can be slow. Use "set sysroot" to
access files locally instead.
Reading /usr/sbin/mariadbd from remote target...
Reading symbols from target:/usr/sbin/mariadbd...

Looking for separate debug info (build-id) for target:/usr/sbin/mariadbd
  Trying
/usr/lib/debug/.build-id/57/c35022dc84dd28479db41cc278f625776c4adc.debug... no,
unable to compute real path
  Trying
target:/usr/lib/debug/.build-id/57/c35022dc84dd28479db41cc278f625776c4adc.debug...Reading
/usr/lib/debug/.build-id/57/c35022dc84dd28479db41cc278f625776c4adc.debug from
remote target...
 yes!
Reading
/usr/lib/debug/.build-id/57/c35022dc84dd28479db41cc278f625776c4adc.debug from
remote target...
Reading symbols from
target:/usr/lib/debug/.build-id/57/c35022dc84dd28479db41cc278f625776c4adc.debug...
Reading /lib/x86_64-linux-gnu/libpcre2-8.so.0 from remote target...
Reading /lib/x86_64-linux-gnu/libcrypt.so.1 from remote target...
Reading /lib/x86_64-linux-gnu/liblz4.so.1 from remote target...
Reading /lib/x86_64-linux-gnu/libsystemd.so.0 from remote target...
Reading /lib/x86_64-linux-gnu/libaio.so.1 from remote target...
Reading /lib/x86_64-linux-gnu/libz.so.1 from remote target...
Reading /lib/x86_64-linux-gnu/libssl.so.1.1 from remote target...
Reading /lib/x86_64-linux-gnu/libcrypto.so.1.1 from remote target...
Reading /lib/x86_64-linux-gnu/libpthread.so.0 from remote target...
Reading /lib/x86_64-linux-gnu/libdl.so.2 from remote target...
Reading /lib/x86_64-linux-gnu/libstdc++.so.6 from remote target...
Reading /lib/x86_64-linux-gnu/libm.so.6 from remote target...
Reading /lib/x86_64-linux-gnu/libc.so.6 from remote target...
Reading /lib64/ld-linux-x86-64.so.2 from remote target...
Reading /lib/x86_64-linux-gnu/librt.so.1 from remote target...
Reading /lib/x86_64-linux-gnu/liblzma.so.5 from remote target...
Reading /lib/x86_64-linux-gnu/libgcrypt.so.20 from remote target...
Reading /lib/x86_64-linux-gnu/libgcc_s.so.1 from remote target...
Reading /lib/x86_64-linux-gnu/libgpg-error.so.0 from remote target...
Reading /lib/x86_64-linux-gnu/libnss_files.so.2 from remote target...
Reading symbols from target:/lib/x86_64-linux-gnu/libpcre2-8.so.0...

Looking for separate debug info (build-id) for
target:/lib/x86_64-linux-gnu/libpcre2-8.so.0
  Trying
/usr/lib/debug/.build-id/dd/36d93c961c18d7164f56a10064cfe3591a7e56.debug... no,
unable to compute real path
  Trying
target:/usr/lib/debug/.build-id/dd/36d93c961c18d7164f56a10064cfe3591a7e56.debug...Reading
/usr/lib/debug/.build-id/dd/36d93c961c18d7164f56a10064cfe3591a7e56.debug from
remote target...
 yes!
Reading
/usr/lib/debug/.build-id/dd/36d93c961c18d7164f56a10064cfe3591a7e56.debug from
remote target...
Reading symbols from
target:/usr/lib/debug/.build-id/dd/36d93c961c18d7164f56a10064cfe3591a7e56.debug...
Reading symbols from target:/lib/x86_64-linux-gnu/libcrypt.so.1...

Looking for separate debug info (build-id) for
target:/lib/x86_64-linux-gnu/libcrypt.so.1
  Trying
/usr/lib/debug/.build-id/46/77a4c8111a47d124c60abe6d1a39f5408875fc.debug... no,
unable to compute real path
  Trying
target:/usr/lib/debug/.build-id/46/77a4c8111a47d124c60abe6d1a39f5408875fc.debug...Reading
/usr/lib/debug/.build-id/46/77a4c8111a47d124c60abe6d1a39f5408875fc.debug from
remote target...
 yes!
Reading
/usr/lib/debug/.build-id/46/77a4c8111a47d124c60abe6d1a39f5408875fc.debug from
remote target...
Reading symbols from
target:/usr/lib/debug/.build-id/46/77a4c8111a47d124c60abe6d1a39f5408875fc.debug...
Reading symbols from target:/lib/x86_64-linux-gnu/liblz4.so.1...

Looking for separate debug info (build-id) for
target:/lib/x86_64-linux-gnu/liblz4.so.1
  Trying
/usr/lib/debug/.build-id/f6/4a53f0f288938d86782cc5547c0ed71063ba3f.debug... no,
unable to compute real path
  Trying
target:/usr/lib/debug/.build-id/f6/4a53f0f288938d86782cc5547c0ed71063ba3f.debug...Reading
/usr/lib/debug/.build-id/f6/4a53f0f288938d86782cc5547c0ed71063ba3f.debug from
remote target...
 yes!
Reading
/usr/lib/debug/.build-id/f6/4a53f0f288938d86782cc5547c0ed71063ba3f.debug from
remote target...
Reading symbols from
target:/usr/lib/debug/.build-id/f6/4a53f0f288938d86782cc5547c0ed71063ba3f.debug...
Reading symbols from target:/lib/x86_64-linux-gnu/libsystemd.so.0...

Looking for separate debug info (build-id) for
target:/lib/x86_64-linux-gnu/libsystemd.so.0
  Trying
/usr/lib/debug/.build-id/af/9d4eb40d9c9a6f95204707e44b67ffe6865f49.debug... no,
unable to compute real path
  Trying
target:/usr/lib/debug/.build-id/af/9d4eb40d9c9a6f95204707e44b67ffe6865f49.debug...Reading
/usr/lib/debug/.build-id/af/9d4eb40d9c9a6f95204707e44b67ffe6865f49.debug from
remote target...
 yes!
Reading
/usr/lib/debug/.build-id/af/9d4eb40d9c9a6f95204707e44b67ffe6865f49.debug from
remote target...

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/17917] gdbserver should lookup build-ids in the remote binaries

[cvs-commit at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=17917

--- Comment #3 from cvs-commit at gcc dot gnu.org <cvs-commit at gcc dot gnu.org> ---
The master branch has been updated by Simon Marchi <simark@sourceware.org>:

https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=4a2f674a688df474639066922f4ce70ce97b30d2

commit 4a2f674a688df474639066922f4ce70ce97b30d2
Author: Daniel Black <daniel@mariadb.org>
Date:   Mon Oct 18 17:06:47 2021 +1100

    Fix PR gdb/17917 Lookup build-id in remote binaries

    GDB doesn't support loading debug files using build-id from remote
    target filesystems.

    This is the case when gdbserver attached to a process and a gdb target
    remote occurs over tcp.

    With this change we make build-id lookups possible:

        (gdb) show debug-file-directory
        The directory where separate debug symbols are searched for is
"/usr/local/lib/debug".
        (gdb) set debug-file-directory /usr/lib/debug
        (gdb) show sysroot
        The current system root is "target:".
        (gdb) target extended-remote :46615
        Remote debugging using :46615
        warning: Can not parse XML target description; XML support was disabled
at compile time
        Reading /usr/sbin/mariadbd from remote target...
        warning: File transfers from remote targets can be slow. Use "set
sysroot" to access files locally instead.
        Reading /usr/sbin/mariadbd from remote target...
        Reading symbols from target:/usr/sbin/mariadbd...
        Reading
/usr/lib/debug/.build-id/6e/0a874dca5a7ff831396ddc0785d939a192efe3.debug from
remote target...
        Reading
/usr/lib/debug/.build-id/6e/0a874dca5a7ff831396ddc0785d939a192efe3.debug from
remote target...
        Reading symbols from
target:/usr/lib/debug/.build-id/6e/0a874dca5a7ff831396ddc0785d939a192efe3.debug...
        Reading /lib/x86_64-linux-gnu/libpcre2-8.so.0 from remote target...
        ...

    Before this change, the lookups would have been (GNU gdb (GDB) Fedora
10.2-3.fc34):

        (gdb) target extended-remote :46615
        Remote debugging using :46615
        Reading /usr/sbin/mariadbd from remote target...
        warning: File transfers from remote targets can be slow. Use "set
sysroot" to access files locally instead.
        Reading /usr/sbin/mariadbd from remote target...
        Reading symbols from target:/usr/sbin/mariadbd...
        Reading /usr/sbin/0a874dca5a7ff831396ddc0785d939a192efe3.debug from
remote target...
        Reading /usr/sbin/.debug/0a874dca5a7ff831396ddc0785d939a192efe3.debug
from remote target...
        Reading
/usr/lib/debug//usr/sbin/0a874dca5a7ff831396ddc0785d939a192efe3.debug from
remote target...
        Reading
/usr/lib/debug/usr/sbin//0a874dca5a7ff831396ddc0785d939a192efe3.debug from
remote target...
        Reading
target:/usr/lib/debug/usr/sbin//0a874dca5a7ff831396ddc0785d939a192efe3.debug
from remote target...
        Missing separate debuginfo for target:/usr/sbin/mariadbd
        Try: dnf --enablerepo='*debug*' install
/usr/lib/debug/.build-id/6e/0a874dca5a7ff831396ddc0785d939a192efe3.debug
        (No debugging symbols found in target:/usr/sbin/mariadbd)

    Observe it didn't look for
    /usr/lib/debug/.build-id/6e/0a874dca5a7ff831396ddc0785d939a192efe3.debug
    on the remote target (where it is) and expected them to be installed
    locally.

    As a minor optimization, this also changes the build-id lookup such that
    if sysroot is empty, no second lookup of the same location is performed.

    Change-Id: I5181696d271c325a25a0805a8defb8ab7f9b3f55
    Bug: https://sourceware.org/bugzilla/show_bug.cgi?id=17917

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/17917] gdbserver should lookup build-ids in the remote binaries

[tromey at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=17917

Tom Tromey <tromey at sourceware dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |tromey at sourceware dot org
         Resolution|---                         |FIXED
             Status|NEW                         |RESOLVED
   Target Milestone|---                         |12.1

--- Comment #4 from Tom Tromey <tromey at sourceware dot org> ---
I think this is fixed now.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/17917] gdbserver should lookup build-ids in the remote binaries

[daniel at mariadb dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=17917

--- Comment #5 from daniel at mariadb dot org <daniel at mariadb dot org> ---
confirmed. Thanks Simon and Tom.

$ podman run --rm -d -e MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1 --cap-add
CAP_SYS_PTRACE --name mdb-debug -p 2345:2345 
quay.io/mariadb-foundation/mariadb-debug:10.5



$ podman exec mdb-debug ls -la
/usr/lib/debug/.build-id/../.build-id/6e/0a874dca5a7ff831396ddc0785d939a192efe3.debug
-rw-r--r--. 1 root root 120877312 Oct  6 15:21
/usr/lib/debug/.build-id/../.build-id/6e/0a874dca5a7ff831396ddc0785d939a192efe3.debug


$ podman exec -ti mdb-debug gosu mysql gdbserver --attach  :2345 1
Attached; pid = 1
Listening on port 2345

$ ./gdb
GNU gdb (GDB) 12.0.50.20211020-git
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".

(gdb) set debug-file-directory /usr/lib/debug

(gdb) target extended-remote  :2345

Remote debugging using :2345
warning: Can not parse XML target description; XML support was disabled at
compile time
Reading /usr/sbin/mariadbd from remote target...
warning: File transfers from remote targets can be slow. Use "set sysroot" to
access files locally instead.
Reading /usr/sbin/mariadbd from remote target...
Reading symbols from target:/usr/sbin/mariadbd...
Reading
/usr/lib/debug/.build-id/6e/0a874dca5a7ff831396ddc0785d939a192efe3.debug from
remote target...
Reading
/usr/lib/debug/.build-id/6e/0a874dca5a7ff831396ddc0785d939a192efe3.debug from
remote target...
Reading symbols from
target:/usr/lib/debug/.build-id/6e/0a874dca5a7ff831396ddc0785d939a192efe3.debug...
...


(gdb) bt
#0  0x00007fc9db431aff in poll () from target:/lib/x86_64-linux-gnu/libc.so.6
#1  0x000055b62ef2593f in poll (__timeout=-1, __nfds=2, __fds=0x7ffe0342f520)
at /usr/include/x86_64-linux-gnu/bits/poll2.h:41
#2  handle_connections_sockets () at ./sql/mysqld.cc:6207
#3  0x000055b62ef2674d in mysqld_main (argc=<optimized out>, argv=<optimized
out>) at ./sql/mysqld.cc:5656
#4  0x00007fc9db3430b3 in __libc_start_main () from
target:/lib/x86_64-linux-gnu/libc.so.6
#5  0x000055b62ef1adae in _start ()


Full line numbers. And the build-id from the target is read.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/17917] gdbserver should lookup build-ids in the remote binaries

[tromey at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=17917

--- Comment #6 from Tom Tromey <tromey at sourceware dot org> ---
(In reply to daniel@mariadb.org from comment #5)

> Reading /usr/sbin/mariadbd from remote target...
> warning: File transfers from remote targets can be slow. Use "set sysroot"
> to access files locally instead.

I've been doing some debugging like this recently and
sometimes I wish gdb just cached these things locally,
like it does for DWARF indices or like debuginfod does.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/17917] gdbserver should lookup build-ids in the remote binaries

[daniel at mariadb dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=17917

--- Comment #7 from daniel at mariadb dot org <daniel at mariadb dot org> ---
Yeh, totally agree. That would look nice. Doesn't seem too hard to reuse the
DWARF info as its already just a build-id cache.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug gdb/17917] gdbserver should lookup build-ids in the remote binaries

[tromey at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=17917

--- Comment #8 from Tom Tromey <tromey at sourceware dot org> ---
FWIW this follow-on idea is bug #28489.

-- 
You are receiving this mail because:
You are on the CC list for the bug.