Local Build ID Directory Lookup (DEBUGINFOD_LOCAL_PATH)

Local Build ID Directory Lookup (DEBUGINFOD_LOCAL_PATH)

[Daniel Thornburgh]

[-- Attachment #1: Type: text/plain, Size: 2276 bytes --]

Hi elfutils-devel@,

I've been working on adding debuginfod support to various LLVM utilities,
while simultaneously setting up production use of debuginfod for a managed
developer environment.

One issue that keeps cropping up is that there's often quite a few places
to get debug files from locally, in addition to various remote backends.
For example, the output of local build systems, prebuilt packages and
tarballs that a developer is working on, etc.

One of the best things about debuginfod is that you can set up a service
once, then point an astonishingly wide array of tools at the server, with
one configuration.

This isn't the case for files available locally, but there is some prior
art: GDB's build ID directory layout (commonly
`/usr/lib/debug/.build-id/<first 2 chars of build ID>/<remainder of build
ID>`). GDB has a flag to control this (`--with-separate-debug-dir`), but
that flag isn't shared by other tools. For example, binutils supports this
layout, but it simply hardcodes a list of directories to search for, with
no way to extend or override it. LLVM also supports this, but has a
differently named flag for it.

Local files can be served to localhost using a debuginfod server, but this
incurs the overhead of transferring the files over a local socket on first
access, and the much worse overhead of storing an additional copy of the
file on disk in the debuginfod cache. With sufficiently large projects,
this gets pretty rough.

It would be really nice if there were, a DEBUGINFOD_LOCAL_PATH environment
variable that provided a colon-separated list of directories in the GDB
`.build-id/<xx>/<xxxxxx>` format to locally fetch files out of before
attempting remote servers. This would allow imbuing local build ID fetching
to tools that wouldn't otherwise support it, with one common configuration
language for how to do it.

I did definitely want to raise this on the mailing list before starting to
implement something like this though, since it increases the scope of
libdebuginfod to include local fetching out of more than just out of its
own cache. It would stretch debuginfod towards a more general "library for
obtaining debug info writ large", rather than a straightforward remote
client library.

Daniel Thornburgh | dthorn@google.com

Re: Local Build ID Directory Lookup (DEBUGINFOD_LOCAL_PATH)

[Paul Smith]

On Wed, 2023-05-31 at 15:17 -0700, Daniel Thornburgh via Elfutils-devel
wrote:
> This isn't the case for files available locally, but there is some
> prior art: GDB's build ID directory layout (commonly
> `/usr/lib/debug/.build-id/<first 2 chars of build ID>/<remainder of
> build ID>`). GDB has a flag to control this (`--with-separate-debug-
> dir`), but that flag isn't shared by other tools. For example,
> binutils supports this layout, but it simply hardcodes a list of
> directories to search for, with no way to extend or override it. LLVM
> also supports this, but has a differently named flag for it.

Having content in /usr/lib is less good.  It's perfect for allowing
package managers to install debug info files for packages, but it's not
good for allowing users to either download or "publish" their own info
files.

Couldn't the standard XDG environment variables be used to generate a
search path or paths, rather than inventing new ones (or, the new
variable can be used to override the default which is computed based on
XDG), to find the user's personal build ID directories?  This location
can be updated without root, can easily be made available inside
containers (flatpak / snap), etc.

If everyone could agree on a naming convention so that all tools that
need debug info could write them to a common location after download
(or even on generation), and check that location for files first before
querying debuginfod, that would be a great "it just works" feature and
avoid a lot of duplication.

Or, maybe I'm just misunderstanding something :).

Re: Local Build ID Directory Lookup (DEBUGINFOD_LOCAL_PATH)

[Frank Ch. Eigler]

Hi -

> [...]
> It would be really nice if there were, a DEBUGINFOD_LOCAL_PATH environment
> variable that provided a colon-separated list of directories in the GDB
> `.build-id/<xx>/<xxxxxx>` [...]

Please note that $DEBUGINFOD_URLS can already include file:// URLs
that identify local filesystems whose directory structure mirrors the
webapi.  The old-school /usr/lib/debug/.build-id/xx/xxxxxxxx.debug
file hierarchy doesn't match that, and of course lacks source code
entirely.  

OTOH a shell script that creates debuginfod-style tree of symlinks
into the old-school hierarchy - for debuginfo/executables only - could
work, and work with current day unmodified debuginfod clients &
servers.  i.e.,:

          find /usr/lib/debug/.build-id -name '*.debug' | while read g; do
               buildid=`echo $g | cut -f6,7 -d/ | tr -d / | cut -f1 -d.`
               mkdir -p buildid/$buildid
               ln -s $g buildid/$buildid/debuginfo
          done

and similar for .../executable

Then the resulting tree is suitable for use as DEBUGINFOD_URLS=file://`pwd`
to a debuginfod server or a client.


- FChE

Re: Local Build ID Directory Lookup (DEBUGINFOD_LOCAL_PATH)

[Daniel Thornburgh]

[-- Attachment #1: Type: text/plain, Size: 1166 bytes --]

On Wed, May 31, 2023 at 3:35 PM Frank Ch. Eigler <fche@redhat.com> wrote:

> > [...]
> > It would be really nice if there were, a DEBUGINFOD_LOCAL_PATH
> environment
> > variable that provided a colon-separated list of directories in the GDB
> > `.build-id/<xx>/<xxxxxx>` [...]
>
> Please note that $DEBUGINFOD_URLS can already include file:// URLs
> that identify local filesystems whose directory structure mirrors the
> webapi.  The old-school /usr/lib/debug/.build-id/xx/xxxxxxxx.debug
> file hierarchy doesn't match that, and of course lacks source code
> entirely.
>

Ah, I had forgotten completely that file URLs worked out of the box
(libcurl, duh).
When we tried that way back when, the issue was just that it actually does
a libcurl fetch
out of that path and saves another copy of the file into the cache.

So I guess, sans the format, the feature request would just be that it
would have a shortcut
for file URLs to produce the path directly in response to e.g. a
debuginfod_find_debuginfo,
rather than making a copy of the file via libcurl.

Does that seem like a reasonable patch?

Daniel Thornburgh | dthorn@google.com

Re: Local Build ID Directory Lookup (DEBUGINFOD_LOCAL_PATH)

[Frank Ch. Eigler]

Hi -

> Ah, I had forgotten completely that file URLs worked out of the box
> (libcurl, duh).

Yeah.

> When we tried that way back when, the issue was just that it
> actually does a libcurl fetch out of that path and saves another
> copy of the file into the cache.

Yes, a copy is made.  This is "traditional" in the sense that the
client code ensures that even if an original upstream server goes
away, cached files stay accessible for a while.  I can also see how
this might be less valuable than just getting at the files with
minimum overhead.  Does anyone think we need to have both as options?

- FChE

Re: Local Build ID Directory Lookup (DEBUGINFOD_LOCAL_PATH)

[Roland McGrath]

[-- Attachment #1: Type: text/plain, Size: 4475 bytes --]

In elfutils, the libdwfl library is responsible for the "library for
obtaining debug info writ large" stuff. It provides <argp.h> API parsers
for command-line arguments that mesh with its library stuff like
Dwfl_Callbacks that set the search path used equivalently to GDB's `set
debug-file-directory` path. eu-* tools as well as things like systemtap use
this to accept `--debuginfo-path=PATH`, but AFAIK none of them support
environment variables.  In the binutils world outside of GDB itself, I
think only objdump and readelf do .build-id/... lookup and those use common
binutils-private code (binutils/dwarf.c) that just has a hard-coded
directory list.  Some more uniformity here would be beneficial to everyone,
I think.

On Wed, May 31, 2023 at 3:17 PM Daniel Thornburgh <dthorn@google.com> wrote:

> Hi elfutils-devel@,
>
> I've been working on adding debuginfod support to various LLVM utilities,
> while simultaneously setting up production use of debuginfod for a managed
> developer environment.
>
> One issue that keeps cropping up is that there's often quite a few places
> to get debug files from locally, in addition to various remote backends.
> For example, the output of local build systems, prebuilt packages and
> tarballs that a developer is working on, etc.
>
> One of the best things about debuginfod is that you can set up a service
> once, then point an astonishingly wide array of tools at the server, with
> one configuration.
>
> This isn't the case for files available locally, but there is some prior
> art: GDB's build ID directory layout (commonly
> `/usr/lib/debug/.build-id/<first 2 chars of build ID>/<remainder of build
> ID>`). GDB has a flag to control this (`--with-separate-debug-dir`), but
> that flag isn't shared by other tools. For example, binutils supports this
> layout, but it simply hardcodes a list of directories to search for, with
> no way to extend or override it. LLVM also supports this, but has a
> differently named flag for it.
>
> Local files can be served to localhost using a debuginfod server, but this
> incurs the overhead of transferring the files over a local socket on first
> access, and the much worse overhead of storing an additional copy of the
> file on disk in the debuginfod cache. With sufficiently large projects,
> this gets pretty rough.
>
> It would be really nice if there were, a DEBUGINFOD_LOCAL_PATH environment
> variable that provided a colon-separated list of directories in the GDB
> `.build-id/<xx>/<xxxxxx>` format to locally fetch files out of before
> attempting remote servers. This would allow imbuing local build ID fetching
> to tools that wouldn't otherwise support it, with one common configuration
> language for how to do it.
>
> I did definitely want to raise this on the mailing list before starting to
> implement something like this though, since it increases the scope of
> libdebuginfod to include local fetching out of more than just out of its
> own cache. It would stretch debuginfod towards a more general "library for
> obtaining debug info writ large", rather than a straightforward remote
> client library.
>
> Daniel Thornburgh | dthorn@google.com
>
> --
> You received this message because you are subscribed to the Google Groups
> "(retired) tq-toolchain-team" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to tq-toolchain-team+unsubscribe@google.com.
> To view this discussion on the web visit
> https://groups.google.com/a/google.com/d/msgid/tq-toolchain-team/CAEdiOyeCSvpUijvZwJ%3DW2ncsGokefdm8z3yKVG%2Bu4jkAgXcumw%40mail.gmail.com
> <https://groups.google.com/a/google.com/d/msgid/tq-toolchain-team/CAEdiOyeCSvpUijvZwJ%3DW2ncsGokefdm8z3yKVG%2Bu4jkAgXcumw%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>
> --
> You received this message because you are subscribed to the Google Groups
> "rvos-toolchain-team" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to rvos-toolchain-team+unsubscribe@google.com.
> To view this discussion on the web visit
> https://groups.google.com/a/google.com/d/msgid/rvos-toolchain-team/CAEdiOyeCSvpUijvZwJ%3DW2ncsGokefdm8z3yKVG%2Bu4jkAgXcumw%40mail.gmail.com
> <https://groups.google.com/a/google.com/d/msgid/rvos-toolchain-team/CAEdiOyeCSvpUijvZwJ%3DW2ncsGokefdm8z3yKVG%2Bu4jkAgXcumw%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>

Re: Local Build ID Directory Lookup (DEBUGINFOD_LOCAL_PATH)

[Frank Ch. Eigler]

Hi -

> So I guess, sans the format, the feature request would just be that
> it would have a shortcut for file URLs to produce the path directly
> in response to e.g. a debuginfod_find_debuginfo, rather than making
> a copy of the file via libcurl.

A compromise solution could be for new code to produce a symlink in
the .cache/debuginfod_client directory that points to the matching
file:// bit, and return that symlink name/fd to the calling app.

At future accesses, the client can determine if the symlink is
broken and reject/unlink it.

- FChE

Re: Local Build ID Directory Lookup (DEBUGINFOD_LOCAL_PATH)

[Daniel Thornburgh]

[-- Attachment #1: Type: text/plain, Size: 1346 bytes --]

Hmm, how would the effective behavior of this differ from directly
returning the path? The symlink could become invalid at any time, and it
would become invalid in precisely the same scenarios that the original file
would. A further request would in both cases stat the original file, and if
it were missing, download a new copy into the cache via the usual
mechanisms.

It would make sense if the cache were made to contain a hard link to the
file if it were on the same filesystem as the cache, but that inherits the
usual problems with hard links. It would persist a copy of the file in the
cache though.

On Thu, Jun 1, 2023 at 7:58 AM Frank Ch. Eigler <fche@redhat.com> wrote:

> Hi -
>
> > So I guess, sans the format, the feature request would just be that
> > it would have a shortcut for file URLs to produce the path directly
> > in response to e.g. a debuginfod_find_debuginfo, rather than making
> > a copy of the file via libcurl.
>
> A compromise solution could be for new code to produce a symlink in
> the .cache/debuginfod_client directory that points to the matching
> file:// bit, and return that symlink name/fd to the calling app.
>
> At future accesses, the client can determine if the symlink is
> broken and reject/unlink it.
>
> - FChE
>
>

-- 

Daniel Thornburgh | dthorn@google.com

Re: Local Build ID Directory Lookup (DEBUGINFOD_LOCAL_PATH)

[Frank Ch. Eigler]

Hi -

> Hmm, how would the effective behavior of this differ from directly
> returning the path? The symlink could become invalid at any time [...]

Effective behaviour is about the same, but code logic and explanation
is simpler.

> It would make sense if the cache were made to contain a hard link to the
> file if it were on the same filesystem as the cache [...]

And this could be a QoI implementation detail: prefer hardlink, fall
back to symlink.

- FChE

Re: Local Build ID Directory Lookup (DEBUGINFOD_LOCAL_PATH)

[Daniel Thornburgh]

[-- Attachment #1: Type: text/plain, Size: 717 bytes --]

SGTM; I'll see if I can write a patch to do this when I get some spare
cycles.

On Thu, Jun 1, 2023 at 2:21 PM Frank Ch. Eigler <fche@redhat.com> wrote:

> Hi -
>
> > Hmm, how would the effective behavior of this differ from directly
> > returning the path? The symlink could become invalid at any time [...]
>
> Effective behaviour is about the same, but code logic and explanation
> is simpler.
>
> > It would make sense if the cache were made to contain a hard link to the
> > file if it were on the same filesystem as the cache [...]
>
> And this could be a QoI implementation detail: prefer hardlink, fall
> back to symlink.
>
> - FChE
>
>

-- 

Daniel Thornburgh | dthorn@google.com

Re: Local Build ID Directory Lookup (DEBUGINFOD_LOCAL_PATH)

[Mark Wielaard]

Hi Roland,

On Wed, 2023-05-31 at 16:44 -0700, Roland McGrath via Elfutils-devel
wrote:
> In elfutils, the libdwfl library is responsible for the "library for
> obtaining debug info writ large" stuff. It provides <argp.h> API parsers
> for command-line arguments that mesh with its library stuff like
> Dwfl_Callbacks that set the search path used equivalently to GDB's `set
> debug-file-directory` path. eu-* tools as well as things like systemtap use
> this to accept `--debuginfo-path=PATH`, but AFAIK none of them support
> environment variables.  In the binutils world outside of GDB itself, I
> think only objdump and readelf do .build-id/... lookup and those use common
> binutils-private code (binutils/dwarf.c) that just has a hard-coded
> directory list.  Some more uniformity here would be beneficial to everyone,
> I think.

One difference is that the elfutils and gdb traditional "search path"
mechanism is mainly based on path/file names (lookups through
.gnu_debuglink, possibly relative .debug dirs/files) while the primary
lookup method for libdebuginfod is based on build-id.

libdebuginfod kind of assumes you already know the traditional lookup
didn't work.

That is also how dwfl_standard_find_debuginfo uses libdebuginfod. It
first gets the build-id from the ELF file, looks for that in the
traditional way (/usr/lib/debug/.build-id/xx/yyyy.debug), if that fails
it does a path based search (using the file name, debuglink file name
and Dwfl_Callbacks debuginfo_path), and if that also fails then it
tries to use libdebuginfod.

Another difference is the lookup of .dwz files. Since they contain an
"artificial" build-id, libdebuginfod uses that to look them up as if a
"normal" .debug file. While the traditional lookup uses the (relative!)
path in the .gnu_debugaltlink section of the .debug file.

I wonder which parts of the "traditional" lookup are still
useful/relevant. Could we get away with pointing at a "traditional"
/usr/lib/debug/ like path that contains a .build-id and (optional) .dwz
subdir to make libdebuginfo that "uniform" debug file lookup library?

Cheers,

Mark

Re: Local Build ID Directory Lookup (DEBUGINFOD_LOCAL_PATH)

[Roland McGrath]

[-- Attachment #1: Type: text/plain, Size: 311 bytes --]

Personally I'm not concerned with any non-build-ID use cases any more. I
don't know if the rest of the world is OK with presuming that build
ID-based lookup is always the only thing you want nowadays.  But it seems
plausible, since we rolled out Build ID in 2008 and it's been pretty
thoroughly adopted by now.

Re: Local Build ID Directory Lookup (DEBUGINFOD_LOCAL_PATH)

[Mark Wielaard]

Hi Roland,

On Wed, 2023-06-14 at 12:43 -0700, Roland McGrath wrote:
> Personally I'm not concerned with any non-build-ID use cases any more.
> I don't know if the rest of the world is OK with presuming that build
> ID-based lookup is always the only thing you want nowadays.
> But it seems plausible, since we rolled out Build ID in 2008 and it's
> been pretty thoroughly adopted by now.

Agreed. And if an application uses libdebuginfod it kind of assumes
build-ids are always there anyway. Because there is no interface for
requesting something without a build-id :)

So that leaves the wrinkle of finding the .dwz files. And how to
connect the debuginfo to the debugsources. When pointing to a
"traditional" build-id based directory like /usr/lib/debug/ and
/usr/src/debug/

Cheers,

Mark