contrasting elfutils debuginfod and clr-debug-info server

contrasting elfutils debuginfod and clr-debug-info server

[Frank Ch. Eigler]

Hi -

Through some comments on phoronix [0], just became (re?)aware of the
neat clear-linux debuginfo server facility [1].  It's similar to the
nixos one we have mentioned before in that it relies on a local fuse
server expose remote debuginfo artifacts under /usr/src,
/usr/lib/debug etc.  to local tools.  The server side relies on a web
server [2] that has a carefully unpacked, unified-namespace tree of
all the artifacts into individual .tar files.  This has some
pros/cons, naturally.

Some contrasts in a tabular form:

                    clear-linux                                elfutils

indexed             by filename                                by buildid
client side sw      fuse server                                apprx. none
server side sw      plain file server                          debuginfod
server side disk    unpacked, unified-namespace layout         orig pkg set
multi-version       maybe, if /usr/src/XXXX filenames unique   no problem
multi-arch          req. separate servers                      no problem
multi-distro        req. separate servers                      one server ok

Neat system.  I wonder how interoperation would work.
(cc:'ing a few clearlinux folks just for their information.)


[0] https://www.phoronix.com/scan.php?page=news_item&px=Red-Hat-FOSDEM-debuginfod
[1] https://docs.01.org/clearlinux/latest/guides/clear/debug.html
[2] https://download.clearlinux.org/debuginfo/

- FChE

Re: contrasting elfutils debuginfod and clr-debug-info server

[Arjan van de Ven]

On 2/5/2020 9:32 AM, Frank Ch. Eigler wrote:
> Hi -
> 
> Through some comments on phoronix [0], just became (re?)aware of the
> neat clear-linux debuginfo server facility [1].  It's similar to the
> nixos one we have mentioned before in that it relies on a local fuse
> server expose remote debuginfo artifacts under /usr/src,
> /usr/lib/debug etc.  to local tools.  The server side relies on a web
> server [2] that has a carefully unpacked, unified-namespace tree of
> all the artifacts into individual .tar files.  This has some
> pros/cons, naturally.
> 
> Some contrasts in a tabular form:
> 
>                      clear-linux                                elfutils
> 
> indexed             by filename                                by buildid

actually the clear side provides both build id and filename, the tools on the client
(gdb and the like) try buildid first, and fall back to filename, and we just do both

one of our requirements was that the server side is just "flat http files" so that no
active components exist on the server (so that CDNs and mirrors scale this out with low
friction)

Re: contrasting elfutils debuginfod and clr-debug-info server

[Frank Ch. Eigler]

Hi, Arjan -

> >                      clear-linux                                elfutils
> > 
> > indexed             by filename                                by buildid

> actually the clear side provides both build id and filename, the
> tools on the client (gdb and the like) try buildid first, and fall
> back to filename, and we just do both

AIUI that works by using the "/usr/lib/debug/.build-id/HE/XCODE" as the
key, i.e., the build-id as a file name.

> one of our requirements was that the server side is just "flat http
> files" so that no active components exist on the server (so that
> CDNs and mirrors scale this out with low friction)

Yup, that makes sense.  In our case, aggressive caching with frontend
CDNs should also be possible, but staging the data whole at a CDN is
not so easy.

- FChE

Re: contrasting elfutils debuginfod and clr-debug-info server

[Arjan van de Ven]

On 2/5/2020 11:14 AM, Frank Ch. Eigler wrote:
> Hi, Arjan -
> 
>>>                       clear-linux                                elfutils
>>>
>>> indexed             by filename                                by buildid
> 
>> actually the clear side provides both build id and filename, the
>> tools on the client (gdb and the like) try buildid first, and fall
>> back to filename, and we just do both
> 
> AIUI that works by using the "/usr/lib/debug/.build-id/HE/XCODE" as the
> key, i.e., the build-id as a file name.

sure but it's not "/usr/bin/true" of which  you might have different versions over
the lifespan of a distro through its updates

(e.g. you can connect gdb to a process that got updated on disk later and still get the right answers)


the disinction of having the "/.build-id/" prefix versus just the code is not a huge difference
(so all the things you can do with just the number you can do with the prefix added as well)

Re: contrasting elfutils debuginfod and clr-debug-info server

[Mark Wielaard]

Hi,

On Wed, Feb 05, 2020 at 12:32:49PM -0500, Frank Ch. Eigler wrote:
> Through some comments on phoronix [0], just became (re?)aware of the
> neat clear-linux debuginfo server facility [1].  It's similar to the
> nixos one we have mentioned before in that it relies on a local fuse
> server expose remote debuginfo artifacts under /usr/src,
> /usr/lib/debug etc.  to local tools.  The server side relies on a web
> server [2] that has a carefully unpacked, unified-namespace tree of
> all the artifacts into individual .tar files.  This has some
> pros/cons, naturally.

This looks nice. I wonder if we can somehow make debuginfod-client a
backend for a similar Fuse system.

> Some contrasts in a tabular form:
> 
>                     clear-linux                                elfutils
> 
> indexed             by filename                                by buildid
> client side sw      fuse server                                apprx. none
> server side sw      plain file server                          debuginfod
> server side disk    unpacked, unified-namespace layout         orig pkg set
> multi-version       maybe, if /usr/src/XXXX filenames unique   no problem

When using newer rpm then the filenames should be unique:
https://fedoraproject.org/wiki/Changes/ParallelInstallableDebuginfo

But I am not sure how it would handle multiple versions of
executables.

> multi-arch          req. separate servers                      no problem
> multi-distro        req. separate servers                      one server ok

One of the use cases of debuginfod is working with local/team builds
for binaries not packaged by a distro. How does clear-linux handle that?

Cheers,

Mark

Re: contrasting elfutils debuginfod and clr-debug-info server

[Mark Wielaard]

Hi,

On Wed, Feb 05, 2020 at 02:05:42PM -0800, Arjan van de Ven wrote:
> On 2/5/2020 11:14 AM, Frank Ch. Eigler wrote:
> > > >                       clear-linux                                elfutils
> > > > 
> > > > indexed             by filename                                by buildid
> > 
> > > actually the clear side provides both build id and filename, the
> > > tools on the client (gdb and the like) try buildid first, and fall
> > > back to filename, and we just do both
> > 
> > AIUI that works by using the "/usr/lib/debug/.build-id/HE/XCODE" as the
> > key, i.e., the build-id as a file name.
> 
> sure but it's not "/usr/bin/true" of which  you might have different versions over
> the lifespan of a distro through its updates
> 
> (e.g. you can connect gdb to a process that got updated on disk later and still get the right answers)

Do you also handle the core file use case? There you might not have
the right /usr/bin/true version on disk.

Recent Fedora releases use /usr/lib/.build-id which you could provide
through the Fuse file system. On Fedora those are symlinks to the
binaries installed, but you could provide the actual binaries
instead. That would be somewhat similar to the debuginfod
/buildid/BUILDID/executable request then.

Cheers,

Mark

Re: contrasting elfutils debuginfod and clr-debug-info server

[Arjan van de Ven]

On 2/7/2020 1:06 AM, Mark Wielaard wrote:
>> Some contrasts in a tabular form:
>>
>>                      clear-linux                                elfutils
>>
>> indexed             by filename                                by buildid
>> client side sw      fuse server                                apprx. none
>> server side sw      plain file server                          debuginfod
>> server side disk    unpacked, unified-namespace layout         orig pkg set
>> multi-version       maybe, if /usr/src/XXXX filenames unique   no problem
> 
> When using newer rpm then the filenames should be unique:
> https://fedoraproject.org/wiki/Changes/ParallelInstallableDebuginfo
> 
> But I am not sure how it would handle multiple versions of
> executables.

since all tools ask for those by buildid those actually work even easier
> 
>> multi-arch          req. separate servers                      no problem
>> multi-distro        req. separate servers                      one server ok
> 
> One of the use cases of debuginfod is working with local/team builds
> for binaries not packaged by a distro. How does clear-linux handle that?

we allow for multiple URLs on the client side
(this is also how we do mirrors/etc)

supporting mirrors/CDN/... for the common files is imho almost more critical as supporting
multi-distro kind of things.. so a table like this should have a line for that.

Re: contrasting elfutils debuginfod and clr-debug-info server

[Arjan van de Ven]

On 2/7/2020 1:13 AM, Mark Wielaard wrote:
> Hi,
> 
> On Wed, Feb 05, 2020 at 02:05:42PM -0800, Arjan van de Ven wrote:
>> On 2/5/2020 11:14 AM, Frank Ch. Eigler wrote:
>>>>>                        clear-linux                                elfutils
>>>>>
>>>>> indexed             by filename                                by buildid
>>>
>>>> actually the clear side provides both build id and filename, the
>>>> tools on the client (gdb and the like) try buildid first, and fall
>>>> back to filename, and we just do both
>>>
>>> AIUI that works by using the "/usr/lib/debug/.build-id/HE/XCODE" as the
>>> key, i.e., the build-id as a file name.
>>
>> sure but it's not "/usr/bin/true" of which  you might have different versions over
>> the lifespan of a distro through its updates
>>
>> (e.g. you can connect gdb to a process that got updated on disk later and still get the right answers)
> 
> Do you also handle the core file use case? There you might not have
> the right /usr/bin/true version on disk.

but the core file has the buildid in it (usually)....

Re: contrasting elfutils debuginfod and clr-debug-info server

[Mark Wielaard]

Hi Arjan,

On Fri, 2020-02-07 at 06:47 -0800, Arjan van de Ven wrote:
> On 2/7/2020 1:13 AM, Mark Wielaard wrote:
> > On Wed, Feb 05, 2020 at 02:05:42PM -0800, Arjan van de Ven wrote:
> > > On 2/5/2020 11:14 AM, Frank Ch. Eigler wrote:
> > > > AIUI that works by using the "/usr/lib/debug/.build-
> > > > id/HE/XCODE" as the
> > > > key, i.e., the build-id as a file name.
> > > 
> > > sure but it's not "/usr/bin/true" of which  you might have
> > > different versions over
> > > the lifespan of a distro through its updates
> > > 
> > > (e.g. you can connect gdb to a process that got updated on disk
> > > later and still get the right answers)
> > 
> > Do you also handle the core file use case? There you might not have
> > the right /usr/bin/true version on disk.
> 
> but the core file has the buildid in it (usually)....

Yes. I just simply don't know how the clr_debug_fuse.service handles
that. I can see how it could handle that. But I don't know if it
currently does.

I assumed that the buildid.tar files on 
https://download.clearlinux.org/debuginfo/lib/.build-id/ would contain
the ELF image corresponding to the build-id. Just like the
buildid.debug.tar contains the separate .debug image. At least, that
would be how I would handle it. But it seems those files contain
symlinks. How do those symlinks get resolved?

The use case here is a core file generated on an older or newer
install. So you might not have the precise binary version installed on
your local system. Does the clr_debug_fuse.service provide the binary
ELF image corresponding to the buildid found in the core file? Or does
it (currently) only provide the .debug file?

Thanks,

Mark

Re: contrasting elfutils debuginfod and clr-debug-info server

[Arjan van de Ven]

On 2/7/2020 3:47 PM, Mark Wielaard wrote:
> Hi Arjan,
> 
> I assumed that the buildid.tar files on
> https://download.clearlinux.org/debuginfo/lib/.build-id/ would contain
> the ELF image corresponding to the build-id. Just like the
> buildid.debug.tar contains the separate .debug image. At least, that
> would be how I would handle it. But it seems those files contain
> symlinks. How do those symlinks get resolved?

I think you just found a bug, and I'll take that up with the current maintainer ;-)
we used to "resolve" symlinks on the server by replacing them at the time of content
preparation with their target, so that they become permanently the right content

seems some shuffle broke that; we'll get that fixed