[Bug remote/28489] New: Allow caching of remote objfiles

[Bug remote/28489] New: Allow caching of remote objfiles

[tromey at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=28489

            Bug ID: 28489
           Summary: Allow caching of remote objfiles
           Product: gdb
           Version: HEAD
            Status: NEW
          Severity: normal
          Priority: P2
         Component: remote
          Assignee: unassigned at sourceware dot org
          Reporter: tromey at sourceware dot org
  Target Milestone: ---

When doing repeated remote debugging with the sysroot set to target:,
it would be handy if gdb could maintain its own cache of remote
objfiles.  Perhaps this could be done by buildid; though even just
maintaining an in-memory cache using the stat info would be good
enough for my purposes.

We discussed this briefly in bug#17917.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug remote/28489] Allow caching of remote objfiles

[tromey at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=28489

--- Comment #1 from Tom Tromey <tromey at sourceware dot org> ---
One possible issue here is that the cache might have to be
per-remote, to deal with the situation where there is
a cross-architecture build-id clash.  Perhaps just encoding
the arch into the cache file name is enough, though.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug remote/28489] Allow caching of remote objfiles

[simark at simark dot ca]

https://sourceware.org/bugzilla/show_bug.cgi?id=28489

Simon Marchi <simark at simark dot ca> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |simark at simark dot ca

--- Comment #2 from Simon Marchi <simark at simark dot ca> ---
(In reply to Tom Tromey from comment #1)
> One possible issue here is that the cache might have to be
> per-remote, to deal with the situation where there is
> a cross-architecture build-id clash.  Perhaps just encoding
> the arch into the cache file name is enough, though.

Isn't the concept of build-id that we can assume they are always unique, for
all practical intents?  I don't understand what situation you imagine where
there would be a clash.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug remote/28489] Allow caching of remote objfiles

[daniel at mariadb dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=28489

daniel at mariadb dot org <daniel at mariadb dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |daniel at mariadb dot org

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug remote/28489] Allow caching of remote objfiles

[mark at klomp dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=28489

Mark Wielaard <mark at klomp dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |mark at klomp dot org

--- Comment #3 from Mark Wielaard <mark at klomp dot org> ---
(In reply to Simon Marchi from comment #2)
> (In reply to Tom Tromey from comment #1)
> > One possible issue here is that the cache might have to be
> > per-remote, to deal with the situation where there is
> > a cross-architecture build-id clash.  Perhaps just encoding
> > the arch into the cache file name is enough, though.
> 
> Isn't the concept of build-id that we can assume they are always unique, for
> all practical intents?  I don't understand what situation you imagine where
> there would be a clash.

yes, the idea is that build-ids are globally unique identifiers. But there is
no mechanism to enforce that. It just relies on the producer/linker to create a
large enough hash.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug remote/28489] Allow caching of remote objfiles

[tromey at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=28489

--- Comment #4 from Tom Tromey <tromey at sourceware dot org> ---
(In reply to Simon Marchi from comment #2)

> Isn't the concept of build-id that we can assume they are always unique, for
> all practical intents?  I don't understand what situation you imagine where
> there would be a clash.

Just bad luck.  I wonder if there's ever been a clash.
Shrug, maybe we don't have to worry about it.  That would make
things simpler.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug remote/28489] Allow caching of remote objfiles

[simark at simark dot ca]

https://sourceware.org/bugzilla/show_bug.cgi?id=28489

--- Comment #5 from Simon Marchi <simark at simark dot ca> ---
(In reply to Tom Tromey from comment #4)
> (In reply to Simon Marchi from comment #2)
> 
> > Isn't the concept of build-id that we can assume they are always unique, for
> > all practical intents?  I don't understand what situation you imagine where
> > there would be a clash.
> 
> Just bad luck.  I wonder if there's ever been a clash.
> Shrug, maybe we don't have to worry about it.  That would make
> things simpler.

I think that many things in the stack use that assumption.  The concept of
build-id would not be very useful if we couldn't assume they are unique, it
means we couldn't use the build-id as the key for anything (for instance, for
the index-cache).

On my system, the build-ids are generated using the sha1 method, for instance:

$ file /bin/ls
/bin/ls: ... BuildID[sha1]=588ca812c340997ca8660ce0e15ee31a542568ad ...

So accidental collisions are not going to happen, for practical intents and
purposes. People will of course point out that there are known attacks on sha1
to create collisions, but then it's a producer problem, if they want to avoid
that they should move to a newer hashing algorithm.

Same thing if the producer fails to consider some important bits of executables
when generating the build-id, resulting in two different binaries having the
same build-id, then it's a producer bug that should be fixed.  I don't think we
should try to protect against that, otherwise we will always work in a
pessimistic fashion and won't be able to optimize anything.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug remote/28489] Allow caching of remote objfiles

[mark at klomp dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=28489

--- Comment #6 from Mark Wielaard <mark at klomp dot org> ---
I agree with Simon. You can just assume there will be no accidental collisions.
So you can design as if each build-id is globally unique.

Any real collisions are either because a user explicitly created it (e.g. ld
can be called with --build-id=0xhexid) or a bug in a producer (e.g. lld has a
--build-id=fast mode which creates a build-id really fast, but using less than
128bits so it cannot guarantee that the build-id is globally unique) which
really is just a bug in such a producer.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug remote/28489] Allow caching of remote objfiles

[tromey at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=28489

--- Comment #7 from Tom Tromey <tromey at sourceware dot org> ---
It occurred to me that to make this work, we'll need a new protocol
extension (no big deal) and gdbserver will have to know how to
extract the build id (a bigger deal).

I guess a different idea would be to construct a cached directory
tree that works like a sysroot, and use the existing remote
protocol stat capability to decide when to update a file.  In this
approach, build ids just would not be used (or, rather, would be
used to locate remote debuginfo files).

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug remote/28489] Allow caching of remote objfiles

[fche at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=28489

Frank Ch. Eigler <fche at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |fche at redhat dot com

--- Comment #8 from Frank Ch. Eigler <fche at redhat dot com> ---
> It occurred to me that to make this work, we'll need a new protocol
> extension (no big deal) and gdbserver will have to know how to
> extract the build id (a bigger deal).

https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=700ca40f6fc1addd7238f4ab57f76c095ad3c99f

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug remote/28489] Allow caching of remote objfiles

[tromey at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=28489

--- Comment #9 from Tom Tromey <tromey at sourceware dot org> ---
(In reply to Frank Ch. Eigler from comment #8)

> https://sourceware.org/git/?p=binutils-gdb.git;a=commit;
> h=700ca40f6fc1addd7238f4ab57f76c095ad3c99f

Thanks.  I had no idea this was in there.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug remote/28489] Allow caching of remote objfiles

[simark at simark dot ca]

https://sourceware.org/bugzilla/show_bug.cgi?id=28489

--- Comment #10 from Simon Marchi <simark at simark dot ca> ---
Note that it was reverted:

https://inbox.sourceware.org/gdb-patches/20150715183346.GA8231@host1.jankratochvil.net/

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug remote/28489] Allow caching of remote objfiles

[simark at simark dot ca]

https://sourceware.org/bugzilla/show_bug.cgi?id=28489

--- Comment #11 from Simon Marchi <simark at simark dot ca> ---
(In reply to Tom Tromey from comment #7)
> It occurred to me that to make this work, we'll need a new protocol
> extension (no big deal) and gdbserver will have to know how to
> extract the build id (a bigger deal).

I think it would be fine for GDBserver to learn how to extract build-ids.  It
should be quite fast, and including a build-id field in the shared library list
does not increase much the data transferred.  I don't know if GDB needs to
download the libraries up front for other reasons though.

Otherwise, in theory, GDB should be able to extract the build-id of remote
libraries by reading just the necessary bits (without downloading the whole
file), using remote file-io.  The upside is that it should work with the
existing protocol, but the downside is that it would do many small requests to
read small bits here and there (reading the ELF header, then looking through
notes), so it adds a lot of back and forth, and the latency adds up.

> I guess a different idea would be to construct a cached directory
> tree that works like a sysroot, and use the existing remote
> protocol stat capability to decide when to update a file. In this
> approach, build ids just would not be used (or, rather, would be
> used to locate remote debuginfo files).

Using modification times to know when cached files are stale, or something
else?  That seems difficult to manage.  Do you make one directory per remote? 
You could connect to 10.0.0.11:8888 and it's some system, and later connect to
10.0.0.11:8888, and it's another system.  How do you detect that?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug remote/28489] Allow caching of remote objfiles

[tromey at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=28489

--- Comment #12 from Tom Tromey <tromey at sourceware dot org> ---
(In reply to Tom Tromey from comment #7)
> It occurred to me that to make this work, we'll need a new protocol
> extension (no big deal) and gdbserver will have to know how to
> extract the build id (a bigger deal).

I was thinking about this again and I did a little experiment.

I instrumented BFD to print how much data it reads from the underlying file.
Then I wrote a small program to open an ELF and extract its build id.
When I coalesce all the reads I end up with it reading 5388 bytes
in 3 different parts of the file.

This doesn't seem so bad -- gdb could just do the reads using the
existing protocol and then switch to a local file (or start one) if the
build-id matches (or is new).

Furthermore, it seems like there isn't really a need to read the
entire file eagerly.  gdb won't ever use most of it, I think.
The idea here would be to have gdb cache pages from the file locally,
and keep a bitmap (in the same file) indicating which pages are valid.
Then when BFD requests some new page, go fetch it from the remote.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug remote/28489] Allow caching of remote objfiles

[fche at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=28489

--- Comment #13 from Frank Ch. Eigler <fche at redhat dot com> ---
Plus: given a buildid, gdb can try consulting debuginfod to fetch the
executable file.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug remote/28489] Allow caching of remote objfiles

[tromey at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=28489

--- Comment #14 from Tom Tromey <tromey at sourceware dot org> ---
I see now this idea was also in comment#11, sorry about that.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug remote/28489] Allow caching of remote objfiles

[tromey at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=28489

--- Comment #15 from Tom Tromey <tromey at sourceware dot org> ---
One thing I noticed is that gdb doesn't reuse a remote BFD even in
the same session.
E.g., gdb may open two BFDs for the exec file, and "maint info bfd"
will show two entries.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug remote/28489] Allow caching of remote objfiles

[tromey at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=28489

--- Comment #16 from Tom Tromey <tromey at sourceware dot org> ---
I found rocm_solib_fd_cache, which could be reused by gdb_bfd.c
to handle this kind of caching.
It is per-inferior, which I guess makes sense, since I suppose
we can't really be sure that different inferiors from the
same target connection really share a filesystem (?).

-- 
You are receiving this mail because:
You are on the CC list for the bug.