debuginfo server progress update

debuginfo server progress update

[Aaron Merey]

Hello,

The debuginfo server, a web service that offers lightweight on-demand
lookups of debuginfo-related data,
along with its elfutils client have begun to take shape so I want to
provide an update on where we currently
are in its development and how to use it.

We plan on shipping debuginfo server with elfutils and it is currently
available on the branch 'fche/dbgserver'
of the elfutils git repo. The http web server itself is able to scan a
directory tree for any debuginfo, executable
and source files and index them by build-id. The elfutils client hooks into
libdwfl and provides a fallback so
that if the library is not able to locate a debuginfo file, the client, if
enabled, will attempt to query debuginfo
servers for debuginfo containing a particular build-id.

As an example, suppose we are trying to trace some program with systemtap.
One of the stap probes we
want to run requires DWARF debuginfo but the executable has been stripped
of this information and the
corresponding debuginfo file is not in any of the default directories that
stap checks.

    $ stap -e 'probe process("/path/to/exec").function("*") { [...] }'
    semantic error: while resolving probe point: identifier 'process' at
t.stp:1:7
            source: probe process("/path/to/exec").function("*") {
    semantic error: no match (similar functions: exit, open, textdomain,
tfind, dup)

Systemtap uses elfutils to locate and read debuginfo files, so it is able
to make use of debuginfo server's
elfutils client. First we run debuginfo server and point it to the
directory containing the stripped debuginfo.

    $ dbgserver -vvv -p 8002 -F ~/debuginfo_dir &
    [Tue Jul  2 15:00:37] (20459/20462): file-path scanning
/home/amerey/debuginfo_dir
    [Tue Jul  2 15:00:37] (20459/20459): Started http server on port 8002,
database path /home/amerey/.dbgserver.sqlite
    [Tue Jul  2 15:00:37] (20459/20462): fts traversing
/home/amerey/debuginfo_dir
    [Tue Jul  2 15:00:37] (20459/20462): fts traversing
/home/amerey/debuginfo_dir/debuginfo
    [Tue Jul  2 15:00:37] (20459/20462): recorded
buildid=0a0cd15e690a378ec77359bb2eeb76ea0f8d67f8 \
        file=/home/amerey/debuginfo_dir/debuginfo mtime=1561060856 as
executable and debuginfo
    [...]

In order to enable the elfutils client, there must be a $DBGSERVER_URLS
environment variable containing
at least one URL of a debuginfo server (multiple URLs should be separated
by spaces). When we run stap
again, elfutils will query the debuginfo server for debuginfo with the
build-id of the executable.

    $ export DBGSERVER_URLS=localhost:8002
    $ stap -v -e 'probe process("/path/to/exec").function("*") { [...] }'
    [...]
    Pass 5: starting run

This time systemtap successfully acquires the debuginfo and begins running
the probe. Although in this
example everything is running locally, the server and the debuginfo could
just as well be on a remote host.

Debuginfo server development is ongoing. Some of the features we have
planned include a gdb client
that could query debuginfo servers for debuginfo or source code and the
ability for the server to
automatically search rpms for any of the files requested by a client. We
also are planning a talk/demo
at GNU Cauldron this September in Montréal, Canada (
https://gcc.gnu.org/wiki/cauldron2019).

Hopefully this helped give you a sense of how to use debuginfo server,
where it is in its development,
and what is to come. Feedback is always appreciated!

Aaron Merey

debuginfo server progress update

[Aaron Merey]

(resending this email, this time with better formatting!)

Hello,

The debuginfo server, a web service that offers lightweight on-demand lookups
of debuginfo-related data, along with its elfutils client have begun to take
shape so I want to provide an update on where we currently are in its
development and how to use it.

We plan on shipping debuginfo server with elfutils and it is currently
available on the branch 'fche/dbgserver' of the elfutils git repo. The
http web server itself is able to scan a directory tree for any debuginfo,
executable and source files and index them by build-id. The elfutils client
hooks into libdwfl and provides a fallback so that if the library is not able
to locate a debuginfo file, the client, if enabled, will attempt to query
debuginfo servers for debuginfo containing a particular build-id.

As an example, suppose we are trying to trace some program with systemtap.
One of the stap probes we want to run requires DWARF debuginfo but the
executable has been stripped of this information and the corresponding
debuginfo file is not in any of the default directories that stap checks.

    $ stap -e 'probe process("/path/to/exec").function("*") { [...] }'
    semantic error: while resolving probe point: \
      identifier 'process' at t.stp:1:7
            source: probe process("/path/to/exec").function("*") {
    semantic error: no match \
      (similar functions: exit, open, textdomain, tfind, dup)

Systemtap uses elfutils to locate and read debuginfo files, so it is able
to make use of debuginfo server's elfutils client. First we run debuginfo
server and point it to the directory containing the stripped debuginfo.

    $ dbgserver -vvv -p 8002 -F ~/debuginfo_dir &
    [Tue Jul  2 15:00:37] (20459/20462): file-path scanning \
      /home/amerey/debuginfo_dir
    [Tue Jul  2 15:00:37] (20459/20459): Started http server on port 8002, \
      database path /home/amerey/.dbgserver.sqlite
    [Tue Jul  2 15:00:37] (20459/20462): fts traversing \
      /home/amerey/debuginfo_dir
    [Tue Jul  2 15:00:37] (20459/20462): fts traversing \
      /home/amerey/debuginfo_dir/debuginfo
    [Tue Jul  2 15:00:37] (20459/20462): recorded \
      buildid=0a0cd15e690a378ec77359bb2eeb76ea0f8d67f8 \
      file=/home/amerey/debuginfo_dir/debuginfo mtime=1561060856 \
      as executable and debuginfo
    [...]

In order to enable the elfutils client, there must be a $DBGSERVER_URLS
environment variable containing at least one URL of a debuginfo server
(multiple URLs should be separated by spaces). When we run stap again,
elfutils will query the debuginfo server for debuginfo with the build-id
of the executable.

    $ export DBGSERVER_URLS=localhost:8002
    $ stap -v -e 'probe process("/path/to/exec").function("*") { [...] }'
    [...]
    Pass 5: starting run

This time systemtap successfully acquires the debuginfo and begins running
the probe. Although in this example everything is running locally, the server
and the debuginfo could just as well be on a remote host.

Debuginfo server development is ongoing. Some of the features we have planned
include a gdb client that could query debuginfo servers for debuginfo or source
code and the ability for the server to automatically search rpms for any of the
files requested by a client. We also are planning a talk/demo at GNU Cauldron
this September in Montréal, Canada (https://gcc.gnu.org/wiki/cauldron2019).

Hopefully this helped give you a sense of how to use debuginfo server, where it
is in its development, and what is to come. Feedback is always appreciated!

Aaron Merey

Re: debuginfo server progress update

[Mark Wielaard]

Hi Aaron,

On Wed, Jul 03, 2019 at 03:59:46PM -0400, Aaron Merey wrote:
> The debuginfo server, a web service that offers lightweight on-demand lookups
> of debuginfo-related data, along with its elfutils client have begun to take
> shape so I want to provide an update on where we currently are in its
> development and how to use it.
> 
> We plan on shipping debuginfo server with elfutils and it is currently
> available on the branch 'fche/dbgserver' of the elfutils git repo.

And now also simply on 'dbgserver'.
https://sourceware.org/git/?p=elfutils.git;a=shortlog;h=refs/heads/dbgserver

> The
> http web server itself is able to scan a directory tree for any debuginfo,
> executable and source files and index them by build-id. The elfutils client
> hooks into libdwfl and provides a fallback so that if the library is not able
> to locate a debuginfo file, the client, if enabled, will attempt to query
> debuginfo servers for debuginfo containing a particular build-id.
> [...]
> This time systemtap successfully acquires the debuginfo and begins running
> the probe. Although in this example everything is running locally, the server
> and the debuginfo could just as well be on a remote host.
> 
> Debuginfo server development is ongoing. Some of the features we have planned
> include a gdb client that could query debuginfo servers for debuginfo or source
> code and the ability for the server to automatically search rpms for any of the
> files requested by a client. We also are planning a talk/demo at GNU Cauldron
> this September in Montréal, Canada (https://gcc.gnu.org/wiki/cauldron2019).
> 
> Hopefully this helped give you a sense of how to use debuginfo server, where it
> is in its development, and what is to come. Feedback is always appreciated!

I think it would be nice if there was a dbgclient binary that could be
reused by multiple clients.  It could do most of what you currently do
in libdwfl/dbgserver-client.c. It would simply take a buildid and type
(exec, dbg, src) and return a file path to the resulting file when
found (or a well defined error number when not found).

Such a dbgclient executable would make writing other clients
easier. For example dbgserver_build_id_find would simply reduce to
spawning dbgclient with the right arguments and opening the resulting
file path. Other clients, like gdb, then also don't need to implement
the caching and quering, but can also simply call dbgclient <build-id>
<type> and just open the resulting file.

Cheers,

Mark

Re: debuginfo server progress update

[Aaron Merey]

Hi Mark,

On Fri, Jul 5, 2019 at 12:46 PM Mark Wielaard <mark@klomp.org> wrote:
>
> Hi Aaron,
>
> On Wed, Jul 03, 2019 at 03:59:46PM -0400, Aaron Merey wrote:
> > The debuginfo server, a web service that offers lightweight on-demand lookups
> > of debuginfo-related data, along with its elfutils client have begun to take
> > shape so I want to provide an update on where we currently are in its
> > development and how to use it.
> >
> > We plan on shipping debuginfo server with elfutils and it is currently
> > available on the branch 'fche/dbgserver' of the elfutils git repo.
>
> And now also simply on 'dbgserver'.
> https://sourceware.org/git/?p=elfutils.git;a=shortlog;h=refs/heads/dbgserver
>
> > The
> > http web server itself is able to scan a directory tree for any debuginfo,
> > executable and source files and index them by build-id. The elfutils client
> > hooks into libdwfl and provides a fallback so that if the library is not able
> > to locate a debuginfo file, the client, if enabled, will attempt to query
> > debuginfo servers for debuginfo containing a particular build-id.
> > [...]
> > This time systemtap successfully acquires the debuginfo and begins running
> > the probe. Although in this example everything is running locally, the server
> > and the debuginfo could just as well be on a remote host.
> >
> > Debuginfo server development is ongoing. Some of the features we have planned
> > include a gdb client that could query debuginfo servers for debuginfo or source
> > code and the ability for the server to automatically search rpms for any of the
> > files requested by a client. We also are planning a talk/demo at GNU Cauldron
> > this September in Montréal, Canada (https://gcc.gnu.org/wiki/cauldron2019).
> >
> > Hopefully this helped give you a sense of how to use debuginfo server, where it
> > is in its development, and what is to come. Feedback is always appreciated!
>
> I think it would be nice if there was a dbgclient binary that could be
> reused by multiple clients.  It could do most of what you currently do
> in libdwfl/dbgserver-client.c. It would simply take a buildid and type
> (exec, dbg, src) and return a file path to the resulting file when
> found (or a well defined error number when not found).
>
> Such a dbgclient executable would make writing other clients
> easier. For example dbgserver_build_id_find would simply reduce to
> spawning dbgclient with the right arguments and opening the resulting
> file path. Other clients, like gdb, then also don't need to implement
> the caching and quering, but can also simply call dbgclient <build-id>
> <type> and just open the resulting file.

I really like this suggestion, even with just elfutils and gdb clients
there will
already be too much code duplication if we don't have the client code in its own
binary. This raises questions such as whether dbgclient should be a shared
library or a command line tool, but regardless of these implementation details
you bring up an issue that needs to be addressed as the client side of debuginfo
server grows. I appreciate the feedback!

Aaron