* [PATCH v2] debuginfod: Support queries for ELF/DWARF sections
@ 2022-10-21 0:06 Aaron Merey
2022-10-22 0:09 ` [PATCH] " Aaron Merey
0 siblings, 1 reply; 8+ messages in thread
From: Aaron Merey @ 2022-10-21 0:06 UTC (permalink / raw)
To: elfutils-devel; +Cc: Aaron Merey
v1 can be found here:
https://sourceware.org/pipermail/elfutils-devel/2022q3/005402.html
v2 simplifies debuginfod_find_section by automatically checking both
debuginfo and executables to extract the target section from. Previously
this function included a parameter for specifying whether the section should
come from a debuginfo or executable file. For sections that are normally
stored in debuginfo or for sections that might differ between debuginfo and
executable (ex. .shstrtab, .gnu.build.attributes), debuginfo files are
searched before executables.
Artifacttype "debuginfo-section" and "executable-section" from v1 have
been replaced with artifacttype "section". An example URL for a
section query is: <HOST>/buildid/0123456789abcdef/section/.gdb_index
Section files in the client cache are named "section" followed by
the section name. Ex: "section.gdb_index".
debuginfod_query_server now attempts to extract the desired section
from a debuginfo or executable in the client cache before querying
any servers.
If debuginfod servers are built without _find_section support, they will
respond to a section query with response code 503. If the client recieves
this code in response to a section query, it will attempt to query servers
for the full debuginfo or the executable (possibly both) in order to extract
the section on the client side. For sections that are normally stored in
debuginfo or for sections that might differ between debuginfo and executable,
debuginfo files are downloaded/searched before executables.
---
ChangeLog | 4 +
NEWS | 2 +
debuginfod/ChangeLog | 22 ++
debuginfod/Makefile.am | 2 +-
debuginfod/debuginfod-client.c | 273 +++++++++++++++++++-
debuginfod/debuginfod-find.c | 21 +-
debuginfod/debuginfod.cxx | 433 +++++++++++++++++++++++++++-----
debuginfod/debuginfod.h.in | 6 +
debuginfod/libdebuginfod.map | 1 +
doc/ChangeLog | 6 +
doc/Makefile.am | 1 +
doc/debuginfod_find_debuginfo.3 | 26 +-
doc/debuginfod_find_section.3 | 1 +
tests/ChangeLog | 5 +
tests/Makefile.am | 4 +-
tests/run-debuginfod-section.sh | 141 +++++++++++
16 files changed, 865 insertions(+), 83 deletions(-)
create mode 100644 doc/debuginfod_find_section.3
create mode 100755 tests/run-debuginfod-section.sh
diff --git a/ChangeLog b/ChangeLog
index 60624183..29b2108e 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,7 @@
+2022-10-20 Aaron Merey <amerey@redhat.com>
+
+ * NEWS: Add debuginfod_find_section.
+
2022-09-13 Aleksei Vetrov <vvvvvv@google.com>
* NEWS (libdwfl): Add dwfl_report_offline_memory.
diff --git a/NEWS b/NEWS
index 6ebd172c..3e290ff7 100644
--- a/NEWS
+++ b/NEWS
@@ -4,6 +4,8 @@ readelf: Add -D, --use-dynamic option.
debuginfod: Add --disable-source-scan option.
+debuginfod-client: Add new function debuginfod_find_section.
+
libdwfl: Add new function dwfl_get_debuginfod_client.
Add new function dwfl_frame_reg.
Add new function dwfl_report_offline_memory.
diff --git a/debuginfod/ChangeLog b/debuginfod/ChangeLog
index 59d50df1..aacf655b 100644
--- a/debuginfod/ChangeLog
+++ b/debuginfod/ChangeLog
@@ -1,3 +1,25 @@
+2022-10-20 Aaron Merey <amerey@redhat.com>
+
+ * Makefile.am (libdebuginfod_so_LDLIBS): Add libelf.
+ * debuginfod-client.c (debuginfod_find_section): New function.
+ (extract_section): New function.
+ (maybe_debuginfo_section): New function.
+ (cache_find_section): New function.
+ (debuginfod_query_server): Add support for section queries.
+ * debuginfod-find.c (main): Add support for section queries.
+ * debuginfod.cxx (extract_section): New function.
+ (handle_buildid_f_match): Add section parameter. When non-empty,
+ try to create response from section contents.
+ (handle_buildid_r_match): Add section parameter. When non-empty,
+ try to create response from section contents.
+ (handle_buildid_match): Add section parameter. Pass to
+ handle_buildid_{f,r}_match.
+ (handle_buildid): Handle section name when artifacttype is set to
+ "section". Query upstream servers via debuginfod_find_section
+ when necessary.
+ (debuginfod.h.in): Add declaration for debuginfod_find_section.
+ (libdebuginfod.map): Add debuginfod_find_section.
+
2022-10-17 Frank Ch. Eigler <fche@redhat.com>
* debuginfod.cxx (main): Report libmicrohttpd version.
diff --git a/debuginfod/Makefile.am b/debuginfod/Makefile.am
index 435cb8a6..f27d6e2e 100644
--- a/debuginfod/Makefile.am
+++ b/debuginfod/Makefile.am
@@ -97,7 +97,7 @@ libdebuginfod_so_LIBS = libdebuginfod_pic.a
if DUMMY_LIBDEBUGINFOD
libdebuginfod_so_LDLIBS =
else
-libdebuginfod_so_LDLIBS = -lpthread $(libcurl_LIBS) $(fts_LIBS)
+libdebuginfod_so_LDLIBS = -lpthread $(libcurl_LIBS) $(fts_LIBS) $(libelf)
endif
$(LIBDEBUGINFOD_SONAME): $(srcdir)/libdebuginfod.map $(libdebuginfod_so_LIBS)
$(AM_V_CCLD)$(LINK) $(dso_LDFLAGS) -o $@ \
diff --git a/debuginfod/debuginfod-client.c b/debuginfod/debuginfod-client.c
index 2a14d9d9..3bd3cc23 100644
--- a/debuginfod/debuginfod-client.c
+++ b/debuginfod/debuginfod-client.c
@@ -44,6 +44,7 @@
#include "system.h"
#include <errno.h>
#include <stdlib.h>
+#include <libdwelf.h>
/* We might be building a bootstrap dummy library, which is really simple. */
#ifdef DUMMY_LIBDEBUGINFOD
@@ -55,6 +56,9 @@ int debuginfod_find_executable (debuginfod_client *c, const unsigned char *b,
int s, char **p) { return -ENOSYS; }
int debuginfod_find_source (debuginfod_client *c, const unsigned char *b,
int s, const char *f, char **p) { return -ENOSYS; }
+int debuginfod_find_section (debuginfod_client *c, const unsigned char *b,
+ int s, const char *scn, char **p)
+ { return -ENOSYS; }
void debuginfod_set_progressfn(debuginfod_client *c,
debuginfod_progressfn_t fn) { }
void debuginfod_set_verbose_fd(debuginfod_client *c, int fd) { }
@@ -536,6 +540,193 @@ header_callback (char * buffer, size_t size, size_t numitems, void * userdata)
return numitems;
}
+/* Attempt to read an ELF/DWARF section with name SECTION from FD and write
+ it to a separate file in the debuginfod cache. If successful the absolute
+ path of the separate file containing SECTION will be stored in USR_PATH.
+ FD_PATH is the absolute path for FD. */
+
+int
+extract_section (int fd, const char *section, char *fd_path, char **usr_path)
+{
+ Elf *elf = elf_begin (fd, ELF_C_READ_MMAP_PRIVATE, NULL);
+ if (elf == NULL)
+ return -EIO;
+
+ size_t shstrndx;
+ int rc = elf_getshdrstrndx (elf, &shstrndx);
+ if (rc < 0)
+ return -EIO;
+
+ /* Try to find the target section and copy the contents into a
+ separate file. */
+ Elf_Scn *scn = NULL;
+ while (true)
+ {
+ scn = elf_nextscn (elf, scn);
+ if (scn == NULL)
+ {
+ rc = -ENOENT;
+ break;
+ }
+ GElf_Shdr shdr_storage;
+ GElf_Shdr *shdr = gelf_getshdr (scn, &shdr_storage);
+ if (shdr == NULL)
+ {
+ rc = -EIO;
+ break;
+ }
+
+ const char *scn_name = elf_strptr (elf, shstrndx, shdr->sh_name);
+ if (scn_name == NULL)
+ {
+ rc = -EIO;
+ break;
+ }
+ if (strcmp (scn_name, section) == 0)
+ {
+ Elf_Data *data = NULL;
+
+ /* We found the desired section. */
+ data = elf_getdata (scn, NULL);
+ if (data == NULL)
+ {
+ rc = -EIO;
+ break;
+ }
+
+ if (data->d_buf == NULL)
+ {
+ rc = -ENOENT;
+ break;
+ }
+
+ /* Compute the absolute filename we'll write the section to. Replace
+ the last component in FD_PATH with the section filename. */
+ int i = strlen (fd_path);
+ while (i >= 0)
+ {
+ if (fd_path[i] == '/')
+ {
+ fd_path[i] = '\0';
+ break;
+ }
+ --i;
+ }
+
+ char *sec_path;
+ if (asprintf (&sec_path, "%s/section%s", fd_path, section) == -1)
+ {
+ rc = -ENOMEM;
+ break;
+ }
+
+ int sec_fd = open (sec_path, O_CREAT | O_EXCL | O_RDWR, S_IRUSR);
+ if (sec_fd < 0)
+ {
+ free (sec_path);
+ rc = -errno;
+ break;
+ }
+
+ ssize_t res = write (sec_fd, data->d_buf, data->d_size);
+ if (res < 0 || (size_t) res != data->d_size)
+ {
+ close (sec_fd);
+ free (sec_path);
+ rc = -EIO;
+ break;
+ }
+
+ /* Success. Update USR_PATH. */
+ if (usr_path != NULL)
+ *usr_path = sec_path;
+ else
+ free (sec_path);
+ rc = sec_fd;
+ break;
+ }
+ }
+
+ elf_end (elf);
+ return rc;
+}
+
+/* Return true if we should search a debuginfo file for the ELF/DWARF section
+ with name SCN_NAME before searching the executable file. */
+
+static bool
+maybe_debuginfo_section (const char *scn_name)
+{
+ return (startswith (scn_name, ".debug")
+ || strcmp (scn_name, ".symtab") == 0
+ || strcmp (scn_name, ".strtab") == 0
+ || strcmp (scn_name, ".shstrtab") == 0
+ || strcmp (scn_name, ".gdb_index") == 0
+ || strcmp (scn_name, ".gnu_debugaltlink") == 0
+ || strcmp (scn_name, ".gnu.build.attributes") == 0);
+}
+
+/* Search TARGET_CACHE_DIR for a debuginfo or executable file containing
+ an ELF/DWARF section with name SCN_NAME. If found, extract the section
+ to a separate file in TARGET_CACHE_DIR and return a file descriptor
+ for the section file. The path for this file will be stored in USR_PATH.
+ Return a negative errno if unsuccessful. */
+
+static int
+cache_find_section (const char *scn_name, const char *target_cache_dir,
+ char **usr_path)
+{
+ int fd;
+ int rc = -ENOENT;
+ bool debug_p = maybe_debuginfo_section (scn_name);
+ char exec_path[PATH_MAX];
+ char debug_path[PATH_MAX];
+ char *fd_path;
+
+ snprintf (exec_path, PATH_MAX, "%s/executable", target_cache_dir);
+ snprintf (debug_path, PATH_MAX, "%s/debuginfo", target_cache_dir);
+
+ if (debug_p)
+ {
+ fd = open (debug_path, O_RDONLY);
+ fd_path = debug_path;
+ }
+ else
+ {
+ fd = open (exec_path, O_RDONLY);
+ fd_path = exec_path;
+ }
+
+ if (fd >= 0)
+ {
+ rc = extract_section (fd, scn_name, fd_path, usr_path);
+ close (fd);
+ }
+
+ /* If the section wasn't found, check the other file where it might be. */
+ if (rc < 0)
+ {
+ if (debug_p)
+ {
+ fd = open (exec_path, O_RDONLY);
+ fd_path = exec_path;
+ }
+ else
+ {
+ fd = open (debug_path, O_RDONLY);
+ fd_path = debug_path;
+ }
+
+ if (fd >= 0)
+ {
+ rc = extract_section (fd, scn_name, fd_path, usr_path);
+ close (fd);
+ }
+ }
+
+ return rc;
+}
+
/* Query each of the server URLs found in $DEBUGINFOD_URLS for the file
with the specified build-id, type (debuginfo, executable or source)
and filename. filename may be NULL. If found, return a file
@@ -546,11 +737,13 @@ debuginfod_query_server (debuginfod_client *c,
const unsigned char *build_id,
int build_id_len,
const char *type,
- const char *filename,
+ const char *type_arg,
char **path)
{
char *server_urls;
char *urls_envvar;
+ const char *section = NULL;
+ const char *filename = NULL;
char *cache_path = NULL;
char *maxage_path = NULL;
char *interval_path = NULL;
@@ -563,6 +756,15 @@ debuginfod_query_server (debuginfod_client *c,
int vfd = c->verbose_fd;
int rc;
+ if (strcmp (type, "source") == 0)
+ filename = type_arg;
+ else if (strcmp (type, "section") == 0)
+ {
+ section = type_arg;
+ if (section != NULL && section[0] != '.')
+ return -EINVAL;
+ }
+
if (vfd >= 0)
{
dprintf (vfd, "debuginfod_find_%s ", type);
@@ -681,6 +883,8 @@ debuginfod_query_server (debuginfod_client *c,
PATH_MAX and truncate/collide. Oh well, that'll teach
them! */
}
+ else if (section != NULL)
+ strncpy (suffix, section, PATH_MAX);
else
suffix[0] = '\0';
@@ -836,6 +1040,15 @@ debuginfod_query_server (debuginfod_client *c,
/* Ensure old 000-permission files are not lingering in the cache. */
unlink(target_cache_path);
+ if (section != NULL)
+ {
+ /* Try to extract the section from a cached file before querying
+ any servers. */
+ rc = cache_find_section (section, target_cache_dir, path);
+ if (rc >= 0)
+ goto out;
+ }
+
long timeout = default_timeout;
const char* timeout_envvar = getenv(DEBUGINFOD_TIMEOUT_ENV_VAR);
if (timeout_envvar != NULL)
@@ -1008,6 +1221,9 @@ debuginfod_query_server (debuginfod_client *c,
snprintf(data[i].url, PATH_MAX, "%s/%s/%s/%s", server_url,
build_id_bytes, type, escaped_string);
}
+ else if (section)
+ snprintf(data[i].url, PATH_MAX, "%s/%s/%s/%s", server_url,
+ build_id_bytes, type, section);
else
snprintf(data[i].url, PATH_MAX, "%s/%s/%s", server_url, build_id_bytes, type);
if (vfd >= 0)
@@ -1279,6 +1495,8 @@ debuginfod_query_server (debuginfod_client *c,
/* 406 signals that the requested file was too large */
if ( ok0 == CURLE_OK && resp_code == 406)
rc = -EFBIG;
+ else if (section != NULL && resp_code == 503)
+ rc = -EINVAL;
else
rc = -ENOENT;
break;
@@ -1525,6 +1743,7 @@ debuginfod_begin (void)
goto out1;
}
+ elf_version (EV_CURRENT);
// extra future initialization
goto out;
@@ -1604,6 +1823,58 @@ int debuginfod_find_source(debuginfod_client *client,
"source", filename, path);
}
+int
+debuginfod_find_section (debuginfod_client *client,
+ const unsigned char *build_id, int build_id_len,
+ const char *section, char **path)
+{
+ int rc = debuginfod_query_server(client, build_id, build_id_len,
+ "section", section, path);
+ if (rc != -EINVAL)
+ return rc;
+
+ /* The servers may have lacked support for section queries. Attempt to
+ download the debuginfo or executable containing the section in order
+ to extract it. */
+ rc = -ENOENT;
+ int fd = -1;
+ char *tmp_path = NULL;
+
+ /* Use the section name as a guide for which file to try first. */
+ bool debug_p = maybe_debuginfo_section (section);
+
+ if (debug_p)
+ fd = debuginfod_find_debuginfo (client, build_id, build_id_len, &tmp_path);
+ else
+ fd = debuginfod_find_executable (client, build_id, build_id_len, &tmp_path);
+
+ if (fd > 0)
+ {
+ rc = extract_section (fd, section, tmp_path, path);
+ close (fd);
+ }
+
+ if (rc == -ENOENT)
+ {
+ /* Still unable to find the section. Try to extract section from the
+ other file type with build_id. */
+ if (debug_p)
+ fd = debuginfod_find_executable (client, build_id,
+ build_id_len, &tmp_path);
+ else
+ fd = debuginfod_find_debuginfo (client, build_id,
+ build_id_len, &tmp_path);
+
+ if (fd > 0)
+ {
+ rc = extract_section (fd, section, tmp_path, path);
+ close (fd);
+ }
+ }
+
+ free (tmp_path);
+ return rc;
+}
/* Add an outgoing HTTP header. */
int debuginfod_add_http_header (debuginfod_client *client, const char* header)
diff --git a/debuginfod/debuginfod-find.c b/debuginfod/debuginfod-find.c
index 778fb09b..197bea6c 100644
--- a/debuginfod/debuginfod-find.c
+++ b/debuginfod/debuginfod-find.c
@@ -190,13 +190,11 @@ main(int argc, char** argv)
debuginfod_find_* function. If FILETYPE is "source"
then ensure a FILENAME was also supplied as an argument. */
if (strcmp(argv[remaining], "debuginfo") == 0)
- rc = debuginfod_find_debuginfo(client,
- build_id, build_id_len,
- &cache_name);
+ rc = debuginfod_find_debuginfo(client, build_id,
+ build_id_len, &cache_name);
else if (strcmp(argv[remaining], "executable") == 0)
- rc = debuginfod_find_executable(client,
- build_id, build_id_len,
- &cache_name);
+ rc = debuginfod_find_executable(client, build_id,
+ build_id_len, &cache_name);
else if (strcmp(argv[remaining], "source") == 0)
{
if (remaining+2 == argc || argv[remaining+2][0] != '/')
@@ -208,6 +206,17 @@ main(int argc, char** argv)
build_id, build_id_len,
argv[remaining+2], &cache_name);
}
+ else if (strcmp(argv[remaining], "section") == 0)
+ {
+ if (remaining+2 == argc || argv[remaining+2][0] != '.')
+ {
+ fprintf(stderr,
+ "A section name starting with '.' must be given\n");
+ return 1;
+ }
+ rc = debuginfod_find_section(client, build_id, build_id_len,
+ argv[remaining+2], &cache_name);
+ }
else
{
argp_help (&argp, stderr, ARGP_HELP_USAGE, argv[0]);
diff --git a/debuginfod/debuginfod.cxx b/debuginfod/debuginfod.cxx
index 9dc4836b..19e35716 100644
--- a/debuginfod/debuginfod.cxx
+++ b/debuginfod/debuginfod.cxx
@@ -249,6 +249,19 @@ static const char DEBUGINFOD_SQLITE_DDL[] =
" where b.id = sr.buildid and f0.id = sd.file and fsref.id = sde.file and f1.id = sd.content\n"
" and sr.artifactsrc = sd.content and sde.buildid = sr.buildid\n"
";"
+ // ... and for I queries
+ "create view if not exists " BUILDIDS "_query_i as \n"
+ "select\n"
+ " b.hex as buildid, n.mtime, 'F' as sourcetype, f0.name as source0, n.mtime as mtime, null as source1,"
+ " n.debuginfo_p as debuginfo_p, n.executable_p as executable_p\n"
+ " from " BUILDIDS "_buildids b, " BUILDIDS "_files f0, " BUILDIDS "_f_de n\n"
+ " where b.id = n.buildid and f0.id = n.file\n"
+ "union all select\n"
+ " b.hex as buildid, n.mtime, 'R' as sourcetype, f0.name as source0, n.mtime as mtime, f1.name as source1,"
+ " n.debuginfo_p as debuginfo_p, n.executable_p as executable_p\n"
+ " from " BUILDIDS "_buildids b, " BUILDIDS "_files f0, " BUILDIDS "_files f1, " BUILDIDS "_r_de n\n"
+ " where b.id = n.buildid and f0.id = n.file and f1.id = n.content\n"
+ ";"
// and for startup overview counts
"drop view if exists " BUILDIDS "_stats;\n"
"create view if not exists " BUILDIDS "_stats as\n"
@@ -1136,66 +1149,6 @@ add_mhd_last_modified (struct MHD_Response *resp, time_t mtime)
add_mhd_response_header (resp, "Cache-Control", "public");
}
-
-
-static struct MHD_Response*
-handle_buildid_f_match (bool internal_req_t,
- int64_t b_mtime,
- const string& b_source0,
- int *result_fd)
-{
- (void) internal_req_t; // ignored
- int fd = open(b_source0.c_str(), O_RDONLY);
- if (fd < 0)
- throw libc_exception (errno, string("open ") + b_source0);
-
- // NB: use manual close(2) in error case instead of defer_dtor, because
- // in the normal case, we want to hand the fd over to libmicrohttpd for
- // file transfer.
-
- struct stat s;
- int rc = fstat(fd, &s);
- if (rc < 0)
- {
- close(fd);
- throw libc_exception (errno, string("fstat ") + b_source0);
- }
-
- if ((int64_t) s.st_mtime != b_mtime)
- {
- if (verbose)
- obatched(clog) << "mtime mismatch for " << b_source0 << endl;
- close(fd);
- return 0;
- }
-
- inc_metric ("http_responses_total","result","file");
- struct MHD_Response* r = MHD_create_response_from_fd ((uint64_t) s.st_size, fd);
- if (r == 0)
- {
- if (verbose)
- obatched(clog) << "cannot create fd-response for " << b_source0 << endl;
- close(fd);
- }
- else
- {
- std::string file = b_source0.substr(b_source0.find_last_of("/")+1, b_source0.length());
- add_mhd_response_header (r, "Content-Type", "application/octet-stream");
- add_mhd_response_header (r, "X-DEBUGINFOD-SIZE",
- to_string(s.st_size).c_str());
- add_mhd_response_header (r, "X-DEBUGINFOD-FILE", file.c_str());
- add_mhd_last_modified (r, s.st_mtime);
- if (verbose > 1)
- obatched(clog) << "serving file " << b_source0 << endl;
- /* libmicrohttpd will close it. */
- if (result_fd)
- *result_fd = fd;
- }
-
- return r;
-}
-
-
// quote all questionable characters of str for safe passage through a sh -c expansion.
static string
shell_escape(const string& str)
@@ -1589,6 +1542,217 @@ public:
};
static libarchive_fdcache fdcache;
+/* Search ELF_FD for an ELF/DWARF section with name SECTION.
+ If found copy the section to a temporary file and return
+ its file descriptor, otherwise return -1.
+
+ The temporary file's mtime will be set to PARENT_MTIME.
+ B_SOURCE should be a description of the parent file suitable
+ for printing to the log. */
+
+static int
+extract_section (int elf_fd, int64_t parent_mtime,
+ const string& b_source, const string& section)
+{
+ /* Search the fdcache. */
+ struct stat fs;
+ int fd = fdcache.lookup (b_source, section);
+ if (fd >= 0)
+ {
+ if (fstat (fd, &fs) != 0)
+ {
+ if (verbose)
+ obatched (clog) << "cannot fstate fdcache "
+ << b_source << " " << section << endl;
+ close (fd);
+ return -1;
+ }
+ if ((int64_t) fs.st_mtime != parent_mtime)
+ {
+ if (verbose)
+ obatched(clog) << "mtime mismatch for "
+ << b_source << " " << section << endl;
+ close (fd);
+ return -1;
+ }
+ /* Success. */
+ return fd;
+ }
+
+ Elf *elf = elf_begin (elf_fd, ELF_C_READ_MMAP_PRIVATE, NULL);
+ if (elf == NULL)
+ return -1;
+
+ /* Try to find the section and copy the contents into a separate file. */
+ try
+ {
+ size_t shstrndx;
+ int rc = elf_getshdrstrndx (elf, &shstrndx);
+ if (rc < 0)
+ throw elfutils_exception (rc, "getshdrstrndx");
+
+ Elf_Scn *scn = NULL;
+ while (true)
+ {
+ scn = elf_nextscn (elf, scn);
+ if (scn == NULL)
+ break;
+ GElf_Shdr shdr_storage;
+ GElf_Shdr *shdr = gelf_getshdr (scn, &shdr_storage);
+ if (shdr == NULL)
+ break;
+
+ const char *scn_name = elf_strptr (elf, shstrndx, shdr->sh_name);
+ if (scn_name == NULL)
+ break;
+ if (scn_name == section)
+ {
+ Elf_Data *data = NULL;
+
+ /* We found the desired section. */
+ data = elf_getdata (scn, NULL);
+ if (data == NULL)
+ throw elfutils_exception (elf_errno (), "getdata");
+ if (data->d_buf == NULL)
+ {
+ obatched(clog) << "section " << section
+ << " is empty" << endl;
+ break;
+ }
+
+ /* Create temporary file containing the section. */
+ char *tmppath = NULL;
+ rc = asprintf (&tmppath, "%s/debuginfod.XXXXXX", tmpdir.c_str());
+ if (rc < 0)
+ throw libc_exception (ENOMEM, "cannot allocate tmppath");
+ defer_dtor<void*,void> tmmpath_freer (tmppath, free);
+ fd = mkstemp (tmppath);
+ if (fd < 0)
+ throw libc_exception (errno, "cannot create temporary file");
+ ssize_t res = write (fd, data->d_buf, data->d_size);
+ if (res < 0 || (size_t) res != data->d_size)
+ throw libc_exception (errno, "cannot write to temporary file");
+
+ /* Set mtime to be the same as the parent file's mtime. */
+ struct timeval tvs[2];
+ if (fstat (elf_fd, &fs) != 0)
+ throw libc_exception (errno, "cannot fstat file");
+
+ tvs[0].tv_sec = tvs[1].tv_sec = fs.st_mtime;
+ tvs[0].tv_usec = tvs[1].tv_usec = 0;
+ (void) futimes (fd, tvs);
+
+ /* Add to fdcache. */
+ fdcache.intern (b_source, section, tmppath, data->d_size, true);
+ break;
+ }
+ }
+ }
+ catch (const reportable_exception &e)
+ {
+ e.report (clog);
+ close (fd);
+ }
+
+ elf_end (elf);
+ return fd;
+}
+
+static struct MHD_Response*
+handle_buildid_f_match (bool internal_req_t,
+ int64_t b_mtime,
+ const string& b_source0,
+ const string& section,
+ int *result_fd)
+{
+ (void) internal_req_t; // ignored
+ int fd = open(b_source0.c_str(), O_RDONLY);
+ if (fd < 0)
+ throw libc_exception (errno, string("open ") + b_source0);
+
+ // NB: use manual close(2) in error case instead of defer_dtor, because
+ // in the normal case, we want to hand the fd over to libmicrohttpd for
+ // file transfer.
+
+ struct stat s;
+ int rc = fstat(fd, &s);
+ if (rc < 0)
+ {
+ close(fd);
+ throw libc_exception (errno, string("fstat ") + b_source0);
+ }
+
+ if ((int64_t) s.st_mtime != b_mtime)
+ {
+ if (verbose)
+ obatched(clog) << "mtime mismatch for " << b_source0 << endl;
+ close(fd);
+ return 0;
+ }
+
+ if (!section.empty ())
+ {
+ int scn_fd = extract_section (fd, s.st_mtime, b_source0, section);
+ close (fd);
+
+ if (scn_fd >= 0)
+ fd = scn_fd;
+ else
+ {
+ if (verbose)
+ obatched (clog) << "cannot find section " << section
+ << " for " << b_source0 << endl;
+ return 0;
+ }
+
+ rc = fstat(fd, &s);
+ if (rc < 0)
+ {
+ close (fd);
+ throw libc_exception (errno, string ("fstat ") + b_source0
+ + string (" ") + section);
+ }
+ }
+
+ struct MHD_Response* r = MHD_create_response_from_fd ((uint64_t) s.st_size, fd);
+ inc_metric ("http_responses_total","result","file");
+ if (r == 0)
+ {
+ if (verbose)
+ {
+ if (!section.empty ())
+ obatched(clog) << "cannot create fd-response for " << b_source0
+ << " section " << section << endl;
+ else
+ obatched(clog) << "cannot create fd-response for " << b_source0 << endl;
+ }
+ close(fd);
+ }
+ else
+ {
+ std::string file = b_source0.substr(b_source0.find_last_of("/")+1, b_source0.length());
+ add_mhd_response_header (r, "Content-Type", "application/octet-stream");
+ add_mhd_response_header (r, "X-DEBUGINFOD-SIZE",
+ to_string(s.st_size).c_str());
+ add_mhd_response_header (r, "X-DEBUGINFOD-FILE", file.c_str());
+ add_mhd_last_modified (r, s.st_mtime);
+ if (!section.empty ())
+ add_mhd_response_header (r, "X-DEBUGINFOD-SECTION", section.c_str ());
+ if (verbose > 1)
+ {
+ if (!section.empty ())
+ obatched(clog) << "serving file " << b_source0
+ << " section " << section << endl;
+ else
+ obatched(clog) << "serving file " << b_source0 << endl;
+ }
+ /* libmicrohttpd will close it. */
+ if (result_fd)
+ *result_fd = fd;
+ }
+
+ return r;
+}
// For security/portability reasons, many distro-package archives have
// a "./" in front of path names; others have nothing, others have
@@ -1614,6 +1778,7 @@ handle_buildid_r_match (bool internal_req_p,
int64_t b_mtime,
const string& b_source0,
const string& b_source1,
+ const string& section,
int *result_fd)
{
struct stat fs;
@@ -1642,6 +1807,33 @@ handle_buildid_r_match (bool internal_req_p,
break; // branch out of if "loop", to try new libarchive fetch attempt
}
+ if (!section.empty ())
+ {
+ int scn_fd = extract_section (fd, fs.st_mtime,
+ b_source0 + ":" + b_source1,
+ section);
+ close (fd);
+ if (scn_fd >= 0)
+ fd = scn_fd;
+ else
+ {
+ if (verbose)
+ obatched (clog) << "cannot find section " << section
+ << " for archive " << b_source0
+ << " file " << b_source1 << endl;
+ return 0;
+ }
+
+ rc = fstat(fd, &fs);
+ if (rc < 0)
+ {
+ close (fd);
+ throw libc_exception (errno,
+ string ("fstat archive ") + b_source0 + string (" file ") + b_source1
+ + string (" section ") + section);
+ }
+ }
+
struct MHD_Response* r = MHD_create_response_from_fd (fs.st_size, fd);
if (r == 0)
{
@@ -1658,9 +1850,19 @@ handle_buildid_r_match (bool internal_req_p,
to_string(fs.st_size).c_str());
add_mhd_response_header (r, "X-DEBUGINFOD-ARCHIVE", b_source0.c_str());
add_mhd_response_header (r, "X-DEBUGINFOD-FILE", b_source1.c_str());
+ if (!section.empty ())
+ add_mhd_response_header (r, "X-DEBUGINFOD-SECTION", section.c_str ());
add_mhd_last_modified (r, fs.st_mtime);
if (verbose > 1)
- obatched(clog) << "serving fdcache archive " << b_source0 << " file " << b_source1 << endl;
+ {
+ if (!section.empty ())
+ obatched(clog) << "serving fdcache archive " << b_source0
+ << " file " << b_source1
+ << " section " << section << endl;
+ else
+ obatched(clog) << "serving fdcache archive " << b_source0
+ << " file " << b_source1 << endl;
+ }
/* libmicrohttpd will close it. */
if (result_fd)
*result_fd = fd;
@@ -1791,8 +1993,36 @@ handle_buildid_r_match (bool internal_req_p,
tmppath, archive_entry_size(e),
true); // requested ones go to the front of lru
+ if (!section.empty ())
+ {
+ int scn_fd = extract_section (fd, b_mtime,
+ b_source0 + ":" + b_source1,
+ section);
+ close (fd);
+ if (scn_fd >= 0)
+ fd = scn_fd;
+ else
+ {
+ if (verbose)
+ obatched (clog) << "cannot find section " << section
+ << " for archive " << b_source0
+ << " file " << b_source1 << endl;
+ return 0;
+ }
+
+ rc = fstat(fd, &fs);
+ if (rc < 0)
+ {
+ close (fd);
+ throw libc_exception (errno,
+ string ("fstat ") + b_source0 + string (" ") + section);
+ }
+ r = MHD_create_response_from_fd (fs.st_size, fd);
+ }
+ else
+ r = MHD_create_response_from_fd (archive_entry_size(e), fd);
+
inc_metric ("http_responses_total","result",archive_extension + " archive");
- r = MHD_create_response_from_fd (archive_entry_size(e), fd);
if (r == 0)
{
if (verbose)
@@ -1810,9 +2040,19 @@ handle_buildid_r_match (bool internal_req_p,
add_mhd_response_header (r, "X-DEBUGINFOD-ARCHIVE",
b_source0.c_str());
add_mhd_response_header (r, "X-DEBUGINFOD-FILE", file.c_str());
+ if (!section.empty ())
+ add_mhd_response_header (r, "X-DEBUGINFOD-SECTION", section.c_str ());
add_mhd_last_modified (r, archive_entry_mtime(e));
if (verbose > 1)
- obatched(clog) << "serving archive " << b_source0 << " file " << b_source1 << endl;
+ {
+ if (!section.empty ())
+ obatched(clog) << "serving archive " << b_source0
+ << " file " << b_source1
+ << " section " << section << endl;
+ else
+ obatched(clog) << "serving archive " << b_source0
+ << " file " << b_source1 << endl;
+ }
/* libmicrohttpd will close it. */
if (result_fd)
*result_fd = fd;
@@ -1831,14 +2071,17 @@ handle_buildid_match (bool internal_req_p,
const string& b_stype,
const string& b_source0,
const string& b_source1,
+ const string& section,
int *result_fd)
{
try
{
if (b_stype == "F")
- return handle_buildid_f_match(internal_req_p, b_mtime, b_source0, result_fd);
+ return handle_buildid_f_match(internal_req_p, b_mtime, b_source0,
+ section, result_fd);
else if (b_stype == "R")
- return handle_buildid_r_match(internal_req_p, b_mtime, b_source0, b_source1, result_fd);
+ return handle_buildid_r_match(internal_req_p, b_mtime, b_source0,
+ b_source1, section, result_fd);
}
catch (const reportable_exception &e)
{
@@ -1913,6 +2156,7 @@ handle_buildid (MHD_Connection* conn,
if (artifacttype == "debuginfo") atype_code = "D";
else if (artifacttype == "executable") atype_code = "E";
else if (artifacttype == "source") atype_code = "S";
+ else if (artifacttype == "section") atype_code = "I";
else {
artifacttype = "invalid"; // PR28242 ensure http_resposes metrics don't propagate unclean user data
throw reportable_exception("invalid artifacttype");
@@ -1920,7 +2164,18 @@ handle_buildid (MHD_Connection* conn,
if (conn != 0)
inc_metric("http_requests_total", "type", artifacttype);
-
+
+ string section;
+ if (atype_code == "I")
+ {
+ // Ensure suffix contains a section name starting with "."
+ if (suffix.size () < 3 || suffix.substr (1, 1) != ".")
+ throw reportable_exception ("invalid section suffix");
+
+ // Remove leading '/'
+ section = suffix.substr(1);
+ }
+
if (atype_code == "S" && suffix == "")
throw reportable_exception("invalid source suffix");
@@ -1972,8 +2227,19 @@ handle_buildid (MHD_Connection* conn,
pp->bind(2, suffix);
pp->bind(3, canon_pathname(suffix));
}
+ else if (atype_code == "I")
+ {
+ pp = new sqlite_ps (thisdb, "mhd-query-i",
+ "select mtime, sourcetype, source0, source1, debuginfo_p, executable_p from " BUILDIDS "_query_i where buildid = ? "
+ "order by mtime desc, debuginfo_p desc");
+ pp->reset();
+ pp->bind(1, buildid);
+ }
unique_ptr<sqlite_ps> ps_closer(pp); // release pp if exception or return
+ int found_exec = false;
+ int found_debug = false;
+
// consume all the rows
while (1)
{
@@ -1994,12 +2260,28 @@ handle_buildid (MHD_Connection* conn,
// Try accessing the located match.
// XXX: in case of multiple matches, attempt them in parallel?
auto r = handle_buildid_match (conn ? false : true,
- b_mtime, b_stype, b_source0, b_source1, result_fd);
+ b_mtime, b_stype, b_source0, b_source1,
+ section, result_fd);
if (r)
return r;
+
+ // If we are handling a section query, record whether we found an
+ // executable or debuginfo containing build-id.
+ if (!section.empty ()
+ && fdcache.probe (b_source0, b_source1))
+ {
+ found_exec |= sqlite3_column_int (*pp, 4);
+ found_debug |= sqlite3_column_int (*pp, 5);
+ }
}
pp->reset();
+ // If the requested section was empty or wasn't found and we checked both
+ // executable and debuginfo containing build-id, then don't bother querying
+ // upstream servers.
+ if (!section.empty () && found_exec && found_debug)
+ throw reportable_exception(MHD_HTTP_NOT_FOUND, "not found");
+
// We couldn't find it in the database. Last ditch effort
// is to defer to other debuginfo servers.
@@ -2074,6 +2356,11 @@ and will not query the upstream servers");
fd = debuginfod_find_source (client,
(const unsigned char*) buildid.c_str(),
0, suffix.c_str(), NULL);
+ else if (artifacttype == "section")
+ fd = debuginfod_find_section (client,
+ (const unsigned char*) buildid.c_str(),
+ 0, section.c_str(), NULL);
+
}
else
fd = -errno; /* Set by debuginfod_begin. */
@@ -2096,7 +2383,17 @@ and will not query the upstream servers");
const char * hdrs = debuginfod_get_headers(client);
string header_dup;
if (hdrs)
- header_dup = string(hdrs);
+ {
+ header_dup = string(hdrs);
+
+ /* It's possible that we queried an upstream server for
+ a debuginfo or executable and extracted a section from
+ it. In this case we have to add the section header. */
+ if (!section.empty ()
+ && strcasestr (hdrs, "x-debuginfod-section") == NULL)
+ header_dup += string ("x-debuginfod-section: ")
+ + section + "\n";
+ }
// Parse the "header: value\n" lines into (h,v) tuples and pass on
while(1)
{
diff --git a/debuginfod/debuginfod.h.in b/debuginfod/debuginfod.h.in
index 40b1ea00..134bb1cd 100644
--- a/debuginfod/debuginfod.h.in
+++ b/debuginfod/debuginfod.h.in
@@ -78,6 +78,12 @@ int debuginfod_find_source (debuginfod_client *client,
const char *filename,
char **path);
+int debuginfod_find_section (debuginfod_client *client,
+ const unsigned char *build_id,
+ int build_id_len,
+ const char *section,
+ char **path);
+
typedef int (*debuginfod_progressfn_t)(debuginfod_client *c, long a, long b);
void debuginfod_set_progressfn(debuginfod_client *c,
debuginfod_progressfn_t fn);
diff --git a/debuginfod/libdebuginfod.map b/debuginfod/libdebuginfod.map
index 93964167..6334373f 100644
--- a/debuginfod/libdebuginfod.map
+++ b/debuginfod/libdebuginfod.map
@@ -20,4 +20,5 @@ ELFUTILS_0.183 {
} ELFUTILS_0.179;
ELFUTILS_0.188 {
debuginfod_get_headers;
+ debuginfod_find_section;
} ELFUTILS_0.183;
diff --git a/doc/ChangeLog b/doc/ChangeLog
index b2bb4890..15279470 100644
--- a/doc/ChangeLog
+++ b/doc/ChangeLog
@@ -1,3 +1,9 @@
+2022-09-27 Aaron Merey <amerey@redhat.com>
+
+ * Makefile.am (notrans_dist_man3_MANS): Add debuginfod_find_section.3.
+ * debuginfod_find_section.3: New file.
+ * debuginfod_find_debuginfo.3: Document debuginfod_find_section.
+
2022-09-02 Frank Ch. Eigler <fche@redhat.com>
* debuginfod_find_debuginfo.3: Tweaked debuginfod_get_headers docs.
diff --git a/doc/Makefile.am b/doc/Makefile.am
index db2506fd..db5a842e 100644
--- a/doc/Makefile.am
+++ b/doc/Makefile.am
@@ -38,6 +38,7 @@ notrans_dist_man3_MANS += debuginfod_end.3
notrans_dist_man3_MANS += debuginfod_find_debuginfo.3
notrans_dist_man3_MANS += debuginfod_find_executable.3
notrans_dist_man3_MANS += debuginfod_find_source.3
+notrans_dist_man3_MANS += debuginfod_find_section.3
notrans_dist_man3_MANS += debuginfod_get_user_data.3
notrans_dist_man3_MANS += debuginfod_get_url.3
notrans_dist_man3_MANS += debuginfod_set_progressfn.3
diff --git a/doc/debuginfod_find_debuginfo.3 b/doc/debuginfod_find_debuginfo.3
index aebbec3f..54503e8a 100644
--- a/doc/debuginfod_find_debuginfo.3
+++ b/doc/debuginfod_find_debuginfo.3
@@ -43,6 +43,12 @@ LOOKUP FUNCTIONS
.BI " int " build_id_len ","
.BI " const char *" filename ","
.BI " char ** " path ");"
+.BI "int debuginfod_find_section(debuginfod_client *" client ","
+.BI " const unsigned char *" build_id ","
+.BI " int " build_id_len ","
+.BI " const char * " section ","
+.BI " char ** " path ");"
+
OPTIONAL FUNCTIONS
@@ -98,6 +104,16 @@ accepts both forms. Specifically, debuginfod canonicalizes path names
according to RFC3986 section 5.2.4 (Remove Dot Segments), plus reducing
any \fB//\fP to \fB/\fP in the path.
+.BR debuginfod_find_section ()
+queries the debuginfod server URLs contained in
+.BR $DEBUGINFOD_URLS
+for the binary contents of an ELF/DWARF section contained in a debuginfo
+or executable file with the given \fIbuild_id\fP. \fIsection\fP should
+be the name of the desired ELF/DWARF section and must begin with ".".
+If a server does not support section queries, debuginfod_find_section
+may query the server for the debuginfo and/or executable with \fIbuild_id\fP
+in order to retrieve the section.
+
If \fIpath\fP is not NULL and the query is successful, \fIpath\fP is set
to the path of the file in the cache. The caller must \fBfree\fP() this value.
@@ -228,11 +244,8 @@ void *debuginfod_so = dlopen(DEBUGINFOD_SONAME, RTLD_LAZY);
.in
.SH "SECURITY"
-.BR debuginfod_find_debuginfo (),
-.BR debuginfod_find_executable (),
-and
-.BR debuginfod_find_source ()
-\fBdo not\fP include any particular security
+.BR debuginfod_find_* ()
+functions \fBdo not\fP include any particular security
features. They trust that the binaries returned by the debuginfod(s)
are accurate. Therefore, the list of servers should include only
trustworthy ones. If accessed across HTTP rather than HTTPS, the
@@ -267,7 +280,8 @@ Unable to resolve remote host.
.BR EINVAL
One or more arguments are incorrectly formatted. \fIbuild_id\fP may
be too long (greater than 256 characters), \fIfilename\fP may not
-be an absolute path or a debuginfod URL is malformed.
+be an absolute path or a debuginfod URL is malformed. \fIsection\fP
+may not begin with ".".
.TP
.BR EIO
diff --git a/doc/debuginfod_find_section.3 b/doc/debuginfod_find_section.3
new file mode 100644
index 00000000..16279936
--- /dev/null
+++ b/doc/debuginfod_find_section.3
@@ -0,0 +1 @@
+.so man3/debuginfod_find_debuginfo.3
diff --git a/tests/ChangeLog b/tests/ChangeLog
index 0ea1df3d..fe3c4dc4 100644
--- a/tests/ChangeLog
+++ b/tests/ChangeLog
@@ -1,3 +1,8 @@
+2022-10-20 Aaron Merey <amerey@redhat.com>
+
+ * Makefile.am (TESTS): Add run-debuginfod-section.sh.
+ * run-debuginfod-section.sh: New test.
+
2022-10-16 Mark Wielaard <mark@klomp.org>
* dwfl-report-offline-memory.c: Include config.h first.
diff --git a/tests/Makefile.am b/tests/Makefile.am
index f680d3e1..f7074075 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -247,7 +247,8 @@ TESTS += run-debuginfod-dlopen.sh \
run-debuginfod-x-forwarded-for.sh \
run-debuginfod-response-headers.sh \
run-debuginfod-extraction-passive.sh \
- run-debuginfod-webapi-concurrency.sh
+ run-debuginfod-webapi-concurrency.sh \
+ run-debuginfod-section.sh
endif
if !OLD_LIBMICROHTTPD
# Will crash on too old libmicrohttpd
@@ -554,6 +555,7 @@ EXTRA_DIST = run-arextract.sh run-arsymtest.sh run-ar.sh \
run-debuginfod-response-headers.sh \
run-debuginfod-extraction-passive.sh \
run-debuginfod-webapi-concurrency.sh \
+ run-debuginfod-section.sh \
debuginfod-rpms/fedora30/hello2-1.0-2.src.rpm \
debuginfod-rpms/fedora30/hello2-1.0-2.x86_64.rpm \
debuginfod-rpms/fedora30/hello2-debuginfo-1.0-2.x86_64.rpm \
diff --git a/tests/run-debuginfod-section.sh b/tests/run-debuginfod-section.sh
new file mode 100755
index 00000000..e640d274
--- /dev/null
+++ b/tests/run-debuginfod-section.sh
@@ -0,0 +1,141 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/debuginfod-subr.sh
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+DB=${PWD}/.debuginfod_tmp.sqlite
+tempfiles $DB
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+
+# Set up directories for scanning
+mkdir F
+mkdir R
+cp -rvp ${abs_srcdir}/debuginfod-rpms R
+
+# This variable is essential and ensures no time-race for claiming ports occurs
+# set base to a unique multiple of 100 not used in any other 'run-debuginfod-*' test
+base=13000
+get_ports
+
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod $VERBOSE \
+ -R R -F F -p $PORT1 -d $DB -t0 -g0 -v F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+# And initial scan should be done
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1
+
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+# We use -t0 and -g0 here to turn off time-based scanning & grooming.
+# For testing purposes, we just sic SIGUSR1 / SIGUSR2 at the process.
+
+########################################################################
+
+# Compile a simple program, strip its debuginfo and save the build-id.
+# Also move the debuginfo into another directory so that elfutils
+# cannot find it without debuginfod.
+echo "int main() { return 0; }" > ${PWD}/prog.c
+tempfiles prog.c
+
+gcc -Wl,--build-id -g -o F/prog ${PWD}/prog.c
+
+testrun ${abs_top_builddir}/src/strip -g -f F/prog.debug ${PWD}/F/prog
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a F/prog | grep 'Build ID' | cut -d ' ' -f 7`
+
+kill -USR1 $PID1
+# Wait till both files are in the index.
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+########################################################################
+
+# Build-id for a file in the one of the testsuite's F31 rpms
+RPM_BUILDID=420e9e3308971f4b817cc5bf83928b41a6909d88
+
+# Download sections from files indexed with -F
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find -vvv section $BUILDID .debug_info > vlog-find.1 2>&1
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find -vvv section $BUILDID .text > vlog-find.2 2>&1
+tempfiles vlog-find.1 vlog-find.2
+
+# Check that the X-DEBUGINFOD-SECTION response header contains the correct value
+testrun grep -i 'X-DEBUGINFOD-SECTION: .debug_info' vlog-find.1
+testrun grep -i 'X-DEBUGINFOD-SECTION: .text' vlog-find.2
+
+# Download sections from files indexed with -R
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find -vvv section $RPM_BUILDID .debug_info > vlog-find.1 2>&1
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find -vvv section $RPM_BUILDID .text > vlog-find.2 2>&1
+
+testrun grep -i 'X-DEBUGINFOD-SECTION: .debug_info' vlog-find.1
+testrun grep -i 'X-DEBUGINFOD-SECTION: .text' vlog-find.2
+
+# Verify that the downloaded files match the contents of the original sections
+objcopy F/prog.debug -O binary --only-section=.debug_info --set-section-flags .debug_info=alloc $BUILDID.debug_info
+tempfiles ${BUILDID}.debug_info
+testrun diff -u ${BUILDID}.debug_info ${DEBUGINFOD_CACHE_PATH}/${BUILDID}/section.debug_info
+
+objcopy F/prog -O binary --only-section=.text ${BUILDID}.text
+tempfiles ${BUILDID}.text
+testrun diff -u ${BUILDID}.text ${DEBUGINFOD_CACHE_PATH}/${BUILDID}/section.text
+
+# Download the original debuginfo/executable files.
+DEBUGFILE=`${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $RPM_BUILDID`
+EXECFILE=`${abs_top_builddir}/debuginfod/debuginfod-find executable $RPM_BUILDID`
+${abs_top_builddir}/debuginfod/debuginfod-find -vvv debuginfo $BUILDID
+${abs_top_builddir}/debuginfod/debuginfod-find -vvv executable $BUILDID
+
+objcopy $DEBUGFILE -O binary --only-section=.debug_info --set-section-flags .debug_info=alloc DEBUGFILE.debug_info
+tempfiles DEBUGFILE.debug_info
+testrun diff -u DEBUGFILE.debug_info ${DEBUGINFOD_CACHE_PATH}/${RPM_BUILDID}/section.debug_info
+
+objcopy $EXECFILE -O binary --only-section=.text EXECFILE.text
+tempfiles EXECFILE.text
+testrun diff -u EXECFILE.text ${DEBUGINFOD_CACHE_PATH}/${RPM_BUILDID}/section.text
+
+# Kill the server.
+kill $PID1
+wait $PID1
+PID1=0
+
+# Delete the section files from the cache.
+rm -f ${DEBUGINFOD_CACHE_PATH}/${RPM_BUILDID}/section.text
+rm -f ${DEBUGINFOD_CACHE_PATH}/${RPM_BUILDID}/section.debug_info
+rm -f ${DEBUGINFOD_CACHE_PATH}/${BUILDID}/section.text
+rm -f ${DEBUGINFOD_CACHE_PATH}/${BUILDID}/section.debug_info
+
+# Verify that the client can extract sections from the debuginfo or executable
+# if they're already in the cache.
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find -vvv section $BUILDID .debug_info
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find -vvv section $BUILDID .text
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find -vvv section $RPM_BUILDID .debug_info
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find -vvv section $RPM_BUILDID .text
+
+exit 0
--
2.37.3
^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH] debuginfod: Support queries for ELF/DWARF sections
2022-10-21 0:06 [PATCH v2] debuginfod: Support queries for ELF/DWARF sections Aaron Merey
@ 2022-10-22 0:09 ` Aaron Merey
2022-10-24 18:38 ` Frank Ch. Eigler
0 siblings, 1 reply; 8+ messages in thread
From: Aaron Merey @ 2022-10-22 0:09 UTC (permalink / raw)
To: elfutils-devel; +Cc: Aaron Merey
I'm resending this patch with a small modification. I added a new
field "progressfn_cancel" to debuginfod_client that indicates whether
the most recent query was cancelled due to progressfn returning 1.
If a server doesn't support section queries and the client begins
downloading a debuginfo or executable in an attempt to extract the
section, progressfn_cancel is used to indicate that if the first
query was cancelled by the progressfn then the second query should
also be skipped.
---
ChangeLog | 4 +
NEWS | 2 +
debuginfod/ChangeLog | 22 ++
debuginfod/Makefile.am | 2 +-
debuginfod/debuginfod-client.c | 295 +++++++++++++++++++++-
debuginfod/debuginfod-find.c | 21 +-
debuginfod/debuginfod.cxx | 433 +++++++++++++++++++++++++++-----
debuginfod/debuginfod.h.in | 6 +
debuginfod/libdebuginfod.map | 1 +
doc/ChangeLog | 6 +
doc/Makefile.am | 1 +
doc/debuginfod_find_debuginfo.3 | 26 +-
doc/debuginfod_find_section.3 | 1 +
tests/ChangeLog | 5 +
tests/Makefile.am | 4 +-
tests/run-debuginfod-section.sh | 141 +++++++++++
16 files changed, 886 insertions(+), 84 deletions(-)
create mode 100644 doc/debuginfod_find_section.3
create mode 100755 tests/run-debuginfod-section.sh
diff --git a/ChangeLog b/ChangeLog
index 60624183..29b2108e 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,7 @@
+2022-10-20 Aaron Merey <amerey@redhat.com>
+
+ * NEWS: Add debuginfod_find_section.
+
2022-09-13 Aleksei Vetrov <vvvvvv@google.com>
* NEWS (libdwfl): Add dwfl_report_offline_memory.
diff --git a/NEWS b/NEWS
index 6ebd172c..3e290ff7 100644
--- a/NEWS
+++ b/NEWS
@@ -4,6 +4,8 @@ readelf: Add -D, --use-dynamic option.
debuginfod: Add --disable-source-scan option.
+debuginfod-client: Add new function debuginfod_find_section.
+
libdwfl: Add new function dwfl_get_debuginfod_client.
Add new function dwfl_frame_reg.
Add new function dwfl_report_offline_memory.
diff --git a/debuginfod/ChangeLog b/debuginfod/ChangeLog
index 59d50df1..aacf655b 100644
--- a/debuginfod/ChangeLog
+++ b/debuginfod/ChangeLog
@@ -1,3 +1,25 @@
+2022-10-20 Aaron Merey <amerey@redhat.com>
+
+ * Makefile.am (libdebuginfod_so_LDLIBS): Add libelf.
+ * debuginfod-client.c (debuginfod_find_section): New function.
+ (extract_section): New function.
+ (maybe_debuginfo_section): New function.
+ (cache_find_section): New function.
+ (debuginfod_query_server): Add support for section queries.
+ * debuginfod-find.c (main): Add support for section queries.
+ * debuginfod.cxx (extract_section): New function.
+ (handle_buildid_f_match): Add section parameter. When non-empty,
+ try to create response from section contents.
+ (handle_buildid_r_match): Add section parameter. When non-empty,
+ try to create response from section contents.
+ (handle_buildid_match): Add section parameter. Pass to
+ handle_buildid_{f,r}_match.
+ (handle_buildid): Handle section name when artifacttype is set to
+ "section". Query upstream servers via debuginfod_find_section
+ when necessary.
+ (debuginfod.h.in): Add declaration for debuginfod_find_section.
+ (libdebuginfod.map): Add debuginfod_find_section.
+
2022-10-17 Frank Ch. Eigler <fche@redhat.com>
* debuginfod.cxx (main): Report libmicrohttpd version.
diff --git a/debuginfod/Makefile.am b/debuginfod/Makefile.am
index 435cb8a6..f27d6e2e 100644
--- a/debuginfod/Makefile.am
+++ b/debuginfod/Makefile.am
@@ -97,7 +97,7 @@ libdebuginfod_so_LIBS = libdebuginfod_pic.a
if DUMMY_LIBDEBUGINFOD
libdebuginfod_so_LDLIBS =
else
-libdebuginfod_so_LDLIBS = -lpthread $(libcurl_LIBS) $(fts_LIBS)
+libdebuginfod_so_LDLIBS = -lpthread $(libcurl_LIBS) $(fts_LIBS) $(libelf)
endif
$(LIBDEBUGINFOD_SONAME): $(srcdir)/libdebuginfod.map $(libdebuginfod_so_LIBS)
$(AM_V_CCLD)$(LINK) $(dso_LDFLAGS) -o $@ \
diff --git a/debuginfod/debuginfod-client.c b/debuginfod/debuginfod-client.c
index 2a14d9d9..c50110b7 100644
--- a/debuginfod/debuginfod-client.c
+++ b/debuginfod/debuginfod-client.c
@@ -44,6 +44,7 @@
#include "system.h"
#include <errno.h>
#include <stdlib.h>
+#include <libdwelf.h>
/* We might be building a bootstrap dummy library, which is really simple. */
#ifdef DUMMY_LIBDEBUGINFOD
@@ -55,6 +56,9 @@ int debuginfod_find_executable (debuginfod_client *c, const unsigned char *b,
int s, char **p) { return -ENOSYS; }
int debuginfod_find_source (debuginfod_client *c, const unsigned char *b,
int s, const char *f, char **p) { return -ENOSYS; }
+int debuginfod_find_section (debuginfod_client *c, const unsigned char *b,
+ int s, const char *scn, char **p)
+ { return -ENOSYS; }
void debuginfod_set_progressfn(debuginfod_client *c,
debuginfod_progressfn_t fn) { }
void debuginfod_set_verbose_fd(debuginfod_client *c, int fd) { }
@@ -129,6 +133,9 @@ struct debuginfod_client
debuginfod_end needs to terminate. */
int default_progressfn_printed_p;
+ /* Indicates whether the last query was cancelled by progressfn. */
+ bool progressfn_cancel;
+
/* File descriptor to output any verbose messages if > 0. */
int verbose_fd;
@@ -536,6 +543,193 @@ header_callback (char * buffer, size_t size, size_t numitems, void * userdata)
return numitems;
}
+/* Attempt to read an ELF/DWARF section with name SECTION from FD and write
+ it to a separate file in the debuginfod cache. If successful the absolute
+ path of the separate file containing SECTION will be stored in USR_PATH.
+ FD_PATH is the absolute path for FD. */
+
+int
+extract_section (int fd, const char *section, char *fd_path, char **usr_path)
+{
+ Elf *elf = elf_begin (fd, ELF_C_READ_MMAP_PRIVATE, NULL);
+ if (elf == NULL)
+ return -EIO;
+
+ size_t shstrndx;
+ int rc = elf_getshdrstrndx (elf, &shstrndx);
+ if (rc < 0)
+ return -EIO;
+
+ /* Try to find the target section and copy the contents into a
+ separate file. */
+ Elf_Scn *scn = NULL;
+ while (true)
+ {
+ scn = elf_nextscn (elf, scn);
+ if (scn == NULL)
+ {
+ rc = -ENOENT;
+ break;
+ }
+ GElf_Shdr shdr_storage;
+ GElf_Shdr *shdr = gelf_getshdr (scn, &shdr_storage);
+ if (shdr == NULL)
+ {
+ rc = -EIO;
+ break;
+ }
+
+ const char *scn_name = elf_strptr (elf, shstrndx, shdr->sh_name);
+ if (scn_name == NULL)
+ {
+ rc = -EIO;
+ break;
+ }
+ if (strcmp (scn_name, section) == 0)
+ {
+ Elf_Data *data = NULL;
+
+ /* We found the desired section. */
+ data = elf_getdata (scn, NULL);
+ if (data == NULL)
+ {
+ rc = -EIO;
+ break;
+ }
+
+ if (data->d_buf == NULL)
+ {
+ rc = -ENOENT;
+ break;
+ }
+
+ /* Compute the absolute filename we'll write the section to. Replace
+ the last component in FD_PATH with the section filename. */
+ int i = strlen (fd_path);
+ while (i >= 0)
+ {
+ if (fd_path[i] == '/')
+ {
+ fd_path[i] = '\0';
+ break;
+ }
+ --i;
+ }
+
+ char *sec_path;
+ if (asprintf (&sec_path, "%s/section%s", fd_path, section) == -1)
+ {
+ rc = -ENOMEM;
+ break;
+ }
+
+ int sec_fd = open (sec_path, O_CREAT | O_EXCL | O_RDWR, S_IRUSR);
+ if (sec_fd < 0)
+ {
+ free (sec_path);
+ rc = -errno;
+ break;
+ }
+
+ ssize_t res = write (sec_fd, data->d_buf, data->d_size);
+ if (res < 0 || (size_t) res != data->d_size)
+ {
+ close (sec_fd);
+ free (sec_path);
+ rc = -EIO;
+ break;
+ }
+
+ /* Success. Update USR_PATH. */
+ if (usr_path != NULL)
+ *usr_path = sec_path;
+ else
+ free (sec_path);
+ rc = sec_fd;
+ break;
+ }
+ }
+
+ elf_end (elf);
+ return rc;
+}
+
+/* Return true if we should search a debuginfo file for the ELF/DWARF section
+ with name SCN_NAME before searching the executable file. */
+
+static bool
+maybe_debuginfo_section (const char *scn_name)
+{
+ return (startswith (scn_name, ".debug")
+ || strcmp (scn_name, ".symtab") == 0
+ || strcmp (scn_name, ".strtab") == 0
+ || strcmp (scn_name, ".shstrtab") == 0
+ || strcmp (scn_name, ".gdb_index") == 0
+ || strcmp (scn_name, ".gnu_debugaltlink") == 0
+ || strcmp (scn_name, ".gnu.build.attributes") == 0);
+}
+
+/* Search TARGET_CACHE_DIR for a debuginfo or executable file containing
+ an ELF/DWARF section with name SCN_NAME. If found, extract the section
+ to a separate file in TARGET_CACHE_DIR and return a file descriptor
+ for the section file. The path for this file will be stored in USR_PATH.
+ Return a negative errno if unsuccessful. */
+
+static int
+cache_find_section (const char *scn_name, const char *target_cache_dir,
+ char **usr_path)
+{
+ int fd;
+ int rc = -ENOENT;
+ bool debug_p = maybe_debuginfo_section (scn_name);
+ char exec_path[PATH_MAX];
+ char debug_path[PATH_MAX];
+ char *fd_path;
+
+ snprintf (exec_path, PATH_MAX, "%s/executable", target_cache_dir);
+ snprintf (debug_path, PATH_MAX, "%s/debuginfo", target_cache_dir);
+
+ if (debug_p)
+ {
+ fd = open (debug_path, O_RDONLY);
+ fd_path = debug_path;
+ }
+ else
+ {
+ fd = open (exec_path, O_RDONLY);
+ fd_path = exec_path;
+ }
+
+ if (fd >= 0)
+ {
+ rc = extract_section (fd, scn_name, fd_path, usr_path);
+ close (fd);
+ }
+
+ /* If the section wasn't found, check the other file where it might be. */
+ if (rc < 0)
+ {
+ if (debug_p)
+ {
+ fd = open (exec_path, O_RDONLY);
+ fd_path = exec_path;
+ }
+ else
+ {
+ fd = open (debug_path, O_RDONLY);
+ fd_path = debug_path;
+ }
+
+ if (fd >= 0)
+ {
+ rc = extract_section (fd, scn_name, fd_path, usr_path);
+ close (fd);
+ }
+ }
+
+ return rc;
+}
+
/* Query each of the server URLs found in $DEBUGINFOD_URLS for the file
with the specified build-id, type (debuginfo, executable or source)
and filename. filename may be NULL. If found, return a file
@@ -546,11 +740,13 @@ debuginfod_query_server (debuginfod_client *c,
const unsigned char *build_id,
int build_id_len,
const char *type,
- const char *filename,
+ const char *type_arg,
char **path)
{
char *server_urls;
char *urls_envvar;
+ const char *section = NULL;
+ const char *filename = NULL;
char *cache_path = NULL;
char *maxage_path = NULL;
char *interval_path = NULL;
@@ -563,6 +759,17 @@ debuginfod_query_server (debuginfod_client *c,
int vfd = c->verbose_fd;
int rc;
+ c->progressfn_cancel = false;
+
+ if (strcmp (type, "source") == 0)
+ filename = type_arg;
+ else if (strcmp (type, "section") == 0)
+ {
+ section = type_arg;
+ if (section != NULL && section[0] != '.')
+ return -EINVAL;
+ }
+
if (vfd >= 0)
{
dprintf (vfd, "debuginfod_find_%s ", type);
@@ -681,6 +888,8 @@ debuginfod_query_server (debuginfod_client *c,
PATH_MAX and truncate/collide. Oh well, that'll teach
them! */
}
+ else if (section != NULL)
+ strncpy (suffix, section, PATH_MAX);
else
suffix[0] = '\0';
@@ -836,6 +1045,15 @@ debuginfod_query_server (debuginfod_client *c,
/* Ensure old 000-permission files are not lingering in the cache. */
unlink(target_cache_path);
+ if (section != NULL)
+ {
+ /* Try to extract the section from a cached file before querying
+ any servers. */
+ rc = cache_find_section (section, target_cache_dir, path);
+ if (rc >= 0)
+ goto out;
+ }
+
long timeout = default_timeout;
const char* timeout_envvar = getenv(DEBUGINFOD_TIMEOUT_ENV_VAR);
if (timeout_envvar != NULL)
@@ -1008,6 +1226,9 @@ debuginfod_query_server (debuginfod_client *c,
snprintf(data[i].url, PATH_MAX, "%s/%s/%s/%s", server_url,
build_id_bytes, type, escaped_string);
}
+ else if (section)
+ snprintf(data[i].url, PATH_MAX, "%s/%s/%s/%s", server_url,
+ build_id_bytes, type, section);
else
snprintf(data[i].url, PATH_MAX, "%s/%s/%s", server_url, build_id_bytes, type);
if (vfd >= 0)
@@ -1212,7 +1433,10 @@ debuginfod_query_server (debuginfod_client *c,
}
if ((*c->progressfn) (c, pa, dl_size))
- break;
+ {
+ c->progressfn_cancel = true;
+ break;
+ }
}
/* Check to see if we are downloading something which exceeds maxsize, if set.*/
@@ -1279,6 +1503,8 @@ debuginfod_query_server (debuginfod_client *c,
/* 406 signals that the requested file was too large */
if ( ok0 == CURLE_OK && resp_code == 406)
rc = -EFBIG;
+ else if (section != NULL && resp_code == 503)
+ rc = -EINVAL;
else
rc = -ENOENT;
break;
@@ -1525,6 +1751,7 @@ debuginfod_begin (void)
goto out1;
}
+ elf_version (EV_CURRENT);
// extra future initialization
goto out;
@@ -1604,6 +1831,70 @@ int debuginfod_find_source(debuginfod_client *client,
"source", filename, path);
}
+int
+debuginfod_find_section (debuginfod_client *client,
+ const unsigned char *build_id, int build_id_len,
+ const char *section, char **path)
+{
+ int rc = debuginfod_query_server(client, build_id, build_id_len,
+ "section", section, path);
+ if (rc != -EINVAL)
+ return rc;
+
+ /* The servers may have lacked support for section queries. Attempt to
+ download the debuginfo or executable containing the section in order
+ to extract it. */
+ rc = -ENOENT;
+ int fd = -1;
+ char *tmp_path = NULL;
+
+ /* Use the section name as a guide for which file to try first. */
+ bool debug_p = maybe_debuginfo_section (section);
+
+ if (debug_p)
+ fd = debuginfod_find_debuginfo (client, build_id, build_id_len, &tmp_path);
+ else
+ fd = debuginfod_find_executable (client, build_id, build_id_len, &tmp_path);
+
+ if (client->progressfn_cancel)
+ {
+ if (fd >= 0)
+ {
+ /* This shouldn't happen, but we'll check this condition
+ just in case. */
+ close (fd);
+ free (tmp_path);
+ }
+ return -ENOENT;
+ }
+
+ if (fd > 0)
+ {
+ rc = extract_section (fd, section, tmp_path, path);
+ close (fd);
+ }
+
+ if (rc == -ENOENT)
+ {
+ /* Still unable to find the section. Try to extract section from the
+ other file type with build_id. */
+ if (debug_p)
+ fd = debuginfod_find_executable (client, build_id,
+ build_id_len, &tmp_path);
+ else
+ fd = debuginfod_find_debuginfo (client, build_id,
+ build_id_len, &tmp_path);
+
+ if (fd > 0)
+ {
+ rc = extract_section (fd, section, tmp_path, path);
+ close (fd);
+ }
+ }
+
+ free (tmp_path);
+ return rc;
+}
/* Add an outgoing HTTP header. */
int debuginfod_add_http_header (debuginfod_client *client, const char* header)
diff --git a/debuginfod/debuginfod-find.c b/debuginfod/debuginfod-find.c
index 778fb09b..197bea6c 100644
--- a/debuginfod/debuginfod-find.c
+++ b/debuginfod/debuginfod-find.c
@@ -190,13 +190,11 @@ main(int argc, char** argv)
debuginfod_find_* function. If FILETYPE is "source"
then ensure a FILENAME was also supplied as an argument. */
if (strcmp(argv[remaining], "debuginfo") == 0)
- rc = debuginfod_find_debuginfo(client,
- build_id, build_id_len,
- &cache_name);
+ rc = debuginfod_find_debuginfo(client, build_id,
+ build_id_len, &cache_name);
else if (strcmp(argv[remaining], "executable") == 0)
- rc = debuginfod_find_executable(client,
- build_id, build_id_len,
- &cache_name);
+ rc = debuginfod_find_executable(client, build_id,
+ build_id_len, &cache_name);
else if (strcmp(argv[remaining], "source") == 0)
{
if (remaining+2 == argc || argv[remaining+2][0] != '/')
@@ -208,6 +206,17 @@ main(int argc, char** argv)
build_id, build_id_len,
argv[remaining+2], &cache_name);
}
+ else if (strcmp(argv[remaining], "section") == 0)
+ {
+ if (remaining+2 == argc || argv[remaining+2][0] != '.')
+ {
+ fprintf(stderr,
+ "A section name starting with '.' must be given\n");
+ return 1;
+ }
+ rc = debuginfod_find_section(client, build_id, build_id_len,
+ argv[remaining+2], &cache_name);
+ }
else
{
argp_help (&argp, stderr, ARGP_HELP_USAGE, argv[0]);
diff --git a/debuginfod/debuginfod.cxx b/debuginfod/debuginfod.cxx
index 9dc4836b..19e35716 100644
--- a/debuginfod/debuginfod.cxx
+++ b/debuginfod/debuginfod.cxx
@@ -249,6 +249,19 @@ static const char DEBUGINFOD_SQLITE_DDL[] =
" where b.id = sr.buildid and f0.id = sd.file and fsref.id = sde.file and f1.id = sd.content\n"
" and sr.artifactsrc = sd.content and sde.buildid = sr.buildid\n"
";"
+ // ... and for I queries
+ "create view if not exists " BUILDIDS "_query_i as \n"
+ "select\n"
+ " b.hex as buildid, n.mtime, 'F' as sourcetype, f0.name as source0, n.mtime as mtime, null as source1,"
+ " n.debuginfo_p as debuginfo_p, n.executable_p as executable_p\n"
+ " from " BUILDIDS "_buildids b, " BUILDIDS "_files f0, " BUILDIDS "_f_de n\n"
+ " where b.id = n.buildid and f0.id = n.file\n"
+ "union all select\n"
+ " b.hex as buildid, n.mtime, 'R' as sourcetype, f0.name as source0, n.mtime as mtime, f1.name as source1,"
+ " n.debuginfo_p as debuginfo_p, n.executable_p as executable_p\n"
+ " from " BUILDIDS "_buildids b, " BUILDIDS "_files f0, " BUILDIDS "_files f1, " BUILDIDS "_r_de n\n"
+ " where b.id = n.buildid and f0.id = n.file and f1.id = n.content\n"
+ ";"
// and for startup overview counts
"drop view if exists " BUILDIDS "_stats;\n"
"create view if not exists " BUILDIDS "_stats as\n"
@@ -1136,66 +1149,6 @@ add_mhd_last_modified (struct MHD_Response *resp, time_t mtime)
add_mhd_response_header (resp, "Cache-Control", "public");
}
-
-
-static struct MHD_Response*
-handle_buildid_f_match (bool internal_req_t,
- int64_t b_mtime,
- const string& b_source0,
- int *result_fd)
-{
- (void) internal_req_t; // ignored
- int fd = open(b_source0.c_str(), O_RDONLY);
- if (fd < 0)
- throw libc_exception (errno, string("open ") + b_source0);
-
- // NB: use manual close(2) in error case instead of defer_dtor, because
- // in the normal case, we want to hand the fd over to libmicrohttpd for
- // file transfer.
-
- struct stat s;
- int rc = fstat(fd, &s);
- if (rc < 0)
- {
- close(fd);
- throw libc_exception (errno, string("fstat ") + b_source0);
- }
-
- if ((int64_t) s.st_mtime != b_mtime)
- {
- if (verbose)
- obatched(clog) << "mtime mismatch for " << b_source0 << endl;
- close(fd);
- return 0;
- }
-
- inc_metric ("http_responses_total","result","file");
- struct MHD_Response* r = MHD_create_response_from_fd ((uint64_t) s.st_size, fd);
- if (r == 0)
- {
- if (verbose)
- obatched(clog) << "cannot create fd-response for " << b_source0 << endl;
- close(fd);
- }
- else
- {
- std::string file = b_source0.substr(b_source0.find_last_of("/")+1, b_source0.length());
- add_mhd_response_header (r, "Content-Type", "application/octet-stream");
- add_mhd_response_header (r, "X-DEBUGINFOD-SIZE",
- to_string(s.st_size).c_str());
- add_mhd_response_header (r, "X-DEBUGINFOD-FILE", file.c_str());
- add_mhd_last_modified (r, s.st_mtime);
- if (verbose > 1)
- obatched(clog) << "serving file " << b_source0 << endl;
- /* libmicrohttpd will close it. */
- if (result_fd)
- *result_fd = fd;
- }
-
- return r;
-}
-
-
// quote all questionable characters of str for safe passage through a sh -c expansion.
static string
shell_escape(const string& str)
@@ -1589,6 +1542,217 @@ public:
};
static libarchive_fdcache fdcache;
+/* Search ELF_FD for an ELF/DWARF section with name SECTION.
+ If found copy the section to a temporary file and return
+ its file descriptor, otherwise return -1.
+
+ The temporary file's mtime will be set to PARENT_MTIME.
+ B_SOURCE should be a description of the parent file suitable
+ for printing to the log. */
+
+static int
+extract_section (int elf_fd, int64_t parent_mtime,
+ const string& b_source, const string& section)
+{
+ /* Search the fdcache. */
+ struct stat fs;
+ int fd = fdcache.lookup (b_source, section);
+ if (fd >= 0)
+ {
+ if (fstat (fd, &fs) != 0)
+ {
+ if (verbose)
+ obatched (clog) << "cannot fstate fdcache "
+ << b_source << " " << section << endl;
+ close (fd);
+ return -1;
+ }
+ if ((int64_t) fs.st_mtime != parent_mtime)
+ {
+ if (verbose)
+ obatched(clog) << "mtime mismatch for "
+ << b_source << " " << section << endl;
+ close (fd);
+ return -1;
+ }
+ /* Success. */
+ return fd;
+ }
+
+ Elf *elf = elf_begin (elf_fd, ELF_C_READ_MMAP_PRIVATE, NULL);
+ if (elf == NULL)
+ return -1;
+
+ /* Try to find the section and copy the contents into a separate file. */
+ try
+ {
+ size_t shstrndx;
+ int rc = elf_getshdrstrndx (elf, &shstrndx);
+ if (rc < 0)
+ throw elfutils_exception (rc, "getshdrstrndx");
+
+ Elf_Scn *scn = NULL;
+ while (true)
+ {
+ scn = elf_nextscn (elf, scn);
+ if (scn == NULL)
+ break;
+ GElf_Shdr shdr_storage;
+ GElf_Shdr *shdr = gelf_getshdr (scn, &shdr_storage);
+ if (shdr == NULL)
+ break;
+
+ const char *scn_name = elf_strptr (elf, shstrndx, shdr->sh_name);
+ if (scn_name == NULL)
+ break;
+ if (scn_name == section)
+ {
+ Elf_Data *data = NULL;
+
+ /* We found the desired section. */
+ data = elf_getdata (scn, NULL);
+ if (data == NULL)
+ throw elfutils_exception (elf_errno (), "getdata");
+ if (data->d_buf == NULL)
+ {
+ obatched(clog) << "section " << section
+ << " is empty" << endl;
+ break;
+ }
+
+ /* Create temporary file containing the section. */
+ char *tmppath = NULL;
+ rc = asprintf (&tmppath, "%s/debuginfod.XXXXXX", tmpdir.c_str());
+ if (rc < 0)
+ throw libc_exception (ENOMEM, "cannot allocate tmppath");
+ defer_dtor<void*,void> tmmpath_freer (tmppath, free);
+ fd = mkstemp (tmppath);
+ if (fd < 0)
+ throw libc_exception (errno, "cannot create temporary file");
+ ssize_t res = write (fd, data->d_buf, data->d_size);
+ if (res < 0 || (size_t) res != data->d_size)
+ throw libc_exception (errno, "cannot write to temporary file");
+
+ /* Set mtime to be the same as the parent file's mtime. */
+ struct timeval tvs[2];
+ if (fstat (elf_fd, &fs) != 0)
+ throw libc_exception (errno, "cannot fstat file");
+
+ tvs[0].tv_sec = tvs[1].tv_sec = fs.st_mtime;
+ tvs[0].tv_usec = tvs[1].tv_usec = 0;
+ (void) futimes (fd, tvs);
+
+ /* Add to fdcache. */
+ fdcache.intern (b_source, section, tmppath, data->d_size, true);
+ break;
+ }
+ }
+ }
+ catch (const reportable_exception &e)
+ {
+ e.report (clog);
+ close (fd);
+ }
+
+ elf_end (elf);
+ return fd;
+}
+
+static struct MHD_Response*
+handle_buildid_f_match (bool internal_req_t,
+ int64_t b_mtime,
+ const string& b_source0,
+ const string& section,
+ int *result_fd)
+{
+ (void) internal_req_t; // ignored
+ int fd = open(b_source0.c_str(), O_RDONLY);
+ if (fd < 0)
+ throw libc_exception (errno, string("open ") + b_source0);
+
+ // NB: use manual close(2) in error case instead of defer_dtor, because
+ // in the normal case, we want to hand the fd over to libmicrohttpd for
+ // file transfer.
+
+ struct stat s;
+ int rc = fstat(fd, &s);
+ if (rc < 0)
+ {
+ close(fd);
+ throw libc_exception (errno, string("fstat ") + b_source0);
+ }
+
+ if ((int64_t) s.st_mtime != b_mtime)
+ {
+ if (verbose)
+ obatched(clog) << "mtime mismatch for " << b_source0 << endl;
+ close(fd);
+ return 0;
+ }
+
+ if (!section.empty ())
+ {
+ int scn_fd = extract_section (fd, s.st_mtime, b_source0, section);
+ close (fd);
+
+ if (scn_fd >= 0)
+ fd = scn_fd;
+ else
+ {
+ if (verbose)
+ obatched (clog) << "cannot find section " << section
+ << " for " << b_source0 << endl;
+ return 0;
+ }
+
+ rc = fstat(fd, &s);
+ if (rc < 0)
+ {
+ close (fd);
+ throw libc_exception (errno, string ("fstat ") + b_source0
+ + string (" ") + section);
+ }
+ }
+
+ struct MHD_Response* r = MHD_create_response_from_fd ((uint64_t) s.st_size, fd);
+ inc_metric ("http_responses_total","result","file");
+ if (r == 0)
+ {
+ if (verbose)
+ {
+ if (!section.empty ())
+ obatched(clog) << "cannot create fd-response for " << b_source0
+ << " section " << section << endl;
+ else
+ obatched(clog) << "cannot create fd-response for " << b_source0 << endl;
+ }
+ close(fd);
+ }
+ else
+ {
+ std::string file = b_source0.substr(b_source0.find_last_of("/")+1, b_source0.length());
+ add_mhd_response_header (r, "Content-Type", "application/octet-stream");
+ add_mhd_response_header (r, "X-DEBUGINFOD-SIZE",
+ to_string(s.st_size).c_str());
+ add_mhd_response_header (r, "X-DEBUGINFOD-FILE", file.c_str());
+ add_mhd_last_modified (r, s.st_mtime);
+ if (!section.empty ())
+ add_mhd_response_header (r, "X-DEBUGINFOD-SECTION", section.c_str ());
+ if (verbose > 1)
+ {
+ if (!section.empty ())
+ obatched(clog) << "serving file " << b_source0
+ << " section " << section << endl;
+ else
+ obatched(clog) << "serving file " << b_source0 << endl;
+ }
+ /* libmicrohttpd will close it. */
+ if (result_fd)
+ *result_fd = fd;
+ }
+
+ return r;
+}
// For security/portability reasons, many distro-package archives have
// a "./" in front of path names; others have nothing, others have
@@ -1614,6 +1778,7 @@ handle_buildid_r_match (bool internal_req_p,
int64_t b_mtime,
const string& b_source0,
const string& b_source1,
+ const string& section,
int *result_fd)
{
struct stat fs;
@@ -1642,6 +1807,33 @@ handle_buildid_r_match (bool internal_req_p,
break; // branch out of if "loop", to try new libarchive fetch attempt
}
+ if (!section.empty ())
+ {
+ int scn_fd = extract_section (fd, fs.st_mtime,
+ b_source0 + ":" + b_source1,
+ section);
+ close (fd);
+ if (scn_fd >= 0)
+ fd = scn_fd;
+ else
+ {
+ if (verbose)
+ obatched (clog) << "cannot find section " << section
+ << " for archive " << b_source0
+ << " file " << b_source1 << endl;
+ return 0;
+ }
+
+ rc = fstat(fd, &fs);
+ if (rc < 0)
+ {
+ close (fd);
+ throw libc_exception (errno,
+ string ("fstat archive ") + b_source0 + string (" file ") + b_source1
+ + string (" section ") + section);
+ }
+ }
+
struct MHD_Response* r = MHD_create_response_from_fd (fs.st_size, fd);
if (r == 0)
{
@@ -1658,9 +1850,19 @@ handle_buildid_r_match (bool internal_req_p,
to_string(fs.st_size).c_str());
add_mhd_response_header (r, "X-DEBUGINFOD-ARCHIVE", b_source0.c_str());
add_mhd_response_header (r, "X-DEBUGINFOD-FILE", b_source1.c_str());
+ if (!section.empty ())
+ add_mhd_response_header (r, "X-DEBUGINFOD-SECTION", section.c_str ());
add_mhd_last_modified (r, fs.st_mtime);
if (verbose > 1)
- obatched(clog) << "serving fdcache archive " << b_source0 << " file " << b_source1 << endl;
+ {
+ if (!section.empty ())
+ obatched(clog) << "serving fdcache archive " << b_source0
+ << " file " << b_source1
+ << " section " << section << endl;
+ else
+ obatched(clog) << "serving fdcache archive " << b_source0
+ << " file " << b_source1 << endl;
+ }
/* libmicrohttpd will close it. */
if (result_fd)
*result_fd = fd;
@@ -1791,8 +1993,36 @@ handle_buildid_r_match (bool internal_req_p,
tmppath, archive_entry_size(e),
true); // requested ones go to the front of lru
+ if (!section.empty ())
+ {
+ int scn_fd = extract_section (fd, b_mtime,
+ b_source0 + ":" + b_source1,
+ section);
+ close (fd);
+ if (scn_fd >= 0)
+ fd = scn_fd;
+ else
+ {
+ if (verbose)
+ obatched (clog) << "cannot find section " << section
+ << " for archive " << b_source0
+ << " file " << b_source1 << endl;
+ return 0;
+ }
+
+ rc = fstat(fd, &fs);
+ if (rc < 0)
+ {
+ close (fd);
+ throw libc_exception (errno,
+ string ("fstat ") + b_source0 + string (" ") + section);
+ }
+ r = MHD_create_response_from_fd (fs.st_size, fd);
+ }
+ else
+ r = MHD_create_response_from_fd (archive_entry_size(e), fd);
+
inc_metric ("http_responses_total","result",archive_extension + " archive");
- r = MHD_create_response_from_fd (archive_entry_size(e), fd);
if (r == 0)
{
if (verbose)
@@ -1810,9 +2040,19 @@ handle_buildid_r_match (bool internal_req_p,
add_mhd_response_header (r, "X-DEBUGINFOD-ARCHIVE",
b_source0.c_str());
add_mhd_response_header (r, "X-DEBUGINFOD-FILE", file.c_str());
+ if (!section.empty ())
+ add_mhd_response_header (r, "X-DEBUGINFOD-SECTION", section.c_str ());
add_mhd_last_modified (r, archive_entry_mtime(e));
if (verbose > 1)
- obatched(clog) << "serving archive " << b_source0 << " file " << b_source1 << endl;
+ {
+ if (!section.empty ())
+ obatched(clog) << "serving archive " << b_source0
+ << " file " << b_source1
+ << " section " << section << endl;
+ else
+ obatched(clog) << "serving archive " << b_source0
+ << " file " << b_source1 << endl;
+ }
/* libmicrohttpd will close it. */
if (result_fd)
*result_fd = fd;
@@ -1831,14 +2071,17 @@ handle_buildid_match (bool internal_req_p,
const string& b_stype,
const string& b_source0,
const string& b_source1,
+ const string& section,
int *result_fd)
{
try
{
if (b_stype == "F")
- return handle_buildid_f_match(internal_req_p, b_mtime, b_source0, result_fd);
+ return handle_buildid_f_match(internal_req_p, b_mtime, b_source0,
+ section, result_fd);
else if (b_stype == "R")
- return handle_buildid_r_match(internal_req_p, b_mtime, b_source0, b_source1, result_fd);
+ return handle_buildid_r_match(internal_req_p, b_mtime, b_source0,
+ b_source1, section, result_fd);
}
catch (const reportable_exception &e)
{
@@ -1913,6 +2156,7 @@ handle_buildid (MHD_Connection* conn,
if (artifacttype == "debuginfo") atype_code = "D";
else if (artifacttype == "executable") atype_code = "E";
else if (artifacttype == "source") atype_code = "S";
+ else if (artifacttype == "section") atype_code = "I";
else {
artifacttype = "invalid"; // PR28242 ensure http_resposes metrics don't propagate unclean user data
throw reportable_exception("invalid artifacttype");
@@ -1920,7 +2164,18 @@ handle_buildid (MHD_Connection* conn,
if (conn != 0)
inc_metric("http_requests_total", "type", artifacttype);
-
+
+ string section;
+ if (atype_code == "I")
+ {
+ // Ensure suffix contains a section name starting with "."
+ if (suffix.size () < 3 || suffix.substr (1, 1) != ".")
+ throw reportable_exception ("invalid section suffix");
+
+ // Remove leading '/'
+ section = suffix.substr(1);
+ }
+
if (atype_code == "S" && suffix == "")
throw reportable_exception("invalid source suffix");
@@ -1972,8 +2227,19 @@ handle_buildid (MHD_Connection* conn,
pp->bind(2, suffix);
pp->bind(3, canon_pathname(suffix));
}
+ else if (atype_code == "I")
+ {
+ pp = new sqlite_ps (thisdb, "mhd-query-i",
+ "select mtime, sourcetype, source0, source1, debuginfo_p, executable_p from " BUILDIDS "_query_i where buildid = ? "
+ "order by mtime desc, debuginfo_p desc");
+ pp->reset();
+ pp->bind(1, buildid);
+ }
unique_ptr<sqlite_ps> ps_closer(pp); // release pp if exception or return
+ int found_exec = false;
+ int found_debug = false;
+
// consume all the rows
while (1)
{
@@ -1994,12 +2260,28 @@ handle_buildid (MHD_Connection* conn,
// Try accessing the located match.
// XXX: in case of multiple matches, attempt them in parallel?
auto r = handle_buildid_match (conn ? false : true,
- b_mtime, b_stype, b_source0, b_source1, result_fd);
+ b_mtime, b_stype, b_source0, b_source1,
+ section, result_fd);
if (r)
return r;
+
+ // If we are handling a section query, record whether we found an
+ // executable or debuginfo containing build-id.
+ if (!section.empty ()
+ && fdcache.probe (b_source0, b_source1))
+ {
+ found_exec |= sqlite3_column_int (*pp, 4);
+ found_debug |= sqlite3_column_int (*pp, 5);
+ }
}
pp->reset();
+ // If the requested section was empty or wasn't found and we checked both
+ // executable and debuginfo containing build-id, then don't bother querying
+ // upstream servers.
+ if (!section.empty () && found_exec && found_debug)
+ throw reportable_exception(MHD_HTTP_NOT_FOUND, "not found");
+
// We couldn't find it in the database. Last ditch effort
// is to defer to other debuginfo servers.
@@ -2074,6 +2356,11 @@ and will not query the upstream servers");
fd = debuginfod_find_source (client,
(const unsigned char*) buildid.c_str(),
0, suffix.c_str(), NULL);
+ else if (artifacttype == "section")
+ fd = debuginfod_find_section (client,
+ (const unsigned char*) buildid.c_str(),
+ 0, section.c_str(), NULL);
+
}
else
fd = -errno; /* Set by debuginfod_begin. */
@@ -2096,7 +2383,17 @@ and will not query the upstream servers");
const char * hdrs = debuginfod_get_headers(client);
string header_dup;
if (hdrs)
- header_dup = string(hdrs);
+ {
+ header_dup = string(hdrs);
+
+ /* It's possible that we queried an upstream server for
+ a debuginfo or executable and extracted a section from
+ it. In this case we have to add the section header. */
+ if (!section.empty ()
+ && strcasestr (hdrs, "x-debuginfod-section") == NULL)
+ header_dup += string ("x-debuginfod-section: ")
+ + section + "\n";
+ }
// Parse the "header: value\n" lines into (h,v) tuples and pass on
while(1)
{
diff --git a/debuginfod/debuginfod.h.in b/debuginfod/debuginfod.h.in
index 40b1ea00..134bb1cd 100644
--- a/debuginfod/debuginfod.h.in
+++ b/debuginfod/debuginfod.h.in
@@ -78,6 +78,12 @@ int debuginfod_find_source (debuginfod_client *client,
const char *filename,
char **path);
+int debuginfod_find_section (debuginfod_client *client,
+ const unsigned char *build_id,
+ int build_id_len,
+ const char *section,
+ char **path);
+
typedef int (*debuginfod_progressfn_t)(debuginfod_client *c, long a, long b);
void debuginfod_set_progressfn(debuginfod_client *c,
debuginfod_progressfn_t fn);
diff --git a/debuginfod/libdebuginfod.map b/debuginfod/libdebuginfod.map
index 93964167..6334373f 100644
--- a/debuginfod/libdebuginfod.map
+++ b/debuginfod/libdebuginfod.map
@@ -20,4 +20,5 @@ ELFUTILS_0.183 {
} ELFUTILS_0.179;
ELFUTILS_0.188 {
debuginfod_get_headers;
+ debuginfod_find_section;
} ELFUTILS_0.183;
diff --git a/doc/ChangeLog b/doc/ChangeLog
index b2bb4890..15279470 100644
--- a/doc/ChangeLog
+++ b/doc/ChangeLog
@@ -1,3 +1,9 @@
+2022-09-27 Aaron Merey <amerey@redhat.com>
+
+ * Makefile.am (notrans_dist_man3_MANS): Add debuginfod_find_section.3.
+ * debuginfod_find_section.3: New file.
+ * debuginfod_find_debuginfo.3: Document debuginfod_find_section.
+
2022-09-02 Frank Ch. Eigler <fche@redhat.com>
* debuginfod_find_debuginfo.3: Tweaked debuginfod_get_headers docs.
diff --git a/doc/Makefile.am b/doc/Makefile.am
index db2506fd..db5a842e 100644
--- a/doc/Makefile.am
+++ b/doc/Makefile.am
@@ -38,6 +38,7 @@ notrans_dist_man3_MANS += debuginfod_end.3
notrans_dist_man3_MANS += debuginfod_find_debuginfo.3
notrans_dist_man3_MANS += debuginfod_find_executable.3
notrans_dist_man3_MANS += debuginfod_find_source.3
+notrans_dist_man3_MANS += debuginfod_find_section.3
notrans_dist_man3_MANS += debuginfod_get_user_data.3
notrans_dist_man3_MANS += debuginfod_get_url.3
notrans_dist_man3_MANS += debuginfod_set_progressfn.3
diff --git a/doc/debuginfod_find_debuginfo.3 b/doc/debuginfod_find_debuginfo.3
index aebbec3f..54503e8a 100644
--- a/doc/debuginfod_find_debuginfo.3
+++ b/doc/debuginfod_find_debuginfo.3
@@ -43,6 +43,12 @@ LOOKUP FUNCTIONS
.BI " int " build_id_len ","
.BI " const char *" filename ","
.BI " char ** " path ");"
+.BI "int debuginfod_find_section(debuginfod_client *" client ","
+.BI " const unsigned char *" build_id ","
+.BI " int " build_id_len ","
+.BI " const char * " section ","
+.BI " char ** " path ");"
+
OPTIONAL FUNCTIONS
@@ -98,6 +104,16 @@ accepts both forms. Specifically, debuginfod canonicalizes path names
according to RFC3986 section 5.2.4 (Remove Dot Segments), plus reducing
any \fB//\fP to \fB/\fP in the path.
+.BR debuginfod_find_section ()
+queries the debuginfod server URLs contained in
+.BR $DEBUGINFOD_URLS
+for the binary contents of an ELF/DWARF section contained in a debuginfo
+or executable file with the given \fIbuild_id\fP. \fIsection\fP should
+be the name of the desired ELF/DWARF section and must begin with ".".
+If a server does not support section queries, debuginfod_find_section
+may query the server for the debuginfo and/or executable with \fIbuild_id\fP
+in order to retrieve the section.
+
If \fIpath\fP is not NULL and the query is successful, \fIpath\fP is set
to the path of the file in the cache. The caller must \fBfree\fP() this value.
@@ -228,11 +244,8 @@ void *debuginfod_so = dlopen(DEBUGINFOD_SONAME, RTLD_LAZY);
.in
.SH "SECURITY"
-.BR debuginfod_find_debuginfo (),
-.BR debuginfod_find_executable (),
-and
-.BR debuginfod_find_source ()
-\fBdo not\fP include any particular security
+.BR debuginfod_find_* ()
+functions \fBdo not\fP include any particular security
features. They trust that the binaries returned by the debuginfod(s)
are accurate. Therefore, the list of servers should include only
trustworthy ones. If accessed across HTTP rather than HTTPS, the
@@ -267,7 +280,8 @@ Unable to resolve remote host.
.BR EINVAL
One or more arguments are incorrectly formatted. \fIbuild_id\fP may
be too long (greater than 256 characters), \fIfilename\fP may not
-be an absolute path or a debuginfod URL is malformed.
+be an absolute path or a debuginfod URL is malformed. \fIsection\fP
+may not begin with ".".
.TP
.BR EIO
diff --git a/doc/debuginfod_find_section.3 b/doc/debuginfod_find_section.3
new file mode 100644
index 00000000..16279936
--- /dev/null
+++ b/doc/debuginfod_find_section.3
@@ -0,0 +1 @@
+.so man3/debuginfod_find_debuginfo.3
diff --git a/tests/ChangeLog b/tests/ChangeLog
index 0ea1df3d..fe3c4dc4 100644
--- a/tests/ChangeLog
+++ b/tests/ChangeLog
@@ -1,3 +1,8 @@
+2022-10-20 Aaron Merey <amerey@redhat.com>
+
+ * Makefile.am (TESTS): Add run-debuginfod-section.sh.
+ * run-debuginfod-section.sh: New test.
+
2022-10-16 Mark Wielaard <mark@klomp.org>
* dwfl-report-offline-memory.c: Include config.h first.
diff --git a/tests/Makefile.am b/tests/Makefile.am
index f680d3e1..f7074075 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -247,7 +247,8 @@ TESTS += run-debuginfod-dlopen.sh \
run-debuginfod-x-forwarded-for.sh \
run-debuginfod-response-headers.sh \
run-debuginfod-extraction-passive.sh \
- run-debuginfod-webapi-concurrency.sh
+ run-debuginfod-webapi-concurrency.sh \
+ run-debuginfod-section.sh
endif
if !OLD_LIBMICROHTTPD
# Will crash on too old libmicrohttpd
@@ -554,6 +555,7 @@ EXTRA_DIST = run-arextract.sh run-arsymtest.sh run-ar.sh \
run-debuginfod-response-headers.sh \
run-debuginfod-extraction-passive.sh \
run-debuginfod-webapi-concurrency.sh \
+ run-debuginfod-section.sh \
debuginfod-rpms/fedora30/hello2-1.0-2.src.rpm \
debuginfod-rpms/fedora30/hello2-1.0-2.x86_64.rpm \
debuginfod-rpms/fedora30/hello2-debuginfo-1.0-2.x86_64.rpm \
diff --git a/tests/run-debuginfod-section.sh b/tests/run-debuginfod-section.sh
new file mode 100755
index 00000000..e640d274
--- /dev/null
+++ b/tests/run-debuginfod-section.sh
@@ -0,0 +1,141 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2019-2021 Red Hat, Inc.
+# This file is part of elfutils.
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# elfutils is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. $srcdir/debuginfod-subr.sh
+
+# for test case debugging, uncomment:
+set -x
+unset VALGRIND_CMD
+
+DB=${PWD}/.debuginfod_tmp.sqlite
+tempfiles $DB
+export DEBUGINFOD_CACHE_PATH=${PWD}/.client_cache
+
+# Set up directories for scanning
+mkdir F
+mkdir R
+cp -rvp ${abs_srcdir}/debuginfod-rpms R
+
+# This variable is essential and ensures no time-race for claiming ports occurs
+# set base to a unique multiple of 100 not used in any other 'run-debuginfod-*' test
+base=13000
+get_ports
+
+env LD_LIBRARY_PATH=$ldpath DEBUGINFOD_URLS= ${abs_builddir}/../debuginfod/debuginfod $VERBOSE \
+ -R R -F F -p $PORT1 -d $DB -t0 -g0 -v F > vlog$PORT1 2>&1 &
+PID1=$!
+tempfiles vlog$PORT1
+errfiles vlog$PORT1
+# Server must become ready
+wait_ready $PORT1 'ready' 1
+# And initial scan should be done
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 1
+
+export DEBUGINFOD_URLS=http://127.0.0.1:$PORT1
+
+# Check thread comm names
+ps -q $PID1 -e -L -o '%p %c %a' | grep groom
+ps -q $PID1 -e -L -o '%p %c %a' | grep scan
+ps -q $PID1 -e -L -o '%p %c %a' | grep traverse
+
+# We use -t0 and -g0 here to turn off time-based scanning & grooming.
+# For testing purposes, we just sic SIGUSR1 / SIGUSR2 at the process.
+
+########################################################################
+
+# Compile a simple program, strip its debuginfo and save the build-id.
+# Also move the debuginfo into another directory so that elfutils
+# cannot find it without debuginfod.
+echo "int main() { return 0; }" > ${PWD}/prog.c
+tempfiles prog.c
+
+gcc -Wl,--build-id -g -o F/prog ${PWD}/prog.c
+
+testrun ${abs_top_builddir}/src/strip -g -f F/prog.debug ${PWD}/F/prog
+BUILDID=`env LD_LIBRARY_PATH=$ldpath ${abs_builddir}/../src/readelf \
+ -a F/prog | grep 'Build ID' | cut -d ' ' -f 7`
+
+kill -USR1 $PID1
+# Wait till both files are in the index.
+wait_ready $PORT1 'thread_work_total{role="traverse"}' 2
+wait_ready $PORT1 'thread_work_pending{role="scan"}' 0
+wait_ready $PORT1 'thread_busy{role="scan"}' 0
+
+########################################################################
+
+# Build-id for a file in the one of the testsuite's F31 rpms
+RPM_BUILDID=420e9e3308971f4b817cc5bf83928b41a6909d88
+
+# Download sections from files indexed with -F
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find -vvv section $BUILDID .debug_info > vlog-find.1 2>&1
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find -vvv section $BUILDID .text > vlog-find.2 2>&1
+tempfiles vlog-find.1 vlog-find.2
+
+# Check that the X-DEBUGINFOD-SECTION response header contains the correct value
+testrun grep -i 'X-DEBUGINFOD-SECTION: .debug_info' vlog-find.1
+testrun grep -i 'X-DEBUGINFOD-SECTION: .text' vlog-find.2
+
+# Download sections from files indexed with -R
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find -vvv section $RPM_BUILDID .debug_info > vlog-find.1 2>&1
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find -vvv section $RPM_BUILDID .text > vlog-find.2 2>&1
+
+testrun grep -i 'X-DEBUGINFOD-SECTION: .debug_info' vlog-find.1
+testrun grep -i 'X-DEBUGINFOD-SECTION: .text' vlog-find.2
+
+# Verify that the downloaded files match the contents of the original sections
+objcopy F/prog.debug -O binary --only-section=.debug_info --set-section-flags .debug_info=alloc $BUILDID.debug_info
+tempfiles ${BUILDID}.debug_info
+testrun diff -u ${BUILDID}.debug_info ${DEBUGINFOD_CACHE_PATH}/${BUILDID}/section.debug_info
+
+objcopy F/prog -O binary --only-section=.text ${BUILDID}.text
+tempfiles ${BUILDID}.text
+testrun diff -u ${BUILDID}.text ${DEBUGINFOD_CACHE_PATH}/${BUILDID}/section.text
+
+# Download the original debuginfo/executable files.
+DEBUGFILE=`${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $RPM_BUILDID`
+EXECFILE=`${abs_top_builddir}/debuginfod/debuginfod-find executable $RPM_BUILDID`
+${abs_top_builddir}/debuginfod/debuginfod-find -vvv debuginfo $BUILDID
+${abs_top_builddir}/debuginfod/debuginfod-find -vvv executable $BUILDID
+
+objcopy $DEBUGFILE -O binary --only-section=.debug_info --set-section-flags .debug_info=alloc DEBUGFILE.debug_info
+tempfiles DEBUGFILE.debug_info
+testrun diff -u DEBUGFILE.debug_info ${DEBUGINFOD_CACHE_PATH}/${RPM_BUILDID}/section.debug_info
+
+objcopy $EXECFILE -O binary --only-section=.text EXECFILE.text
+tempfiles EXECFILE.text
+testrun diff -u EXECFILE.text ${DEBUGINFOD_CACHE_PATH}/${RPM_BUILDID}/section.text
+
+# Kill the server.
+kill $PID1
+wait $PID1
+PID1=0
+
+# Delete the section files from the cache.
+rm -f ${DEBUGINFOD_CACHE_PATH}/${RPM_BUILDID}/section.text
+rm -f ${DEBUGINFOD_CACHE_PATH}/${RPM_BUILDID}/section.debug_info
+rm -f ${DEBUGINFOD_CACHE_PATH}/${BUILDID}/section.text
+rm -f ${DEBUGINFOD_CACHE_PATH}/${BUILDID}/section.debug_info
+
+# Verify that the client can extract sections from the debuginfo or executable
+# if they're already in the cache.
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find -vvv section $BUILDID .debug_info
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find -vvv section $BUILDID .text
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find -vvv section $RPM_BUILDID .debug_info
+testrun ${abs_top_builddir}/debuginfod/debuginfod-find -vvv section $RPM_BUILDID .text
+
+exit 0
--
2.37.3
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] debuginfod: Support queries for ELF/DWARF sections
2022-10-22 0:09 ` [PATCH] " Aaron Merey
@ 2022-10-24 18:38 ` Frank Ch. Eigler
2022-10-26 14:57 ` Mark Wielaard
` (2 more replies)
0 siblings, 3 replies; 8+ messages in thread
From: Frank Ch. Eigler @ 2022-10-24 18:38 UTC (permalink / raw)
To: Aaron Merey; +Cc: elfutils-devel
Hi -
Generally looks fine, thanks a lot.
A few nits:
- use of write(2) to put files onto disk is not quite right; write(2) can
be partial, so you need a loop (or a macro wrapping a loop)
- not sure I understand why the code worries about dots in or not in
section names. Why not just pass them verbatim throughout the code
base, and not worry about whether or not there's a dot? Does the
ELF standard even require a dot?
- not sure whether/why the I queries require a new _query_i view, as
opposed to running the _query_d & _query_e views union'd together.
I see an ORDER BY that's different here but not sure why bother;
if anything, the server could prefer one or the other type, based
on the same section-name heuristic as the client
- don't really see a need for the X-DEBUGINFOD-SECTION response
header, which simply echoes back the very same parameter the client
just requested; the other X-DEBUGINFOD-* headers are novel metadata
- re. verbose logging in the section vs non-section case, suggest
just keeping the code simple (even if it makes the logs more verbose),
i.e., not duplicating if (...) clog << STUFF else clog << STUFF;
no biggie tho
- the webapi docs in debuginfod.8 should document the new query type
Otherwise lgtm. Lots of nice work.
- FChE
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] debuginfod: Support queries for ELF/DWARF sections
2022-10-24 18:38 ` Frank Ch. Eigler
@ 2022-10-26 14:57 ` Mark Wielaard
2022-10-26 15:28 ` Frank Ch. Eigler
2022-10-26 17:57 ` Aaron Merey
2022-10-26 15:25 ` Mark Wielaard
2022-10-26 17:55 ` Aaron Merey
2 siblings, 2 replies; 8+ messages in thread
From: Mark Wielaard @ 2022-10-26 14:57 UTC (permalink / raw)
To: Frank Ch. Eigler, Aaron Merey; +Cc: elfutils-devel
Hi,
On Mon, 2022-10-24 at 14:38 -0400, Frank Ch. Eigler via Elfutils-devel
wrote:
> - not sure I understand why the code worries about dots in or not in
> section names. Why not just pass them verbatim throughout the code
> base, and not worry about whether or not there's a dot? Does the
> ELF standard even require a dot?
I agree that just passing them through as is might be better. The ELF
standard doesn't say much about section names, just:
Section names with a dot (.) prefix are reserved for the system,
although applications may use these sections if their existing
meanings are satisfactory. Applications may use names without the
prefix to avoid conflicts with system sections.
Is/should the section name be URL-encoded?
I would drop the maybe_debuginfo_section heuristics. There are some
sections like .strtab/.symtab that are probably in the debug file, but
might be in the executable. I would assume that a named section can
normally be found in the debugfile and only use the executable as
fallback.
So see if you can find the .debug file, if you can, then look for the
section by name. If it isn't SHT_NOBITS you found it. If it is
SHT_NOBITS the section should be in the exe. If the section cannot be
found by name (in the .debug file) you can stop searching, it also
won't be in the exe. If you cannot find the .debug file, or the section
was in the .debug file, but had type SHT_NOBITS then search for the exe
file and the named section in there.
Finally, if the section comes from a file in the cache or if we have to
download it in full anyway, then extracting the section into its own
file seems slightly wasteful. It would be great if we could just report
back "here is the full exe/debug file which does contain the requested
section name". But that might make the interface a little ugly.
int
debuginfod_find_section (debuginfod_client *client,
const unsigned char *build_id,
int build_id_len,
const char *section, char **path,
bool *file_is_elf)
Maybe that is over-designed to avoid a little bit of disk waste?
Cheers,
Mark
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] debuginfod: Support queries for ELF/DWARF sections
2022-10-24 18:38 ` Frank Ch. Eigler
2022-10-26 14:57 ` Mark Wielaard
@ 2022-10-26 15:25 ` Mark Wielaard
2022-10-26 17:55 ` Aaron Merey
2 siblings, 0 replies; 8+ messages in thread
From: Mark Wielaard @ 2022-10-26 15:25 UTC (permalink / raw)
To: Frank Ch. Eigler, Aaron Merey; +Cc: elfutils-devel
Hi,
On Mon, 2022-10-24 at 14:38 -0400, Frank Ch. Eigler via Elfutils-devel
wrote:
> - use of write(2) to put files onto disk is not quite right; write(2)
> can
> be partial, so you need a loop (or a macro wrapping a loop)
Since debuginfod-client.c already includes system.h it can use:
static inline ssize_t
write_retry (int fd, const void *buf, size_t len)
Which takes care of partial and/or interrupted write calls.
Cheers,
Mark
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] debuginfod: Support queries for ELF/DWARF sections
2022-10-26 14:57 ` Mark Wielaard
@ 2022-10-26 15:28 ` Frank Ch. Eigler
2022-10-26 17:57 ` Aaron Merey
1 sibling, 0 replies; 8+ messages in thread
From: Frank Ch. Eigler @ 2022-10-26 15:28 UTC (permalink / raw)
To: Mark Wielaard; +Cc: Aaron Merey, elfutils-devel
Hi -
> Is/should the section name be URL-encoded?
Yes!
> I would drop the maybe_debuginfo_section heuristics. There are some
> sections like .strtab/.symtab that are probably in the debug file, but
> might be in the executable. I would assume that a named section can
> normally be found in the debugfile and only use the executable as
> fallback.
That heuristic would work fine for the case of .gdb_index, that
motivated this whole piece of work. Sure.
> Finally, if the section comes from a file in the cache or if we have to
> download it in full anyway, then extracting the section into its own
> file seems slightly wasteful. It would be great if we could just report
> back "here is the full exe/debug file which does contain the requested
> section name". [...]
It'd be fine to pass back the extracted section content anyway, even
if the full elf and/or dwarf file is already there. Consider
federated debuginfod servers. Intermediate servers may be
willing/able to do this extraction on behalf of clients who really
only want the section in question. And if they cache the result, as
in amerey's draft code, then this will also help accelerate other
future clients. That's just the usage scenario (gdb acceleration).
> int
> debuginfod_find_section (debuginfod_client *client,
> const unsigned char *build_id,
> int build_id_len,
> const char *section, char **path,
> bool *file_is_elf)
>
> Maybe that is over-designed to avoid a little bit of disk waste?
(Then the client code would have to learn elfutils API internals in
order to extract the section it was actually interested in.)
- FChE
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] debuginfod: Support queries for ELF/DWARF sections
2022-10-24 18:38 ` Frank Ch. Eigler
2022-10-26 14:57 ` Mark Wielaard
2022-10-26 15:25 ` Mark Wielaard
@ 2022-10-26 17:55 ` Aaron Merey
2 siblings, 0 replies; 8+ messages in thread
From: Aaron Merey @ 2022-10-26 17:55 UTC (permalink / raw)
To: Frank Ch. Eigler; +Cc: elfutils-devel
Hi Frank,
On Mon, Oct 24, 2022 at 2:38 PM Frank Ch. Eigler <fche@redhat.com> wrote:
> - use of write(2) to put files onto disk is not quite right; write(2) can
> be partial, so you need a loop (or a macro wrapping a loop)
Fixed.
> - not sure I understand why the code worries about dots in or not in
> section names. Why not just pass them verbatim throughout the code
> base, and not worry about whether or not there's a dot? Does the
> ELF standard even require a dot?
Fair point. The motivation for providing a general section query
API as opposed to an API for specific indices was to be flexible and
accommodate a wide range of use cases. Insisting on the dot is contrary
to this goal.
> - not sure whether/why the I queries require a new _query_i view, as
> opposed to running the _query_d & _query_e views union'd together.
> I see an ORDER BY that's different here but not sure why bother;
> if anything, the server could prefer one or the other type, based
> on the same section-name heuristic as the client
The idea here was to prevent cases where a section would be extracted
from the executable on one server and from the debuginfo on another server.
Granted this is only relevant when the file mtimes differ between servers
and for sections that differ between debuginfo and executable. This
situation seems unlikely and these particular sections aren't of much
interest (ex. .shstrtab). I will remove this view and just use the existing
ones.
> - don't really see a need for the X-DEBUGINFOD-SECTION response
> header, which simply echoes back the very same parameter the client
> just requested; the other X-DEBUGINFOD-* headers are novel metadata
>
> - re. verbose logging in the section vs non-section case, suggest
> just keeping the code simple (even if it makes the logs more verbose),
> i.e., not duplicating if (...) clog << STUFF else clog << STUFF;
> no biggie tho
>
> - the webapi docs in debuginfod.8 should document the new query type
Fixed.
Aaron
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] debuginfod: Support queries for ELF/DWARF sections
2022-10-26 14:57 ` Mark Wielaard
2022-10-26 15:28 ` Frank Ch. Eigler
@ 2022-10-26 17:57 ` Aaron Merey
1 sibling, 0 replies; 8+ messages in thread
From: Aaron Merey @ 2022-10-26 17:57 UTC (permalink / raw)
To: Mark Wielaard; +Cc: Frank Ch. Eigler, elfutils-devel
Hi Mark,
On Wed, Oct 26, 2022 at 11:06 AM Mark Wielaard <mark@klomp.org> wrote:
> On Mon, 2022-10-24 at 14:38 -0400, Frank Ch. Eigler via Elfutils-devel
> wrote:
> > - not sure I understand why the code worries about dots in or not in
> > section names. Why not just pass them verbatim throughout the code
> > base, and not worry about whether or not there's a dot? Does the
> > ELF standard even require a dot?
>
> I agree that just passing them through as is might be better. The ELF
> standard doesn't say much about section names, just:
>
> Section names with a dot (.) prefix are reserved for the system,
> although applications may use these sections if their existing
> meanings are satisfactory. Applications may use names without the
> prefix to avoid conflicts with system sections.
Agreed, will fix.
> I would drop the maybe_debuginfo_section heuristics. There are some
> sections like .strtab/.symtab that are probably in the debug file, but
> might be in the executable. I would assume that a named section can
> normally be found in the debugfile and only use the executable as
> fallback.
>
> So see if you can find the .debug file, if you can, then look for the
> section by name. If it isn't SHT_NOBITS you found it. If it is
> SHT_NOBITS the section should be in the exe. If the section cannot be
> found by name (in the .debug file) you can stop searching, it also
> won't be in the exe. If you cannot find the .debug file, or the section
> was in the .debug file, but had type SHT_NOBITS then search for the exe
> file and the named section in there.
I like this heuristic. It's simpler and we don't have to update anything
if/when a new section becomes common.
> Finally, if the section comes from a file in the cache or if we have to
> download it in full anyway, then extracting the section into its own
> file seems slightly wasteful. It would be great if we could just report
> back "here is the full exe/debug file which does contain the requested
> section name". But that might make the interface a little ugly.
>
> int
> debuginfod_find_section (debuginfod_client *client,
> const unsigned char *build_id,
> int build_id_len,
> const char *section, char **path,
> bool *file_is_elf)
>
> Maybe that is over-designed to avoid a little bit of disk waste?
We'd save a bit of disk space but complicate the API and often
cause client tools to have to do more work to get at the section
contents. In the case of .gdb_index, gdb already knows how to read
the index from a separate file. Of course it can read the section
from an ELF file too but I suspect there might need to be some
changes to teach it how to handle "unusual" ELF files that only
contain a single section.
> Since debuginfod-client.c already includes system.h it can use:
>
> static inline ssize_t
> write_retry (int fd, const void *buf, size_t len)
>
> Which takes care of partial and/or interrupted write calls.
Thanks that's exactly what I'm looking for.
Aaron
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2022-10-26 17:57 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-10-21 0:06 [PATCH v2] debuginfod: Support queries for ELF/DWARF sections Aaron Merey
2022-10-22 0:09 ` [PATCH] " Aaron Merey
2022-10-24 18:38 ` Frank Ch. Eigler
2022-10-26 14:57 ` Mark Wielaard
2022-10-26 15:28 ` Frank Ch. Eigler
2022-10-26 17:57 ` Aaron Merey
2022-10-26 15:25 ` Mark Wielaard
2022-10-26 17:55 ` Aaron Merey
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).