[PATCH 6/6] sysroot: _stp_umodule_relocate needs receive target file path

[PATCH 6/6] sysroot: _stp_umodule_relocate needs receive target file path

[Victor Kamensky]

Strip of sysroot from module name is required when
_stp_umodule_relocate call is generated. Otherwise path won't
match path on target and could will fail to calculated address
within the file. Note in another place in tapsets.cxx where
_stp_umodule_relocate is emitted path_remove_sysroot is already
applied.

Here are steps how to reproduce the issue. Note <sysroot> is produced by
yocto poky build.

[kamensky@coreos-lnx2 tests]$ cat od.stp
probe process("/usr/bin/od.coreutils").function("main").call {
  printf ("%s: flag_dump_strings=0x%x\n", ppfunc(), $flag_dump_strings)
}
[kamensky@coreos-lnx2 tests]$ /home/wd8/systemtap/20180208_2/packages/bin/stap --sysroot=/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs -a x86_64 -r 4.9.78-yocto-standard -B CROSS_COMPILE=x86_64-poky-linux- --sysenv=PATH=/usr/bin:/bin:/usr/sbin:/sbin --sysenv=LD_LIBRARY_PATH=/lib:/usr/lib -p4 -k -m od od.stp
od.ko
Keeping temporary directory "/tmp/stapdnaaGF"
[kamensky@coreos-lnx2 tests]$ grep _stp_umodule_relocate /tmp/stapdnaaGF/od_src.c
  l->__retvalue = uderef(1, ((((int64_t) (/* pragma:vma */ ({ unsigned long addr = 0; addr = _stp_umodule_relocate ("/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/usr/bin/od.coreutils", 0x212212, current); addr; }))))));

Note above _stp_umodule_relocate call was emitted with file path
on the host, it will fail once executed on target.

Signed-off-by: Victor Kamensky <kamensky@cisco.com>
---
 loc2stap.cxx | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/loc2stap.cxx b/loc2stap.cxx
index 4818ee088..e2339a45e 100644
--- a/loc2stap.cxx
+++ b/loc2stap.cxx
@@ -17,6 +17,7 @@
 
 #include "loc2stap.h"
 #include "dwflpp.h"
+#include "tapsets.h"
 
 #if ! _ELFUTILS_PREREQ(0, 153)
 #define DW_OP_GNU_entry_value 0xf3
@@ -106,7 +107,7 @@ location_context::translate_address(Dwarf_Addr addr)
           c = "/* pragma:vma */ "
               "({ unsigned long addr = 0; "
               "addr = _stp_umodule_relocate (\""
-              + resolve_path(dw->module_name.c_str()) + "\", "
+              + path_remove_sysroot(dw->sess, resolve_path(dw->module_name.c_str())) + "\", "
               + lex_cast_hex (addr)
 	      + ", current); addr; })";
 	}
-- 
2.14.3

[PATCH 3/6] sysroot: debuginfo lookup with sysroot case do not remove sysroot from file_name

[Victor Kamensky]

If sysroot option is passed, and debug symbols reside in sysroot along
with executable <foo> in <foo_dir>/.debug/<foo_file> directory, stap
fails to find debuginfo because it strips out sysroot path from file_name
so dwfl_standard_find_debuginfo ends up looking at host
<foo_dir>/.debug/<foo_file> rather then checking
<sysroot>/<foo_dir>/.debug/<foo_file>.

Note in cross compile environment, it is good idea to set and export
proper SYSTEMTAP_DEBUGINFO_PATH variable because usual built defaults that
work for native distros very often not applicable to cross sysroot
based environment. For example for yocto poky/OE build the following
setting seems proper "+:.debug:build".

Here are steps how to reproduce the issue. Note <sysroot> is produced by
yocto poky build.

[kamensky@coreos-lnx2 tests]$ ls /home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/mkdir.coreutils
/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/mkdir.coreutils
[kamensky@coreos-lnx2 tests]$ ls /home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/.debug/mkdir.coreutils
/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/.debug/mkdir.coreutils
[kamensky@coreos-lnx2 tests]$ /home/wd8/systemtap/20180208_2/packages/bin/stap --sysroot=/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs -a x86_64 -r 4.9.78-yocto-standard -B CROSS_COMPILE=x86_64-poky-linux- --sysenv=PATH=/usr/bin:/bin:/usr/sbin:/sbin --sysenv=LD_LIBRARY_PATH=/lib:/usr/lib -v -l 'process("/bin/mkdir.coreutils").function("*")'
Pass 1: parsed user script and 480 library scripts using 230184virt/93416res/5344shr/88612data kb, in 350usr/30sys/378real ms.
process("/bin/mkdir.coreutils").function("_fini")
process("/bin/mkdir.coreutils").function("_init")
process("/bin/mkdir.coreutils").function("fts_children")
process("/bin/mkdir.coreutils").function("fts_close")
process("/bin/mkdir.coreutils").function("fts_open")
process("/bin/mkdir.coreutils").function("fts_read")
process("/bin/mkdir.coreutils").function("fts_set")
Pass 2: analyzed script: 7 probes, 0 functions, 0 embeds, 0 globals using 233484virt/97928res/6212shr/91912data kb, in 20usr/0sys/26real ms.

running above example under strace shows that stap tries to access
/bin/.debug/mkdir.coreutils on host, rather then sysroot

13064 openat(AT_FDCWD, "/bin/mkdir.coreutils", O_RDONLY) = -1 ENOENT (No such file or directory)
13064 openat(AT_FDCWD, "/bin/.debug/mkdir.coreutils", O_RDONLY) = -1 ENOENT (No such file or directory)
13064 openat(AT_FDCWD, "/bin/mkdir.coreutils", O_RDONLY) = -1 ENOENT (No such file or directory)
13064 openat(AT_FDCWD, "/bin/build/mkdir.coreutils", O_RDONLY) = -1 ENOENT (No such file or directory)
13064 openat(AT_FDCWD, "/bin/mkdir.coreutils", O_RDONLY) = -1 ENOENT (No such file or directory)

After fix applied stap is able to find /bin/.debug/mkdir.coreutils under
sysroot:

[kamensky@coreos-lnx2 tests]$ /home/wd8/systemtap/20180208_2/packages/bin/stap --sysroot=/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs -a x86_64 -r 4.9.78-yocto-standard -B CROSS_COMPILE=x86_64-poky-linux- --sysenv=PATH=/usr/bin:/bin:/usr/sbin:/sbin --sysenv=LD_LIBRARY_PATH=/lib:/usr/lib -v -l 'process("/bin/mkdir.coreutils").function("*")' | wc
Pass 1: parsed user script and 480 library scripts using 230184virt/93440res/5368shr/88612data kb, in 330usr/40sys/371real ms.
Pass 2: analyzed script: 401 probes, 0 functions, 0 embeds, 0 globals using 235092virt/99484res/6256shr/93520data kb, in 30usr/0sys/35real ms.
    249     249   29671

Also verified on target that script like example below, once compiled
against sysroot on host and copied back on target can trace 'mkdir'
invocation functions:

[kamensky@coreos-lnx2 tests]$ cat mkdir1.stp
probe process("/bin/mkdir.coreutils").function("*").call {
  printf ("%s -> %s\n", thread_indent(1), ppfunc())
}
probe process("/bin/mkdir.coreutils").function("*").return {
  printf ("%s <- %s\n", thread_indent(-1), ppfunc())
}

Signed-off-by: Victor Kamensky <kamensky@cisco.com>
---
 setupdwfl.cxx | 12 ------------
 1 file changed, 12 deletions(-)

diff --git a/setupdwfl.cxx b/setupdwfl.cxx
index f00cf755b..e5bfa28f0 100644
--- a/setupdwfl.cxx
+++ b/setupdwfl.cxx
@@ -637,18 +637,6 @@ internal_find_debuginfo (Dwfl_Module *mod,
 
   call_dwfl_standard_find_debuginfo:
 
-  if (current_session_for_find_debuginfo)
-    {
-      string sysroot = current_session_for_find_debuginfo->sysroot + "/*";
-      int    found   = fnmatch(sysroot.c_str(), file_name, 0);
-
-      if (found)
-	{
-	  file_name = file_name
-	    + current_session_for_find_debuginfo->sysroot.length() - 1;
-	}
-    }
-
   /* Call the original dwfl_standard_find_debuginfo */
   return dwfl_standard_find_debuginfo(mod, userdata, modname, base,
               file_name, debuglink_file,
-- 
2.14.3

[PATCH 5/6] sysroot: handle symbolic links with absolute name relative to sysroot

[Victor Kamensky]

In case of symbolic link found under sysroot point to absolute path,
instead of trying to look for such absolute path in host system,
apply sysroot prefix first.

Here are steps how to reproduce the issue. Note <sysroot> is produced by
yocto poky build.

kamensky@coreos-lnx2 tests]$ ls -l /home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/mkdir
lrwxrwxrwx. 1 kamensky kamensky 20 Jan 30 18:29 /home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/mkdir -> /bin/mkdir.coreutils
[kamensky@coreos-lnx2 tests]$ ls -l /home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/mkdir.coreutils
-rwxr-xr-x. 1 kamensky kamensky 88232 Jan 29 09:58 /home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/mkdir.coreutils
[kamensky@coreos-lnx2 tests]$ ls -l /home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/.debug/mkdir.coreutils
-rwxr-xr-x. 1 kamensky kamensky 383456 Jan 29 09:58 /home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/.debug/mkdir.coreutils
[kamensky@coreos-lnx2 tests]$ cat mkdir4.stp
probe process("/bin/mkdir").function("*").call {
  printf ("%s -> %s\n", thread_indent(1), ppfunc())
}
probe process("/bin/mkdir").function("*").return {
  printf ("%s <- %s\n", thread_indent(-1), ppfunc())
}
[kamensky@coreos-lnx2 tests]$ /home/wd8/systemtap/20180208_2/packages/bin/stap --sysroot=/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs -a x86_64 -r 4.9.78-yocto-standard -B CROSS_COMPILE=x86_64-poky-linux- --sysenv=PATH=/usr/bin:/bin:/usr/sbin:/sbin --sysenv=LD_LIBRARY_PATH=/lib:/usr/lib -v -p4 -m mkdir4 mkdir4.stp
Pass 1: parsed user script and 480 library scripts using 230196virt/93484res/5396shr/88624data kb, in 340usr/30sys/370real ms.
semantic error: resolution failed in DWARF builder

semantic error: while resolving probe point: identifier 'process' at mkdir4.stp:1:7
        source: probe process("/bin/mkdir").function("*").call {
                      ^

semantic error: no match

semantic error: resolution failed in DWARF builder

semantic error: while resolving probe point: identifier 'process' at :4:7
        source: probe process("/bin/mkdir").function("*").return {
                      ^

semantic error: no match

Pass 2: analyzed script: 0 probes, 0 functions, 0 embeds, 0 globals using 233496virt/96980res/5468shr/91924data kb, in 20usr/0sys/25real ms.
Pass 2: analysis failed.  [man error::pass2]

After the fix above sript works fine verified that it traces mkdir
functions on target.

Signed-off-by: Victor Kamensky <kamensky@cisco.com>
---
 util.cxx | 76 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 76 insertions(+)

diff --git a/util.cxx b/util.cxx
index 2724be956..f48a8884f 100644
--- a/util.cxx
+++ b/util.cxx
@@ -443,6 +443,64 @@ split_lines(const char *buf, size_t n)
   return lines;
 }
 
+static string
+follow_link(const string& name, const string& sysroot)
+{
+  char *linkname;
+  ssize_t r;
+  string retpath;
+  struct stat st;
+
+  const char *f = name.c_str();
+
+  lstat(f, &st);
+
+  linkname = (char *) malloc(st.st_size + 1);
+
+  if (linkname)
+    {
+      r = readlink(f, linkname, st.st_size + 1);
+      linkname[st.st_size] = '\0';
+      /*
+       * If we have non-empty sysroot and we got link that
+       * points to absolute path name, we need to look at
+       * this path relative to sysroot itself. access and
+       * stat will follow symbolic links correctly only in
+       * case with empty sysroot.
+       */
+      while (r != -1 && linkname && linkname[0] == '/')
+	{
+	  string fname1 = sysroot + linkname;
+	  const char *f1 = fname1.c_str();
+	  if (access(f1, X_OK) == 0
+	      && stat(f1, &st) == 0
+	      && S_ISREG(st.st_mode))
+	    {
+	      retpath = fname1;
+	      break;
+	    }
+	  else if (lstat(f1, &st) == 0
+		   && S_ISLNK(st.st_mode))
+	    {
+	      free(linkname);
+	      linkname = (char *) malloc(st.st_size + 1);
+	      if (linkname)
+		{
+		  r = readlink(f1, linkname, st.st_size + 1);
+		  linkname[st.st_size] = '\0';
+		}
+	    }
+	  else
+	    {
+	      break;
+	    }
+	}
+    }
+  free(linkname);
+
+  return retpath;
+}
+
 // Resolve an executable name to a canonical full path name, with the
 // same policy as execvp().  A program name not containing a slash
 // will be searched along the $PATH.
@@ -467,6 +525,14 @@ string find_executable(const string& name, const string& sysroot,
   if (name.find('/') != string::npos) // slash in the path already?
     {
       retpath = sysroot + name;
+
+      const char *f = retpath.c_str();
+      if (sysroot != ""
+	  && lstat(f, &st) == 0
+	  && S_ISLNK(st.st_mode))
+	{
+	  retpath = follow_link(f, sysroot);
+	}
     }
   else // Nope, search $PATH.
     {
@@ -495,6 +561,16 @@ string find_executable(const string& name, const string& sysroot,
                   retpath = fname;
                   break;
                 }
+              else if (sysroot != ""
+                       && lstat(f, &st) == 0
+                       && S_ISLNK(st.st_mode))
+		{
+		  retpath = follow_link(f, sysroot);
+		  if (retpath != "")
+		    {
+		      break;
+		    }
+		}
             }
         }
     }
-- 
2.14.3

[PATCH 4/6] sysroot: delay adding sysroot path to module name in case of non absolute executable

[Victor Kamensky]

Current stap code adds sysroot prematurely for probes that specify
non absolute path name, i.e like "foo", so when find_executable called
it receives full path as <sysroot>/foo and find_executable does not
search PATH while applying sysroot.

Fix delays adding sysroot till path inside of sysroot is searched first.

Also fix missing sysroot addition in glob expansion case.

Note in case of sysroot cross compile environment it is highly recommended
to pass --sysenv=PATH=xxx:yyy and --sysenv=LD_LIBRARY_PATH=zzz to use
search path appropriate for target system, rather then host setting on
system where stap runs.

Here are steps how to reproduce the issue. Note <sysroot> is produced by
yocto poky build.

[kamensky@coreos-lnx2 tests]$ cat mkdir2.stp
probe process("mkdir.coreutils").function("*").call {
  printf ("%s -> %s\n", thread_indent(1), ppfunc())
}
probe process("mkdir.coreutils").function("*").return {
  printf ("%s <- %s\n", thread_indent(-1), ppfunc())
}
[kamensky@coreos-lnx2 tests]$ ls /home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/mkdir.coreutils
/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/mkdir.coreutils
[kamensky@coreos-lnx2 tests]$ ls /home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/.debug/mkdir.coreutils
/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/.debug/mkdir.coreutils
[kamensky@coreos-lnx2 tests]$ /home/wd8/systemtap/20180208_2/packages/bin/stap --sysroot=/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs -a x86_64 -r 4.9.78-yocto-standard -B CROSS_COMPILE=x86_64-poky-linux- --sysenv=PATH=/usr/bin:/bin:/usr/sbin:/sbin --sysenv=LD_LIBRARY_PATH=/lib:/usr/lib -v -p4 -m mkdir2 mkdir2.stp
Pass 1: parsed user script and 480 library scripts using 230196virt/93536res/5448shr/88624data kb, in 330usr/30sys/364real ms.
semantic error: resolution failed in DWARF builder

semantic error: while resolving probe point: identifier 'process' at mkdir2.stp:1:7
        source: probe process("mkdir.coreutils").function("*").call {
                      ^

semantic error: no match

semantic error: resolution failed in DWARF builder

semantic error: while resolving probe point: identifier 'process' at :4:7
        source: probe process("mkdir.coreutils").function("*").return {
                      ^

semantic error: no match

Pass 2: analyzed script: 0 probes, 0 functions, 0 embeds, 0 globals using 233496virt/97032res/5520shr/91924data kb, in 20usr/0sys/25real ms.
Pass 2: analysis failed.  [man error::pass2]

Under strace it shows that stap is not trying to lookup mkdir.coreutils
inside of sysroot accoring to PATH that was passed through --sysenv:

16048 openat(AT_FDCWD, "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/mkdir.coreutils", O_RDONLY) = -1 ENOENT (No such file or directory)

i.e it rather just tries <sysroot> + "mkdir.coreutils"

After the fix veried that that above example able to find mkdir.coreutils
executable in sysroot and corresponding symbols

Also verified that glob matching in sysroot works too for example for
this test case:

[kamensky@coreos-lnx2 tests]$ cat mkdir3.stp
probe process("/bin/mkdi*").function("*").call {
  printf ("%s -> %s\n", thread_indent(1), ppfunc())
}
probe process("/bin/mkdi*").function("*").return {
  printf ("%s <- %s\n", thread_indent(-1), ppfunc())
}

Signed-off-by: Victor Kamensky <kamensky@cisco.com>
---
 tapsets.cxx | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/tapsets.cxx b/tapsets.cxx
index 28296b14f..c664df0f9 100644
--- a/tapsets.cxx
+++ b/tapsets.cxx
@@ -746,7 +746,7 @@ base_query::base_query(dwflpp & dw, literal_map_t const & params):
               pid_val = 0;
               get_string_param(params, TOK_PROCESS, module_val);
             }
-          module_val = find_executable (module_val, "", sess.sysenv);
+          module_val = find_executable (module_val, sess.sysroot, sess.sysenv);
           if (!is_fully_resolved(module_val, "", sess.sysenv))
             throw SEMANTIC_ERROR(_F("cannot find executable '%s'",
                                     module_val.to_string().c_str()));
@@ -8293,7 +8293,6 @@ dwarf_builder::build(systemtap_session & sess,
             }
           else
             {
-              module_name = (string)sess.sysroot + (string)module_name;
               filled_parameters[TOK_PROCESS] = new literal_string(module_name);
             }
         }
@@ -8327,7 +8326,7 @@ dwarf_builder::build(systemtap_session & sess,
           assert (lit);
 
           // Evaluate glob here, and call derive_probes recursively with each match.
-          const auto& globs = glob_executable (module_name);
+          const auto& globs = glob_executable (sess.sysroot + string(module_name));
           unsigned results_pre = finished_results.size();
           for (auto it = globs.begin(); it != globs.end(); ++it)
             {
@@ -8418,7 +8417,8 @@ dwarf_builder::build(systemtap_session & sess,
 
       // PR13338: unquote glob results
       module_name = unescape_glob_chars (module_name);
-      user_path = find_executable (module_name, "", sess.sysenv); // canonicalize it
+      user_path = find_executable (module_name, sess.sysroot,
+				   sess.sysenv); // canonicalize it
       if (!is_fully_resolved(user_path, "", sess.sysenv))
         throw SEMANTIC_ERROR(_F("cannot find executable '%s'",
                                 user_path.to_string().c_str()));
-- 
2.14.3

[PATCH 2/6] sysroot: fix short release -r option handling in case of sysroot pass proper kernel modules

[Victor Kamensky]

In case of -r option getting short release (opposite to full path of
kernel build tree) stap passes short release name to
dwfl_linux_kernel_report_offline function and this function tries to
look up symbols in the host file system ignoring sysroot setting.

Fix: in case of non empty, non "/" sysroot construct full path to
kernel modules directory and store it into elfutils_kernel_path variable.

Here are steps how to reproduce the issue. Note <sysroot> is produced by
yocto poky build.

[kamensky@coreos-lnx2 tests]$ cat meminfo.stp
probe kernel.function("meminfo_proc_show") {
        println("meminfo_proc_show called")
}
[kamensky@coreos-lnx2 tests]$ /home/wd8/systemtap/20180208_2/packages/bin/stap --sysroot=/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs -a x86_64 -r 4.9.78-yocto-standard -B CROSS_COMPILE=x86_64-poky-linux- --sysenv=PATH=/usr/bin:/bin:/usr/sbin:/sbin --sysenv=LD_LIBRARY_PATH=/lib:/usr/lib -p4 -v -m meminfo meminfo.stp
Pass 1: parsed user script and 480 library scripts using 230196virt/93452res/5364shr/88624data kb, in 340usr/30sys/375real ms.
semantic error: while resolving probe point: identifier 'kernel' at meminfo.stp:1:7
        source: probe kernel.function("meminfo_proc_show") {
                      ^

Running command under strace shows that stap is trying to open linux symbol file
on host file system, instead of looking at sysroot location:

19517 openat(AT_FDCWD, "/boot/vmlinux-4.9.78-yocto-standard.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
19517 openat(AT_FDCWD, "/boot/vmlinux-4.9.78-yocto-standard", O_RDONLY) = -1 ENOENT (No such file or directory)

Signed-off-by: Victor Kamensky <kamensky@cisco.com>
---
 setupdwfl.cxx | 16 +++++++++++++---
 1 file changed, 13 insertions(+), 3 deletions(-)

diff --git a/setupdwfl.cxx b/setupdwfl.cxx
index 11e0bb2af..f00cf755b 100644
--- a/setupdwfl.cxx
+++ b/setupdwfl.cxx
@@ -359,9 +359,19 @@ setup_dwfl_kernel (unsigned *modules_found, systemtap_session &s)
   // passs the plain kernel_release here.  So instead we have to
   // hard-code this magic here.
    string lib_path = "/lib/modules/" + s.kernel_release + "/build";
-   if (s.kernel_build_tree == string(s.sysroot + lib_path) ||
-       (s.kernel_build_tree == lib_path
-	&& s.sysroot == "/"))
+   if (s.kernel_build_tree == string(s.sysroot + lib_path))
+     {
+       // If we have sysroot set does not make sense to pass
+       // short release to dwfl, it won't take a sysroot into
+       // account. Let's construct full path in such case.
+       if (s.sysroot != "" && s.sysroot != "/")
+	 elfutils_kernel_path = string(s.sysroot + "/lib/modules/" + s.kernel_release);
+       else
+	 elfutils_kernel_path = s.kernel_release;
+     }
+   else
+     if (s.kernel_build_tree == lib_path
+	 && s.sysroot == "/")
       elfutils_kernel_path = s.kernel_release;
    else
       elfutils_kernel_path = s.kernel_build_tree;
-- 
2.14.3

[PATCH 0/6] systemtap set of --sysroot option handling fixes

[Victor Kamensky]

Hi Folks,

Here is set of fixes for SystemTap --sysroot option handling.

Please look at individual patches decsription for issues observed and fixed.

If commit messages are too details and big for SystemTap project
guidelines feel free to trim them. Commit message structured in such
way that text after "Here are steps how to reproduce the issue." could
be removed if needed.

I did not run full SystemTap regression with --sysroot option since SystemTap
DejaGnu testsuites are not friendly to remote execution mode. I did run
regular 'make installcheck' on the same baseline with and without changes
and made sure that results are identical.

If you would like to try stap in cross compiled environment with --sysroot
option I've set up openembedded-core repository where these changes are
integrated and since openembedded-core support virtual qemu based target
for different CPU types stap with --sysroot option could be tested.

Setup instructions are here:
    https://github.com/victorkamensky/systemtap-oe-sysroot-manifest

Victor Kamensky (6):
  sysroot: fix short release -r option handling in case of sysroot read
    System.map symbols
  sysroot: fix short release -r option handling in case of sysroot pass
    proper kernel modules
  sysroot: debuginfo lookup with sysroot case do not remove sysroot from
    file_name
  sysroot: delay adding sysroot path to module name in case of non
    absolute executable
  sysroot: handle symbolic links with absolute name relative to sysroot
  sysroot: _stp_umodule_relocate needs receive target file path

 loc2stap.cxx  |  3 ++-
 session.cxx   |  2 +-
 setupdwfl.cxx | 28 ++++++++++------------
 tapsets.cxx   |  8 +++----
 util.cxx      | 76 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 96 insertions(+), 21 deletions(-)

-- 
2.14.3

[PATCH 1/6] sysroot: fix short release -r option handling in case of sysroot read System.map symbols

[Victor Kamensky]

Add sysroot to system_map_path + "/boot/System.map" case. Otherwise
stap tries to look symbols on host system and it produce error like this:

> Kernel symbol table /boot/System.map-4.9.78-yocto-standard unavailable, (No such file or directory)

Here are steps to reproduce the issue. Note <sysroot> is produced by yocto
poky build. Under <sysroot>/lib/modules/4.9.78-yocto-standard/build there
is no System.map file; instead it is installed under <sysroot>/boot directory.

[kamensky@coreos-lnx2 tests]$ cat meminfo.stp
probe kernel.function("meminfo_proc_show") {
        println("meminfo_proc_show called")
}
[kamensky@coreos-lnx2 tests]$ export SYSTEMTAP_DEBUGINFO_PATH=+:.debug:build
[kamensky@coreos-lnx2 tests]$ /home/wd8/systemtap/20180208_2/packages/bin/stap --sysroot=/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs -a x86_64 -r 4.9.78-yocto-standard -B CROSS_COMPILE=x86_64-poky-linux- --sysenv=PATH=/usr/bin:/bin:/usr/sbin:/sbin --sysenv=LD_LIBRARY_PATH=/lib:/usr/lib -p4 -v -v -m meminfo meminfo.stp
Systemtap translator/driver (version 3.3/0.170, commit release-3.2-92-g26c5e3c03c94 + changes)
Copyright (C) 2005-2017 Red Hat, Inc. and others
This is free software; see the source for copying conditions.
tested kernel versions: 2.6.18 ... 4.14-rc4
enabled features: BPF PYTHON2 PYTHON3 LIBXML2 NLS
Created temporary directory "/tmp/staprq4IGE"
Session arch: x86_64 release: 4.9.78-yocto-standard
Kernel symbol table /home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs//lib/modules/4.9.78-yocto-standard/build/System.map unavailable, (No such file or directory)
Kernel symbol table /boot/System.map-4.9.78-yocto-standard unavailable, (No such file or directory)
WARNING: Kernel function symbol table missing [man warning::symbols]
<snip>
semantic error: while resolving probe point: identifier 'kernel' at meminfo.stp:1:7
   thrown from: ../systemtap/elaborate.cxx:1080
        source: probe kernel.function("meminfo_proc_show") {
                      ^

semantic error: missing x86_64 kernel/module debuginfo [man warning::debuginfo] under '/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs//lib/modules/4.9.78-yocto-standard/build'
   thrown from: ../systemtap/dwflpp.cxx:340

Signed-off-by: Victor Kamensky <kamensky@cisco.com>
---
 session.cxx | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/session.cxx b/session.cxx
index 5800f28fd..ebec94edb 100644
--- a/session.cxx
+++ b/session.cxx
@@ -2038,7 +2038,7 @@ systemtap_session::parse_kernel_functions ()
 	clog << _F("Kernel symbol table %s unavailable, (%s)",
 		   system_map_path.c_str(), strerror(errno)) << endl;
 
-      system_map_path = "/boot/System.map-" + kernel_release;
+      system_map_path = sysroot + "/boot/System.map-" + kernel_release;
       system_map.clear();
       system_map.open(system_map_path.c_str(), ifstream::in);
       if (! system_map.is_open())
-- 
2.14.3

Re: [PATCH 1/6] sysroot: fix short release -r option handling in case of sysroot read System.map symbols

[David Smith]

Thanks! Checked in as commit 436063d5e (although now I realize I
forgot to use --author so you didn't get credit for this one - sorry).

On Mon, Mar 5, 2018 at 11:37 AM, Victor Kamensky <kamensky@cisco.com> wrote:
> Add sysroot to system_map_path + "/boot/System.map" case. Otherwise
> stap tries to look symbols on host system and it produce error like this:
>
>> Kernel symbol table /boot/System.map-4.9.78-yocto-standard unavailable, (No such file or directory)
>
> Here are steps to reproduce the issue. Note <sysroot> is produced by yocto
> poky build. Under <sysroot>/lib/modules/4.9.78-yocto-standard/build there
> is no System.map file; instead it is installed under <sysroot>/boot directory.
>
> [kamensky@coreos-lnx2 tests]$ cat meminfo.stp
> probe kernel.function("meminfo_proc_show") {
>         println("meminfo_proc_show called")
> }
> [kamensky@coreos-lnx2 tests]$ export SYSTEMTAP_DEBUGINFO_PATH=+:.debug:build
> [kamensky@coreos-lnx2 tests]$ /home/wd8/systemtap/20180208_2/packages/bin/stap --sysroot=/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs -a x86_64 -r 4.9.78-yocto-standard -B CROSS_COMPILE=x86_64-poky-linux- --sysenv=PATH=/usr/bin:/bin:/usr/sbin:/sbin --sysenv=LD_LIBRARY_PATH=/lib:/usr/lib -p4 -v -v -m meminfo meminfo.stp
> Systemtap translator/driver (version 3.3/0.170, commit release-3.2-92-g26c5e3c03c94 + changes)
> Copyright (C) 2005-2017 Red Hat, Inc. and others
> This is free software; see the source for copying conditions.
> tested kernel versions: 2.6.18 ... 4.14-rc4
> enabled features: BPF PYTHON2 PYTHON3 LIBXML2 NLS
> Created temporary directory "/tmp/staprq4IGE"
> Session arch: x86_64 release: 4.9.78-yocto-standard
> Kernel symbol table /home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs//lib/modules/4.9.78-yocto-standard/build/System.map unavailable, (No such file or directory)
> Kernel symbol table /boot/System.map-4.9.78-yocto-standard unavailable, (No such file or directory)
> WARNING: Kernel function symbol table missing [man warning::symbols]
> <snip>
> semantic error: while resolving probe point: identifier 'kernel' at meminfo.stp:1:7
>    thrown from: ../systemtap/elaborate.cxx:1080
>         source: probe kernel.function("meminfo_proc_show") {
>                       ^
>
> semantic error: missing x86_64 kernel/module debuginfo [man warning::debuginfo] under '/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs//lib/modules/4.9.78-yocto-standard/build'
>    thrown from: ../systemtap/dwflpp.cxx:340
>
> Signed-off-by: Victor Kamensky <kamensky@cisco.com>
> ---
>  session.cxx | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/session.cxx b/session.cxx
> index 5800f28fd..ebec94edb 100644
> --- a/session.cxx
> +++ b/session.cxx
> @@ -2038,7 +2038,7 @@ systemtap_session::parse_kernel_functions ()
>         clog << _F("Kernel symbol table %s unavailable, (%s)",
>                    system_map_path.c_str(), strerror(errno)) << endl;
>
> -      system_map_path = "/boot/System.map-" + kernel_release;
> +      system_map_path = sysroot + "/boot/System.map-" + kernel_release;
>        system_map.clear();
>        system_map.open(system_map_path.c_str(), ifstream::in);
>        if (! system_map.is_open())
> --
> 2.14.3
>



-- 
David Smith
Associate Manager
Red Hat

Re: [PATCH 3/6] sysroot: debuginfo lookup with sysroot case do not remove sysroot from file_name

[David Smith]

Hmm, the problem I have here is the code your patch wants to delete
was added to fix a sysroot problem in the following commit:

====
commit df932a3175284fa75a4c6bdb3aa7af1a34c15cd6
Author: Torsten Polle <Torsten.Polle@gmx.de>
Date:   Thu Mar 6 21:38:49 2014 +0100

    Fix: Debug links are not found if the sysroot is used.

    Signed-off-by: Torsten Polle <Torsten.Polle@gmx.de>
====

Torsten, have you tried using --sysroot recently?

On Mon, Mar 5, 2018 at 11:37 AM, Victor Kamensky <kamensky@cisco.com> wrote:
> If sysroot option is passed, and debug symbols reside in sysroot along
> with executable <foo> in <foo_dir>/.debug/<foo_file> directory, stap
> fails to find debuginfo because it strips out sysroot path from file_name
> so dwfl_standard_find_debuginfo ends up looking at host
> <foo_dir>/.debug/<foo_file> rather then checking
> <sysroot>/<foo_dir>/.debug/<foo_file>.
>
> Note in cross compile environment, it is good idea to set and export
> proper SYSTEMTAP_DEBUGINFO_PATH variable because usual built defaults that
> work for native distros very often not applicable to cross sysroot
> based environment. For example for yocto poky/OE build the following
> setting seems proper "+:.debug:build".
>
> Here are steps how to reproduce the issue. Note <sysroot> is produced by
> yocto poky build.
>
> [kamensky@coreos-lnx2 tests]$ ls /home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/mkdir.coreutils
> /home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/mkdir.coreutils
> [kamensky@coreos-lnx2 tests]$ ls /home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/.debug/mkdir.coreutils
> /home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/.debug/mkdir.coreutils
> [kamensky@coreos-lnx2 tests]$ /home/wd8/systemtap/20180208_2/packages/bin/stap --sysroot=/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs -a x86_64 -r 4.9.78-yocto-standard -B CROSS_COMPILE=x86_64-poky-linux- --sysenv=PATH=/usr/bin:/bin:/usr/sbin:/sbin --sysenv=LD_LIBRARY_PATH=/lib:/usr/lib -v -l 'process("/bin/mkdir.coreutils").function("*")'
> Pass 1: parsed user script and 480 library scripts using 230184virt/93416res/5344shr/88612data kb, in 350usr/30sys/378real ms.
> process("/bin/mkdir.coreutils").function("_fini")
> process("/bin/mkdir.coreutils").function("_init")
> process("/bin/mkdir.coreutils").function("fts_children")
> process("/bin/mkdir.coreutils").function("fts_close")
> process("/bin/mkdir.coreutils").function("fts_open")
> process("/bin/mkdir.coreutils").function("fts_read")
> process("/bin/mkdir.coreutils").function("fts_set")
> Pass 2: analyzed script: 7 probes, 0 functions, 0 embeds, 0 globals using 233484virt/97928res/6212shr/91912data kb, in 20usr/0sys/26real ms.
>
> running above example under strace shows that stap tries to access
> /bin/.debug/mkdir.coreutils on host, rather then sysroot
>
> 13064 openat(AT_FDCWD, "/bin/mkdir.coreutils", O_RDONLY) = -1 ENOENT (No such file or directory)
> 13064 openat(AT_FDCWD, "/bin/.debug/mkdir.coreutils", O_RDONLY) = -1 ENOENT (No such file or directory)
> 13064 openat(AT_FDCWD, "/bin/mkdir.coreutils", O_RDONLY) = -1 ENOENT (No such file or directory)
> 13064 openat(AT_FDCWD, "/bin/build/mkdir.coreutils", O_RDONLY) = -1 ENOENT (No such file or directory)
> 13064 openat(AT_FDCWD, "/bin/mkdir.coreutils", O_RDONLY) = -1 ENOENT (No such file or directory)
>
> After fix applied stap is able to find /bin/.debug/mkdir.coreutils under
> sysroot:
>
> [kamensky@coreos-lnx2 tests]$ /home/wd8/systemtap/20180208_2/packages/bin/stap --sysroot=/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs -a x86_64 -r 4.9.78-yocto-standard -B CROSS_COMPILE=x86_64-poky-linux- --sysenv=PATH=/usr/bin:/bin:/usr/sbin:/sbin --sysenv=LD_LIBRARY_PATH=/lib:/usr/lib -v -l 'process("/bin/mkdir.coreutils").function("*")' | wc
> Pass 1: parsed user script and 480 library scripts using 230184virt/93440res/5368shr/88612data kb, in 330usr/40sys/371real ms.
> Pass 2: analyzed script: 401 probes, 0 functions, 0 embeds, 0 globals using 235092virt/99484res/6256shr/93520data kb, in 30usr/0sys/35real ms.
>     249     249   29671
>
> Also verified on target that script like example below, once compiled
> against sysroot on host and copied back on target can trace 'mkdir'
> invocation functions:
>
> [kamensky@coreos-lnx2 tests]$ cat mkdir1.stp
> probe process("/bin/mkdir.coreutils").function("*").call {
>   printf ("%s -> %s\n", thread_indent(1), ppfunc())
> }
> probe process("/bin/mkdir.coreutils").function("*").return {
>   printf ("%s <- %s\n", thread_indent(-1), ppfunc())
> }
>
> Signed-off-by: Victor Kamensky <kamensky@cisco.com>
> ---
>  setupdwfl.cxx | 12 ------------
>  1 file changed, 12 deletions(-)
>
> diff --git a/setupdwfl.cxx b/setupdwfl.cxx
> index f00cf755b..e5bfa28f0 100644
> --- a/setupdwfl.cxx
> +++ b/setupdwfl.cxx
> @@ -637,18 +637,6 @@ internal_find_debuginfo (Dwfl_Module *mod,
>
>    call_dwfl_standard_find_debuginfo:
>
> -  if (current_session_for_find_debuginfo)
> -    {
> -      string sysroot = current_session_for_find_debuginfo->sysroot + "/*";
> -      int    found   = fnmatch(sysroot.c_str(), file_name, 0);
> -
> -      if (found)
> -       {
> -         file_name = file_name
> -           + current_session_for_find_debuginfo->sysroot.length() - 1;
> -       }
> -    }
> -
>    /* Call the original dwfl_standard_find_debuginfo */
>    return dwfl_standard_find_debuginfo(mod, userdata, modname, base,
>                file_name, debuglink_file,
> --
> 2.14.3
>



-- 
David Smith
Associate Manager
Red Hat

Re: [PATCH 3/6] sysroot: debuginfo lookup with sysroot case do not remove sysroot from file_name

[Victor Kamensky]

Hi David,

On Thu, 8 Mar 2018, David Smith wrote:

> Hmm, the problem I have here is the code your patch wants to delete
> was added to fix a sysroot problem in the following commit:
>
> ====
> commit df932a3175284fa75a4c6bdb3aa7af1a34c15cd6
> Author: Torsten Polle <Torsten.Polle@gmx.de>
> Date:   Thu Mar 6 21:38:49 2014 +0100
>
>    Fix: Debug links are not found if the sysroot is used.
>
>    Signed-off-by: Torsten Polle <Torsten.Polle@gmx.de>
> ====

Thank you for bring my attention to it. Unfortunately above
commit message is too terse, and it is not clear what was the
use case that did not work.

But I looked closely into it again, and I think I know what
is happening:

It boils down to three different types of entries that
SYSTEMTAP_DEBUGINFO_PATH environment (or its default) may have. 
SYSTEMTAP_DEBUGINFO_PATH value is passed through debuginfo_usr_path
to mod->dwfl->callbacks.debuginfo_path and that is passed
to dwfl_standard_find_debuginfo function of elfutils.

Note fila_name passed to function may have sysroot stripped
from it (current code), or may include sysroot as it was
before df932a3175284 and after my fix.

debuginfo_path may contain the following entries, each
may preceeded by optional "+" or "-"

1) empty entry, in this case fila_name passed to the function
is checked directly. If file_name does not have sysroot in
it dwfl_standard_find_debuginfo will try to look for file
in current root. Which is clearly very wrong thing to do in
case of sysroot build, since it may pick up file with the
same name in host. Current code with default debuginfo_path
has this issue

2) Local path like ".debug". In this case function will look
for `dirname file_name`/.debug/`basename file_name'. Again
if file_name does not have sysroot in it, host files system
will be looked and it is wrong. That is actually my case,
in for /bin/mkdir.coreutils in my sysroot there is
/bin/.debug/mkdir.coreutils with symbols. So if file_name
does have sysroot in it, correct symbols file is picked and
that is what I made this fix.

3) Absolute path like '/<sysroot>/usr/lib/debug'. In this
case function will look at
'/<sysroot>/usr/lib/debug'/file_name or
'/<sysroot>/usr/lib/debug'/`basename file_name`. And in
this case if file_name contains sysroot it will not work.
I believe it is what df932a317528 has fixed and that is
what my fix would break again. But note df932a317528 broke
lookup for case 2).

I am not sure how to fix it to work in all cases. I will
think about it tomorrow. Any suggestions are welcome.

For now it is clear in sysroot case, passing empty entry
is wrong. Instead sysroot itself should be passed.

Local path and absolute path debuginfo_path contradict
each other wrt expectation whether file_name contains
sysroot or not. Maybe debuginfo_path in case of
non-empty sysroot should be split into two diferent
sets: one with absolute entries and other with local entries
and dwfl_standard_find_debuginfo should be called twice
with different variants of file_name.

If interseted please find below my gdb session that
illustrates behavior of dwfl_standard_find_debuginfo described
above. Gdb session has run against current code.

Thanks,
Victor

In this example /bin/hello executable compiled in such way that it's
.gnu_debuglink has value hello.debug, so it could be distingushable
from executable itself.

(gdb) show args
Argument list to give program being debugged when it is started is "--sysroot=/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs -a x86_64 -r /home/wd8/yocto/20180128/build_x86_64/tmp/work/corei7-64-intel-common-poky-linux/linux-yocto/4.9.78+gitAUTOINC+ef2f5d9a0a_f7a6d45fff-r0/linux-corei7-64-intel-common-standard-build -B CROSS_COMPILE=/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/usr/bin/x86_64-poky-linux- --sysenv=PATH=/usr/bin:/bin:/usr/sbin:/sbin --sysenv=LD_LIBRARY_PATH=/lib:/usr/lib -v -l 'process("/bin/hello").function("*")'".
...

(gdb) bt
#0  find_debuginfo_in_path (mod=mod@entry=0x62d6950, file_name=file_name@entry=0x62d6bf2 "/bin/hello", debuglink_file=debuglink_file@entry=0x7ffff7fed044 "hello.debug",
     debuglink_crc=debuglink_crc@entry=611974856, debuginfo_file_name=debuginfo_file_name@entry=0x62d69a8) at find-debuginfo.c:165
#1  0x00007ffff7bb2855 in dwfl_standard_find_debuginfo (mod=0x62d6950, userdata=<optimized out>, modname=<optimized out>, base=<optimized out>, file_name=0x62d6bf2 "/bin/hello",
     debuglink_file=0x7ffff7fed044 "hello.debug", debuglink_crc=611974856, debuginfo_file_name=0x62d69a8) at find-debuginfo.c:383
#2  0x0000000000657719 in internal_find_debuginfo (mod=0x62d6950, userdata=0x62d6960,
     modname=0x62d6af0 "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/hello", base=65536,
     file_name=0x62d6bf2 "/bin/hello", debuglink_file=0x7ffff7fed044 "hello.debug", debuglink_crc=611974856, debuginfo_file_name=0x62d69a8) at ../systemtap/setupdwfl.cxx:645
#3  0x00007ffff7bb0153 in find_debuginfo (mod=0x62d6950) at dwfl_module_getdwarf.c:539
#4  0x00007ffff7bb1393 in find_dw (mod=0x62d6950) at dwfl_module_getdwarf.c:1400
#5  dwfl_module_getdwarf (mod=0x62d6950, bias=0x7fffffff79d8) at dwfl_module_getdwarf.c:1434
#6  0x0000000000530b2c in validate_module_elf (mod=0x62d6950,
     name=0x62d6af0 "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/hello", q=0x7fffffff8110)
     at ../systemtap/tapsets.cxx:2471
#7  0x000000000053169e in query_module (mod=0x62d6950,
     name=0x62d6af0 "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/hello", addr=65536, q=0x7fffffff8110)
     at ../systemtap/tapsets.cxx:2605
#8  0x00007ffff7baed71 in dwfl_getmodules (dwfl=0x62d5020, callback=0x53137c <query_module(Dwfl_Module*, void**, char const*, Dwarf_Addr, base_query*)>, arg=0x7fffffff8110, offset=0)
     at dwfl_getmodules.c:86
#9  0x0000000000623215 in dwflpp::iterate_over_modules<void> (this=0x62d4b80, callback=0x53137c <query_module(Dwfl_Module*, void**, char const*, Dwarf_Addr, base_query*)>,
     data=0x7fffffff8110) at ../systemtap/dwflpp.cxx:409
#10 0x000000000058b7a3 in dwflpp::iterate_over_modules<base_query> (this=0x62d4b80, callback=0x53137c <query_module(Dwfl_Module*, void**, char const*, Dwarf_Addr, base_query*)>,
     data=0x7fffffff8110) at ../systemtap/dwflpp.h:238
#11 0x00000000005664f7 in dwarf_builder::build (this=0x62c7960, sess=..., base=0x4b538d0, location=0x1028eb0, parameters=std::map with 2 elements = {...},
     finished_results=std::vector of length 0, capacity 0) at ../systemtap/tapsets.cxx:8616
#12 0x000000000049dcbf in match_node::find_and_build (this=0x62cc660, s=..., p=0x4b538d0, loc=0x1028eb0, pos=2, results=std::vector of length 0, capacity 0, builders=std::set with 0 elements)
     at ../systemtap/elaborate.cxx:474
#13 0x000000000049ee12 in match_node::find_and_build (this=0x4b4d920, s=..., p=0x4b538d0, loc=0x1028eb0, pos=1, results=std::vector of length 0, capacity 0, builders=std::set with 0 elements)
     at ../systemtap/elaborate.cxx:648
#14 0x000000000049ee12 in match_node::find_and_build (this=0xa1b2d0, s=..., p=0x4b538d0, loc=0x1028eb0, pos=0, results=std::vector of length 0, capacity 0, builders=std::set with 0 elements)
     at ../systemtap/elaborate.cxx:648
#15 0x00000000004a0ca5 in derive_probes (s=..., p=0x4b538d0, dps=std::vector of length 0, capacity 0, optional=false, rethrow_errors=false) at ../systemtap/elaborate.cxx:1022
#16 0x00000000004a4332 in semantic_pass_symbols (s=...) at ../systemtap/elaborate.cxx:1890
#17 0x00000000004a88c5 in semantic_pass (s=...) at ../systemtap/elaborate.cxx:2441
#18 0x0000000000413a48 in passes_0_4 (s=...) at ../systemtap/main.cxx:914
#19 0x00000000004161a8 in main (argc=13, argv=0x7fffffffdd68) at ../systemtap/main.cxx:1381
(gdb) p *mod
$43 = {dwfl = 0x62d5020, next = 0x0, userdata = 0x0,
   name = 0x62d6af0 "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/hello", low_addr = 65536, high_addr = 2166840,
   main = {name = 0x62d6b80 "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/hello", fd = -1, valid = false,
     relocated = false, elf = 0x62d50b0, vaddr = 0, address_sync = 2304}, debug = {name = 0x0, fd = 0, valid = false, relocated = false, elf = 0x0, vaddr = 0, address_sync = 0}, aux_sym = {
     name = 0x0, fd = 0, valid = false, relocated = false, elf = 0x0, vaddr = 0, address_sync = 0}, main_bias = 65536, ebl = 0x0, e_type = 3, elferr = DWFL_E_NOERROR, reloc_info = 0x0,
   symfile = 0x0, symdata = 0x0, aux_symdata = 0x0, syments = 0, aux_syments = 0, first_global = 0, aux_first_global = 0, symstrdata = 0x0, aux_symstrdata = 0x0, symxndxdata = 0x0,
   aux_symxndxdata = 0x0, dw = 0x0, alt = 0x0, alt_fd = 0, alt_elf = 0x0, symerr = DWFL_E_NOERROR, dwerr = DWFL_E_NO_DWARF, first_cu = 0x0, cu = 0x0, lazy_cu_root = 0x0, aranges = 0x0,
   build_id_bits = 0x62d4b60, build_id_vaddr = 66180, build_id_len = 20, ncu = 0, lazycu = 0, naranges = 0, dwarf_cfi = 0x0, eh_cfi = 0x0, segment = 0, gc = false, is_executable = false}
(gdb) p *mod->dwfl->callbacks
$44 = {find_elf = 0x0, find_debuginfo = 0x657103 <internal_find_debuginfo(Dwfl_Module*, void**, char const*, unsigned long, char const*, char const*, unsigned int, char**)>,
   section_address = 0x0, debuginfo_path = 0xa07db8 <debuginfo_usr_path>}
(gdb) p debuginfo_usr_path
$45 = 0xa28bc0 "+:.debug:/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/usr/lib/debug:/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/var/cache/abrt-di/usr/lib/debug"
(gdb) c

Breakpoint 5, try_open (dir=dir@entry=0x62d8c50 "/bin", subdir=subdir@entry=0x0, debuglink=debuglink@entry=0x7ffff7fed044 "hello.debug",
     debuginfo_file_name=debuginfo_file_name@entry=0x7fffffff7340, main_stat=<optimized out>, main_stat=<optimized out>) at find-debuginfo.c:44
44	try_open (const struct stat *main_stat,
(gdb) c
Continuing.

Breakpoint 6, try_open (dir=dir@entry=0x62d8c50 "/bin", subdir=subdir@entry=0x0, debuglink=debuglink@entry=0x7ffff7fed044 "hello.debug",
     debuginfo_file_name=debuginfo_file_name@entry=0x7fffffff7340, main_stat=<optimized out>, main_stat=<optimized out>) at find-debuginfo.c:61
61	  int fd = TEMP_FAILURE_RETRY (open (fname, O_RDONLY));
(gdb) p fname
$46 = 0x62d9320 "/bin/hello.debug"
(gdb) c
Continuing.

Breakpoint 5, try_open (dir=dir@entry=0x62d8c50 "/bin", subdir=subdir@entry=0xa28e42 ".debug", debuglink=debuglink@entry=0x7ffff7fed044 "hello.debug",
     debuginfo_file_name=debuginfo_file_name@entry=0x7fffffff7340, main_stat=<optimized out>, main_stat=<optimized out>) at find-debuginfo.c:44
44	try_open (const struct stat *main_stat,
(gdb) c
Continuing.

Breakpoint 6, try_open (dir=dir@entry=0x62d8c50 "/bin", subdir=subdir@entry=0xa28e42 ".debug", debuglink=debuglink@entry=0x7ffff7fed044 "hello.debug",
     debuginfo_file_name=debuginfo_file_name@entry=0x7fffffff7340, main_stat=<optimized out>, main_stat=<optimized out>) at find-debuginfo.c:61
61	  int fd = TEMP_FAILURE_RETRY (open (fname, O_RDONLY));
(gdb) p fname
$47 = 0x62d9320 "/bin/.debug/hello.debug"
(gdb) c
Continuing.

Breakpoint 5, try_open (dir=dir@entry=0x62d8c50 "/bin", subdir=subdir@entry=0x0, debuglink=debuglink@entry=0x7ffff7fed044 "hello.debug",
     debuginfo_file_name=debuginfo_file_name@entry=0x7fffffff7340, main_stat=<optimized out>, main_stat=<optimized out>) at find-debuginfo.c:44
44	try_open (const struct stat *main_stat,
(gdb) c
Continuing.

Breakpoint 6, try_open (dir=dir@entry=0x62d8c50 "/bin", subdir=subdir@entry=0x0, debuglink=debuglink@entry=0x7ffff7fed044 "hello.debug",
     debuginfo_file_name=debuginfo_file_name@entry=0x7fffffff7340, main_stat=<optimized out>, main_stat=<optimized out>) at find-debuginfo.c:61
61	  int fd = TEMP_FAILURE_RETRY (open (fname, O_RDONLY));
(gdb) p fname
$48 = 0x62d9320 "/bin/hello.debug"
(gdb) c
Continuing.

Breakpoint 5, try_open (dir=dir@entry=0xa28e49 "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/usr/lib/debug",
     subdir=subdir@entry=0x62d8c51 "bin", debuglink=debuglink@entry=0x7ffff7fed044 "hello.debug", debuginfo_file_name=debuginfo_file_name@entry=0x7fffffff7340, main_stat=<optimized out>,
     main_stat=<optimized out>) at find-debuginfo.c:44
44	try_open (const struct stat *main_stat,
(gdb) c
Continuing.

Breakpoint 6, try_open (dir=dir@entry=0xa28e49 "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/usr/lib/debug",
     subdir=subdir@entry=0x62d8c51 "bin", debuglink=debuglink@entry=0x7ffff7fed044 "hello.debug", debuginfo_file_name=debuginfo_file_name@entry=0x7fffffff7340, main_stat=<optimized out>,
     main_stat=<optimized out>) at find-debuginfo.c:61
61	  int fd = TEMP_FAILURE_RETRY (open (fname, O_RDONLY));
(gdb) p fname
$49 = 0x62d9fd0 "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/usr/lib/debug/bin/hello.debug"
(gdb) c
Continuing.

Breakpoint 5, try_open (dir=dir@entry=0xa28e49 "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/usr/lib/debug",
     subdir=subdir@entry=0x0, debuglink=debuglink@entry=0x7ffff7fed044 "hello.debug", debuginfo_file_name=debuginfo_file_name@entry=0x7fffffff7340, main_stat=<optimized out>,
     main_stat=<optimized out>) at find-debuginfo.c:44
44	try_open (const struct stat *main_stat,
(gdb) c
Continuing.

Breakpoint 6, try_open (dir=dir@entry=0xa28e49 "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/usr/lib/debug",
     subdir=subdir@entry=0x0, debuglink=debuglink@entry=0x7ffff7fed044 "hello.debug", debuginfo_file_name=debuginfo_file_name@entry=0x7fffffff7340, main_stat=<optimized out>,
     main_stat=<optimized out>) at find-debuginfo.c:61
61	  int fd = TEMP_FAILURE_RETRY (open (fname, O_RDONLY));
(gdb) p fname
$50 = 0x62d9fd0 "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/usr/lib/debug/hello.debug"
(gdb) c
Continuing.

Breakpoint 5, try_open (
     dir=dir@entry=0xa28eca "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/var/cache/abrt-di/usr/lib/debug",
     subdir=subdir@entry=0x62d8c51 "bin", debuglink=debuglink@entry=0x7ffff7fed044 "hello.debug", debuginfo_file_name=debuginfo_file_name@entry=0x7fffffff7340, main_stat=<optimized out>,
     main_stat=<optimized out>) at find-debuginfo.c:44
44	try_open (const struct stat *main_stat,
(gdb) c
Continuing.

Breakpoint 6, try_open (
     dir=dir@entry=0xa28eca "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/var/cache/abrt-di/usr/lib/debug",
     subdir=subdir@entry=0x62d8c51 "bin", debuglink=debuglink@entry=0x7ffff7fed044 "hello.debug", debuginfo_file_name=debuginfo_file_name@entry=0x7fffffff7340, main_stat=<optimized out>,
     main_stat=<optimized out>) at find-debuginfo.c:61
61	  int fd = TEMP_FAILURE_RETRY (open (fname, O_RDONLY));
(gdb) p fname
$51 = 0x62d9e90 "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/var/cache/abrt-di/usr/lib/debug/bin/hello.debug"
(gdb) c
Continuing.

Breakpoint 5, try_open (
     dir=dir@entry=0xa28eca "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/var/cache/abrt-di/usr/lib/debug",
     subdir=subdir@entry=0x0, debuglink=debuglink@entry=0x7ffff7fed044 "hello.debug", debuginfo_file_name=debuginfo_file_name@entry=0x7fffffff7340, main_stat=<optimized out>,
     main_stat=<optimized out>) at find-debuginfo.c:44
44	try_open (const struct stat *main_stat,
(gdb) c
Continuing.

Breakpoint 6, try_open (
     dir=dir@entry=0xa28eca "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/var/cache/abrt-di/usr/lib/debug",
     subdir=subdir@entry=0x0, debuglink=debuglink@entry=0x7ffff7fed044 "hello.debug", debuginfo_file_name=debuginfo_file_name@entry=0x7fffffff7340, main_stat=<optimized out>,
     main_stat=<optimized out>) at find-debuginfo.c:61
61	  int fd = TEMP_FAILURE_RETRY (open (fname, O_RDONLY));
(gdb) p fname
$52 = 0x62da070 "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/var/cache/abrt-di/usr/lib/debug/hello.debug"


> Torsten, have you tried using --sysroot recently?
>
> On Mon, Mar 5, 2018 at 11:37 AM, Victor Kamensky <kamensky@cisco.com> wrote:
>> If sysroot option is passed, and debug symbols reside in sysroot along
>> with executable <foo> in <foo_dir>/.debug/<foo_file> directory, stap
>> fails to find debuginfo because it strips out sysroot path from file_name
>> so dwfl_standard_find_debuginfo ends up looking at host
>> <foo_dir>/.debug/<foo_file> rather then checking
>> <sysroot>/<foo_dir>/.debug/<foo_file>.
>>
>> Note in cross compile environment, it is good idea to set and export
>> proper SYSTEMTAP_DEBUGINFO_PATH variable because usual built defaults that
>> work for native distros very often not applicable to cross sysroot
>> based environment. For example for yocto poky/OE build the following
>> setting seems proper "+:.debug:build".
>>
>> Here are steps how to reproduce the issue. Note <sysroot> is produced by
>> yocto poky build.
>>
>> [kamensky@coreos-lnx2 tests]$ ls /home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/mkdir.coreutils
>> /home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/mkdir.coreutils
>> [kamensky@coreos-lnx2 tests]$ ls /home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/.debug/mkdir.coreutils
>> /home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/.debug/mkdir.coreutils
>> [kamensky@coreos-lnx2 tests]$ /home/wd8/systemtap/20180208_2/packages/bin/stap --sysroot=/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs -a x86_64 -r 4.9.78-yocto-standard -B CROSS_COMPILE=x86_64-poky-linux- --sysenv=PATH=/usr/bin:/bin:/usr/sbin:/sbin --sysenv=LD_LIBRARY_PATH=/lib:/usr/lib -v -l 'process("/bin/mkdir.coreutils").function("*")'
>> Pass 1: parsed user script and 480 library scripts using 230184virt/93416res/5344shr/88612data kb, in 350usr/30sys/378real ms.
>> process("/bin/mkdir.coreutils").function("_fini")
>> process("/bin/mkdir.coreutils").function("_init")
>> process("/bin/mkdir.coreutils").function("fts_children")
>> process("/bin/mkdir.coreutils").function("fts_close")
>> process("/bin/mkdir.coreutils").function("fts_open")
>> process("/bin/mkdir.coreutils").function("fts_read")
>> process("/bin/mkdir.coreutils").function("fts_set")
>> Pass 2: analyzed script: 7 probes, 0 functions, 0 embeds, 0 globals using 233484virt/97928res/6212shr/91912data kb, in 20usr/0sys/26real ms.
>>
>> running above example under strace shows that stap tries to access
>> /bin/.debug/mkdir.coreutils on host, rather then sysroot
>>
>> 13064 openat(AT_FDCWD, "/bin/mkdir.coreutils", O_RDONLY) = -1 ENOENT (No such file or directory)
>> 13064 openat(AT_FDCWD, "/bin/.debug/mkdir.coreutils", O_RDONLY) = -1 ENOENT (No such file or directory)
>> 13064 openat(AT_FDCWD, "/bin/mkdir.coreutils", O_RDONLY) = -1 ENOENT (No such file or directory)
>> 13064 openat(AT_FDCWD, "/bin/build/mkdir.coreutils", O_RDONLY) = -1 ENOENT (No such file or directory)
>> 13064 openat(AT_FDCWD, "/bin/mkdir.coreutils", O_RDONLY) = -1 ENOENT (No such file or directory)
>>
>> After fix applied stap is able to find /bin/.debug/mkdir.coreutils under
>> sysroot:
>>
>> [kamensky@coreos-lnx2 tests]$ /home/wd8/systemtap/20180208_2/packages/bin/stap --sysroot=/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs -a x86_64 -r 4.9.78-yocto-standard -B CROSS_COMPILE=x86_64-poky-linux- --sysenv=PATH=/usr/bin:/bin:/usr/sbin:/sbin --sysenv=LD_LIBRARY_PATH=/lib:/usr/lib -v -l 'process("/bin/mkdir.coreutils").function("*")' | wc
>> Pass 1: parsed user script and 480 library scripts using 230184virt/93440res/5368shr/88612data kb, in 330usr/40sys/371real ms.
>> Pass 2: analyzed script: 401 probes, 0 functions, 0 embeds, 0 globals using 235092virt/99484res/6256shr/93520data kb, in 30usr/0sys/35real ms.
>>     249     249   29671
>>
>> Also verified on target that script like example below, once compiled
>> against sysroot on host and copied back on target can trace 'mkdir'
>> invocation functions:
>>
>> [kamensky@coreos-lnx2 tests]$ cat mkdir1.stp
>> probe process("/bin/mkdir.coreutils").function("*").call {
>>   printf ("%s -> %s\n", thread_indent(1), ppfunc())
>> }
>> probe process("/bin/mkdir.coreutils").function("*").return {
>>   printf ("%s <- %s\n", thread_indent(-1), ppfunc())
>> }
>>
>> Signed-off-by: Victor Kamensky <kamensky@cisco.com>
>> ---
>>  setupdwfl.cxx | 12 ------------
>>  1 file changed, 12 deletions(-)
>>
>> diff --git a/setupdwfl.cxx b/setupdwfl.cxx
>> index f00cf755b..e5bfa28f0 100644
>> --- a/setupdwfl.cxx
>> +++ b/setupdwfl.cxx
>> @@ -637,18 +637,6 @@ internal_find_debuginfo (Dwfl_Module *mod,
>>
>>    call_dwfl_standard_find_debuginfo:
>>
>> -  if (current_session_for_find_debuginfo)
>> -    {
>> -      string sysroot = current_session_for_find_debuginfo->sysroot + "/*";
>> -      int    found   = fnmatch(sysroot.c_str(), file_name, 0);
>> -
>> -      if (found)
>> -       {
>> -         file_name = file_name
>> -           + current_session_for_find_debuginfo->sysroot.length() - 1;
>> -       }
>> -    }
>> -
>>    /* Call the original dwfl_standard_find_debuginfo */
>>    return dwfl_standard_find_debuginfo(mod, userdata, modname, base,
>>                file_name, debuglink_file,
>> --
>> 2.14.3
>>
>
>
>
> -- 
> David Smith
> Associate Manager
> Red Hat
>

Re: [PATCH 3/6] sysroot: debuginfo lookup with sysroot case do not remove sysroot from file_name

[Victor Kamensky]

On Thu, 8 Mar 2018, Victor Kamensky wrote:

> Hi David,
>
> On Thu, 8 Mar 2018, David Smith wrote:
>
>> Hmm, the problem I have here is the code your patch wants to delete
>> was added to fix a sysroot problem in the following commit:
>> 
>> ====
>> commit df932a3175284fa75a4c6bdb3aa7af1a34c15cd6
>> Author: Torsten Polle <Torsten.Polle@gmx.de>
>> Date:   Thu Mar 6 21:38:49 2014 +0100
>>
>>    Fix: Debug links are not found if the sysroot is used.
>>
>>    Signed-off-by: Torsten Polle <Torsten.Polle@gmx.de>
>> ====
>
> Thank you for bring my attention to it. Unfortunately above
> commit message is too terse, and it is not clear what was the
> use case that did not work.
>
> But I looked closely into it again, and I think I know what
> is happening:
>
> It boils down to three different types of entries that
> SYSTEMTAP_DEBUGINFO_PATH environment (or its default) may have. 
> SYSTEMTAP_DEBUGINFO_PATH value is passed through debuginfo_usr_path
> to mod->dwfl->callbacks.debuginfo_path and that is passed
> to dwfl_standard_find_debuginfo function of elfutils.
>
> Note fila_name passed to function may have sysroot stripped
> from it (current code), or may include sysroot as it was
> before df932a3175284 and after my fix.
>
> debuginfo_path may contain the following entries, each
> may preceeded by optional "+" or "-"
>
> 1) empty entry, in this case fila_name passed to the function
> is checked directly. If file_name does not have sysroot in
> it dwfl_standard_find_debuginfo will try to look for file
> in current root. Which is clearly very wrong thing to do in
> case of sysroot build, since it may pick up file with the
> same name in host. Current code with default debuginfo_path
> has this issue
>
> 2) Local path like ".debug". In this case function will look
> for `dirname file_name`/.debug/`basename file_name'. Again
> if file_name does not have sysroot in it, host files system
> will be looked and it is wrong. That is actually my case,
> in for /bin/mkdir.coreutils in my sysroot there is
> /bin/.debug/mkdir.coreutils with symbols. So if file_name
> does have sysroot in it, correct symbols file is picked and
> that is what I made this fix.
>
> 3) Absolute path like '/<sysroot>/usr/lib/debug'. In this
> case function will look at
> '/<sysroot>/usr/lib/debug'/file_name or
> '/<sysroot>/usr/lib/debug'/`basename file_name`. And in
> this case if file_name contains sysroot it will not work.
> I believe it is what df932a317528 has fixed and that is
> what my fix would break again. But note df932a317528 broke
> lookup for case 2).

I've tried to implement fix as per my understanding
above, but I discovered
that I missed important piece yesterday wrt how
dwfl_standard_find_debuginfo function handles absolute
entries. It turns out if function receives file_name it
will try one by one chipping of directories from the front of
file_name so eventually it comes to short name within
sysroot. I.e it gets file_name as /my/sys/root/bin/hello.debug
it will try:

<absolute_debuginfo>/my/sys/root/bin/hello.debug

<absolute_debuginfo>/sys/root/bin/hello.debug

<absolute_debuginfo>/root/bin/hello.debug

<absolute_debuginfo>/bin/hello.debug

<absolute_debuginfo>/hello.debug

So now, I stand by my original patch - df932a317528 should be
reverted. And reverted case works for all debuginof_path entries
types. I've unit tested either local or absolute entries all
work with my originally proposed patch.

df932a317528 was done in 'Thu Mar 6 21:38:49 2014' but
dwfl_standard_find_debuginfo logic to explore all sub-subdirs
for absolute entries was added in elfutils in 2015-08-13
by b901b5e7. I think that explains it.

Please merge my original patch or revert df932a317528.

Here is snipet from 'git blame ./libdwfl/find-debuginfo.c'
around explore all sub-subdirs in dwfl_standard_find_debuginfo code

b901b5e7 (Dodji Seketeli 2015-08-13 15:59:41 +0200 268) 
subdir = file_dirname;
b901b5e7 (Dodji Seketeli 2015-08-13 15:59:41 +0200 269)               /* 
We want to explore all sub-subdirs.  Chop off one slash
b901b5e7 (Dodji Seketeli 2015-08-13 15:59:41 +0200 270) 
at a time.  */
b901b5e7 (Dodji Seketeli 2015-08-13 15:59:41 +0200 271) 
explore_dir:
b901b5e7 (Dodji Seketeli 2015-08-13 15:59:41 +0200 272) 
subdir = strchr (subdir, '/');
b901b5e7 (Dodji Seketeli 2015-08-13 15:59:41 +0200 273)               if 
(subdir != NULL)
b901b5e7 (Dodji Seketeli 2015-08-13 15:59:41 +0200 274) 
subdir = subdir + 1;
b901b5e7 (Dodji Seketeli 2015-08-13 15:59:41 +0200 275)               if 
(subdir && *subdir == 0)
b901b5e7 (Dodji Seketeli 2015-08-13 15:59:41 +0200 276) 
continue;

Thanks,
Victor

> I am not sure how to fix it to work in all cases. I will
> think about it tomorrow. Any suggestions are welcome.
>
> For now it is clear in sysroot case, passing empty entry
> is wrong. Instead sysroot itself should be passed.
>
> Local path and absolute path debuginfo_path contradict
> each other wrt expectation whether file_name contains
> sysroot or not. Maybe debuginfo_path in case of
> non-empty sysroot should be split into two diferent
> sets: one with absolute entries and other with local entries
> and dwfl_standard_find_debuginfo should be called twice
> with different variants of file_name.
>
> If interseted please find below my gdb session that
> illustrates behavior of dwfl_standard_find_debuginfo described
> above. Gdb session has run against current code.
>
> Thanks,
> Victor
>
> In this example /bin/hello executable compiled in such way that it's
> .gnu_debuglink has value hello.debug, so it could be distingushable
> from executable itself.
>
> (gdb) show args
> Argument list to give program being debugged when it is started is 
> "--sysroot=/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs 
> -a x86_64 -r 
> /home/wd8/yocto/20180128/build_x86_64/tmp/work/corei7-64-intel-common-poky-linux/linux-yocto/4.9.78+gitAUTOINC+ef2f5d9a0a_f7a6d45fff-r0/linux-corei7-64-intel-common-standard-build 
> -B 
> CROSS_COMPILE=/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/usr/bin/x86_64-poky-linux- 
> --sysenv=PATH=/usr/bin:/bin:/usr/sbin:/sbin 
> --sysenv=LD_LIBRARY_PATH=/lib:/usr/lib -v -l 
> 'process("/bin/hello").function("*")'".
> ...
>
> (gdb) bt
> #0  find_debuginfo_in_path (mod=mod@entry=0x62d6950, 
> file_name=file_name@entry=0x62d6bf2 "/bin/hello", 
> debuglink_file=debuglink_file@entry=0x7ffff7fed044 "hello.debug",
>    debuglink_crc=debuglink_crc@entry=611974856, 
> debuginfo_file_name=debuginfo_file_name@entry=0x62d69a8) at 
> find-debuginfo.c:165
> #1  0x00007ffff7bb2855 in dwfl_standard_find_debuginfo (mod=0x62d6950, 
> userdata=<optimized out>, modname=<optimized out>, base=<optimized out>, 
> file_name=0x62d6bf2 "/bin/hello",
>    debuglink_file=0x7ffff7fed044 "hello.debug", debuglink_crc=611974856, 
> debuginfo_file_name=0x62d69a8) at find-debuginfo.c:383
> #2  0x0000000000657719 in internal_find_debuginfo (mod=0x62d6950, 
> userdata=0x62d6960,
>    modname=0x62d6af0 
> "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/hello", 
> base=65536,
>    file_name=0x62d6bf2 "/bin/hello", debuglink_file=0x7ffff7fed044 
> "hello.debug", debuglink_crc=611974856, debuginfo_file_name=0x62d69a8) at 
> ../systemtap/setupdwfl.cxx:645
> #3  0x00007ffff7bb0153 in find_debuginfo (mod=0x62d6950) at 
> dwfl_module_getdwarf.c:539
> #4  0x00007ffff7bb1393 in find_dw (mod=0x62d6950) at 
> dwfl_module_getdwarf.c:1400
> #5  dwfl_module_getdwarf (mod=0x62d6950, bias=0x7fffffff79d8) at 
> dwfl_module_getdwarf.c:1434
> #6  0x0000000000530b2c in validate_module_elf (mod=0x62d6950,
>    name=0x62d6af0 
> "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/hello", 
> q=0x7fffffff8110)
>    at ../systemtap/tapsets.cxx:2471
> #7  0x000000000053169e in query_module (mod=0x62d6950,
>    name=0x62d6af0 
> "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/hello", 
> addr=65536, q=0x7fffffff8110)
>    at ../systemtap/tapsets.cxx:2605
> #8  0x00007ffff7baed71 in dwfl_getmodules (dwfl=0x62d5020, callback=0x53137c 
> <query_module(Dwfl_Module*, void**, char const*, Dwarf_Addr, base_query*)>, 
> arg=0x7fffffff8110, offset=0)
>    at dwfl_getmodules.c:86
> #9  0x0000000000623215 in dwflpp::iterate_over_modules<void> (this=0x62d4b80, 
> callback=0x53137c <query_module(Dwfl_Module*, void**, char const*, 
> Dwarf_Addr, base_query*)>,
>    data=0x7fffffff8110) at ../systemtap/dwflpp.cxx:409
> #10 0x000000000058b7a3 in dwflpp::iterate_over_modules<base_query> 
> (this=0x62d4b80, callback=0x53137c <query_module(Dwfl_Module*, void**, char 
> const*, Dwarf_Addr, base_query*)>,
>    data=0x7fffffff8110) at ../systemtap/dwflpp.h:238
> #11 0x00000000005664f7 in dwarf_builder::build (this=0x62c7960, sess=..., 
> base=0x4b538d0, location=0x1028eb0, parameters=std::map with 2 elements = 
> {...},
>    finished_results=std::vector of length 0, capacity 0) at 
> ../systemtap/tapsets.cxx:8616
> #12 0x000000000049dcbf in match_node::find_and_build (this=0x62cc660, s=..., 
> p=0x4b538d0, loc=0x1028eb0, pos=2, results=std::vector of length 0, capacity 
> 0, builders=std::set with 0 elements)
>    at ../systemtap/elaborate.cxx:474
> #13 0x000000000049ee12 in match_node::find_and_build (this=0x4b4d920, s=..., 
> p=0x4b538d0, loc=0x1028eb0, pos=1, results=std::vector of length 0, capacity 
> 0, builders=std::set with 0 elements)
>    at ../systemtap/elaborate.cxx:648
> #14 0x000000000049ee12 in match_node::find_and_build (this=0xa1b2d0, s=..., 
> p=0x4b538d0, loc=0x1028eb0, pos=0, results=std::vector of length 0, capacity 
> 0, builders=std::set with 0 elements)
>    at ../systemtap/elaborate.cxx:648
> #15 0x00000000004a0ca5 in derive_probes (s=..., p=0x4b538d0, dps=std::vector 
> of length 0, capacity 0, optional=false, rethrow_errors=false) at 
> ../systemtap/elaborate.cxx:1022
> #16 0x00000000004a4332 in semantic_pass_symbols (s=...) at 
> ../systemtap/elaborate.cxx:1890
> #17 0x00000000004a88c5 in semantic_pass (s=...) at 
> ../systemtap/elaborate.cxx:2441
> #18 0x0000000000413a48 in passes_0_4 (s=...) at ../systemtap/main.cxx:914
> #19 0x00000000004161a8 in main (argc=13, argv=0x7fffffffdd68) at 
> ../systemtap/main.cxx:1381
> (gdb) p *mod
> $43 = {dwfl = 0x62d5020, next = 0x0, userdata = 0x0,
>  name = 0x62d6af0 
> "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/hello", 
> low_addr = 65536, high_addr = 2166840,
>  main = {name = 0x62d6b80 
> "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/hello", 
> fd = -1, valid = false,
>    relocated = false, elf = 0x62d50b0, vaddr = 0, address_sync = 2304}, 
> debug = {name = 0x0, fd = 0, valid = false, relocated = false, elf = 0x0, 
> vaddr = 0, address_sync = 0}, aux_sym = {
>    name = 0x0, fd = 0, valid = false, relocated = false, elf = 0x0, vaddr = 
> 0, address_sync = 0}, main_bias = 65536, ebl = 0x0, e_type = 3, elferr = 
> DWFL_E_NOERROR, reloc_info = 0x0,
>  symfile = 0x0, symdata = 0x0, aux_symdata = 0x0, syments = 0, aux_syments = 
> 0, first_global = 0, aux_first_global = 0, symstrdata = 0x0, aux_symstrdata = 
> 0x0, symxndxdata = 0x0,
>  aux_symxndxdata = 0x0, dw = 0x0, alt = 0x0, alt_fd = 0, alt_elf = 0x0, 
> symerr = DWFL_E_NOERROR, dwerr = DWFL_E_NO_DWARF, first_cu = 0x0, cu = 0x0, 
> lazy_cu_root = 0x0, aranges = 0x0,
>  build_id_bits = 0x62d4b60, build_id_vaddr = 66180, build_id_len = 20, ncu = 
> 0, lazycu = 0, naranges = 0, dwarf_cfi = 0x0, eh_cfi = 0x0, segment = 0, gc = 
> false, is_executable = false}
> (gdb) p *mod->dwfl->callbacks
> $44 = {find_elf = 0x0, find_debuginfo = 0x657103 
> <internal_find_debuginfo(Dwfl_Module*, void**, char const*, unsigned long, 
> char const*, char const*, unsigned int, char**)>,
>  section_address = 0x0, debuginfo_path = 0xa07db8 <debuginfo_usr_path>}
> (gdb) p debuginfo_usr_path
> $45 = 0xa28bc0 
> "+:.debug:/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/usr/lib/debug:/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/var/cache/abrt-di/usr/lib/debug"
> (gdb) c
>
> Breakpoint 5, try_open (dir=dir@entry=0x62d8c50 "/bin", 
> subdir=subdir@entry=0x0, debuglink=debuglink@entry=0x7ffff7fed044 
> "hello.debug",
>    debuginfo_file_name=debuginfo_file_name@entry=0x7fffffff7340, 
> main_stat=<optimized out>, main_stat=<optimized out>) at find-debuginfo.c:44
> 44	try_open (const struct stat *main_stat,
> (gdb) c
> Continuing.
>
> Breakpoint 6, try_open (dir=dir@entry=0x62d8c50 "/bin", 
> subdir=subdir@entry=0x0, debuglink=debuglink@entry=0x7ffff7fed044 
> "hello.debug",
>    debuginfo_file_name=debuginfo_file_name@entry=0x7fffffff7340, 
> main_stat=<optimized out>, main_stat=<optimized out>) at find-debuginfo.c:61
> 61	  int fd = TEMP_FAILURE_RETRY (open (fname, O_RDONLY));
> (gdb) p fname
> $46 = 0x62d9320 "/bin/hello.debug"
> (gdb) c
> Continuing.
>
> Breakpoint 5, try_open (dir=dir@entry=0x62d8c50 "/bin", 
> subdir=subdir@entry=0xa28e42 ".debug", 
> debuglink=debuglink@entry=0x7ffff7fed044 "hello.debug",
>    debuginfo_file_name=debuginfo_file_name@entry=0x7fffffff7340, 
> main_stat=<optimized out>, main_stat=<optimized out>) at find-debuginfo.c:44
> 44	try_open (const struct stat *main_stat,
> (gdb) c
> Continuing.
>
> Breakpoint 6, try_open (dir=dir@entry=0x62d8c50 "/bin", 
> subdir=subdir@entry=0xa28e42 ".debug", 
> debuglink=debuglink@entry=0x7ffff7fed044 "hello.debug",
>    debuginfo_file_name=debuginfo_file_name@entry=0x7fffffff7340, 
> main_stat=<optimized out>, main_stat=<optimized out>) at find-debuginfo.c:61
> 61	  int fd = TEMP_FAILURE_RETRY (open (fname, O_RDONLY));
> (gdb) p fname
> $47 = 0x62d9320 "/bin/.debug/hello.debug"
> (gdb) c
> Continuing.
>
> Breakpoint 5, try_open (dir=dir@entry=0x62d8c50 "/bin", 
> subdir=subdir@entry=0x0, debuglink=debuglink@entry=0x7ffff7fed044 
> "hello.debug",
>    debuginfo_file_name=debuginfo_file_name@entry=0x7fffffff7340, 
> main_stat=<optimized out>, main_stat=<optimized out>) at find-debuginfo.c:44
> 44	try_open (const struct stat *main_stat,
> (gdb) c
> Continuing.
>
> Breakpoint 6, try_open (dir=dir@entry=0x62d8c50 "/bin", 
> subdir=subdir@entry=0x0, debuglink=debuglink@entry=0x7ffff7fed044 
> "hello.debug",
>    debuginfo_file_name=debuginfo_file_name@entry=0x7fffffff7340, 
> main_stat=<optimized out>, main_stat=<optimized out>) at find-debuginfo.c:61
> 61	  int fd = TEMP_FAILURE_RETRY (open (fname, O_RDONLY));
> (gdb) p fname
> $48 = 0x62d9320 "/bin/hello.debug"
> (gdb) c
> Continuing.
>
> Breakpoint 5, try_open (dir=dir@entry=0xa28e49 
> "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/usr/lib/debug",
>    subdir=subdir@entry=0x62d8c51 "bin", 
> debuglink=debuglink@entry=0x7ffff7fed044 "hello.debug", 
> debuginfo_file_name=debuginfo_file_name@entry=0x7fffffff7340, 
> main_stat=<optimized out>,
>    main_stat=<optimized out>) at find-debuginfo.c:44
> 44	try_open (const struct stat *main_stat,
> (gdb) c
> Continuing.
>
> Breakpoint 6, try_open (dir=dir@entry=0xa28e49 
> "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/usr/lib/debug",
>    subdir=subdir@entry=0x62d8c51 "bin", 
> debuglink=debuglink@entry=0x7ffff7fed044 "hello.debug", 
> debuginfo_file_name=debuginfo_file_name@entry=0x7fffffff7340, 
> main_stat=<optimized out>,
>    main_stat=<optimized out>) at find-debuginfo.c:61
> 61	  int fd = TEMP_FAILURE_RETRY (open (fname, O_RDONLY));
> (gdb) p fname
> $49 = 0x62d9fd0 
> "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/usr/lib/debug/bin/hello.debug"
> (gdb) c
> Continuing.
>
> Breakpoint 5, try_open (dir=dir@entry=0xa28e49 
> "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/usr/lib/debug",
>    subdir=subdir@entry=0x0, debuglink=debuglink@entry=0x7ffff7fed044 
> "hello.debug", debuginfo_file_name=debuginfo_file_name@entry=0x7fffffff7340, 
> main_stat=<optimized out>,
>    main_stat=<optimized out>) at find-debuginfo.c:44
> 44	try_open (const struct stat *main_stat,
> (gdb) c
> Continuing.
>
> Breakpoint 6, try_open (dir=dir@entry=0xa28e49 
> "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/usr/lib/debug",
>    subdir=subdir@entry=0x0, debuglink=debuglink@entry=0x7ffff7fed044 
> "hello.debug", debuginfo_file_name=debuginfo_file_name@entry=0x7fffffff7340, 
> main_stat=<optimized out>,
>    main_stat=<optimized out>) at find-debuginfo.c:61
> 61	  int fd = TEMP_FAILURE_RETRY (open (fname, O_RDONLY));
> (gdb) p fname
> $50 = 0x62d9fd0 
> "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/usr/lib/debug/hello.debug"
> (gdb) c
> Continuing.
>
> Breakpoint 5, try_open (
>    dir=dir@entry=0xa28eca 
> "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/var/cache/abrt-di/usr/lib/debug",
>    subdir=subdir@entry=0x62d8c51 "bin", 
> debuglink=debuglink@entry=0x7ffff7fed044 "hello.debug", 
> debuginfo_file_name=debuginfo_file_name@entry=0x7fffffff7340, 
> main_stat=<optimized out>,
>    main_stat=<optimized out>) at find-debuginfo.c:44
> 44	try_open (const struct stat *main_stat,
> (gdb) c
> Continuing.
>
> Breakpoint 6, try_open (
>    dir=dir@entry=0xa28eca 
> "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/var/cache/abrt-di/usr/lib/debug",
>    subdir=subdir@entry=0x62d8c51 "bin", 
> debuglink=debuglink@entry=0x7ffff7fed044 "hello.debug", 
> debuginfo_file_name=debuginfo_file_name@entry=0x7fffffff7340, 
> main_stat=<optimized out>,
>    main_stat=<optimized out>) at find-debuginfo.c:61
> 61	  int fd = TEMP_FAILURE_RETRY (open (fname, O_RDONLY));
> (gdb) p fname
> $51 = 0x62d9e90 
> "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/var/cache/abrt-di/usr/lib/debug/bin/hello.debug"
> (gdb) c
> Continuing.
>
> Breakpoint 5, try_open (
>    dir=dir@entry=0xa28eca 
> "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/var/cache/abrt-di/usr/lib/debug",
>    subdir=subdir@entry=0x0, debuglink=debuglink@entry=0x7ffff7fed044 
> "hello.debug", debuginfo_file_name=debuginfo_file_name@entry=0x7fffffff7340, 
> main_stat=<optimized out>,
>    main_stat=<optimized out>) at find-debuginfo.c:44
> 44	try_open (const struct stat *main_stat,
> (gdb) c
> Continuing.
>
> Breakpoint 6, try_open (
>    dir=dir@entry=0xa28eca 
> "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/var/cache/abrt-di/usr/lib/debug",
>    subdir=subdir@entry=0x0, debuglink=debuglink@entry=0x7ffff7fed044 
> "hello.debug", debuginfo_file_name=debuginfo_file_name@entry=0x7fffffff7340, 
> main_stat=<optimized out>,
>    main_stat=<optimized out>) at find-debuginfo.c:61
> 61	  int fd = TEMP_FAILURE_RETRY (open (fname, O_RDONLY));
> (gdb) p fname
> $52 = 0x62da070 
> "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/var/cache/abrt-di/usr/lib/debug/hello.debug"
>
>
>> Torsten, have you tried using --sysroot recently?
>> 
>> On Mon, Mar 5, 2018 at 11:37 AM, Victor Kamensky <kamensky@cisco.com> 
>> wrote:
>>> If sysroot option is passed, and debug symbols reside in sysroot along
>>> with executable <foo> in <foo_dir>/.debug/<foo_file> directory, stap
>>> fails to find debuginfo because it strips out sysroot path from file_name
>>> so dwfl_standard_find_debuginfo ends up looking at host
>>> <foo_dir>/.debug/<foo_file> rather then checking
>>> <sysroot>/<foo_dir>/.debug/<foo_file>.
>>> 
>>> Note in cross compile environment, it is good idea to set and export
>>> proper SYSTEMTAP_DEBUGINFO_PATH variable because usual built defaults that
>>> work for native distros very often not applicable to cross sysroot
>>> based environment. For example for yocto poky/OE build the following
>>> setting seems proper "+:.debug:build".
>>> 
>>> Here are steps how to reproduce the issue. Note <sysroot> is produced by
>>> yocto poky build.
>>> 
>>> [kamensky@coreos-lnx2 tests]$ ls 
>>> /home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/mkdir.coreutils
>>> /home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/mkdir.coreutils
>>> [kamensky@coreos-lnx2 tests]$ ls 
>>> /home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/.debug/mkdir.coreutils
>>> /home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/.debug/mkdir.coreutils
>>> [kamensky@coreos-lnx2 tests]$ 
>>> /home/wd8/systemtap/20180208_2/packages/bin/stap 
>>> --sysroot=/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs 
>>> -a x86_64 -r 4.9.78-yocto-standard -B CROSS_COMPILE=x86_64-poky-linux- 
>>> --sysenv=PATH=/usr/bin:/bin:/usr/sbin:/sbin 
>>> --sysenv=LD_LIBRARY_PATH=/lib:/usr/lib -v -l 
>>> 'process("/bin/mkdir.coreutils").function("*")'
>>> Pass 1: parsed user script and 480 library scripts using 
>>> 230184virt/93416res/5344shr/88612data kb, in 350usr/30sys/378real ms.
>>> process("/bin/mkdir.coreutils").function("_fini")
>>> process("/bin/mkdir.coreutils").function("_init")
>>> process("/bin/mkdir.coreutils").function("fts_children")
>>> process("/bin/mkdir.coreutils").function("fts_close")
>>> process("/bin/mkdir.coreutils").function("fts_open")
>>> process("/bin/mkdir.coreutils").function("fts_read")
>>> process("/bin/mkdir.coreutils").function("fts_set")
>>> Pass 2: analyzed script: 7 probes, 0 functions, 0 embeds, 0 globals using 
>>> 233484virt/97928res/6212shr/91912data kb, in 20usr/0sys/26real ms.
>>> 
>>> running above example under strace shows that stap tries to access
>>> /bin/.debug/mkdir.coreutils on host, rather then sysroot
>>> 
>>> 13064 openat(AT_FDCWD, "/bin/mkdir.coreutils", O_RDONLY) = -1 ENOENT (No 
>>> such file or directory)
>>> 13064 openat(AT_FDCWD, "/bin/.debug/mkdir.coreutils", O_RDONLY) = -1 
>>> ENOENT (No such file or directory)
>>> 13064 openat(AT_FDCWD, "/bin/mkdir.coreutils", O_RDONLY) = -1 ENOENT (No 
>>> such file or directory)
>>> 13064 openat(AT_FDCWD, "/bin/build/mkdir.coreutils", O_RDONLY) = -1 ENOENT 
>>> (No such file or directory)
>>> 13064 openat(AT_FDCWD, "/bin/mkdir.coreutils", O_RDONLY) = -1 ENOENT (No 
>>> such file or directory)
>>> 
>>> After fix applied stap is able to find /bin/.debug/mkdir.coreutils under
>>> sysroot:
>>> 
>>> [kamensky@coreos-lnx2 tests]$ 
>>> /home/wd8/systemtap/20180208_2/packages/bin/stap 
>>> --sysroot=/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs 
>>> -a x86_64 -r 4.9.78-yocto-standard -B CROSS_COMPILE=x86_64-poky-linux- 
>>> --sysenv=PATH=/usr/bin:/bin:/usr/sbin:/sbin 
>>> --sysenv=LD_LIBRARY_PATH=/lib:/usr/lib -v -l 
>>> 'process("/bin/mkdir.coreutils").function("*")' | wc
>>> Pass 1: parsed user script and 480 library scripts using 
>>> 230184virt/93440res/5368shr/88612data kb, in 330usr/40sys/371real ms.
>>> Pass 2: analyzed script: 401 probes, 0 functions, 0 embeds, 0 globals 
>>> using 235092virt/99484res/6256shr/93520data kb, in 30usr/0sys/35real ms.
>>>     249     249   29671
>>> 
>>> Also verified on target that script like example below, once compiled
>>> against sysroot on host and copied back on target can trace 'mkdir'
>>> invocation functions:
>>> 
>>> [kamensky@coreos-lnx2 tests]$ cat mkdir1.stp
>>> probe process("/bin/mkdir.coreutils").function("*").call {
>>>   printf ("%s -> %s\n", thread_indent(1), ppfunc())
>>> }
>>> probe process("/bin/mkdir.coreutils").function("*").return {
>>>   printf ("%s <- %s\n", thread_indent(-1), ppfunc())
>>> }
>>> 
>>> Signed-off-by: Victor Kamensky <kamensky@cisco.com>
>>> ---
>>>  setupdwfl.cxx | 12 ------------
>>>  1 file changed, 12 deletions(-)
>>> 
>>> diff --git a/setupdwfl.cxx b/setupdwfl.cxx
>>> index f00cf755b..e5bfa28f0 100644
>>> --- a/setupdwfl.cxx
>>> +++ b/setupdwfl.cxx
>>> @@ -637,18 +637,6 @@ internal_find_debuginfo (Dwfl_Module *mod,
>>>
>>>    call_dwfl_standard_find_debuginfo:
>>> 
>>> -  if (current_session_for_find_debuginfo)
>>> -    {
>>> -      string sysroot = current_session_for_find_debuginfo->sysroot + 
>>> "/*";
>>> -      int    found   = fnmatch(sysroot.c_str(), file_name, 0);
>>> -
>>> -      if (found)
>>> -       {
>>> -         file_name = file_name
>>> -           + current_session_for_find_debuginfo->sysroot.length() - 1;
>>> -       }
>>> -    }
>>> -
>>>    /* Call the original dwfl_standard_find_debuginfo */
>>>    return dwfl_standard_find_debuginfo(mod, userdata, modname, base,
>>>                file_name, debuglink_file,
>>> --
>>> 2.14.3
>>> 
>> 
>> 
>> 
>> -- 
>> David Smith
>> Associate Manager
>> Red Hat
>> 
>

Re: [PATCH 3/6] sysroot: debuginfo lookup with sysroot case do not remove sysroot from file_name

[David Smith]

After looking into it, I believe you here. Your patched checked in as
commit b29e448e1.

On Fri, Mar 9, 2018 at 2:20 PM, Victor Kamensky <kamensky@cisco.com> wrote:
>
>
> On Thu, 8 Mar 2018, Victor Kamensky wrote:
>
>> Hi David,
>>
>> On Thu, 8 Mar 2018, David Smith wrote:
>>
>>> Hmm, the problem I have here is the code your patch wants to delete
>>> was added to fix a sysroot problem in the following commit:
>>>
>>> ====
>>> commit df932a3175284fa75a4c6bdb3aa7af1a34c15cd6
>>> Author: Torsten Polle <Torsten.Polle@gmx.de>
>>> Date:   Thu Mar 6 21:38:49 2014 +0100
>>>
>>>    Fix: Debug links are not found if the sysroot is used.
>>>
>>>    Signed-off-by: Torsten Polle <Torsten.Polle@gmx.de>
>>> ====
>>
>>
>> Thank you for bring my attention to it. Unfortunately above
>> commit message is too terse, and it is not clear what was the
>> use case that did not work.
>>
>> But I looked closely into it again, and I think I know what
>> is happening:
>>
>> It boils down to three different types of entries that
>> SYSTEMTAP_DEBUGINFO_PATH environment (or its default) may have.
>> SYSTEMTAP_DEBUGINFO_PATH value is passed through debuginfo_usr_path
>> to mod->dwfl->callbacks.debuginfo_path and that is passed
>> to dwfl_standard_find_debuginfo function of elfutils.
>>
>> Note fila_name passed to function may have sysroot stripped
>> from it (current code), or may include sysroot as it was
>> before df932a3175284 and after my fix.
>>
>> debuginfo_path may contain the following entries, each
>> may preceeded by optional "+" or "-"
>>
>> 1) empty entry, in this case fila_name passed to the function
>> is checked directly. If file_name does not have sysroot in
>> it dwfl_standard_find_debuginfo will try to look for file
>> in current root. Which is clearly very wrong thing to do in
>> case of sysroot build, since it may pick up file with the
>> same name in host. Current code with default debuginfo_path
>> has this issue
>>
>> 2) Local path like ".debug". In this case function will look
>> for `dirname file_name`/.debug/`basename file_name'. Again
>> if file_name does not have sysroot in it, host files system
>> will be looked and it is wrong. That is actually my case,
>> in for /bin/mkdir.coreutils in my sysroot there is
>> /bin/.debug/mkdir.coreutils with symbols. So if file_name
>> does have sysroot in it, correct symbols file is picked and
>> that is what I made this fix.
>>
>> 3) Absolute path like '/<sysroot>/usr/lib/debug'. In this
>> case function will look at
>> '/<sysroot>/usr/lib/debug'/file_name or
>> '/<sysroot>/usr/lib/debug'/`basename file_name`. And in
>> this case if file_name contains sysroot it will not work.
>> I believe it is what df932a317528 has fixed and that is
>> what my fix would break again. But note df932a317528 broke
>> lookup for case 2).
>
>
> I've tried to implement fix as per my understanding
> above, but I discovered
> that I missed important piece yesterday wrt how
> dwfl_standard_find_debuginfo function handles absolute
> entries. It turns out if function receives file_name it
> will try one by one chipping of directories from the front of
> file_name so eventually it comes to short name within
> sysroot. I.e it gets file_name as /my/sys/root/bin/hello.debug
> it will try:
>
> <absolute_debuginfo>/my/sys/root/bin/hello.debug
>
> <absolute_debuginfo>/sys/root/bin/hello.debug
>
> <absolute_debuginfo>/root/bin/hello.debug
>
> <absolute_debuginfo>/bin/hello.debug
>
> <absolute_debuginfo>/hello.debug
>
> So now, I stand by my original patch - df932a317528 should be
> reverted. And reverted case works for all debuginof_path entries
> types. I've unit tested either local or absolute entries all
> work with my originally proposed patch.
>
> df932a317528 was done in 'Thu Mar 6 21:38:49 2014' but
> dwfl_standard_find_debuginfo logic to explore all sub-subdirs
> for absolute entries was added in elfutils in 2015-08-13
> by b901b5e7. I think that explains it.
>
> Please merge my original patch or revert df932a317528.
>
> Here is snipet from 'git blame ./libdwfl/find-debuginfo.c'
> around explore all sub-subdirs in dwfl_standard_find_debuginfo code
>
> b901b5e7 (Dodji Seketeli 2015-08-13 15:59:41 +0200 268) subdir =
> file_dirname;
> b901b5e7 (Dodji Seketeli 2015-08-13 15:59:41 +0200 269)               /* We
> want to explore all sub-subdirs.  Chop off one slash
> b901b5e7 (Dodji Seketeli 2015-08-13 15:59:41 +0200 270) at a time.  */
> b901b5e7 (Dodji Seketeli 2015-08-13 15:59:41 +0200 271) explore_dir:
> b901b5e7 (Dodji Seketeli 2015-08-13 15:59:41 +0200 272) subdir = strchr
> (subdir, '/');
> b901b5e7 (Dodji Seketeli 2015-08-13 15:59:41 +0200 273)               if
> (subdir != NULL)
> b901b5e7 (Dodji Seketeli 2015-08-13 15:59:41 +0200 274) subdir = subdir + 1;
> b901b5e7 (Dodji Seketeli 2015-08-13 15:59:41 +0200 275)               if
> (subdir && *subdir == 0)
> b901b5e7 (Dodji Seketeli 2015-08-13 15:59:41 +0200 276) continue;
>
> Thanks,
> Victor
>
>
>> I am not sure how to fix it to work in all cases. I will
>> think about it tomorrow. Any suggestions are welcome.
>>
>> For now it is clear in sysroot case, passing empty entry
>> is wrong. Instead sysroot itself should be passed.
>>
>> Local path and absolute path debuginfo_path contradict
>> each other wrt expectation whether file_name contains
>> sysroot or not. Maybe debuginfo_path in case of
>> non-empty sysroot should be split into two diferent
>> sets: one with absolute entries and other with local entries
>> and dwfl_standard_find_debuginfo should be called twice
>> with different variants of file_name.
>>
>> If interseted please find below my gdb session that
>> illustrates behavior of dwfl_standard_find_debuginfo described
>> above. Gdb session has run against current code.
>>
>> Thanks,
>> Victor
>>
>> In this example /bin/hello executable compiled in such way that it's
>> .gnu_debuglink has value hello.debug, so it could be distingushable
>> from executable itself.
>>
>> (gdb) show args
>> Argument list to give program being debugged when it is started is
>> "--sysroot=/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs
>> -a x86_64 -r
>> /home/wd8/yocto/20180128/build_x86_64/tmp/work/corei7-64-intel-common-poky-linux/linux-yocto/4.9.78+gitAUTOINC+ef2f5d9a0a_f7a6d45fff-r0/linux-corei7-64-intel-common-standard-build
>> -B
>> CROSS_COMPILE=/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/usr/bin/x86_64-poky-linux-
>> --sysenv=PATH=/usr/bin:/bin:/usr/sbin:/sbin
>> --sysenv=LD_LIBRARY_PATH=/lib:/usr/lib -v -l
>> 'process("/bin/hello").function("*")'".
>> ...
>>
>> (gdb) bt
>> #0  find_debuginfo_in_path (mod=mod@entry=0x62d6950,
>> file_name=file_name@entry=0x62d6bf2 "/bin/hello",
>> debuglink_file=debuglink_file@entry=0x7ffff7fed044 "hello.debug",
>>    debuglink_crc=debuglink_crc@entry=611974856,
>> debuginfo_file_name=debuginfo_file_name@entry=0x62d69a8) at
>> find-debuginfo.c:165
>> #1  0x00007ffff7bb2855 in dwfl_standard_find_debuginfo (mod=0x62d6950,
>> userdata=<optimized out>, modname=<optimized out>, base=<optimized out>,
>> file_name=0x62d6bf2 "/bin/hello",
>>    debuglink_file=0x7ffff7fed044 "hello.debug", debuglink_crc=611974856,
>> debuginfo_file_name=0x62d69a8) at find-debuginfo.c:383
>> #2  0x0000000000657719 in internal_find_debuginfo (mod=0x62d6950,
>> userdata=0x62d6960,
>>    modname=0x62d6af0
>> "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/hello",
>> base=65536,
>>    file_name=0x62d6bf2 "/bin/hello", debuglink_file=0x7ffff7fed044
>> "hello.debug", debuglink_crc=611974856, debuginfo_file_name=0x62d69a8) at
>> ../systemtap/setupdwfl.cxx:645
>> #3  0x00007ffff7bb0153 in find_debuginfo (mod=0x62d6950) at
>> dwfl_module_getdwarf.c:539
>> #4  0x00007ffff7bb1393 in find_dw (mod=0x62d6950) at
>> dwfl_module_getdwarf.c:1400
>> #5  dwfl_module_getdwarf (mod=0x62d6950, bias=0x7fffffff79d8) at
>> dwfl_module_getdwarf.c:1434
>> #6  0x0000000000530b2c in validate_module_elf (mod=0x62d6950,
>>    name=0x62d6af0
>> "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/hello",
>> q=0x7fffffff8110)
>>    at ../systemtap/tapsets.cxx:2471
>> #7  0x000000000053169e in query_module (mod=0x62d6950,
>>    name=0x62d6af0
>> "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/hello",
>> addr=65536, q=0x7fffffff8110)
>>    at ../systemtap/tapsets.cxx:2605
>> #8  0x00007ffff7baed71 in dwfl_getmodules (dwfl=0x62d5020,
>> callback=0x53137c <query_module(Dwfl_Module*, void**, char const*,
>> Dwarf_Addr, base_query*)>, arg=0x7fffffff8110, offset=0)
>>    at dwfl_getmodules.c:86
>> #9  0x0000000000623215 in dwflpp::iterate_over_modules<void>
>> (this=0x62d4b80, callback=0x53137c <query_module(Dwfl_Module*, void**, char
>> const*, Dwarf_Addr, base_query*)>,
>>    data=0x7fffffff8110) at ../systemtap/dwflpp.cxx:409
>> #10 0x000000000058b7a3 in dwflpp::iterate_over_modules<base_query>
>> (this=0x62d4b80, callback=0x53137c <query_module(Dwfl_Module*, void**, char
>> const*, Dwarf_Addr, base_query*)>,
>>    data=0x7fffffff8110) at ../systemtap/dwflpp.h:238
>> #11 0x00000000005664f7 in dwarf_builder::build (this=0x62c7960, sess=...,
>> base=0x4b538d0, location=0x1028eb0, parameters=std::map with 2 elements =
>> {...},
>>    finished_results=std::vector of length 0, capacity 0) at
>> ../systemtap/tapsets.cxx:8616
>> #12 0x000000000049dcbf in match_node::find_and_build (this=0x62cc660,
>> s=..., p=0x4b538d0, loc=0x1028eb0, pos=2, results=std::vector of length 0,
>> capacity 0, builders=std::set with 0 elements)
>>    at ../systemtap/elaborate.cxx:474
>> #13 0x000000000049ee12 in match_node::find_and_build (this=0x4b4d920,
>> s=..., p=0x4b538d0, loc=0x1028eb0, pos=1, results=std::vector of length 0,
>> capacity 0, builders=std::set with 0 elements)
>>    at ../systemtap/elaborate.cxx:648
>> #14 0x000000000049ee12 in match_node::find_and_build (this=0xa1b2d0,
>> s=..., p=0x4b538d0, loc=0x1028eb0, pos=0, results=std::vector of length 0,
>> capacity 0, builders=std::set with 0 elements)
>>    at ../systemtap/elaborate.cxx:648
>> #15 0x00000000004a0ca5 in derive_probes (s=..., p=0x4b538d0,
>> dps=std::vector of length 0, capacity 0, optional=false,
>> rethrow_errors=false) at ../systemtap/elaborate.cxx:1022
>> #16 0x00000000004a4332 in semantic_pass_symbols (s=...) at
>> ../systemtap/elaborate.cxx:1890
>> #17 0x00000000004a88c5 in semantic_pass (s=...) at
>> ../systemtap/elaborate.cxx:2441
>> #18 0x0000000000413a48 in passes_0_4 (s=...) at ../systemtap/main.cxx:914
>> #19 0x00000000004161a8 in main (argc=13, argv=0x7fffffffdd68) at
>> ../systemtap/main.cxx:1381
>> (gdb) p *mod
>> $43 = {dwfl = 0x62d5020, next = 0x0, userdata = 0x0,
>>  name = 0x62d6af0
>> "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/hello",
>> low_addr = 65536, high_addr = 2166840,
>>  main = {name = 0x62d6b80
>> "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/hello",
>> fd = -1, valid = false,
>>    relocated = false, elf = 0x62d50b0, vaddr = 0, address_sync = 2304},
>> debug = {name = 0x0, fd = 0, valid = false, relocated = false, elf = 0x0,
>> vaddr = 0, address_sync = 0}, aux_sym = {
>>    name = 0x0, fd = 0, valid = false, relocated = false, elf = 0x0, vaddr
>> = 0, address_sync = 0}, main_bias = 65536, ebl = 0x0, e_type = 3, elferr =
>> DWFL_E_NOERROR, reloc_info = 0x0,
>>  symfile = 0x0, symdata = 0x0, aux_symdata = 0x0, syments = 0, aux_syments
>> = 0, first_global = 0, aux_first_global = 0, symstrdata = 0x0,
>> aux_symstrdata = 0x0, symxndxdata = 0x0,
>>  aux_symxndxdata = 0x0, dw = 0x0, alt = 0x0, alt_fd = 0, alt_elf = 0x0,
>> symerr = DWFL_E_NOERROR, dwerr = DWFL_E_NO_DWARF, first_cu = 0x0, cu = 0x0,
>> lazy_cu_root = 0x0, aranges = 0x0,
>>  build_id_bits = 0x62d4b60, build_id_vaddr = 66180, build_id_len = 20, ncu
>> = 0, lazycu = 0, naranges = 0, dwarf_cfi = 0x0, eh_cfi = 0x0, segment = 0,
>> gc = false, is_executable = false}
>> (gdb) p *mod->dwfl->callbacks
>> $44 = {find_elf = 0x0, find_debuginfo = 0x657103
>> <internal_find_debuginfo(Dwfl_Module*, void**, char const*, unsigned long,
>> char const*, char const*, unsigned int, char**)>,
>>  section_address = 0x0, debuginfo_path = 0xa07db8 <debuginfo_usr_path>}
>> (gdb) p debuginfo_usr_path
>> $45 = 0xa28bc0
>> "+:.debug:/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/usr/lib/debug:/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/var/cache/abrt-di/usr/lib/debug"
>> (gdb) c
>>
>> Breakpoint 5, try_open (dir=dir@entry=0x62d8c50 "/bin",
>> subdir=subdir@entry=0x0, debuglink=debuglink@entry=0x7ffff7fed044
>> "hello.debug",
>>    debuginfo_file_name=debuginfo_file_name@entry=0x7fffffff7340,
>> main_stat=<optimized out>, main_stat=<optimized out>) at find-debuginfo.c:44
>> 44      try_open (const struct stat *main_stat,
>> (gdb) c
>> Continuing.
>>
>> Breakpoint 6, try_open (dir=dir@entry=0x62d8c50 "/bin",
>> subdir=subdir@entry=0x0, debuglink=debuglink@entry=0x7ffff7fed044
>> "hello.debug",
>>    debuginfo_file_name=debuginfo_file_name@entry=0x7fffffff7340,
>> main_stat=<optimized out>, main_stat=<optimized out>) at find-debuginfo.c:61
>> 61        int fd = TEMP_FAILURE_RETRY (open (fname, O_RDONLY));
>> (gdb) p fname
>> $46 = 0x62d9320 "/bin/hello.debug"
>> (gdb) c
>> Continuing.
>>
>> Breakpoint 5, try_open (dir=dir@entry=0x62d8c50 "/bin",
>> subdir=subdir@entry=0xa28e42 ".debug",
>> debuglink=debuglink@entry=0x7ffff7fed044 "hello.debug",
>>    debuginfo_file_name=debuginfo_file_name@entry=0x7fffffff7340,
>> main_stat=<optimized out>, main_stat=<optimized out>) at find-debuginfo.c:44
>> 44      try_open (const struct stat *main_stat,
>> (gdb) c
>> Continuing.
>>
>> Breakpoint 6, try_open (dir=dir@entry=0x62d8c50 "/bin",
>> subdir=subdir@entry=0xa28e42 ".debug",
>> debuglink=debuglink@entry=0x7ffff7fed044 "hello.debug",
>>    debuginfo_file_name=debuginfo_file_name@entry=0x7fffffff7340,
>> main_stat=<optimized out>, main_stat=<optimized out>) at find-debuginfo.c:61
>> 61        int fd = TEMP_FAILURE_RETRY (open (fname, O_RDONLY));
>> (gdb) p fname
>> $47 = 0x62d9320 "/bin/.debug/hello.debug"
>> (gdb) c
>> Continuing.
>>
>> Breakpoint 5, try_open (dir=dir@entry=0x62d8c50 "/bin",
>> subdir=subdir@entry=0x0, debuglink=debuglink@entry=0x7ffff7fed044
>> "hello.debug",
>>    debuginfo_file_name=debuginfo_file_name@entry=0x7fffffff7340,
>> main_stat=<optimized out>, main_stat=<optimized out>) at find-debuginfo.c:44
>> 44      try_open (const struct stat *main_stat,
>> (gdb) c
>> Continuing.
>>
>> Breakpoint 6, try_open (dir=dir@entry=0x62d8c50 "/bin",
>> subdir=subdir@entry=0x0, debuglink=debuglink@entry=0x7ffff7fed044
>> "hello.debug",
>>    debuginfo_file_name=debuginfo_file_name@entry=0x7fffffff7340,
>> main_stat=<optimized out>, main_stat=<optimized out>) at find-debuginfo.c:61
>> 61        int fd = TEMP_FAILURE_RETRY (open (fname, O_RDONLY));
>> (gdb) p fname
>> $48 = 0x62d9320 "/bin/hello.debug"
>> (gdb) c
>> Continuing.
>>
>> Breakpoint 5, try_open (dir=dir@entry=0xa28e49
>> "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/usr/lib/debug",
>>    subdir=subdir@entry=0x62d8c51 "bin",
>> debuglink=debuglink@entry=0x7ffff7fed044 "hello.debug",
>> debuginfo_file_name=debuginfo_file_name@entry=0x7fffffff7340,
>> main_stat=<optimized out>,
>>    main_stat=<optimized out>) at find-debuginfo.c:44
>> 44      try_open (const struct stat *main_stat,
>> (gdb) c
>> Continuing.
>>
>> Breakpoint 6, try_open (dir=dir@entry=0xa28e49
>> "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/usr/lib/debug",
>>    subdir=subdir@entry=0x62d8c51 "bin",
>> debuglink=debuglink@entry=0x7ffff7fed044 "hello.debug",
>> debuginfo_file_name=debuginfo_file_name@entry=0x7fffffff7340,
>> main_stat=<optimized out>,
>>    main_stat=<optimized out>) at find-debuginfo.c:61
>> 61        int fd = TEMP_FAILURE_RETRY (open (fname, O_RDONLY));
>> (gdb) p fname
>> $49 = 0x62d9fd0
>> "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/usr/lib/debug/bin/hello.debug"
>> (gdb) c
>> Continuing.
>>
>> Breakpoint 5, try_open (dir=dir@entry=0xa28e49
>> "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/usr/lib/debug",
>>    subdir=subdir@entry=0x0, debuglink=debuglink@entry=0x7ffff7fed044
>> "hello.debug", debuginfo_file_name=debuginfo_file_name@entry=0x7fffffff7340,
>> main_stat=<optimized out>,
>>    main_stat=<optimized out>) at find-debuginfo.c:44
>> 44      try_open (const struct stat *main_stat,
>> (gdb) c
>> Continuing.
>>
>> Breakpoint 6, try_open (dir=dir@entry=0xa28e49
>> "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/usr/lib/debug",
>>    subdir=subdir@entry=0x0, debuglink=debuglink@entry=0x7ffff7fed044
>> "hello.debug", debuginfo_file_name=debuginfo_file_name@entry=0x7fffffff7340,
>> main_stat=<optimized out>,
>>    main_stat=<optimized out>) at find-debuginfo.c:61
>> 61        int fd = TEMP_FAILURE_RETRY (open (fname, O_RDONLY));
>> (gdb) p fname
>> $50 = 0x62d9fd0
>> "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/usr/lib/debug/hello.debug"
>> (gdb) c
>> Continuing.
>>
>> Breakpoint 5, try_open (
>>    dir=dir@entry=0xa28eca
>> "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/var/cache/abrt-di/usr/lib/debug",
>>    subdir=subdir@entry=0x62d8c51 "bin",
>> debuglink=debuglink@entry=0x7ffff7fed044 "hello.debug",
>> debuginfo_file_name=debuginfo_file_name@entry=0x7fffffff7340,
>> main_stat=<optimized out>,
>>    main_stat=<optimized out>) at find-debuginfo.c:44
>> 44      try_open (const struct stat *main_stat,
>> (gdb) c
>> Continuing.
>>
>> Breakpoint 6, try_open (
>>    dir=dir@entry=0xa28eca
>> "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/var/cache/abrt-di/usr/lib/debug",
>>    subdir=subdir@entry=0x62d8c51 "bin",
>> debuglink=debuglink@entry=0x7ffff7fed044 "hello.debug",
>> debuginfo_file_name=debuginfo_file_name@entry=0x7fffffff7340,
>> main_stat=<optimized out>,
>>    main_stat=<optimized out>) at find-debuginfo.c:61
>> 61        int fd = TEMP_FAILURE_RETRY (open (fname, O_RDONLY));
>> (gdb) p fname
>> $51 = 0x62d9e90
>> "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/var/cache/abrt-di/usr/lib/debug/bin/hello.debug"
>> (gdb) c
>> Continuing.
>>
>> Breakpoint 5, try_open (
>>    dir=dir@entry=0xa28eca
>> "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/var/cache/abrt-di/usr/lib/debug",
>>    subdir=subdir@entry=0x0, debuglink=debuglink@entry=0x7ffff7fed044
>> "hello.debug", debuginfo_file_name=debuginfo_file_name@entry=0x7fffffff7340,
>> main_stat=<optimized out>,
>>    main_stat=<optimized out>) at find-debuginfo.c:44
>> 44      try_open (const struct stat *main_stat,
>> (gdb) c
>> Continuing.
>>
>> Breakpoint 6, try_open (
>>    dir=dir@entry=0xa28eca
>> "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/var/cache/abrt-di/usr/lib/debug",
>>    subdir=subdir@entry=0x0, debuglink=debuglink@entry=0x7ffff7fed044
>> "hello.debug", debuginfo_file_name=debuginfo_file_name@entry=0x7fffffff7340,
>> main_stat=<optimized out>,
>>    main_stat=<optimized out>) at find-debuginfo.c:61
>> 61        int fd = TEMP_FAILURE_RETRY (open (fname, O_RDONLY));
>> (gdb) p fname
>> $52 = 0x62da070
>> "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/var/cache/abrt-di/usr/lib/debug/hello.debug"
>>
>>
>>> Torsten, have you tried using --sysroot recently?
>>>
>>> On Mon, Mar 5, 2018 at 11:37 AM, Victor Kamensky <kamensky@cisco.com>
>>> wrote:
>>>>
>>>> If sysroot option is passed, and debug symbols reside in sysroot along
>>>> with executable <foo> in <foo_dir>/.debug/<foo_file> directory, stap
>>>> fails to find debuginfo because it strips out sysroot path from
>>>> file_name
>>>> so dwfl_standard_find_debuginfo ends up looking at host
>>>> <foo_dir>/.debug/<foo_file> rather then checking
>>>> <sysroot>/<foo_dir>/.debug/<foo_file>.
>>>>
>>>> Note in cross compile environment, it is good idea to set and export
>>>> proper SYSTEMTAP_DEBUGINFO_PATH variable because usual built defaults
>>>> that
>>>> work for native distros very often not applicable to cross sysroot
>>>> based environment. For example for yocto poky/OE build the following
>>>> setting seems proper "+:.debug:build".
>>>>
>>>> Here are steps how to reproduce the issue. Note <sysroot> is produced by
>>>> yocto poky build.
>>>>
>>>> [kamensky@coreos-lnx2 tests]$ ls
>>>> /home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/mkdir.coreutils
>>>>
>>>> /home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/mkdir.coreutils
>>>> [kamensky@coreos-lnx2 tests]$ ls
>>>> /home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/.debug/mkdir.coreutils
>>>>
>>>> /home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/.debug/mkdir.coreutils
>>>> [kamensky@coreos-lnx2 tests]$
>>>> /home/wd8/systemtap/20180208_2/packages/bin/stap
>>>> --sysroot=/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs
>>>> -a x86_64 -r 4.9.78-yocto-standard -B CROSS_COMPILE=x86_64-poky-linux-
>>>> --sysenv=PATH=/usr/bin:/bin:/usr/sbin:/sbin
>>>> --sysenv=LD_LIBRARY_PATH=/lib:/usr/lib -v -l
>>>> 'process("/bin/mkdir.coreutils").function("*")'
>>>> Pass 1: parsed user script and 480 library scripts using
>>>> 230184virt/93416res/5344shr/88612data kb, in 350usr/30sys/378real ms.
>>>> process("/bin/mkdir.coreutils").function("_fini")
>>>> process("/bin/mkdir.coreutils").function("_init")
>>>> process("/bin/mkdir.coreutils").function("fts_children")
>>>> process("/bin/mkdir.coreutils").function("fts_close")
>>>> process("/bin/mkdir.coreutils").function("fts_open")
>>>> process("/bin/mkdir.coreutils").function("fts_read")
>>>> process("/bin/mkdir.coreutils").function("fts_set")
>>>> Pass 2: analyzed script: 7 probes, 0 functions, 0 embeds, 0 globals
>>>> using 233484virt/97928res/6212shr/91912data kb, in 20usr/0sys/26real ms.
>>>>
>>>> running above example under strace shows that stap tries to access
>>>> /bin/.debug/mkdir.coreutils on host, rather then sysroot
>>>>
>>>> 13064 openat(AT_FDCWD, "/bin/mkdir.coreutils", O_RDONLY) = -1 ENOENT (No
>>>> such file or directory)
>>>> 13064 openat(AT_FDCWD, "/bin/.debug/mkdir.coreutils", O_RDONLY) = -1
>>>> ENOENT (No such file or directory)
>>>> 13064 openat(AT_FDCWD, "/bin/mkdir.coreutils", O_RDONLY) = -1 ENOENT (No
>>>> such file or directory)
>>>> 13064 openat(AT_FDCWD, "/bin/build/mkdir.coreutils", O_RDONLY) = -1
>>>> ENOENT (No such file or directory)
>>>> 13064 openat(AT_FDCWD, "/bin/mkdir.coreutils", O_RDONLY) = -1 ENOENT (No
>>>> such file or directory)
>>>>
>>>> After fix applied stap is able to find /bin/.debug/mkdir.coreutils under
>>>> sysroot:
>>>>
>>>> [kamensky@coreos-lnx2 tests]$
>>>> /home/wd8/systemtap/20180208_2/packages/bin/stap
>>>> --sysroot=/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs
>>>> -a x86_64 -r 4.9.78-yocto-standard -B CROSS_COMPILE=x86_64-poky-linux-
>>>> --sysenv=PATH=/usr/bin:/bin:/usr/sbin:/sbin
>>>> --sysenv=LD_LIBRARY_PATH=/lib:/usr/lib -v -l
>>>> 'process("/bin/mkdir.coreutils").function("*")' | wc
>>>> Pass 1: parsed user script and 480 library scripts using
>>>> 230184virt/93440res/5368shr/88612data kb, in 330usr/40sys/371real ms.
>>>> Pass 2: analyzed script: 401 probes, 0 functions, 0 embeds, 0 globals
>>>> using 235092virt/99484res/6256shr/93520data kb, in 30usr/0sys/35real ms.
>>>>     249     249   29671
>>>>
>>>> Also verified on target that script like example below, once compiled
>>>> against sysroot on host and copied back on target can trace 'mkdir'
>>>> invocation functions:
>>>>
>>>> [kamensky@coreos-lnx2 tests]$ cat mkdir1.stp
>>>> probe process("/bin/mkdir.coreutils").function("*").call {
>>>>   printf ("%s -> %s\n", thread_indent(1), ppfunc())
>>>> }
>>>> probe process("/bin/mkdir.coreutils").function("*").return {
>>>>   printf ("%s <- %s\n", thread_indent(-1), ppfunc())
>>>> }
>>>>
>>>> Signed-off-by: Victor Kamensky <kamensky@cisco.com>
>>>> ---
>>>>  setupdwfl.cxx | 12 ------------
>>>>  1 file changed, 12 deletions(-)
>>>>
>>>> diff --git a/setupdwfl.cxx b/setupdwfl.cxx
>>>> index f00cf755b..e5bfa28f0 100644
>>>> --- a/setupdwfl.cxx
>>>> +++ b/setupdwfl.cxx
>>>> @@ -637,18 +637,6 @@ internal_find_debuginfo (Dwfl_Module *mod,
>>>>
>>>>    call_dwfl_standard_find_debuginfo:
>>>>
>>>> -  if (current_session_for_find_debuginfo)
>>>> -    {
>>>> -      string sysroot = current_session_for_find_debuginfo->sysroot +
>>>> "/*";
>>>> -      int    found   = fnmatch(sysroot.c_str(), file_name, 0);
>>>> -
>>>> -      if (found)
>>>> -       {
>>>> -         file_name = file_name
>>>> -           + current_session_for_find_debuginfo->sysroot.length() - 1;
>>>> -       }
>>>> -    }
>>>> -
>>>>    /* Call the original dwfl_standard_find_debuginfo */
>>>>    return dwfl_standard_find_debuginfo(mod, userdata, modname, base,
>>>>                file_name, debuglink_file,
>>>> --
>>>> 2.14.3
>>>>
>>>
>>>
>>>
>>> --
>>> David Smith
>>> Associate Manager
>>> Red Hat
>>>
>>
>



-- 
David Smith
Associate Manager
Red Hat

Re: [PATCH 2/6] sysroot: fix short release -r option handling in case of sysroot pass proper kernel modules

[David Smith]

I tweaked this one. First, in commit 41efad047 I made sure that
sysroot paths don't end in a '/'. That then simplifies this patch,
since we don't have to special case sysroot == "/". I checked this in
as commit a9ce89bcd.

On Mon, Mar 5, 2018 at 11:37 AM, Victor Kamensky <kamensky@cisco.com> wrote:
> In case of -r option getting short release (opposite to full path of
> kernel build tree) stap passes short release name to
> dwfl_linux_kernel_report_offline function and this function tries to
> look up symbols in the host file system ignoring sysroot setting.
>
> Fix: in case of non empty, non "/" sysroot construct full path to
> kernel modules directory and store it into elfutils_kernel_path variable.
>
> Here are steps how to reproduce the issue. Note <sysroot> is produced by
> yocto poky build.
>
> [kamensky@coreos-lnx2 tests]$ cat meminfo.stp
> probe kernel.function("meminfo_proc_show") {
>         println("meminfo_proc_show called")
> }
> [kamensky@coreos-lnx2 tests]$ /home/wd8/systemtap/20180208_2/packages/bin/stap --sysroot=/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs -a x86_64 -r 4.9.78-yocto-standard -B CROSS_COMPILE=x86_64-poky-linux- --sysenv=PATH=/usr/bin:/bin:/usr/sbin:/sbin --sysenv=LD_LIBRARY_PATH=/lib:/usr/lib -p4 -v -m meminfo meminfo.stp
> Pass 1: parsed user script and 480 library scripts using 230196virt/93452res/5364shr/88624data kb, in 340usr/30sys/375real ms.
> semantic error: while resolving probe point: identifier 'kernel' at meminfo.stp:1:7
>         source: probe kernel.function("meminfo_proc_show") {
>                       ^
>
> Running command under strace shows that stap is trying to open linux symbol file
> on host file system, instead of looking at sysroot location:
>
> 19517 openat(AT_FDCWD, "/boot/vmlinux-4.9.78-yocto-standard.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
> 19517 openat(AT_FDCWD, "/boot/vmlinux-4.9.78-yocto-standard", O_RDONLY) = -1 ENOENT (No such file or directory)
>
> Signed-off-by: Victor Kamensky <kamensky@cisco.com>
> ---
>  setupdwfl.cxx | 16 +++++++++++++---
>  1 file changed, 13 insertions(+), 3 deletions(-)
>
> diff --git a/setupdwfl.cxx b/setupdwfl.cxx
> index 11e0bb2af..f00cf755b 100644
> --- a/setupdwfl.cxx
> +++ b/setupdwfl.cxx
> @@ -359,9 +359,19 @@ setup_dwfl_kernel (unsigned *modules_found, systemtap_session &s)
>    // passs the plain kernel_release here.  So instead we have to
>    // hard-code this magic here.
>     string lib_path = "/lib/modules/" + s.kernel_release + "/build";
> -   if (s.kernel_build_tree == string(s.sysroot + lib_path) ||
> -       (s.kernel_build_tree == lib_path
> -       && s.sysroot == "/"))
> +   if (s.kernel_build_tree == string(s.sysroot + lib_path))
> +     {
> +       // If we have sysroot set does not make sense to pass
> +       // short release to dwfl, it won't take a sysroot into
> +       // account. Let's construct full path in such case.
> +       if (s.sysroot != "" && s.sysroot != "/")
> +        elfutils_kernel_path = string(s.sysroot + "/lib/modules/" + s.kernel_release);
> +       else
> +        elfutils_kernel_path = s.kernel_release;
> +     }
> +   else
> +     if (s.kernel_build_tree == lib_path
> +        && s.sysroot == "/")
>        elfutils_kernel_path = s.kernel_release;
>     else
>        elfutils_kernel_path = s.kernel_build_tree;
> --
> 2.14.3
>



-- 
David Smith
Associate Manager
Red Hat

Re: [PATCH 2/6] sysroot: fix short release -r option handling in case of sysroot pass proper kernel modules

[Victor Kamensky]

On Wed, 14 Mar 2018, David Smith wrote:

> I tweaked this one. First, in commit 41efad047 I made sure that
> sysroot paths don't end in a '/'. That then simplifies this patch,
> since we don't have to special case sysroot == "/". I checked this in
> as commit a9ce89bcd.

Thanks David! Appreciate progress on accepting patches.

Patches 4, 5, 6 from the series are still in queue, right?

PATCH 4: https://sourceware.org/ml/systemtap/2018-q1/msg00071.html
PATCH 5: https://sourceware.org/ml/systemtap/2018-q1/msg00066.html
PATCH 6: https://sourceware.org/ml/systemtap/2018-q1/msg00070.html

Thanks,
Victor

> On Mon, Mar 5, 2018 at 11:37 AM, Victor Kamensky <kamensky@cisco.com> wrote:
>> In case of -r option getting short release (opposite to full path of
>> kernel build tree) stap passes short release name to
>> dwfl_linux_kernel_report_offline function and this function tries to
>> look up symbols in the host file system ignoring sysroot setting.
>>
>> Fix: in case of non empty, non "/" sysroot construct full path to
>> kernel modules directory and store it into elfutils_kernel_path variable.
>>
>> Here are steps how to reproduce the issue. Note <sysroot> is produced by
>> yocto poky build.
>>
>> [kamensky@coreos-lnx2 tests]$ cat meminfo.stp
>> probe kernel.function("meminfo_proc_show") {
>>         println("meminfo_proc_show called")
>> }
>> [kamensky@coreos-lnx2 tests]$ /home/wd8/systemtap/20180208_2/packages/bin/stap --sysroot=/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs -a x86_64 -r 4.9.78-yocto-standard -B CROSS_COMPILE=x86_64-poky-linux- --sysenv=PATH=/usr/bin:/bin:/usr/sbin:/sbin --sysenv=LD_LIBRARY_PATH=/lib:/usr/lib -p4 -v -m meminfo meminfo.stp
>> Pass 1: parsed user script and 480 library scripts using 230196virt/93452res/5364shr/88624data kb, in 340usr/30sys/375real ms.
>> semantic error: while resolving probe point: identifier 'kernel' at meminfo.stp:1:7
>>         source: probe kernel.function("meminfo_proc_show") {
>>                       ^
>>
>> Running command under strace shows that stap is trying to open linux symbol file
>> on host file system, instead of looking at sysroot location:
>>
>> 19517 openat(AT_FDCWD, "/boot/vmlinux-4.9.78-yocto-standard.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
>> 19517 openat(AT_FDCWD, "/boot/vmlinux-4.9.78-yocto-standard", O_RDONLY) = -1 ENOENT (No such file or directory)
>>
>> Signed-off-by: Victor Kamensky <kamensky@cisco.com>
>> ---
>>  setupdwfl.cxx | 16 +++++++++++++---
>>  1 file changed, 13 insertions(+), 3 deletions(-)
>>
>> diff --git a/setupdwfl.cxx b/setupdwfl.cxx
>> index 11e0bb2af..f00cf755b 100644
>> --- a/setupdwfl.cxx
>> +++ b/setupdwfl.cxx
>> @@ -359,9 +359,19 @@ setup_dwfl_kernel (unsigned *modules_found, systemtap_session &s)
>>    // passs the plain kernel_release here.  So instead we have to
>>    // hard-code this magic here.
>>     string lib_path = "/lib/modules/" + s.kernel_release + "/build";
>> -   if (s.kernel_build_tree == string(s.sysroot + lib_path) ||
>> -       (s.kernel_build_tree == lib_path
>> -       && s.sysroot == "/"))
>> +   if (s.kernel_build_tree == string(s.sysroot + lib_path))
>> +     {
>> +       // If we have sysroot set does not make sense to pass
>> +       // short release to dwfl, it won't take a sysroot into
>> +       // account. Let's construct full path in such case.
>> +       if (s.sysroot != "" && s.sysroot != "/")
>> +        elfutils_kernel_path = string(s.sysroot + "/lib/modules/" + s.kernel_release);
>> +       else
>> +        elfutils_kernel_path = s.kernel_release;
>> +     }
>> +   else
>> +     if (s.kernel_build_tree == lib_path
>> +        && s.sysroot == "/")
>>        elfutils_kernel_path = s.kernel_release;
>>     else
>>        elfutils_kernel_path = s.kernel_build_tree;
>> --
>> 2.14.3
>>
>
>
>
> -- 
> David Smith
> Associate Manager
> Red Hat
>

Re: [PATCH 4/6] sysroot: delay adding sysroot path to module name in case of non absolute executable

[David Smith]

Checked in as commit 4ffecddf5. Thanks.

On Mon, Mar 5, 2018 at 11:37 AM, Victor Kamensky <kamensky@cisco.com> wrote:
> Current stap code adds sysroot prematurely for probes that specify
> non absolute path name, i.e like "foo", so when find_executable called
> it receives full path as <sysroot>/foo and find_executable does not
> search PATH while applying sysroot.
>
> Fix delays adding sysroot till path inside of sysroot is searched first.
>
> Also fix missing sysroot addition in glob expansion case.
>
> Note in case of sysroot cross compile environment it is highly recommended
> to pass --sysenv=PATH=xxx:yyy and --sysenv=LD_LIBRARY_PATH=zzz to use
> search path appropriate for target system, rather then host setting on
> system where stap runs.
>
> Here are steps how to reproduce the issue. Note <sysroot> is produced by
> yocto poky build.
>
> [kamensky@coreos-lnx2 tests]$ cat mkdir2.stp
> probe process("mkdir.coreutils").function("*").call {
>   printf ("%s -> %s\n", thread_indent(1), ppfunc())
> }
> probe process("mkdir.coreutils").function("*").return {
>   printf ("%s <- %s\n", thread_indent(-1), ppfunc())
> }
> [kamensky@coreos-lnx2 tests]$ ls /home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/mkdir.coreutils
> /home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/mkdir.coreutils
> [kamensky@coreos-lnx2 tests]$ ls /home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/.debug/mkdir.coreutils
> /home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/.debug/mkdir.coreutils
> [kamensky@coreos-lnx2 tests]$ /home/wd8/systemtap/20180208_2/packages/bin/stap --sysroot=/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs -a x86_64 -r 4.9.78-yocto-standard -B CROSS_COMPILE=x86_64-poky-linux- --sysenv=PATH=/usr/bin:/bin:/usr/sbin:/sbin --sysenv=LD_LIBRARY_PATH=/lib:/usr/lib -v -p4 -m mkdir2 mkdir2.stp
> Pass 1: parsed user script and 480 library scripts using 230196virt/93536res/5448shr/88624data kb, in 330usr/30sys/364real ms.
> semantic error: resolution failed in DWARF builder
>
> semantic error: while resolving probe point: identifier 'process' at mkdir2.stp:1:7
>         source: probe process("mkdir.coreutils").function("*").call {
>                       ^
>
> semantic error: no match
>
> semantic error: resolution failed in DWARF builder
>
> semantic error: while resolving probe point: identifier 'process' at :4:7
>         source: probe process("mkdir.coreutils").function("*").return {
>                       ^
>
> semantic error: no match
>
> Pass 2: analyzed script: 0 probes, 0 functions, 0 embeds, 0 globals using 233496virt/97032res/5520shr/91924data kb, in 20usr/0sys/25real ms.
> Pass 2: analysis failed.  [man error::pass2]
>
> Under strace it shows that stap is not trying to lookup mkdir.coreutils
> inside of sysroot accoring to PATH that was passed through --sysenv:
>
> 16048 openat(AT_FDCWD, "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/mkdir.coreutils", O_RDONLY) = -1 ENOENT (No such file or directory)
>
> i.e it rather just tries <sysroot> + "mkdir.coreutils"
>
> After the fix veried that that above example able to find mkdir.coreutils
> executable in sysroot and corresponding symbols
>
> Also verified that glob matching in sysroot works too for example for
> this test case:
>
> [kamensky@coreos-lnx2 tests]$ cat mkdir3.stp
> probe process("/bin/mkdi*").function("*").call {
>   printf ("%s -> %s\n", thread_indent(1), ppfunc())
> }
> probe process("/bin/mkdi*").function("*").return {
>   printf ("%s <- %s\n", thread_indent(-1), ppfunc())
> }
>
> Signed-off-by: Victor Kamensky <kamensky@cisco.com>
> ---
>  tapsets.cxx | 8 ++++----
>  1 file changed, 4 insertions(+), 4 deletions(-)
>
> diff --git a/tapsets.cxx b/tapsets.cxx
> index 28296b14f..c664df0f9 100644
> --- a/tapsets.cxx
> +++ b/tapsets.cxx
> @@ -746,7 +746,7 @@ base_query::base_query(dwflpp & dw, literal_map_t const & params):
>                pid_val = 0;
>                get_string_param(params, TOK_PROCESS, module_val);
>              }
> -          module_val = find_executable (module_val, "", sess.sysenv);
> +          module_val = find_executable (module_val, sess.sysroot, sess.sysenv);
>            if (!is_fully_resolved(module_val, "", sess.sysenv))
>              throw SEMANTIC_ERROR(_F("cannot find executable '%s'",
>                                      module_val.to_string().c_str()));
> @@ -8293,7 +8293,6 @@ dwarf_builder::build(systemtap_session & sess,
>              }
>            else
>              {
> -              module_name = (string)sess.sysroot + (string)module_name;
>                filled_parameters[TOK_PROCESS] = new literal_string(module_name);
>              }
>          }
> @@ -8327,7 +8326,7 @@ dwarf_builder::build(systemtap_session & sess,
>            assert (lit);
>
>            // Evaluate glob here, and call derive_probes recursively with each match.
> -          const auto& globs = glob_executable (module_name);
> +          const auto& globs = glob_executable (sess.sysroot + string(module_name));
>            unsigned results_pre = finished_results.size();
>            for (auto it = globs.begin(); it != globs.end(); ++it)
>              {
> @@ -8418,7 +8417,8 @@ dwarf_builder::build(systemtap_session & sess,
>
>        // PR13338: unquote glob results
>        module_name = unescape_glob_chars (module_name);
> -      user_path = find_executable (module_name, "", sess.sysenv); // canonicalize it
> +      user_path = find_executable (module_name, sess.sysroot,
> +                                  sess.sysenv); // canonicalize it
>        if (!is_fully_resolved(user_path, "", sess.sysenv))
>          throw SEMANTIC_ERROR(_F("cannot find executable '%s'",
>                                  user_path.to_string().c_str()));
> --
> 2.14.3
>



-- 
David Smith
Associate Manager
Red Hat

Re: [PATCH 4/6] sysroot: delay adding sysroot path to module name in case of non absolute executable

[David Smith]

Sigh. I had just made your patch changes by hand, and somehow I
reversed the last one. Commit a71465872 fixes that, and adds a small
cosmetic fix to path_remove_sysroot().

On Mon, Mar 19, 2018 at 8:54 AM, David Smith <dsmith@redhat.com> wrote:
> Checked in as commit 4ffecddf5. Thanks.
>
> On Mon, Mar 5, 2018 at 11:37 AM, Victor Kamensky <kamensky@cisco.com> wrote:
>> Current stap code adds sysroot prematurely for probes that specify
>> non absolute path name, i.e like "foo", so when find_executable called
>> it receives full path as <sysroot>/foo and find_executable does not
>> search PATH while applying sysroot.
>>
>> Fix delays adding sysroot till path inside of sysroot is searched first.
>>
>> Also fix missing sysroot addition in glob expansion case.
>>
>> Note in case of sysroot cross compile environment it is highly recommended
>> to pass --sysenv=PATH=xxx:yyy and --sysenv=LD_LIBRARY_PATH=zzz to use
>> search path appropriate for target system, rather then host setting on
>> system where stap runs.
>>
>> Here are steps how to reproduce the issue. Note <sysroot> is produced by
>> yocto poky build.
>>
>> [kamensky@coreos-lnx2 tests]$ cat mkdir2.stp
>> probe process("mkdir.coreutils").function("*").call {
>>   printf ("%s -> %s\n", thread_indent(1), ppfunc())
>> }
>> probe process("mkdir.coreutils").function("*").return {
>>   printf ("%s <- %s\n", thread_indent(-1), ppfunc())
>> }
>> [kamensky@coreos-lnx2 tests]$ ls /home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/mkdir.coreutils
>> /home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/mkdir.coreutils
>> [kamensky@coreos-lnx2 tests]$ ls /home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/.debug/mkdir.coreutils
>> /home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/.debug/mkdir.coreutils
>> [kamensky@coreos-lnx2 tests]$ /home/wd8/systemtap/20180208_2/packages/bin/stap --sysroot=/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs -a x86_64 -r 4.9.78-yocto-standard -B CROSS_COMPILE=x86_64-poky-linux- --sysenv=PATH=/usr/bin:/bin:/usr/sbin:/sbin --sysenv=LD_LIBRARY_PATH=/lib:/usr/lib -v -p4 -m mkdir2 mkdir2.stp
>> Pass 1: parsed user script and 480 library scripts using 230196virt/93536res/5448shr/88624data kb, in 330usr/30sys/364real ms.
>> semantic error: resolution failed in DWARF builder
>>
>> semantic error: while resolving probe point: identifier 'process' at mkdir2.stp:1:7
>>         source: probe process("mkdir.coreutils").function("*").call {
>>                       ^
>>
>> semantic error: no match
>>
>> semantic error: resolution failed in DWARF builder
>>
>> semantic error: while resolving probe point: identifier 'process' at :4:7
>>         source: probe process("mkdir.coreutils").function("*").return {
>>                       ^
>>
>> semantic error: no match
>>
>> Pass 2: analyzed script: 0 probes, 0 functions, 0 embeds, 0 globals using 233496virt/97032res/5520shr/91924data kb, in 20usr/0sys/25real ms.
>> Pass 2: analysis failed.  [man error::pass2]
>>
>> Under strace it shows that stap is not trying to lookup mkdir.coreutils
>> inside of sysroot accoring to PATH that was passed through --sysenv:
>>
>> 16048 openat(AT_FDCWD, "/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/mkdir.coreutils", O_RDONLY) = -1 ENOENT (No such file or directory)
>>
>> i.e it rather just tries <sysroot> + "mkdir.coreutils"
>>
>> After the fix veried that that above example able to find mkdir.coreutils
>> executable in sysroot and corresponding symbols
>>
>> Also verified that glob matching in sysroot works too for example for
>> this test case:
>>
>> [kamensky@coreos-lnx2 tests]$ cat mkdir3.stp
>> probe process("/bin/mkdi*").function("*").call {
>>   printf ("%s -> %s\n", thread_indent(1), ppfunc())
>> }
>> probe process("/bin/mkdi*").function("*").return {
>>   printf ("%s <- %s\n", thread_indent(-1), ppfunc())
>> }
>>
>> Signed-off-by: Victor Kamensky <kamensky@cisco.com>
>> ---
>>  tapsets.cxx | 8 ++++----
>>  1 file changed, 4 insertions(+), 4 deletions(-)
>>
>> diff --git a/tapsets.cxx b/tapsets.cxx
>> index 28296b14f..c664df0f9 100644
>> --- a/tapsets.cxx
>> +++ b/tapsets.cxx
>> @@ -746,7 +746,7 @@ base_query::base_query(dwflpp & dw, literal_map_t const & params):
>>                pid_val = 0;
>>                get_string_param(params, TOK_PROCESS, module_val);
>>              }
>> -          module_val = find_executable (module_val, "", sess.sysenv);
>> +          module_val = find_executable (module_val, sess.sysroot, sess.sysenv);
>>            if (!is_fully_resolved(module_val, "", sess.sysenv))
>>              throw SEMANTIC_ERROR(_F("cannot find executable '%s'",
>>                                      module_val.to_string().c_str()));
>> @@ -8293,7 +8293,6 @@ dwarf_builder::build(systemtap_session & sess,
>>              }
>>            else
>>              {
>> -              module_name = (string)sess.sysroot + (string)module_name;
>>                filled_parameters[TOK_PROCESS] = new literal_string(module_name);
>>              }
>>          }
>> @@ -8327,7 +8326,7 @@ dwarf_builder::build(systemtap_session & sess,
>>            assert (lit);
>>
>>            // Evaluate glob here, and call derive_probes recursively with each match.
>> -          const auto& globs = glob_executable (module_name);
>> +          const auto& globs = glob_executable (sess.sysroot + string(module_name));
>>            unsigned results_pre = finished_results.size();
>>            for (auto it = globs.begin(); it != globs.end(); ++it)
>>              {
>> @@ -8418,7 +8417,8 @@ dwarf_builder::build(systemtap_session & sess,
>>
>>        // PR13338: unquote glob results
>>        module_name = unescape_glob_chars (module_name);
>> -      user_path = find_executable (module_name, "", sess.sysenv); // canonicalize it
>> +      user_path = find_executable (module_name, sess.sysroot,
>> +                                  sess.sysenv); // canonicalize it
>>        if (!is_fully_resolved(user_path, "", sess.sysenv))
>>          throw SEMANTIC_ERROR(_F("cannot find executable '%s'",
>>                                  user_path.to_string().c_str()));
>> --
>> 2.14.3
>>
>
>
>
> --
> David Smith
> Associate Manager
> Red Hat



-- 
David Smith
Associate Manager
Red Hat

Re: [PATCH 5/6] sysroot: handle symbolic links with absolute name relative to sysroot

[David Smith]

Fix checked in as commit 2041085d1. Thanks!

On Mon, Mar 5, 2018 at 11:37 AM, Victor Kamensky <kamensky@cisco.com> wrote:
> In case of symbolic link found under sysroot point to absolute path,
> instead of trying to look for such absolute path in host system,
> apply sysroot prefix first.
>
> Here are steps how to reproduce the issue. Note <sysroot> is produced by
> yocto poky build.
>
> kamensky@coreos-lnx2 tests]$ ls -l /home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/mkdir
> lrwxrwxrwx. 1 kamensky kamensky 20 Jan 30 18:29 /home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/mkdir -> /bin/mkdir.coreutils
> [kamensky@coreos-lnx2 tests]$ ls -l /home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/mkdir.coreutils
> -rwxr-xr-x. 1 kamensky kamensky 88232 Jan 29 09:58 /home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/mkdir.coreutils
> [kamensky@coreos-lnx2 tests]$ ls -l /home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/.debug/mkdir.coreutils
> -rwxr-xr-x. 1 kamensky kamensky 383456 Jan 29 09:58 /home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs/bin/.debug/mkdir.coreutils
> [kamensky@coreos-lnx2 tests]$ cat mkdir4.stp
> probe process("/bin/mkdir").function("*").call {
>   printf ("%s -> %s\n", thread_indent(1), ppfunc())
> }
> probe process("/bin/mkdir").function("*").return {
>   printf ("%s <- %s\n", thread_indent(-1), ppfunc())
> }
> [kamensky@coreos-lnx2 tests]$ /home/wd8/systemtap/20180208_2/packages/bin/stap --sysroot=/home/wd8/yocto/20180128/build_x86_64/tmp/work/intel_corei7_64-poky-linux/kdevel-console-devel-image/1.0-r0/rootfs -a x86_64 -r 4.9.78-yocto-standard -B CROSS_COMPILE=x86_64-poky-linux- --sysenv=PATH=/usr/bin:/bin:/usr/sbin:/sbin --sysenv=LD_LIBRARY_PATH=/lib:/usr/lib -v -p4 -m mkdir4 mkdir4.stp
> Pass 1: parsed user script and 480 library scripts using 230196virt/93484res/5396shr/88624data kb, in 340usr/30sys/370real ms.
> semantic error: resolution failed in DWARF builder
>
> semantic error: while resolving probe point: identifier 'process' at mkdir4.stp:1:7
>         source: probe process("/bin/mkdir").function("*").call {
>                       ^
>
> semantic error: no match
>
> semantic error: resolution failed in DWARF builder
>
> semantic error: while resolving probe point: identifier 'process' at :4:7
>         source: probe process("/bin/mkdir").function("*").return {
>                       ^
>
> semantic error: no match
>
> Pass 2: analyzed script: 0 probes, 0 functions, 0 embeds, 0 globals using 233496virt/96980res/5468shr/91924data kb, in 20usr/0sys/25real ms.
> Pass 2: analysis failed.  [man error::pass2]
>
> After the fix above sript works fine verified that it traces mkdir
> functions on target.
>
> Signed-off-by: Victor Kamensky <kamensky@cisco.com>
> ---
>  util.cxx | 76 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 76 insertions(+)
>
> diff --git a/util.cxx b/util.cxx
> index 2724be956..f48a8884f 100644
> --- a/util.cxx
> +++ b/util.cxx
> @@ -443,6 +443,64 @@ split_lines(const char *buf, size_t n)
>    return lines;
>  }
>
> +static string
> +follow_link(const string& name, const string& sysroot)
> +{
> +  char *linkname;
> +  ssize_t r;
> +  string retpath;
> +  struct stat st;
> +
> +  const char *f = name.c_str();
> +
> +  lstat(f, &st);
> +
> +  linkname = (char *) malloc(st.st_size + 1);
> +
> +  if (linkname)
> +    {
> +      r = readlink(f, linkname, st.st_size + 1);
> +      linkname[st.st_size] = '\0';
> +      /*
> +       * If we have non-empty sysroot and we got link that
> +       * points to absolute path name, we need to look at
> +       * this path relative to sysroot itself. access and
> +       * stat will follow symbolic links correctly only in
> +       * case with empty sysroot.
> +       */
> +      while (r != -1 && linkname && linkname[0] == '/')
> +       {
> +         string fname1 = sysroot + linkname;
> +         const char *f1 = fname1.c_str();
> +         if (access(f1, X_OK) == 0
> +             && stat(f1, &st) == 0
> +             && S_ISREG(st.st_mode))
> +           {
> +             retpath = fname1;
> +             break;
> +           }
> +         else if (lstat(f1, &st) == 0
> +                  && S_ISLNK(st.st_mode))
> +           {
> +             free(linkname);
> +             linkname = (char *) malloc(st.st_size + 1);
> +             if (linkname)
> +               {
> +                 r = readlink(f1, linkname, st.st_size + 1);
> +                 linkname[st.st_size] = '\0';
> +               }
> +           }
> +         else
> +           {
> +             break;
> +           }
> +       }
> +    }
> +  free(linkname);
> +
> +  return retpath;
> +}
> +
>  // Resolve an executable name to a canonical full path name, with the
>  // same policy as execvp().  A program name not containing a slash
>  // will be searched along the $PATH.
> @@ -467,6 +525,14 @@ string find_executable(const string& name, const string& sysroot,
>    if (name.find('/') != string::npos) // slash in the path already?
>      {
>        retpath = sysroot + name;
> +
> +      const char *f = retpath.c_str();
> +      if (sysroot != ""
> +         && lstat(f, &st) == 0
> +         && S_ISLNK(st.st_mode))
> +       {
> +         retpath = follow_link(f, sysroot);
> +       }
>      }
>    else // Nope, search $PATH.
>      {
> @@ -495,6 +561,16 @@ string find_executable(const string& name, const string& sysroot,
>                    retpath = fname;
>                    break;
>                  }
> +              else if (sysroot != ""
> +                       && lstat(f, &st) == 0
> +                       && S_ISLNK(st.st_mode))
> +               {
> +                 retpath = follow_link(f, sysroot);
> +                 if (retpath != "")
> +                   {
> +                     break;
> +                   }
> +               }
>              }
>          }
>      }
> --
> 2.14.3
>



-- 
David Smith
Associate Manager
Red Hat

Re: [PATCH 0/6] systemtap set of --sysroot option handling fixes

[David Smith]

Victor,

I've checked in all your patches at this point. If anything still
doesn't work, please let me know.

In addition, I tried to add test cases for all the items you found in
commit c96dfba71:

<https://sourceware.org/git/gitweb.cgi?p=systemtap.git;a=commit;h=c96dfba71f882b6f3bfcc0a844c514e10d210fcc>

If you get a chance, I'd appreciate you looking it over and see if
there is anything else you think we should test.

Thanks again for all the patches!


On Mon, Mar 5, 2018 at 11:37 AM, Victor Kamensky <kamensky@cisco.com> wrote:
> Hi Folks,
>
> Here is set of fixes for SystemTap --sysroot option handling.
>
> Please look at individual patches decsription for issues observed and fixed.
>
> If commit messages are too details and big for SystemTap project
> guidelines feel free to trim them. Commit message structured in such
> way that text after "Here are steps how to reproduce the issue." could
> be removed if needed.
>
> I did not run full SystemTap regression with --sysroot option since SystemTap
> DejaGnu testsuites are not friendly to remote execution mode. I did run
> regular 'make installcheck' on the same baseline with and without changes
> and made sure that results are identical.
>
> If you would like to try stap in cross compiled environment with --sysroot
> option I've set up openembedded-core repository where these changes are
> integrated and since openembedded-core support virtual qemu based target
> for different CPU types stap with --sysroot option could be tested.
>
> Setup instructions are here:
>     https://github.com/victorkamensky/systemtap-oe-sysroot-manifest
>
> Victor Kamensky (6):
>   sysroot: fix short release -r option handling in case of sysroot read
>     System.map symbols
>   sysroot: fix short release -r option handling in case of sysroot pass
>     proper kernel modules
>   sysroot: debuginfo lookup with sysroot case do not remove sysroot from
>     file_name
>   sysroot: delay adding sysroot path to module name in case of non
>     absolute executable
>   sysroot: handle symbolic links with absolute name relative to sysroot
>   sysroot: _stp_umodule_relocate needs receive target file path
>
>  loc2stap.cxx  |  3 ++-
>  session.cxx   |  2 +-
>  setupdwfl.cxx | 28 ++++++++++------------
>  tapsets.cxx   |  8 +++----
>  util.cxx      | 76 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  5 files changed, 96 insertions(+), 21 deletions(-)
>
> --
> 2.14.3
>



-- 
David Smith
Associate Manager
Red Hat

Re: [PATCH 0/6] systemtap set of --sysroot option handling fixes

[Victor Kamensky]

Hi David,

On Wed, 21 Mar 2018, David Smith wrote:

> Victor,
>
> I've checked in all your patches at this point. If anything still
> doesn't work, please let me know.

Appreciate very much time you spent integrating patches.

I went through all my test cases on latest systemtap tree.
All fixes except test case for 'PATCH 2' [1] looks good.

'PATCH 2' [1] for sysroot and short -r option did not come out
quite right. For this combination systemtap code still calls
dwfl_linux_kernel_report_offline function with short release
version stored into elfutils_kernel_path and it cannot find
kernel or kernel modules.

In the case of sysroot
and short -r option absolute path should be constructed
instead, in order for dwfl_linux_kernel_report_offline to
pick up kernel modules inside of sysroot.

I will post follow up adjustment for PATCH 2 issue right
after this email. If you would like to tweak it please send
patch candidate back and I will give it a spin on my test
case.

> In addition, I tried to add test cases for all the items you found in
> commit c96dfba71:
>
> <https://sourceware.org/git/gitweb.cgi?p=systemtap.git;a=commit;h=c96dfba71f882b6f3bfcc0a844c514e10d210fcc>

Yes, I saw it. It looks great.

> If you get a chance, I'd appreciate you looking it over and see if
> there is anything else you think we should test.

Will do.

[1] https://sourceware.org/ml/systemtap/2018-q1/msg00067.html

Thanks,
Victor

> Thanks again for all the patches!
>
>
> On Mon, Mar 5, 2018 at 11:37 AM, Victor Kamensky <kamensky@cisco.com> wrote:
>> Hi Folks,
>>
>> Here is set of fixes for SystemTap --sysroot option handling.
>>
>> Please look at individual patches decsription for issues observed and fixed.
>>
>> If commit messages are too details and big for SystemTap project
>> guidelines feel free to trim them. Commit message structured in such
>> way that text after "Here are steps how to reproduce the issue." could
>> be removed if needed.
>>
>> I did not run full SystemTap regression with --sysroot option since SystemTap
>> DejaGnu testsuites are not friendly to remote execution mode. I did run
>> regular 'make installcheck' on the same baseline with and without changes
>> and made sure that results are identical.
>>
>> If you would like to try stap in cross compiled environment with --sysroot
>> option I've set up openembedded-core repository where these changes are
>> integrated and since openembedded-core support virtual qemu based target
>> for different CPU types stap with --sysroot option could be tested.
>>
>> Setup instructions are here:
>>     https://github.com/victorkamensky/systemtap-oe-sysroot-manifest
>>
>> Victor Kamensky (6):
>>   sysroot: fix short release -r option handling in case of sysroot read
>>     System.map symbols
>>   sysroot: fix short release -r option handling in case of sysroot pass
>>     proper kernel modules
>>   sysroot: debuginfo lookup with sysroot case do not remove sysroot from
>>     file_name
>>   sysroot: delay adding sysroot path to module name in case of non
>>     absolute executable
>>   sysroot: handle symbolic links with absolute name relative to sysroot
>>   sysroot: _stp_umodule_relocate needs receive target file path
>>
>>  loc2stap.cxx  |  3 ++-
>>  session.cxx   |  2 +-
>>  setupdwfl.cxx | 28 ++++++++++------------
>>  tapsets.cxx   |  8 +++----
>>  util.cxx      | 76 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>  5 files changed, 96 insertions(+), 21 deletions(-)
>>
>> --
>> 2.14.3
>>
>
>
>
> -- 
> David Smith
> Associate Manager
> Red Hat
>