Re: [RFC] elfutils: Checks for debuginfo file without .debug extension as well

Re: [RFC] elfutils: Checks for debuginfo file without .debug extension as well

[Ravi Bangoria]

[-- Attachment #1: Type: text/plain, Size: 2576 bytes --]

HI Mark,

Thanks for giving detailed analysis. Please find my comments.

On Wednesday 17 February 2016 09:19 PM, Mark Wielaard wrote:
> Hi Ravi,
>
> My apologies for being slightly confused. I think what you are proposing
> with your patch is the right thing. I am just slightly confused because
> I thought the current code already did what you are proposing. So the
> current code doesn't actually do what I thought it did :)
>
> I have attached a simplified version of what I believe systemtap is
> doing. It is a two step process. First it calls
> dwfl_linux_kernel_report_offline to get the kernel file itself. Second
> it gets the Dwarf for the kernel module, which should trigger
> find_debuginfo to find the separate debug file, if needed.

Thanks for the sample program. I can see it tries to simulate what 
systemtap do.
Here is the output on ubunut powerpc.

Without patch:

   # ./dwfl_find_kernel
   Finding kernel release: 3.13.0-76-generic
   Debuginfo path: +:.debug:/usr/lib/debug
   setup report modname: kernel, filename: /boot/vmlinux-3.13.0-76-generic
   setup report modname: xfrm_user, filename: 
/lib/modules/3.13.0-76-generic/kernel/net/xfrm/xfrm_user.ko
   Found kernel file: /boot/vmlinux-3.13.0-76-generic
   getdwarf name: kernel
   kernel without DWARF
   Couldn't get kernel DWARF


With patch:

   # ./dwfl_find_kernel
   Finding kernel release: 3.13.0-76-generic
   Debuginfo path: +:.debug:/usr/lib/debug
   setup report modname: kernel, filename: /boot/vmlinux-3.13.0-76-generic
   setup report modname: xfrm_user, filename: 
/lib/modules/3.13.0-76-generic/kernel/net/xfrm/xfrm_user.ko
   Found kernel file: /boot/vmlinux-3.13.0-76-generic
   getdwarf name: kernel
   mainfile: /boot/vmlinux-3.13.0-76-generic, debugfile: 
/usr/lib/debug/boot/vmlinux-3.13.0-76-generic
   Found kernel DWARF file: /usr/lib/debug/boot/vmlinux-3.13.0-76-generic


> It still doesn't show the full search path (for that we should hack
> find_debuginfo_in_path and try_open in libdwfl/find-debuginfo.c a bit),
> but it should be a start to better understand why the current search
> isn't finding the separate kernel debug file.

Sry, I'm little bit confused in your last two paragraphs.

1. Does this means change I've proposed is at wrong place i.e. 
find_debuginfo
is not correct place to change?

2. I'm not able to understand why kernel module came into picture here. 
when we do
         probe kernel.function("do_fork") {...}
stap looks for kernel debuginfo only, right?

Regards,
Ravi

Re: [RFC] elfutils: Checks for debuginfo file without .debug extension as well

[Mark Wielaard]

[-- Attachment #1: Type: text/plain, Size: 406 bytes --]

On Mon, Feb 22, 2016 at 10:42:47PM +0530, Ravi Bangoria wrote:
> >Updated patch attached. This time with updated commit message and
> >ChangeLog entry. Does this look correct to you?
> 
> This patch looks fine to me. I've tested it on Ubuntu. I think we can go
> ahead with this.

Great, thanks for testing and the various reviews and suggestions.
I have pushed it to master now.

Cheers,

Mark

Re: [RFC] elfutils: Checks for debuginfo file without .debug extension as well

[Ravi Bangoria]

[-- Attachment #1: Type: text/plain, Size: 3606 bytes --]

Thanks Mark,

Please find my comments.

On Monday 22 February 2016 07:15 PM, Mark Wielaard wrote:
> Hi Ravi,
>
> On Sat, 2016-02-20 at 19:10 +0530, Ravi Bangoria wrote:
>> Thanks for the patch. Indeed this is optimized one compared to what I've
>> proposed.
>>
>> On Friday 19 February 2016 08:41 PM, Mark Wielaard wrote:
>>> Thanks, that looks like what I expected.
>>> I also got access to a somewhat newer version of ubuntu ppc64le and that
>>> has an additional issue. It has the vmlinux file installed unreadable
>>> (except for root). That causes almost the same issue, but slightly
>>> differently, now no kernel at all is found... Although in that case it
>>> can be worked around be using /usr/lib/debug as base. But the main issue
>>> is exactly as you describe.
>> Can you please provide me system configurations like ubuntu versrion,
>> kernel version, stap version, elfutils version etc. I'll check if I'll
>> be able to get similar system.
> Ubuntu 15.10 wily. 4.2.0-27-generic #32-Ubuntu SMP ppc64le
> elfutils-0.163-4ubuntu1 stap from systemtap git.

Yes you are right. I checked it. stap is not working on system with
above configurations.

>
>>> Could you try out if this variant of the patch works for you?
>> I've tested your patch and it's working fine with Ubuntu 14.04 with
>> kernel 3.13,
>> stap 2.3 and elfutils 0.165.
> Great, thanks for testing.
>
>> But I've a little doubt here. Here is the code snip of try_kernel_name
>> from libdwfl/linux-kernel-modules.c
>>
>>       static int
>>       try_kernel_name (Dwfl *dwfl, char **fname, bool try_debug)
>>       {
>> LINE A:
>>         int fd = ((((dwfl->callbacks->debuginfo_path
>>                      ? *dwfl->callbacks->debuginfo_path : NULL)
>>                     ?: DEFAULT_DEBUGINFO_PATH)[0] == ':') ? -1
>>                   : TEMP_FAILURE_RETRY (open64 (*fname, O_RDONLY)));
>>
>>         if (fd < 0)
>>           {
>> LINE B:
>>             /* look for "vmlinux" files.  */
>>             fd = INTUSE(dwfl_standard_find_debuginfo) (&fakemod, NULL,
>> NULL, 0,
>>                                                        *fname, basename
>> (*fname), 0,
>> &fakemod.debug.name);
>>             if (fd < 0 && try_debug)
>> LINE C:
>>               /* look for "vmlinux.debug" files.  */
>>               fd = INTUSE(dwfl_standard_find_debuginfo) (&fakemod, NULL,
>> NULL, 0,
>>                                                          *fname, NULL, 0,
>> &fakemod.debug.name);
>>
>> try_kernel_name is doing almost same thing what I have proposed. Now let's
>> say we want to go ahead with your patch, than call to dwfl_standard_find_debuginfo
>> in LINE C will look for both vmlinux and vmlinux.debug right? But it has
>> already checked for vmlinux in LINE B. So, in this case we have to modify
>> try_kernel_name as well.
> Interesting find. And I think you are right.
>
> In our case on ubuntu ppc64le, LINE A would find the vmlinux image
> already and we would never reach LINE B or C. The separate debuginfo
> would then be found, through dwfl_standard_find_debuginfo, when we want
> to get the DWARF for the kernel.
>
> But on other arches where the main image isn't called vmlinux we would
> indeed hit them. When try_debug == true then we only need to do C now,
> otherwise we only need to do B.
>
> Updated patch attached. This time with updated commit message and
> ChangeLog entry. Does this look correct to you?

This patch looks fine to me. I've tested it on Ubuntu. I think we can go 
ahead
with this.

Regards,
Ravi

Re: [RFC] elfutils: Checks for debuginfo file without .debug extension as well

[Mark Wielaard]

[-- Attachment #1: Type: text/plain, Size: 3241 bytes --]

Hi Ravi,

On Sat, 2016-02-20 at 19:10 +0530, Ravi Bangoria wrote:
> Thanks for the patch. Indeed this is optimized one compared to what I've
> proposed.
> 
> On Friday 19 February 2016 08:41 PM, Mark Wielaard wrote:
> > Thanks, that looks like what I expected.
> > I also got access to a somewhat newer version of ubuntu ppc64le and that
> > has an additional issue. It has the vmlinux file installed unreadable
> > (except for root). That causes almost the same issue, but slightly
> > differently, now no kernel at all is found... Although in that case it
> > can be worked around be using /usr/lib/debug as base. But the main issue
> > is exactly as you describe.
> 
> Can you please provide me system configurations like ubuntu versrion,
> kernel version, stap version, elfutils version etc. I'll check if I'll 
> be able to get similar system.

Ubuntu 15.10 wily. 4.2.0-27-generic #32-Ubuntu SMP ppc64le
elfutils-0.163-4ubuntu1 stap from systemtap git.

> > Could you try out if this variant of the patch works for you?
> 
> I've tested your patch and it's working fine with Ubuntu 14.04 with 
> kernel 3.13,
> stap 2.3 and elfutils 0.165.

Great, thanks for testing.

> But I've a little doubt here. Here is the code snip of try_kernel_name
> from libdwfl/linux-kernel-modules.c
> 
>      static int
>      try_kernel_name (Dwfl *dwfl, char **fname, bool try_debug)
>      {
> LINE A:
>        int fd = ((((dwfl->callbacks->debuginfo_path
>                     ? *dwfl->callbacks->debuginfo_path : NULL)
>                    ?: DEFAULT_DEBUGINFO_PATH)[0] == ':') ? -1
>                  : TEMP_FAILURE_RETRY (open64 (*fname, O_RDONLY)));
> 
>        if (fd < 0)
>          {
> LINE B:
>            /* look for "vmlinux" files.  */
>            fd = INTUSE(dwfl_standard_find_debuginfo) (&fakemod, NULL, 
> NULL, 0,
>                                                       *fname, basename 
> (*fname), 0,
> &fakemod.debug.name);
>            if (fd < 0 && try_debug)
> LINE C:
>              /* look for "vmlinux.debug" files.  */
>              fd = INTUSE(dwfl_standard_find_debuginfo) (&fakemod, NULL, 
> NULL, 0,
>                                                         *fname, NULL, 0,
> &fakemod.debug.name);
> 
> try_kernel_name is doing almost same thing what I have proposed. Now let's
> say we want to go ahead with your patch, than call to dwfl_standard_find_debuginfo
> in LINE C will look for both vmlinux and vmlinux.debug right? But it has 
> already checked for vmlinux in LINE B. So, in this case we have to modify 
> try_kernel_name as well.

Interesting find. And I think you are right.

In our case on ubuntu ppc64le, LINE A would find the vmlinux image
already and we would never reach LINE B or C. The separate debuginfo
would then be found, through dwfl_standard_find_debuginfo, when we want
to get the DWARF for the kernel.

But on other arches where the main image isn't called vmlinux we would
indeed hit them. When try_debug == true then we only need to do C now,
otherwise we only need to do B.

Updated patch attached. This time with updated commit message and
ChangeLog entry. Does this look correct to you?

Thanks,

Mark

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-libdwfl-Checks-for-kernel-debuginfo-file-without-.de.patch --]
[-- Type: text/x-patch, Size: 5447 bytes --]

From 786d65d8848efaa9d62b4121790c520d3a6782fb Mon Sep 17 00:00:00 2001
From: Ravi Bangoria <ravi.bangoria@linux.vnet.ibm.com>
Date: Tue, 16 Feb 2016 21:51:35 +0530
Subject: [PATCH] libdwfl: Checks for (kernel) debuginfo file without .debug
 extension as well

Elfutils, looking for kernel debuginfo file, tries to find it at
various places. If elfutils finds /boot/vmlinu*x* file, it checks
for debufginfo section. If debuginfo is not present, it saves it as
'main elf' and continue looking for debuginfo file having .debug
extension i.e. vmlinux-RELEASE.debug.

'Ubuntu on powerpc' installs kernel as /boot/vmlinux and installs
debuginfo without any extension as /usr/lib/debug/boot/vmlinux-RELEASE
and hence, elfutils is not able to find the debuginfo file.

Here is the lunchpad bug for the same:
  https://bugs.launchpad.net/ubuntu/+source/systemtap/+bug/1537125

This patch adds functionality to search for a kernel or debuginfo file
both with and without .debug extension.

Signed-off-by: Ravi Bangoria <ravi.bangoria@linux.vnet.ibm.com>
Signed-off-by: Mark Wielaard <mjw@redhat.com>
---
 libdwfl/ChangeLog              | 10 ++++++++++
 libdwfl/find-debuginfo.c       | 15 ++++++++++++++-
 libdwfl/linux-kernel-modules.c | 20 ++++++++++++--------
 3 files changed, 36 insertions(+), 9 deletions(-)

diff --git a/libdwfl/ChangeLog b/libdwfl/ChangeLog
index 69fd233..462a61d 100644
--- a/libdwfl/ChangeLog
+++ b/libdwfl/ChangeLog
@@ -1,3 +1,13 @@
+2016-02-22  Ravi Bangoria <ravi.bangoria@linux.vnet.ibm.com>
+	    Mark Wielaard  <mjw@redhat.com>
+
+	* find-debuginfo.c (find_debuginfo_in_path): Remember whether
+	debuglink_file is NULL. Try file basename (without .debug) ofr
+	absolute and relative path if debug_file was NULL.
+	* linux-kernel-modules.c (try_kernel_name): If try_debug is true call
+	dwfl_standard_find_debuginfo with NULL debuglink_file, otherwise with
+	basename of fname.
+
 2016-02-13  Mark Wielaard  <mjw@redhat.com>
 
 	* linux-proc-maps.c (proc_maps_report): Free last_file when ENOEXEC.
diff --git a/libdwfl/find-debuginfo.c b/libdwfl/find-debuginfo.c
index 72461bc..80515db 100644
--- a/libdwfl/find-debuginfo.c
+++ b/libdwfl/find-debuginfo.c
@@ -163,7 +163,11 @@ find_debuginfo_in_path (Dwfl_Module *mod, const char *file_name,
 
   const char *file_basename = file_name == NULL ? NULL : basename (file_name);
   char *localname = NULL;
-  if (debuglink_file == NULL)
+
+  /* We invent a debuglink .debug name if NULL, but then want to try the
+     basename too.  */
+  bool debuglink_null = debuglink_file == NULL;
+  if (debuglink_null)
     {
       /* For a alt debug multi file we need a name, for a separate debug
 	 name we may be able to fall back on file_basename.debug.  */
@@ -231,6 +235,10 @@ find_debuginfo_in_path (Dwfl_Module *mod, const char *file_name,
 	check = *p++ == '+';
       check = check && cancheck;
 
+      /* Try the basename too, if we made up the debuglink name and this
+	 is not the main directory.  */
+      bool try_file_basename;
+
       const char *dir, *subdir, *file;
       switch (p[0])
 	{
@@ -239,6 +247,7 @@ find_debuginfo_in_path (Dwfl_Module *mod, const char *file_name,
 	  dir = file_dirname;
 	  subdir = NULL;
 	  file = debuglink_file;
+	  try_file_basename = false;
 	  break;
 	case '/':
 	  /* An absolute path says to look there for a subdirectory
@@ -268,6 +277,7 @@ find_debuginfo_in_path (Dwfl_Module *mod, const char *file_name,
 	      subdir = NULL;
 	      file = basename (debuglink_file);
 	    }
+	  try_file_basename = debuglink_null;
 	  break;
 	default:
 	  /* A relative path says to try a subdirectory of that name
@@ -275,11 +285,14 @@ find_debuginfo_in_path (Dwfl_Module *mod, const char *file_name,
 	  dir = file_dirname;
 	  subdir = p;
 	  file = debuglink_file;
+	  try_file_basename = debuglink_null;
 	  break;
 	}
 
       char *fname = NULL;
       int fd = try_open (&main_stat, dir, subdir, file, &fname);
+      if (fd < 0 && try_file_basename)
+	fd = try_open (&main_stat, dir, subdir, file_basename, &fname);
       if (fd < 0)
 	switch (errno)
 	  {
diff --git a/libdwfl/linux-kernel-modules.c b/libdwfl/linux-kernel-modules.c
index 79faf99..54c0b90 100644
--- a/libdwfl/linux-kernel-modules.c
+++ b/libdwfl/linux-kernel-modules.c
@@ -92,17 +92,21 @@ try_kernel_name (Dwfl *dwfl, char **fname, bool try_debug)
   if (fd < 0)
     {
       Dwfl_Module fakemod = { .dwfl = dwfl };
-      /* First try the file's unadorned basename as DEBUGLINK_FILE,
-	 to look for "vmlinux" files.  */
-      fd = INTUSE(dwfl_standard_find_debuginfo) (&fakemod, NULL, NULL, 0,
-						 *fname, basename (*fname), 0,
-						 &fakemod.debug.name);
-      if (fd < 0 && try_debug)
-	/* Next, let the call use the default of basename + ".debug",
-	   to look for "vmlinux.debug" files.  */
+
+      if (try_debug)
+	/* Passing NULL for DEBUGLINK_FILE searches for both the basenamer
+	   "vmlinux" and the default of basename + ".debug", to look for
+	   "vmlinux.debug" files.  */
 	fd = INTUSE(dwfl_standard_find_debuginfo) (&fakemod, NULL, NULL, 0,
 						   *fname, NULL, 0,
 						   &fakemod.debug.name);
+      else
+	/* Try the file's unadorned basename as DEBUGLINK_FILE,
+	   to look only for "vmlinux" files.  */
+	fd = INTUSE(dwfl_standard_find_debuginfo) (&fakemod, NULL, NULL, 0,
+						   *fname, basename (*fname),
+						   0, &fakemod.debug.name);
+
       if (fakemod.debug.name != NULL)
 	{
 	  free (*fname);
-- 
1.8.3.1

Re: [RFC] elfutils: Checks for debuginfo file without .debug extension as well

[Ravi Bangoria]

[-- Attachment #1: Type: text/plain, Size: 5180 bytes --]

Hi Mark,

Thanks for the patch. Indeed this is optimized one compared to what I've
proposed.

On Friday 19 February 2016 08:41 PM, Mark Wielaard wrote:
> Hi Ravi,
>
> On Fri, 2016-02-19 at 15:07 +0530, Ravi Bangoria wrote:
>> Thanks for the sample program. I can see it tries to simulate what
>> systemtap do.
>> Here is the output on ubunut powerpc.
>>
>> Without patch:
>>
>>     # ./dwfl_find_kernel
>>     Finding kernel release: 3.13.0-76-generic
>>     Debuginfo path: +:.debug:/usr/lib/debug
>>     setup report modname: kernel, filename: /boot/vmlinux-3.13.0-76-generic
>>     setup report modname: xfrm_user, filename:
>> /lib/modules/3.13.0-76-generic/kernel/net/xfrm/xfrm_user.ko
>>     Found kernel file: /boot/vmlinux-3.13.0-76-generic
>>     getdwarf name: kernel
>>     kernel without DWARF
>>     Couldn't get kernel DWARF
>>
>> With patch:
>>
>>     # ./dwfl_find_kernel
>>     Finding kernel release: 3.13.0-76-generic
>>     Debuginfo path: +:.debug:/usr/lib/debug
>>     setup report modname: kernel, filename: /boot/vmlinux-3.13.0-76-generic
>>     setup report modname: xfrm_user, filename:
>> /lib/modules/3.13.0-76-generic/kernel/net/xfrm/xfrm_user.ko
>>     Found kernel file: /boot/vmlinux-3.13.0-76-generic
>>     getdwarf name: kernel
>>     mainfile: /boot/vmlinux-3.13.0-76-generic, debugfile:
>> /usr/lib/debug/boot/vmlinux-3.13.0-76-generic
>>     Found kernel DWARF file: /usr/lib/debug/boot/vmlinux-3.13.0-76-generic
> Thanks, that looks like what I expected.
> I also got access to a somewhat newer version of ubuntu ppc64le and that
> has an additional issue. It has the vmlinux file installed unreadable
> (except for root). That causes almost the same issue, but slightly
> differently, now no kernel at all is found... Although in that case it
> can be worked around be using /usr/lib/debug as base. But the main issue
> is exactly as you describe.

Can you please provide me system configurations like ubuntu versrion,
kernel version, stap version, elfutils version etc. I'll check if I'll 
be able
to get similar system.

>>> It still doesn't show the full search path (for that we should hack
>>> find_debuginfo_in_path and try_open in libdwfl/find-debuginfo.c a bit),
>>> but it should be a start to better understand why the current search
>>> isn't finding the separate kernel debug file.
>> Sry, I'm little bit confused in your last two paragraphs.
> That is probably because I was still a little confused myself :)
>
>> 1. Does this means change I've proposed is at wrong place i.e.
>> find_debuginfo
> Not wrong. It correctly finds the kernel. But it could be done a little
> more efficient I believe. With your change we do the search through all
> possible directories (and subdirs) twice. We could do it slightly more
> efficient by moving the logic into find_debuginfo_in_path itself.
>
> The issue is really that if find_debuginfo_in_path is passed in a "null"
> debuglink_name then it invents one (basename + .debug). But if it is
> "null" then that is really just one guess. We should as you indicate
> also try the basename itself. So that is what the attached patch does,
> it checks if debuglink_name is NULL and if we are checking in a debug
> directory (or subdirectory), but not the main file location, then we try
> both basename.debug and basename.
>
> That should do the same thing as your suggested patch, but makes it
> slightly more efficient. We try less directories/files. And if the
> kernel image exists we should find it earlier.
>
> Could you try out if this variant of the patch works for you?

I've tested your patch and it's working fine with Ubuntu 14.04 with 
kernel 3.13,
stap 2.3 and elfutils 0.165.

But I've a little doubt here. Here is the code snip of try_kernel_name
from libdwfl/linux-kernel-modules.c

     static int
     try_kernel_name (Dwfl *dwfl, char **fname, bool try_debug)
     {
LINE A:
       int fd = ((((dwfl->callbacks->debuginfo_path
                    ? *dwfl->callbacks->debuginfo_path : NULL)
                   ?: DEFAULT_DEBUGINFO_PATH)[0] == ':') ? -1
                 : TEMP_FAILURE_RETRY (open64 (*fname, O_RDONLY)));

       if (fd < 0)
         {
LINE B:
           /* look for "vmlinux" files.  */
           fd = INTUSE(dwfl_standard_find_debuginfo) (&fakemod, NULL, 
NULL, 0,
                                                      *fname, basename 
(*fname), 0,
&fakemod.debug.name);
           if (fd < 0 && try_debug)
LINE C:
             /* look for "vmlinux.debug" files.  */
             fd = INTUSE(dwfl_standard_find_debuginfo) (&fakemod, NULL, 
NULL, 0,
                                                        *fname, NULL, 0,
&fakemod.debug.name);

try_kernel_name is doing almost same thing what I have proposed. Now let's
say we want to go ahead with your patch, than call to 
dwfl_standard_find_debuginfo
in LINE C will look for both vmlinux and vmlinux.debug right? But it has 
already
checked for vmlinux in LINE B. So, in this case we have to modify 
try_kernel_name
as well.

Please correct me if I'm wrong.

Regards,
Ravi

Re: [RFC] elfutils: Checks for debuginfo file without .debug extension as well

[Mark Wielaard]

[-- Attachment #1: Type: text/plain, Size: 3710 bytes --]

Hi Ravi,

On Fri, 2016-02-19 at 15:07 +0530, Ravi Bangoria wrote:
> Thanks for the sample program. I can see it tries to simulate what 
> systemtap do.
> Here is the output on ubunut powerpc.
> 
> Without patch:
> 
>    # ./dwfl_find_kernel
>    Finding kernel release: 3.13.0-76-generic
>    Debuginfo path: +:.debug:/usr/lib/debug
>    setup report modname: kernel, filename: /boot/vmlinux-3.13.0-76-generic
>    setup report modname: xfrm_user, filename: 
> /lib/modules/3.13.0-76-generic/kernel/net/xfrm/xfrm_user.ko
>    Found kernel file: /boot/vmlinux-3.13.0-76-generic
>    getdwarf name: kernel
>    kernel without DWARF
>    Couldn't get kernel DWARF
>
> With patch:
> 
>    # ./dwfl_find_kernel
>    Finding kernel release: 3.13.0-76-generic
>    Debuginfo path: +:.debug:/usr/lib/debug
>    setup report modname: kernel, filename: /boot/vmlinux-3.13.0-76-generic
>    setup report modname: xfrm_user, filename: 
> /lib/modules/3.13.0-76-generic/kernel/net/xfrm/xfrm_user.ko
>    Found kernel file: /boot/vmlinux-3.13.0-76-generic
>    getdwarf name: kernel
>    mainfile: /boot/vmlinux-3.13.0-76-generic, debugfile: 
> /usr/lib/debug/boot/vmlinux-3.13.0-76-generic
>    Found kernel DWARF file: /usr/lib/debug/boot/vmlinux-3.13.0-76-generic

Thanks, that looks like what I expected.
I also got access to a somewhat newer version of ubuntu ppc64le and that
has an additional issue. It has the vmlinux file installed unreadable
(except for root). That causes almost the same issue, but slightly
differently, now no kernel at all is found... Although in that case it
can be worked around be using /usr/lib/debug as base. But the main issue
is exactly as you describe.

> > It still doesn't show the full search path (for that we should hack
> > find_debuginfo_in_path and try_open in libdwfl/find-debuginfo.c a bit),
> > but it should be a start to better understand why the current search
> > isn't finding the separate kernel debug file.
> 
> Sry, I'm little bit confused in your last two paragraphs.

That is probably because I was still a little confused myself :)

> 1. Does this means change I've proposed is at wrong place i.e. 
> find_debuginfo

Not wrong. It correctly finds the kernel. But it could be done a little
more efficient I believe. With your change we do the search through all
possible directories (and subdirs) twice. We could do it slightly more
efficient by moving the logic into find_debuginfo_in_path itself.

The issue is really that if find_debuginfo_in_path is passed in a "null"
debuglink_name then it invents one (basename + .debug). But if it is
"null" then that is really just one guess. We should as you indicate
also try the basename itself. So that is what the attached patch does,
it checks if debuglink_name is NULL and if we are checking in a debug
directory (or subdirectory), but not the main file location, then we try
both basename.debug and basename.

That should do the same thing as your suggested patch, but makes it
slightly more efficient. We try less directories/files. And if the
kernel image exists we should find it earlier.

Could you try out if this variant of the patch works for you?

> 2. I'm not able to understand why kernel module came into picture here. 

Yes sorry, that was just an artifact of my test program.
setup_report_kernel gets called for all "modules" of the kernel. That
includes the kernel image itself. We return 0 if we are not interested
in that module, 1 if we want it and -1 to report we don't want any more.
That means we see at least one kernel module. The test program prints
that one out, but that is just confusing.

Cheers,

Mark

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: find-debuginfo.patch --]
[-- Type: text/x-patch, Size: 2057 bytes --]

diff --git a/libdwfl/find-debuginfo.c b/libdwfl/find-debuginfo.c
index 72461bc..80515db 100644
--- a/libdwfl/find-debuginfo.c
+++ b/libdwfl/find-debuginfo.c
@@ -163,7 +163,11 @@ find_debuginfo_in_path (Dwfl_Module *mod, const char *file_name,
 
   const char *file_basename = file_name == NULL ? NULL : basename (file_name);
   char *localname = NULL;
-  if (debuglink_file == NULL)
+
+  /* We invent a debuglink .debug name if NULL, but then want to try the
+     basename too.  */
+  bool debuglink_null = debuglink_file == NULL;
+  if (debuglink_null)
     {
       /* For a alt debug multi file we need a name, for a separate debug
 	 name we may be able to fall back on file_basename.debug.  */
@@ -231,6 +235,10 @@ find_debuginfo_in_path (Dwfl_Module *mod, const char *file_name,
 	check = *p++ == '+';
       check = check && cancheck;
 
+      /* Try the basename too, if we made up the debuglink name and this
+	 is not the main directory.  */
+      bool try_file_basename;
+
       const char *dir, *subdir, *file;
       switch (p[0])
 	{
@@ -239,6 +247,7 @@ find_debuginfo_in_path (Dwfl_Module *mod, const char *file_name,
 	  dir = file_dirname;
 	  subdir = NULL;
 	  file = debuglink_file;
+	  try_file_basename = false;
 	  break;
 	case '/':
 	  /* An absolute path says to look there for a subdirectory
@@ -268,6 +277,7 @@ find_debuginfo_in_path (Dwfl_Module *mod, const char *file_name,
 	      subdir = NULL;
 	      file = basename (debuglink_file);
 	    }
+	  try_file_basename = debuglink_null;
 	  break;
 	default:
 	  /* A relative path says to try a subdirectory of that name
@@ -275,11 +285,14 @@ find_debuginfo_in_path (Dwfl_Module *mod, const char *file_name,
 	  dir = file_dirname;
 	  subdir = p;
 	  file = debuglink_file;
+	  try_file_basename = debuglink_null;
 	  break;
 	}
 
       char *fname = NULL;
       int fd = try_open (&main_stat, dir, subdir, file, &fname);
+      if (fd < 0 && try_file_basename)
+	fd = try_open (&main_stat, dir, subdir, file_basename, &fname);
       if (fd < 0)
 	switch (errno)
 	  {

Re: [RFC] elfutils: Checks for debuginfo file without .debug extension as well

[Mark Wielaard]

[-- Attachment #1: Type: text/plain, Size: 4115 bytes --]

Hi Ravi,

My apologies for being slightly confused. I think what you are proposing
with your patch is the right thing. I am just slightly confused because
I thought the current code already did what you are proposing. So the
current code doesn't actually do what I thought it did :)

I have attached a simplified version of what I believe systemtap is
doing. It is a two step process. First it calls
dwfl_linux_kernel_report_offline to get the kernel file itself. Second
it gets the Dwarf for the kernel module, which should trigger
find_debuginfo to find the separate debug file, if needed.

It still doesn't show the full search path (for that we should hack
find_debuginfo_in_path and try_open in libdwfl/find-debuginfo.c a bit),
but it should be a start to better understand why the current search
isn't finding the separate kernel debug file.

On Wed, 2016-02-17 at 13:50 +0530, Ravi Bangoria wrote:
> On Tuesday 16 February 2016 10:15 PM, Mark Wielaard wrote:
> >> This patch adds functionality to search for file without any extension
> >> followed by searching file having .debug extension.
> > Thanks for the analysis. I do have a question though.
> >
> > The reason we are looking for the vmlinux-3.13.0-76-generic.debug
> > filename instead of the plain vmlinux-3.13.0-76-generic filename seems
> > to be because /boot/vmlinux contains a .gnu_debuginfo link with the name
> > of the debug file. But that name doesn't match?
> 
> So I should put it this way:
> 
> 'Ubuntu on powerpc' installs 'stripped' kernel 'without compressing it'
> as /boot/vmlinux. Most of the time it's assumed that vmlinux will have
> .debuginfo of .debuglink section. But this is not true here. See outputs
> below.
> 
> > What does eu-readelf --strings=.gnu_debuglink /boot/vmlinux* say?
> 
> # eu-readelf --strings=.gnu_debuglink /boot/vmlinux*
> 
> /boot/vmlinux:
> 
> eu-readelf:
> section '.gnu_debuglink' does not exist
> 
> /boot/vmlinux-3.13.0-76-generic:
> 
> eu-readelf:
> section '.gnu_debuglink' does not exist
> 
> /boot/vmlinux-4.2.0-27-generic:
> 
> eu-readelf:
> section '.gnu_debuglink' does not exist
> 
> /boot/vmlinux.old:
> 
> eu-readelf:
> section '.gnu_debuglink' does not exist

Aha, that is indeed not what I expected. Thanks.

> > Would it make sense to not put in the "correct" (without .debug suffix)
> > debugfile name in .gnu_debuglink
> 
> Yes, that should solve the problem. But changing structure of ubuntu
> kernel elf to solve stap problem doesn't look feasible to me. IIRC no
> other distro do that. I've confirmed this on RHEL. RHEL distro-kernel
> does not have .debuglink section.

OK, thanks. That is actually not what I expected. I had expected that if
there is no .debuglink section we already search for both the plain
and .debug file name. Apparently I got this wrong.

> Then you may ask how this is working in RHEL then. RHEL / Fedora on
> powerpc installs stripped uncompressed kernel as /boot/vmlinuz.
> elfutils looking for vmlinux, doesn't find it from /boot/, so it continue
> looking for file without any suffix(.debug) and it gets proper file. This is
> the case with ubunut x86 as well, except on x86, /boot/vmlinuz is
> compressed stripped image.

Aha. So Fedora/RHEL "cheats" by calling it vmlinuz while it isn't
actually compressed.

And another thing I missed is that the vmlinux file is the full
unstripped kernel image, even though it is put under /usr/debug...
So it contains the whole kernel plus the debuginfo.

> So I thought about this solution as well, renaming /boot/vmlinux to
> /boot/vmlinuz will solve the problem. But this is not appropriate. Because
> vmlinu*z* is for compressed image and we are just renaming file without
> compressing to solve stap problem.

Yeah, it is a bit cheating.

> >   or completely leaving .gnu_debuginfo
> > out (or is nothing found then?)
> 
> Not sure what does this mean. Can you please explain a bit more.

Sorry, silly confusing typo. I meant leaving out .gnu_debuglink. But it
is already not there.

Thanks,

Mark

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: dwfl_find_kernel.c --]
[-- Type: text/x-csrc, Size: 3297 bytes --]

// gcc -Wall -o dwfl_find_kernel -ldw dwfl_find_kernel.c

#include <inttypes.h>
#include <stdbool.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/utsname.h>
#include <elfutils/libdwfl.h>

static char *kernel_file = NULL;
static int
setup_report_kernel (const char* modname, const char* filename)
{
  printf ("setup report modname: %s, filename: %s\n", modname, filename);

  /* We are only interested in the kernel.  */
  if (modname != NULL && strcmp (modname, "kernel") == 0)
    {
      /* We will get a null filename for modname == "kernel" if the
	 kernel couldn't be found.  We still want to stop, not
	 interested in any modules being found.  */
      if (filename == NULL)
	return -1;

      /* Copy the kernel file name and tell dwfl we want this module.  */
      kernel_file = strdup (filename);
      return 1;
    }

  /* Once we have seen the kernel we don't need any more modules.  We
     are not interested in any kernel modules.  */
  if (kernel_file != NULL)
    return -1;

  /* We don't need this module.  But we haven't seen the kernel yet,
     continue.  */
  return 0;
}

static char *kernel_dwarf_file = NULL;
static int
getdwarf_kernel (Dwfl_Module *mod, void **userdata,
		 const char *name, Dwarf_Addr base,
		 Dwarf *dw, Dwarf_Addr bias,
		 void *arg)
{
  printf ("getdwarf name: %s\n", name);

  if (name != NULL && strcmp (name, "kernel") == 0)
    {
      if (dw == NULL)
	{
	  printf ("kernel without DWARF\n");
	  return DWARF_CB_ABORT;
	}

      const char *mainfile, *debugfile;
      dwfl_module_info (mod, NULL, NULL, NULL, NULL, NULL,
			&mainfile, &debugfile);
      printf ("mainfile: %s, debugfile: %s\n", mainfile, debugfile);
      if (debugfile != NULL)
	kernel_dwarf_file = strdup (debugfile);
      else
	kernel_dwarf_file = strdup (mainfile);
    }

  return DWARF_CB_OK;
}

int
main (int argc, char **argv)
{
  char *release;
  if (argc > 1)
    release = argv[1];
  else
    {
      struct utsname uts;
      if (uname (&uts) != 0)
	{
	  perror ("uname failed");
	  return -1;
	}
      release = uts.release;
    }

  char *debuginfo_path;
  if (argc > 2)
    debuginfo_path = argv[2];
  else
    debuginfo_path = "+:.debug:/usr/lib/debug";

  printf ("Finding kernel release: %s\n", release);
  printf ("Debuginfo path: %s\n", debuginfo_path);

  Dwfl_Callbacks kernel_callbacks =
  {
    dwfl_linux_kernel_find_elf,
    dwfl_standard_find_debuginfo,
    dwfl_offline_section_address,
    (char **) & debuginfo_path
  };

  Dwfl *dwfl = dwfl_begin (&kernel_callbacks);
  if (dwfl == NULL)
    {
      printf ("dwfl_begin: %s\n", dwfl_errmsg (-1));
      return -1;
    }

  dwfl_report_begin (dwfl);
  dwfl_linux_kernel_report_offline (dwfl, release, &setup_report_kernel);
  dwfl_report_end(dwfl, NULL, NULL);

  if (kernel_file == NULL)
    {
      printf ("Couldn't find kernel\n");
      return -1;
    }

  printf ("Found kernel file: %s\n", kernel_file);

  ptrdiff_t p = 0;
  do
    p = dwfl_getdwarf (dwfl, &getdwarf_kernel, NULL, p);
  while (p > 0);

  if (kernel_dwarf_file == NULL)
    {
      printf ("Couldn't get kernel DWARF\n");
      return -1;
    }

  printf ("Found kernel DWARF file: %s\n", kernel_dwarf_file);

  dwfl_end (dwfl);
  free (kernel_file);
  free (kernel_dwarf_file);
  return 0;
}

Re: [RFC] elfutils: Checks for debuginfo file without .debug extension as well

[Ravi Bangoria]

[-- Attachment #1: Type: text/plain, Size: 3205 bytes --]

Hi Mark,

Thanks for reviewing. Please find my comments.

On Tuesday 16 February 2016 10:15 PM, Mark Wielaard wrote:
> Hi Ravi,
>
> On Tue, 2016-02-16 at 21:51 +0530, Ravi Bangoria wrote:
>> Elfutils, looking for kernel debuginfo file, tries to find it at
>> various places. If elfutils finds /boot/vmlinu*x* file, it checks
>> for debufginfo section. If debuginfo is not present, it saves it as
>> 'main elf' and continue looking for debuginfo file having .debug
>> extension i.e. vmlinux-RELEASE.debug.
>>
>> 'Ubuntu on powerpc' installs kernel as /boot/vmlinux and installs
>> debuginfo without any extension as /usr/lib/debug/boot/vmlinux-RELEASE
>> and hence, elfutils is not able to find the debuginfo file.
>>
>> Here is the lunchpad bug for the same:
>>    https://bugs.launchpad.net/ubuntu/+source/systemtap/+bug/1537125
>>
>> This patch adds functionality to search for file without any extension
>> followed by searching file having .debug extension.
> Thanks for the analysis. I do have a question though.
>
> The reason we are looking for the vmlinux-3.13.0-76-generic.debug
> filename instead of the plain vmlinux-3.13.0-76-generic filename seems
> to be because /boot/vmlinux contains a .gnu_debuginfo link with the name
> of the debug file. But that name doesn't match?

So I should put it this way:

'Ubuntu on powerpc' installs 'stripped' kernel 'without compressing it'
as /boot/vmlinux. Most of the time it's assumed that vmlinux will have
.debuginfo of .debuglink section. But this is not true here. See outputs
below.

> What does eu-readelf --strings=.gnu_debuglink /boot/vmlinux* say?

# eu-readelf --strings=.gnu_debuglink /boot/vmlinux*

/boot/vmlinux:

eu-readelf:
section '.gnu_debuglink' does not exist

/boot/vmlinux-3.13.0-76-generic:

eu-readelf:
section '.gnu_debuglink' does not exist

/boot/vmlinux-4.2.0-27-generic:

eu-readelf:
section '.gnu_debuglink' does not exist

/boot/vmlinux.old:

eu-readelf:
section '.gnu_debuglink' does not exist

> Would it make sense to not put in the "correct" (without .debug suffix)
> debugfile name in .gnu_debuglink

Yes, that should solve the problem. But changing structure of ubuntu
kernel elf to solve stap problem doesn't look feasible to me. IIRC no
other distro do that. I've confirmed this on RHEL. RHEL distro-kernel
does not have .debuglink section.

Then you may ask how this is working in RHEL then. RHEL / Fedora on
powerpc installs stripped uncompressed kernel as /boot/vmlinuz.
elfutils looking for vmlinux, doesn't find it from /boot/, so it continue
looking for file without any suffix(.debug) and it gets proper file. This is
the case with ubunut x86 as well, except on x86, /boot/vmlinuz is
compressed stripped image.

So I thought about this solution as well, renaming /boot/vmlinux to
/boot/vmlinuz will solve the problem. But this is not appropriate. Because
vmlinu*z* is for compressed image and we are just renaming file without
compressing to solve stap problem.

>   or completely leaving .gnu_debuginfo
> out (or is nothing found then?)

Not sure what does this mean. Can you please explain a bit more.

Regards,
Ravi

Re: [RFC] elfutils: Checks for debuginfo file without .debug extension as well

[Mark Wielaard]

[-- Attachment #1: Type: text/plain, Size: 1431 bytes --]

Hi Ravi,

On Tue, 2016-02-16 at 21:51 +0530, Ravi Bangoria wrote:
> Elfutils, looking for kernel debuginfo file, tries to find it at
> various places. If elfutils finds /boot/vmlinu*x* file, it checks
> for debufginfo section. If debuginfo is not present, it saves it as
> 'main elf' and continue looking for debuginfo file having .debug
> extension i.e. vmlinux-RELEASE.debug.
> 
> 'Ubuntu on powerpc' installs kernel as /boot/vmlinux and installs
> debuginfo without any extension as /usr/lib/debug/boot/vmlinux-RELEASE
> and hence, elfutils is not able to find the debuginfo file.
> 
> Here is the lunchpad bug for the same:
>   https://bugs.launchpad.net/ubuntu/+source/systemtap/+bug/1537125
> 
> This patch adds functionality to search for file without any extension
> followed by searching file having .debug extension.

Thanks for the analysis. I do have a question though.

The reason we are looking for the vmlinux-3.13.0-76-generic.debug
filename instead of the plain vmlinux-3.13.0-76-generic filename seems
to be because /boot/vmlinux contains a .gnu_debuginfo link with the name
of the debug file. But that name doesn't match?

What does eu-readelf --strings=.gnu_debuglink /boot/vmlinux* say?

Would it make sense to not put in the "correct" (without .debug suffix)
debugfile name in .gnu_debuglink or completely leaving .gnu_debuginfo
out (or is nothing found then?)

Thanks,

Mark

[RFC] elfutils: Checks for debuginfo file without .debug extension as well

[Ravi Bangoria]

[-- Attachment #1: Type: text/plain, Size: 5557 bytes --]

Elfutils, looking for kernel debuginfo file, tries to find it at
various places. If elfutils finds /boot/vmlinu*x* file, it checks
for debufginfo section. If debuginfo is not present, it saves it as
'main elf' and continue looking for debuginfo file having .debug
extension i.e. vmlinux-RELEASE.debug.

'Ubuntu on powerpc' installs kernel as /boot/vmlinux and installs
debuginfo without any extension as /usr/lib/debug/boot/vmlinux-RELEASE
and hence, elfutils is not able to find the debuginfo file.

Here is the lunchpad bug for the same:
  https://bugs.launchpad.net/ubuntu/+source/systemtap/+bug/1537125

This patch adds functionality to search for file without any extension
followed by searching file having .debug extension.

I've formatted this patch from elfutils-0.165

Before applying patch:

  # strace -o out -e open ./stap -v probe.stp
  Pass 1: parsed user script and 95 library script(s) using 42176virt/32128res/5952shr/25472data kb, in 190usr/20sys/222real ms.
  WARNING: cannot find module kernel debuginfo: No DWARF information found [man warning::debuginfo]
  semantic error: while resolving probe point: identifier 'kernel' at probe.stp:1:7
          source: probe kernel.function("do_fork")
                      ^

  semantic error: no match
  Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0 global(s) using 66688virt/34496res/7296shr/26432data kb, in 0usr/0sys/28real ms.
  Pass 2: analysis failed.  [man error::pass2]
  Tip: /usr/share/doc/systemtap/README.Debian should help you get started.

  # cat out | grep vmlinu
  open("/boot/vmlinux-3.13.0-76-generic", O_RDONLY) = 3
  open("/lib/modules/3.13.0-76-generic/build/vmlinux.id", O_RDONLY) = -1 ENOENT (No such file or directory)
  open("/boot/vmlinux-3.13.0-76-generic.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
  open("/boot/.debug/vmlinux-3.13.0-76-generic.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
  open("/usr/lib/debug/boot/vmlinux-3.13.0-76-generic.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
  open("/var/cache/abrt-di/usr/lib/debug/boot/vmlinux-3.13.0-76-generic.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
  open("/boot/build/vmlinux-3.13.0-76-generic.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
  open("/boot/vmlinux-3.13.0-76-generic.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
  open("/boot/.debug/vmlinux-3.13.0-76-generic.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
  open("/usr/lib/debug/boot/vmlinux-3.13.0-76-generic.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
  open("/var/cache/abrt-di/usr/lib/debug/boot/vmlinux-3.13.0-76-generic.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
  open("/boot/build/vmlinux-3.13.0-76-generic.debug", O_RDONLY) = -1 ENOENT (No such file or directory)

After applying patch:

  # strace -o out1 -e open ./stap -v probe.stp
  Pass 1: parsed user script and 95 library script(s) using 41856virt/32128res/5952shr/25472data kb, in 170usr/10sys/189real ms.
  Pass 2: analyzed script: 1 probe(s), 0 function(s), 0 embed(s), 0 global(s) using 242496virt/166912res/105088shr/60608data kb, in 700usr/100sys/1518real ms.
  Pass 3: translated to C into "/tmp/stapbbBu6y/stap_a18833407fc7a78b1251d743383f3fef_989_src.c" using 242496virt/167040res/105216shr/60608data kb, in 0usr/0sys/9real ms.
  Pass 4: compiled C into "stap_a18833407fc7a78b1251d743383f3fef_989.ko" in 6710usr/820sys/8338real ms.
  Pass 5: starting run.
  ^CPass 5: run completed in 0usr/10sys/3477real ms.

  # cat out1 | grep vmlinu
  open("/boot/vmlinux-3.13.0-76-generic", O_RDONLY) = 3
  open("/lib/modules/3.13.0-76-generic/build/vmlinux.id", O_RDONLY) = -1 ENOENT (No such file or directory)
  open("/boot/vmlinux-3.13.0-76-generic", O_RDONLY) = 4
  open("/boot/.debug/vmlinux-3.13.0-76-generic", O_RDONLY) = -1 ENOENT (No such file or directory)
  open("/boot/vmlinux-3.13.0-76-generic", O_RDONLY) = 4
  open("/usr/lib/debug/boot/vmlinux-3.13.0-76-generic", O_RDONLY) = 4

Signed-off-by: Ravi Bangoria <ravi.bangoria@linux.vnet.ibm.com>
---
 libdwfl/dwfl_module_getdwarf.c | 21 ++++++++++++++++-----
 1 file changed, 16 insertions(+), 5 deletions(-)

diff --git a/libdwfl/dwfl_module_getdwarf.c b/libdwfl/dwfl_module_getdwarf.c
index 0e8810b..ad5054b 100644
--- a/libdwfl/dwfl_module_getdwarf.c
+++ b/libdwfl/dwfl_module_getdwarf.c
@@ -529,11 +529,22 @@ find_debuginfo (Dwfl_Module *mod)
   debuglink_file = INTUSE(dwelf_elf_gnu_debuglink) (mod->main.elf,
 						    &debuglink_crc);
 
-  mod->debug.fd = (*mod->dwfl->callbacks->find_debuginfo) (MODCB_ARGS (mod),
-							   mod->main.name,
-							   debuglink_file,
-							   debuglink_crc,
-							   &mod->debug.name);
+  /* First try to look for vmlinux file */
+  if (debuglink_file == NULL)
+    mod->debug.fd = (*mod->dwfl->callbacks->find_debuginfo) (MODCB_ARGS (mod),
+							     mod->main.name,
+							     basename(mod->main.name),
+							     debuglink_crc,
+							     &mod->debug.name);
+
+  /* Try to look for vmlinux.debug file */
+  if (mod->debug.fd <= 0)
+    mod->debug.fd = (*mod->dwfl->callbacks->find_debuginfo) (MODCB_ARGS (mod),
+							     mod->main.name,
+							     debuglink_file,
+							     debuglink_crc,
+							     &mod->debug.name);
+
   Dwfl_Error result = open_elf (mod, &mod->debug);
   if (result == DWFL_E_NOERROR && mod->debug.address_sync != 0)
     result = find_prelink_address_sync (mod, &mod->debug);
-- 
1.9.1