* [RFC] elfutils: Checks for debuginfo file without .debug extension as well @ 2016-02-16 16:22 Ravi Bangoria 2016-02-16 16:45 ` Mark Wielaard 0 siblings, 1 reply; 10+ messages in thread From: Ravi Bangoria @ 2016-02-16 16:22 UTC (permalink / raw) To: elfutils-devel; +Cc: systemtap, naveen.n.rao, hemant, Ravi Bangoria Elfutils, looking for kernel debuginfo file, tries to find it at various places. If elfutils finds /boot/vmlinu*x* file, it checks for debufginfo section. If debuginfo is not present, it saves it as 'main elf' and continue looking for debuginfo file having .debug extension i.e. vmlinux-RELEASE.debug. 'Ubuntu on powerpc' installs kernel as /boot/vmlinux and installs debuginfo without any extension as /usr/lib/debug/boot/vmlinux-RELEASE and hence, elfutils is not able to find the debuginfo file. Here is the lunchpad bug for the same: https://bugs.launchpad.net/ubuntu/+source/systemtap/+bug/1537125 This patch adds functionality to search for file without any extension followed by searching file having .debug extension. I've formatted this patch from elfutils-0.165 Before applying patch: # strace -o out -e open ./stap -v probe.stp Pass 1: parsed user script and 95 library script(s) using 42176virt/32128res/5952shr/25472data kb, in 190usr/20sys/222real ms. WARNING: cannot find module kernel debuginfo: No DWARF information found [man warning::debuginfo] semantic error: while resolving probe point: identifier 'kernel' at probe.stp:1:7 source: probe kernel.function("do_fork") ^ semantic error: no match Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0 global(s) using 66688virt/34496res/7296shr/26432data kb, in 0usr/0sys/28real ms. Pass 2: analysis failed. [man error::pass2] Tip: /usr/share/doc/systemtap/README.Debian should help you get started. # cat out | grep vmlinu open("/boot/vmlinux-3.13.0-76-generic", O_RDONLY) = 3 open("/lib/modules/3.13.0-76-generic/build/vmlinux.id", O_RDONLY) = -1 ENOENT (No such file or directory) open("/boot/vmlinux-3.13.0-76-generic.debug", O_RDONLY) = -1 ENOENT (No such file or directory) open("/boot/.debug/vmlinux-3.13.0-76-generic.debug", O_RDONLY) = -1 ENOENT (No such file or directory) open("/usr/lib/debug/boot/vmlinux-3.13.0-76-generic.debug", O_RDONLY) = -1 ENOENT (No such file or directory) open("/var/cache/abrt-di/usr/lib/debug/boot/vmlinux-3.13.0-76-generic.debug", O_RDONLY) = -1 ENOENT (No such file or directory) open("/boot/build/vmlinux-3.13.0-76-generic.debug", O_RDONLY) = -1 ENOENT (No such file or directory) open("/boot/vmlinux-3.13.0-76-generic.debug", O_RDONLY) = -1 ENOENT (No such file or directory) open("/boot/.debug/vmlinux-3.13.0-76-generic.debug", O_RDONLY) = -1 ENOENT (No such file or directory) open("/usr/lib/debug/boot/vmlinux-3.13.0-76-generic.debug", O_RDONLY) = -1 ENOENT (No such file or directory) open("/var/cache/abrt-di/usr/lib/debug/boot/vmlinux-3.13.0-76-generic.debug", O_RDONLY) = -1 ENOENT (No such file or directory) open("/boot/build/vmlinux-3.13.0-76-generic.debug", O_RDONLY) = -1 ENOENT (No such file or directory) After applying patch: # strace -o out1 -e open ./stap -v probe.stp Pass 1: parsed user script and 95 library script(s) using 41856virt/32128res/5952shr/25472data kb, in 170usr/10sys/189real ms. Pass 2: analyzed script: 1 probe(s), 0 function(s), 0 embed(s), 0 global(s) using 242496virt/166912res/105088shr/60608data kb, in 700usr/100sys/1518real ms. Pass 3: translated to C into "/tmp/stapbbBu6y/stap_a18833407fc7a78b1251d743383f3fef_989_src.c" using 242496virt/167040res/105216shr/60608data kb, in 0usr/0sys/9real ms. Pass 4: compiled C into "stap_a18833407fc7a78b1251d743383f3fef_989.ko" in 6710usr/820sys/8338real ms. Pass 5: starting run. ^CPass 5: run completed in 0usr/10sys/3477real ms. # cat out1 | grep vmlinu open("/boot/vmlinux-3.13.0-76-generic", O_RDONLY) = 3 open("/lib/modules/3.13.0-76-generic/build/vmlinux.id", O_RDONLY) = -1 ENOENT (No such file or directory) open("/boot/vmlinux-3.13.0-76-generic", O_RDONLY) = 4 open("/boot/.debug/vmlinux-3.13.0-76-generic", O_RDONLY) = -1 ENOENT (No such file or directory) open("/boot/vmlinux-3.13.0-76-generic", O_RDONLY) = 4 open("/usr/lib/debug/boot/vmlinux-3.13.0-76-generic", O_RDONLY) = 4 Signed-off-by: Ravi Bangoria <ravi.bangoria@linux.vnet.ibm.com> --- libdwfl/dwfl_module_getdwarf.c | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-) diff --git a/libdwfl/dwfl_module_getdwarf.c b/libdwfl/dwfl_module_getdwarf.c index 0e8810b..ad5054b 100644 --- a/libdwfl/dwfl_module_getdwarf.c +++ b/libdwfl/dwfl_module_getdwarf.c @@ -529,11 +529,22 @@ find_debuginfo (Dwfl_Module *mod) debuglink_file = INTUSE(dwelf_elf_gnu_debuglink) (mod->main.elf, &debuglink_crc); - mod->debug.fd = (*mod->dwfl->callbacks->find_debuginfo) (MODCB_ARGS (mod), - mod->main.name, - debuglink_file, - debuglink_crc, - &mod->debug.name); + /* First try to look for vmlinux file */ + if (debuglink_file == NULL) + mod->debug.fd = (*mod->dwfl->callbacks->find_debuginfo) (MODCB_ARGS (mod), + mod->main.name, + basename(mod->main.name), + debuglink_crc, + &mod->debug.name); + + /* Try to look for vmlinux.debug file */ + if (mod->debug.fd <= 0) + mod->debug.fd = (*mod->dwfl->callbacks->find_debuginfo) (MODCB_ARGS (mod), + mod->main.name, + debuglink_file, + debuglink_crc, + &mod->debug.name); + Dwfl_Error result = open_elf (mod, &mod->debug); if (result == DWFL_E_NOERROR && mod->debug.address_sync != 0) result = find_prelink_address_sync (mod, &mod->debug); -- 1.9.1 ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [RFC] elfutils: Checks for debuginfo file without .debug extension as well 2016-02-16 16:22 [RFC] elfutils: Checks for debuginfo file without .debug extension as well Ravi Bangoria @ 2016-02-16 16:45 ` Mark Wielaard 2016-02-17 8:21 ` Ravi Bangoria 0 siblings, 1 reply; 10+ messages in thread From: Mark Wielaard @ 2016-02-16 16:45 UTC (permalink / raw) To: Ravi Bangoria; +Cc: elfutils-devel, systemtap, naveen.n.rao, hemant Hi Ravi, On Tue, 2016-02-16 at 21:51 +0530, Ravi Bangoria wrote: > Elfutils, looking for kernel debuginfo file, tries to find it at > various places. If elfutils finds /boot/vmlinu*x* file, it checks > for debufginfo section. If debuginfo is not present, it saves it as > 'main elf' and continue looking for debuginfo file having .debug > extension i.e. vmlinux-RELEASE.debug. > > 'Ubuntu on powerpc' installs kernel as /boot/vmlinux and installs > debuginfo without any extension as /usr/lib/debug/boot/vmlinux-RELEASE > and hence, elfutils is not able to find the debuginfo file. > > Here is the lunchpad bug for the same: > https://bugs.launchpad.net/ubuntu/+source/systemtap/+bug/1537125 > > This patch adds functionality to search for file without any extension > followed by searching file having .debug extension. Thanks for the analysis. I do have a question though. The reason we are looking for the vmlinux-3.13.0-76-generic.debug filename instead of the plain vmlinux-3.13.0-76-generic filename seems to be because /boot/vmlinux contains a .gnu_debuginfo link with the name of the debug file. But that name doesn't match? What does eu-readelf --strings=.gnu_debuglink /boot/vmlinux* say? Would it make sense to not put in the "correct" (without .debug suffix) debugfile name in .gnu_debuglink or completely leaving .gnu_debuginfo out (or is nothing found then?) Thanks, Mark ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [RFC] elfutils: Checks for debuginfo file without .debug extension as well 2016-02-16 16:45 ` Mark Wielaard @ 2016-02-17 8:21 ` Ravi Bangoria 2016-02-17 15:49 ` Mark Wielaard 0 siblings, 1 reply; 10+ messages in thread From: Ravi Bangoria @ 2016-02-17 8:21 UTC (permalink / raw) To: Mark Wielaard Cc: elfutils-devel, systemtap, naveen.n.rao, hemant, Srikar Dronamraju Hi Mark, Thanks for reviewing. Please find my comments. On Tuesday 16 February 2016 10:15 PM, Mark Wielaard wrote: > Hi Ravi, > > On Tue, 2016-02-16 at 21:51 +0530, Ravi Bangoria wrote: >> Elfutils, looking for kernel debuginfo file, tries to find it at >> various places. If elfutils finds /boot/vmlinu*x* file, it checks >> for debufginfo section. If debuginfo is not present, it saves it as >> 'main elf' and continue looking for debuginfo file having .debug >> extension i.e. vmlinux-RELEASE.debug. >> >> 'Ubuntu on powerpc' installs kernel as /boot/vmlinux and installs >> debuginfo without any extension as /usr/lib/debug/boot/vmlinux-RELEASE >> and hence, elfutils is not able to find the debuginfo file. >> >> Here is the lunchpad bug for the same: >> https://bugs.launchpad.net/ubuntu/+source/systemtap/+bug/1537125 >> >> This patch adds functionality to search for file without any extension >> followed by searching file having .debug extension. > Thanks for the analysis. I do have a question though. > > The reason we are looking for the vmlinux-3.13.0-76-generic.debug > filename instead of the plain vmlinux-3.13.0-76-generic filename seems > to be because /boot/vmlinux contains a .gnu_debuginfo link with the name > of the debug file. But that name doesn't match? So I should put it this way: 'Ubuntu on powerpc' installs 'stripped' kernel 'without compressing it' as /boot/vmlinux. Most of the time it's assumed that vmlinux will have .debuginfo of .debuglink section. But this is not true here. See outputs below. > What does eu-readelf --strings=.gnu_debuglink /boot/vmlinux* say? # eu-readelf --strings=.gnu_debuglink /boot/vmlinux* /boot/vmlinux: eu-readelf: section '.gnu_debuglink' does not exist /boot/vmlinux-3.13.0-76-generic: eu-readelf: section '.gnu_debuglink' does not exist /boot/vmlinux-4.2.0-27-generic: eu-readelf: section '.gnu_debuglink' does not exist /boot/vmlinux.old: eu-readelf: section '.gnu_debuglink' does not exist > Would it make sense to not put in the "correct" (without .debug suffix) > debugfile name in .gnu_debuglink Yes, that should solve the problem. But changing structure of ubuntu kernel elf to solve stap problem doesn't look feasible to me. IIRC no other distro do that. I've confirmed this on RHEL. RHEL distro-kernel does not have .debuglink section. Then you may ask how this is working in RHEL then. RHEL / Fedora on powerpc installs stripped uncompressed kernel as /boot/vmlinuz. elfutils looking for vmlinux, doesn't find it from /boot/, so it continue looking for file without any suffix(.debug) and it gets proper file. This is the case with ubunut x86 as well, except on x86, /boot/vmlinuz is compressed stripped image. So I thought about this solution as well, renaming /boot/vmlinux to /boot/vmlinuz will solve the problem. But this is not appropriate. Because vmlinu*z* is for compressed image and we are just renaming file without compressing to solve stap problem. > or completely leaving .gnu_debuginfo > out (or is nothing found then?) Not sure what does this mean. Can you please explain a bit more. Regards, Ravi ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [RFC] elfutils: Checks for debuginfo file without .debug extension as well 2016-02-17 8:21 ` Ravi Bangoria @ 2016-02-17 15:49 ` Mark Wielaard 2016-02-19 9:37 ` Ravi Bangoria 0 siblings, 1 reply; 10+ messages in thread From: Mark Wielaard @ 2016-02-17 15:49 UTC (permalink / raw) To: Ravi Bangoria Cc: elfutils-devel, systemtap, naveen.n.rao, hemant, Srikar Dronamraju [-- Attachment #1: Type: text/plain, Size: 4115 bytes --] Hi Ravi, My apologies for being slightly confused. I think what you are proposing with your patch is the right thing. I am just slightly confused because I thought the current code already did what you are proposing. So the current code doesn't actually do what I thought it did :) I have attached a simplified version of what I believe systemtap is doing. It is a two step process. First it calls dwfl_linux_kernel_report_offline to get the kernel file itself. Second it gets the Dwarf for the kernel module, which should trigger find_debuginfo to find the separate debug file, if needed. It still doesn't show the full search path (for that we should hack find_debuginfo_in_path and try_open in libdwfl/find-debuginfo.c a bit), but it should be a start to better understand why the current search isn't finding the separate kernel debug file. On Wed, 2016-02-17 at 13:50 +0530, Ravi Bangoria wrote: > On Tuesday 16 February 2016 10:15 PM, Mark Wielaard wrote: > >> This patch adds functionality to search for file without any extension > >> followed by searching file having .debug extension. > > Thanks for the analysis. I do have a question though. > > > > The reason we are looking for the vmlinux-3.13.0-76-generic.debug > > filename instead of the plain vmlinux-3.13.0-76-generic filename seems > > to be because /boot/vmlinux contains a .gnu_debuginfo link with the name > > of the debug file. But that name doesn't match? > > So I should put it this way: > > 'Ubuntu on powerpc' installs 'stripped' kernel 'without compressing it' > as /boot/vmlinux. Most of the time it's assumed that vmlinux will have > .debuginfo of .debuglink section. But this is not true here. See outputs > below. > > > What does eu-readelf --strings=.gnu_debuglink /boot/vmlinux* say? > > # eu-readelf --strings=.gnu_debuglink /boot/vmlinux* > > /boot/vmlinux: > > eu-readelf: > section '.gnu_debuglink' does not exist > > /boot/vmlinux-3.13.0-76-generic: > > eu-readelf: > section '.gnu_debuglink' does not exist > > /boot/vmlinux-4.2.0-27-generic: > > eu-readelf: > section '.gnu_debuglink' does not exist > > /boot/vmlinux.old: > > eu-readelf: > section '.gnu_debuglink' does not exist Aha, that is indeed not what I expected. Thanks. > > Would it make sense to not put in the "correct" (without .debug suffix) > > debugfile name in .gnu_debuglink > > Yes, that should solve the problem. But changing structure of ubuntu > kernel elf to solve stap problem doesn't look feasible to me. IIRC no > other distro do that. I've confirmed this on RHEL. RHEL distro-kernel > does not have .debuglink section. OK, thanks. That is actually not what I expected. I had expected that if there is no .debuglink section we already search for both the plain and .debug file name. Apparently I got this wrong. > Then you may ask how this is working in RHEL then. RHEL / Fedora on > powerpc installs stripped uncompressed kernel as /boot/vmlinuz. > elfutils looking for vmlinux, doesn't find it from /boot/, so it continue > looking for file without any suffix(.debug) and it gets proper file. This is > the case with ubunut x86 as well, except on x86, /boot/vmlinuz is > compressed stripped image. Aha. So Fedora/RHEL "cheats" by calling it vmlinuz while it isn't actually compressed. And another thing I missed is that the vmlinux file is the full unstripped kernel image, even though it is put under /usr/debug... So it contains the whole kernel plus the debuginfo. > So I thought about this solution as well, renaming /boot/vmlinux to > /boot/vmlinuz will solve the problem. But this is not appropriate. Because > vmlinu*z* is for compressed image and we are just renaming file without > compressing to solve stap problem. Yeah, it is a bit cheating. > > or completely leaving .gnu_debuginfo > > out (or is nothing found then?) > > Not sure what does this mean. Can you please explain a bit more. Sorry, silly confusing typo. I meant leaving out .gnu_debuglink. But it is already not there. Thanks, Mark [-- Attachment #2: dwfl_find_kernel.c --] [-- Type: text/x-csrc, Size: 3297 bytes --] // gcc -Wall -o dwfl_find_kernel -ldw dwfl_find_kernel.c #include <inttypes.h> #include <stdbool.h> #include <stdio.h> #include <stdlib.h> #include <string.h> #include <sys/utsname.h> #include <elfutils/libdwfl.h> static char *kernel_file = NULL; static int setup_report_kernel (const char* modname, const char* filename) { printf ("setup report modname: %s, filename: %s\n", modname, filename); /* We are only interested in the kernel. */ if (modname != NULL && strcmp (modname, "kernel") == 0) { /* We will get a null filename for modname == "kernel" if the kernel couldn't be found. We still want to stop, not interested in any modules being found. */ if (filename == NULL) return -1; /* Copy the kernel file name and tell dwfl we want this module. */ kernel_file = strdup (filename); return 1; } /* Once we have seen the kernel we don't need any more modules. We are not interested in any kernel modules. */ if (kernel_file != NULL) return -1; /* We don't need this module. But we haven't seen the kernel yet, continue. */ return 0; } static char *kernel_dwarf_file = NULL; static int getdwarf_kernel (Dwfl_Module *mod, void **userdata, const char *name, Dwarf_Addr base, Dwarf *dw, Dwarf_Addr bias, void *arg) { printf ("getdwarf name: %s\n", name); if (name != NULL && strcmp (name, "kernel") == 0) { if (dw == NULL) { printf ("kernel without DWARF\n"); return DWARF_CB_ABORT; } const char *mainfile, *debugfile; dwfl_module_info (mod, NULL, NULL, NULL, NULL, NULL, &mainfile, &debugfile); printf ("mainfile: %s, debugfile: %s\n", mainfile, debugfile); if (debugfile != NULL) kernel_dwarf_file = strdup (debugfile); else kernel_dwarf_file = strdup (mainfile); } return DWARF_CB_OK; } int main (int argc, char **argv) { char *release; if (argc > 1) release = argv[1]; else { struct utsname uts; if (uname (&uts) != 0) { perror ("uname failed"); return -1; } release = uts.release; } char *debuginfo_path; if (argc > 2) debuginfo_path = argv[2]; else debuginfo_path = "+:.debug:/usr/lib/debug"; printf ("Finding kernel release: %s\n", release); printf ("Debuginfo path: %s\n", debuginfo_path); Dwfl_Callbacks kernel_callbacks = { dwfl_linux_kernel_find_elf, dwfl_standard_find_debuginfo, dwfl_offline_section_address, (char **) & debuginfo_path }; Dwfl *dwfl = dwfl_begin (&kernel_callbacks); if (dwfl == NULL) { printf ("dwfl_begin: %s\n", dwfl_errmsg (-1)); return -1; } dwfl_report_begin (dwfl); dwfl_linux_kernel_report_offline (dwfl, release, &setup_report_kernel); dwfl_report_end(dwfl, NULL, NULL); if (kernel_file == NULL) { printf ("Couldn't find kernel\n"); return -1; } printf ("Found kernel file: %s\n", kernel_file); ptrdiff_t p = 0; do p = dwfl_getdwarf (dwfl, &getdwarf_kernel, NULL, p); while (p > 0); if (kernel_dwarf_file == NULL) { printf ("Couldn't get kernel DWARF\n"); return -1; } printf ("Found kernel DWARF file: %s\n", kernel_dwarf_file); dwfl_end (dwfl); free (kernel_file); free (kernel_dwarf_file); return 0; } ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [RFC] elfutils: Checks for debuginfo file without .debug extension as well 2016-02-17 15:49 ` Mark Wielaard @ 2016-02-19 9:37 ` Ravi Bangoria 2016-02-19 15:12 ` Mark Wielaard 0 siblings, 1 reply; 10+ messages in thread From: Ravi Bangoria @ 2016-02-19 9:37 UTC (permalink / raw) To: Mark Wielaard Cc: elfutils-devel, systemtap, naveen.n.rao, hemant, Srikar Dronamraju HI Mark, Thanks for giving detailed analysis. Please find my comments. On Wednesday 17 February 2016 09:19 PM, Mark Wielaard wrote: > Hi Ravi, > > My apologies for being slightly confused. I think what you are proposing > with your patch is the right thing. I am just slightly confused because > I thought the current code already did what you are proposing. So the > current code doesn't actually do what I thought it did :) > > I have attached a simplified version of what I believe systemtap is > doing. It is a two step process. First it calls > dwfl_linux_kernel_report_offline to get the kernel file itself. Second > it gets the Dwarf for the kernel module, which should trigger > find_debuginfo to find the separate debug file, if needed. Thanks for the sample program. I can see it tries to simulate what systemtap do. Here is the output on ubunut powerpc. Without patch: # ./dwfl_find_kernel Finding kernel release: 3.13.0-76-generic Debuginfo path: +:.debug:/usr/lib/debug setup report modname: kernel, filename: /boot/vmlinux-3.13.0-76-generic setup report modname: xfrm_user, filename: /lib/modules/3.13.0-76-generic/kernel/net/xfrm/xfrm_user.ko Found kernel file: /boot/vmlinux-3.13.0-76-generic getdwarf name: kernel kernel without DWARF Couldn't get kernel DWARF With patch: # ./dwfl_find_kernel Finding kernel release: 3.13.0-76-generic Debuginfo path: +:.debug:/usr/lib/debug setup report modname: kernel, filename: /boot/vmlinux-3.13.0-76-generic setup report modname: xfrm_user, filename: /lib/modules/3.13.0-76-generic/kernel/net/xfrm/xfrm_user.ko Found kernel file: /boot/vmlinux-3.13.0-76-generic getdwarf name: kernel mainfile: /boot/vmlinux-3.13.0-76-generic, debugfile: /usr/lib/debug/boot/vmlinux-3.13.0-76-generic Found kernel DWARF file: /usr/lib/debug/boot/vmlinux-3.13.0-76-generic > It still doesn't show the full search path (for that we should hack > find_debuginfo_in_path and try_open in libdwfl/find-debuginfo.c a bit), > but it should be a start to better understand why the current search > isn't finding the separate kernel debug file. Sry, I'm little bit confused in your last two paragraphs. 1. Does this means change I've proposed is at wrong place i.e. find_debuginfo is not correct place to change? 2. I'm not able to understand why kernel module came into picture here. when we do probe kernel.function("do_fork") {...} stap looks for kernel debuginfo only, right? Regards, Ravi ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [RFC] elfutils: Checks for debuginfo file without .debug extension as well 2016-02-19 9:37 ` Ravi Bangoria @ 2016-02-19 15:12 ` Mark Wielaard 2016-02-20 13:41 ` Ravi Bangoria 0 siblings, 1 reply; 10+ messages in thread From: Mark Wielaard @ 2016-02-19 15:12 UTC (permalink / raw) To: Ravi Bangoria Cc: elfutils-devel, systemtap, naveen.n.rao, hemant, Srikar Dronamraju [-- Attachment #1: Type: text/plain, Size: 3710 bytes --] Hi Ravi, On Fri, 2016-02-19 at 15:07 +0530, Ravi Bangoria wrote: > Thanks for the sample program. I can see it tries to simulate what > systemtap do. > Here is the output on ubunut powerpc. > > Without patch: > > # ./dwfl_find_kernel > Finding kernel release: 3.13.0-76-generic > Debuginfo path: +:.debug:/usr/lib/debug > setup report modname: kernel, filename: /boot/vmlinux-3.13.0-76-generic > setup report modname: xfrm_user, filename: > /lib/modules/3.13.0-76-generic/kernel/net/xfrm/xfrm_user.ko > Found kernel file: /boot/vmlinux-3.13.0-76-generic > getdwarf name: kernel > kernel without DWARF > Couldn't get kernel DWARF > > With patch: > > # ./dwfl_find_kernel > Finding kernel release: 3.13.0-76-generic > Debuginfo path: +:.debug:/usr/lib/debug > setup report modname: kernel, filename: /boot/vmlinux-3.13.0-76-generic > setup report modname: xfrm_user, filename: > /lib/modules/3.13.0-76-generic/kernel/net/xfrm/xfrm_user.ko > Found kernel file: /boot/vmlinux-3.13.0-76-generic > getdwarf name: kernel > mainfile: /boot/vmlinux-3.13.0-76-generic, debugfile: > /usr/lib/debug/boot/vmlinux-3.13.0-76-generic > Found kernel DWARF file: /usr/lib/debug/boot/vmlinux-3.13.0-76-generic Thanks, that looks like what I expected. I also got access to a somewhat newer version of ubuntu ppc64le and that has an additional issue. It has the vmlinux file installed unreadable (except for root). That causes almost the same issue, but slightly differently, now no kernel at all is found... Although in that case it can be worked around be using /usr/lib/debug as base. But the main issue is exactly as you describe. > > It still doesn't show the full search path (for that we should hack > > find_debuginfo_in_path and try_open in libdwfl/find-debuginfo.c a bit), > > but it should be a start to better understand why the current search > > isn't finding the separate kernel debug file. > > Sry, I'm little bit confused in your last two paragraphs. That is probably because I was still a little confused myself :) > 1. Does this means change I've proposed is at wrong place i.e. > find_debuginfo Not wrong. It correctly finds the kernel. But it could be done a little more efficient I believe. With your change we do the search through all possible directories (and subdirs) twice. We could do it slightly more efficient by moving the logic into find_debuginfo_in_path itself. The issue is really that if find_debuginfo_in_path is passed in a "null" debuglink_name then it invents one (basename + .debug). But if it is "null" then that is really just one guess. We should as you indicate also try the basename itself. So that is what the attached patch does, it checks if debuglink_name is NULL and if we are checking in a debug directory (or subdirectory), but not the main file location, then we try both basename.debug and basename. That should do the same thing as your suggested patch, but makes it slightly more efficient. We try less directories/files. And if the kernel image exists we should find it earlier. Could you try out if this variant of the patch works for you? > 2. I'm not able to understand why kernel module came into picture here. Yes sorry, that was just an artifact of my test program. setup_report_kernel gets called for all "modules" of the kernel. That includes the kernel image itself. We return 0 if we are not interested in that module, 1 if we want it and -1 to report we don't want any more. That means we see at least one kernel module. The test program prints that one out, but that is just confusing. Cheers, Mark [-- Attachment #2: find-debuginfo.patch --] [-- Type: text/x-patch, Size: 2057 bytes --] diff --git a/libdwfl/find-debuginfo.c b/libdwfl/find-debuginfo.c index 72461bc..80515db 100644 --- a/libdwfl/find-debuginfo.c +++ b/libdwfl/find-debuginfo.c @@ -163,7 +163,11 @@ find_debuginfo_in_path (Dwfl_Module *mod, const char *file_name, const char *file_basename = file_name == NULL ? NULL : basename (file_name); char *localname = NULL; - if (debuglink_file == NULL) + + /* We invent a debuglink .debug name if NULL, but then want to try the + basename too. */ + bool debuglink_null = debuglink_file == NULL; + if (debuglink_null) { /* For a alt debug multi file we need a name, for a separate debug name we may be able to fall back on file_basename.debug. */ @@ -231,6 +235,10 @@ find_debuginfo_in_path (Dwfl_Module *mod, const char *file_name, check = *p++ == '+'; check = check && cancheck; + /* Try the basename too, if we made up the debuglink name and this + is not the main directory. */ + bool try_file_basename; + const char *dir, *subdir, *file; switch (p[0]) { @@ -239,6 +247,7 @@ find_debuginfo_in_path (Dwfl_Module *mod, const char *file_name, dir = file_dirname; subdir = NULL; file = debuglink_file; + try_file_basename = false; break; case '/': /* An absolute path says to look there for a subdirectory @@ -268,6 +277,7 @@ find_debuginfo_in_path (Dwfl_Module *mod, const char *file_name, subdir = NULL; file = basename (debuglink_file); } + try_file_basename = debuglink_null; break; default: /* A relative path says to try a subdirectory of that name @@ -275,11 +285,14 @@ find_debuginfo_in_path (Dwfl_Module *mod, const char *file_name, dir = file_dirname; subdir = p; file = debuglink_file; + try_file_basename = debuglink_null; break; } char *fname = NULL; int fd = try_open (&main_stat, dir, subdir, file, &fname); + if (fd < 0 && try_file_basename) + fd = try_open (&main_stat, dir, subdir, file_basename, &fname); if (fd < 0) switch (errno) { ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [RFC] elfutils: Checks for debuginfo file without .debug extension as well 2016-02-19 15:12 ` Mark Wielaard @ 2016-02-20 13:41 ` Ravi Bangoria 2016-02-22 13:45 ` Mark Wielaard 0 siblings, 1 reply; 10+ messages in thread From: Ravi Bangoria @ 2016-02-20 13:41 UTC (permalink / raw) To: Mark Wielaard Cc: elfutils-devel, systemtap, naveen.n.rao, hemant, Srikar Dronamraju Hi Mark, Thanks for the patch. Indeed this is optimized one compared to what I've proposed. On Friday 19 February 2016 08:41 PM, Mark Wielaard wrote: > Hi Ravi, > > On Fri, 2016-02-19 at 15:07 +0530, Ravi Bangoria wrote: >> Thanks for the sample program. I can see it tries to simulate what >> systemtap do. >> Here is the output on ubunut powerpc. >> >> Without patch: >> >> # ./dwfl_find_kernel >> Finding kernel release: 3.13.0-76-generic >> Debuginfo path: +:.debug:/usr/lib/debug >> setup report modname: kernel, filename: /boot/vmlinux-3.13.0-76-generic >> setup report modname: xfrm_user, filename: >> /lib/modules/3.13.0-76-generic/kernel/net/xfrm/xfrm_user.ko >> Found kernel file: /boot/vmlinux-3.13.0-76-generic >> getdwarf name: kernel >> kernel without DWARF >> Couldn't get kernel DWARF >> >> With patch: >> >> # ./dwfl_find_kernel >> Finding kernel release: 3.13.0-76-generic >> Debuginfo path: +:.debug:/usr/lib/debug >> setup report modname: kernel, filename: /boot/vmlinux-3.13.0-76-generic >> setup report modname: xfrm_user, filename: >> /lib/modules/3.13.0-76-generic/kernel/net/xfrm/xfrm_user.ko >> Found kernel file: /boot/vmlinux-3.13.0-76-generic >> getdwarf name: kernel >> mainfile: /boot/vmlinux-3.13.0-76-generic, debugfile: >> /usr/lib/debug/boot/vmlinux-3.13.0-76-generic >> Found kernel DWARF file: /usr/lib/debug/boot/vmlinux-3.13.0-76-generic > Thanks, that looks like what I expected. > I also got access to a somewhat newer version of ubuntu ppc64le and that > has an additional issue. It has the vmlinux file installed unreadable > (except for root). That causes almost the same issue, but slightly > differently, now no kernel at all is found... Although in that case it > can be worked around be using /usr/lib/debug as base. But the main issue > is exactly as you describe. Can you please provide me system configurations like ubuntu versrion, kernel version, stap version, elfutils version etc. I'll check if I'll be able to get similar system. >>> It still doesn't show the full search path (for that we should hack >>> find_debuginfo_in_path and try_open in libdwfl/find-debuginfo.c a bit), >>> but it should be a start to better understand why the current search >>> isn't finding the separate kernel debug file. >> Sry, I'm little bit confused in your last two paragraphs. > That is probably because I was still a little confused myself :) > >> 1. Does this means change I've proposed is at wrong place i.e. >> find_debuginfo > Not wrong. It correctly finds the kernel. But it could be done a little > more efficient I believe. With your change we do the search through all > possible directories (and subdirs) twice. We could do it slightly more > efficient by moving the logic into find_debuginfo_in_path itself. > > The issue is really that if find_debuginfo_in_path is passed in a "null" > debuglink_name then it invents one (basename + .debug). But if it is > "null" then that is really just one guess. We should as you indicate > also try the basename itself. So that is what the attached patch does, > it checks if debuglink_name is NULL and if we are checking in a debug > directory (or subdirectory), but not the main file location, then we try > both basename.debug and basename. > > That should do the same thing as your suggested patch, but makes it > slightly more efficient. We try less directories/files. And if the > kernel image exists we should find it earlier. > > Could you try out if this variant of the patch works for you? I've tested your patch and it's working fine with Ubuntu 14.04 with kernel 3.13, stap 2.3 and elfutils 0.165. But I've a little doubt here. Here is the code snip of try_kernel_name from libdwfl/linux-kernel-modules.c static int try_kernel_name (Dwfl *dwfl, char **fname, bool try_debug) { LINE A: int fd = ((((dwfl->callbacks->debuginfo_path ? *dwfl->callbacks->debuginfo_path : NULL) ?: DEFAULT_DEBUGINFO_PATH)[0] == ':') ? -1 : TEMP_FAILURE_RETRY (open64 (*fname, O_RDONLY))); if (fd < 0) { LINE B: /* look for "vmlinux" files. */ fd = INTUSE(dwfl_standard_find_debuginfo) (&fakemod, NULL, NULL, 0, *fname, basename (*fname), 0, &fakemod.debug.name); if (fd < 0 && try_debug) LINE C: /* look for "vmlinux.debug" files. */ fd = INTUSE(dwfl_standard_find_debuginfo) (&fakemod, NULL, NULL, 0, *fname, NULL, 0, &fakemod.debug.name); try_kernel_name is doing almost same thing what I have proposed. Now let's say we want to go ahead with your patch, than call to dwfl_standard_find_debuginfo in LINE C will look for both vmlinux and vmlinux.debug right? But it has already checked for vmlinux in LINE B. So, in this case we have to modify try_kernel_name as well. Please correct me if I'm wrong. Regards, Ravi ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [RFC] elfutils: Checks for debuginfo file without .debug extension as well 2016-02-20 13:41 ` Ravi Bangoria @ 2016-02-22 13:45 ` Mark Wielaard 2016-02-22 17:13 ` Ravi Bangoria 0 siblings, 1 reply; 10+ messages in thread From: Mark Wielaard @ 2016-02-22 13:45 UTC (permalink / raw) To: Ravi Bangoria; +Cc: elfutils-devel, hemant, naveen.n.rao, systemtap [-- Attachment #1: Type: text/plain, Size: 3241 bytes --] Hi Ravi, On Sat, 2016-02-20 at 19:10 +0530, Ravi Bangoria wrote: > Thanks for the patch. Indeed this is optimized one compared to what I've > proposed. > > On Friday 19 February 2016 08:41 PM, Mark Wielaard wrote: > > Thanks, that looks like what I expected. > > I also got access to a somewhat newer version of ubuntu ppc64le and that > > has an additional issue. It has the vmlinux file installed unreadable > > (except for root). That causes almost the same issue, but slightly > > differently, now no kernel at all is found... Although in that case it > > can be worked around be using /usr/lib/debug as base. But the main issue > > is exactly as you describe. > > Can you please provide me system configurations like ubuntu versrion, > kernel version, stap version, elfutils version etc. I'll check if I'll > be able to get similar system. Ubuntu 15.10 wily. 4.2.0-27-generic #32-Ubuntu SMP ppc64le elfutils-0.163-4ubuntu1 stap from systemtap git. > > Could you try out if this variant of the patch works for you? > > I've tested your patch and it's working fine with Ubuntu 14.04 with > kernel 3.13, > stap 2.3 and elfutils 0.165. Great, thanks for testing. > But I've a little doubt here. Here is the code snip of try_kernel_name > from libdwfl/linux-kernel-modules.c > > static int > try_kernel_name (Dwfl *dwfl, char **fname, bool try_debug) > { > LINE A: > int fd = ((((dwfl->callbacks->debuginfo_path > ? *dwfl->callbacks->debuginfo_path : NULL) > ?: DEFAULT_DEBUGINFO_PATH)[0] == ':') ? -1 > : TEMP_FAILURE_RETRY (open64 (*fname, O_RDONLY))); > > if (fd < 0) > { > LINE B: > /* look for "vmlinux" files. */ > fd = INTUSE(dwfl_standard_find_debuginfo) (&fakemod, NULL, > NULL, 0, > *fname, basename > (*fname), 0, > &fakemod.debug.name); > if (fd < 0 && try_debug) > LINE C: > /* look for "vmlinux.debug" files. */ > fd = INTUSE(dwfl_standard_find_debuginfo) (&fakemod, NULL, > NULL, 0, > *fname, NULL, 0, > &fakemod.debug.name); > > try_kernel_name is doing almost same thing what I have proposed. Now let's > say we want to go ahead with your patch, than call to dwfl_standard_find_debuginfo > in LINE C will look for both vmlinux and vmlinux.debug right? But it has > already checked for vmlinux in LINE B. So, in this case we have to modify > try_kernel_name as well. Interesting find. And I think you are right. In our case on ubuntu ppc64le, LINE A would find the vmlinux image already and we would never reach LINE B or C. The separate debuginfo would then be found, through dwfl_standard_find_debuginfo, when we want to get the DWARF for the kernel. But on other arches where the main image isn't called vmlinux we would indeed hit them. When try_debug == true then we only need to do C now, otherwise we only need to do B. Updated patch attached. This time with updated commit message and ChangeLog entry. Does this look correct to you? Thanks, Mark [-- Attachment #2: 0001-libdwfl-Checks-for-kernel-debuginfo-file-without-.de.patch --] [-- Type: text/x-patch, Size: 5447 bytes --] From 786d65d8848efaa9d62b4121790c520d3a6782fb Mon Sep 17 00:00:00 2001 From: Ravi Bangoria <ravi.bangoria@linux.vnet.ibm.com> Date: Tue, 16 Feb 2016 21:51:35 +0530 Subject: [PATCH] libdwfl: Checks for (kernel) debuginfo file without .debug extension as well Elfutils, looking for kernel debuginfo file, tries to find it at various places. If elfutils finds /boot/vmlinu*x* file, it checks for debufginfo section. If debuginfo is not present, it saves it as 'main elf' and continue looking for debuginfo file having .debug extension i.e. vmlinux-RELEASE.debug. 'Ubuntu on powerpc' installs kernel as /boot/vmlinux and installs debuginfo without any extension as /usr/lib/debug/boot/vmlinux-RELEASE and hence, elfutils is not able to find the debuginfo file. Here is the lunchpad bug for the same: https://bugs.launchpad.net/ubuntu/+source/systemtap/+bug/1537125 This patch adds functionality to search for a kernel or debuginfo file both with and without .debug extension. Signed-off-by: Ravi Bangoria <ravi.bangoria@linux.vnet.ibm.com> Signed-off-by: Mark Wielaard <mjw@redhat.com> --- libdwfl/ChangeLog | 10 ++++++++++ libdwfl/find-debuginfo.c | 15 ++++++++++++++- libdwfl/linux-kernel-modules.c | 20 ++++++++++++-------- 3 files changed, 36 insertions(+), 9 deletions(-) diff --git a/libdwfl/ChangeLog b/libdwfl/ChangeLog index 69fd233..462a61d 100644 --- a/libdwfl/ChangeLog +++ b/libdwfl/ChangeLog @@ -1,3 +1,13 @@ +2016-02-22 Ravi Bangoria <ravi.bangoria@linux.vnet.ibm.com> + Mark Wielaard <mjw@redhat.com> + + * find-debuginfo.c (find_debuginfo_in_path): Remember whether + debuglink_file is NULL. Try file basename (without .debug) ofr + absolute and relative path if debug_file was NULL. + * linux-kernel-modules.c (try_kernel_name): If try_debug is true call + dwfl_standard_find_debuginfo with NULL debuglink_file, otherwise with + basename of fname. + 2016-02-13 Mark Wielaard <mjw@redhat.com> * linux-proc-maps.c (proc_maps_report): Free last_file when ENOEXEC. diff --git a/libdwfl/find-debuginfo.c b/libdwfl/find-debuginfo.c index 72461bc..80515db 100644 --- a/libdwfl/find-debuginfo.c +++ b/libdwfl/find-debuginfo.c @@ -163,7 +163,11 @@ find_debuginfo_in_path (Dwfl_Module *mod, const char *file_name, const char *file_basename = file_name == NULL ? NULL : basename (file_name); char *localname = NULL; - if (debuglink_file == NULL) + + /* We invent a debuglink .debug name if NULL, but then want to try the + basename too. */ + bool debuglink_null = debuglink_file == NULL; + if (debuglink_null) { /* For a alt debug multi file we need a name, for a separate debug name we may be able to fall back on file_basename.debug. */ @@ -231,6 +235,10 @@ find_debuginfo_in_path (Dwfl_Module *mod, const char *file_name, check = *p++ == '+'; check = check && cancheck; + /* Try the basename too, if we made up the debuglink name and this + is not the main directory. */ + bool try_file_basename; + const char *dir, *subdir, *file; switch (p[0]) { @@ -239,6 +247,7 @@ find_debuginfo_in_path (Dwfl_Module *mod, const char *file_name, dir = file_dirname; subdir = NULL; file = debuglink_file; + try_file_basename = false; break; case '/': /* An absolute path says to look there for a subdirectory @@ -268,6 +277,7 @@ find_debuginfo_in_path (Dwfl_Module *mod, const char *file_name, subdir = NULL; file = basename (debuglink_file); } + try_file_basename = debuglink_null; break; default: /* A relative path says to try a subdirectory of that name @@ -275,11 +285,14 @@ find_debuginfo_in_path (Dwfl_Module *mod, const char *file_name, dir = file_dirname; subdir = p; file = debuglink_file; + try_file_basename = debuglink_null; break; } char *fname = NULL; int fd = try_open (&main_stat, dir, subdir, file, &fname); + if (fd < 0 && try_file_basename) + fd = try_open (&main_stat, dir, subdir, file_basename, &fname); if (fd < 0) switch (errno) { diff --git a/libdwfl/linux-kernel-modules.c b/libdwfl/linux-kernel-modules.c index 79faf99..54c0b90 100644 --- a/libdwfl/linux-kernel-modules.c +++ b/libdwfl/linux-kernel-modules.c @@ -92,17 +92,21 @@ try_kernel_name (Dwfl *dwfl, char **fname, bool try_debug) if (fd < 0) { Dwfl_Module fakemod = { .dwfl = dwfl }; - /* First try the file's unadorned basename as DEBUGLINK_FILE, - to look for "vmlinux" files. */ - fd = INTUSE(dwfl_standard_find_debuginfo) (&fakemod, NULL, NULL, 0, - *fname, basename (*fname), 0, - &fakemod.debug.name); - if (fd < 0 && try_debug) - /* Next, let the call use the default of basename + ".debug", - to look for "vmlinux.debug" files. */ + + if (try_debug) + /* Passing NULL for DEBUGLINK_FILE searches for both the basenamer + "vmlinux" and the default of basename + ".debug", to look for + "vmlinux.debug" files. */ fd = INTUSE(dwfl_standard_find_debuginfo) (&fakemod, NULL, NULL, 0, *fname, NULL, 0, &fakemod.debug.name); + else + /* Try the file's unadorned basename as DEBUGLINK_FILE, + to look only for "vmlinux" files. */ + fd = INTUSE(dwfl_standard_find_debuginfo) (&fakemod, NULL, NULL, 0, + *fname, basename (*fname), + 0, &fakemod.debug.name); + if (fakemod.debug.name != NULL) { free (*fname); -- 1.8.3.1 ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [RFC] elfutils: Checks for debuginfo file without .debug extension as well 2016-02-22 13:45 ` Mark Wielaard @ 2016-02-22 17:13 ` Ravi Bangoria 2016-02-22 19:54 ` Mark Wielaard 0 siblings, 1 reply; 10+ messages in thread From: Ravi Bangoria @ 2016-02-22 17:13 UTC (permalink / raw) To: Mark Wielaard; +Cc: elfutils-devel, hemant, naveen.n.rao, systemtap Thanks Mark, Please find my comments. On Monday 22 February 2016 07:15 PM, Mark Wielaard wrote: > Hi Ravi, > > On Sat, 2016-02-20 at 19:10 +0530, Ravi Bangoria wrote: >> Thanks for the patch. Indeed this is optimized one compared to what I've >> proposed. >> >> On Friday 19 February 2016 08:41 PM, Mark Wielaard wrote: >>> Thanks, that looks like what I expected. >>> I also got access to a somewhat newer version of ubuntu ppc64le and that >>> has an additional issue. It has the vmlinux file installed unreadable >>> (except for root). That causes almost the same issue, but slightly >>> differently, now no kernel at all is found... Although in that case it >>> can be worked around be using /usr/lib/debug as base. But the main issue >>> is exactly as you describe. >> Can you please provide me system configurations like ubuntu versrion, >> kernel version, stap version, elfutils version etc. I'll check if I'll >> be able to get similar system. > Ubuntu 15.10 wily. 4.2.0-27-generic #32-Ubuntu SMP ppc64le > elfutils-0.163-4ubuntu1 stap from systemtap git. Yes you are right. I checked it. stap is not working on system with above configurations. > >>> Could you try out if this variant of the patch works for you? >> I've tested your patch and it's working fine with Ubuntu 14.04 with >> kernel 3.13, >> stap 2.3 and elfutils 0.165. > Great, thanks for testing. > >> But I've a little doubt here. Here is the code snip of try_kernel_name >> from libdwfl/linux-kernel-modules.c >> >> static int >> try_kernel_name (Dwfl *dwfl, char **fname, bool try_debug) >> { >> LINE A: >> int fd = ((((dwfl->callbacks->debuginfo_path >> ? *dwfl->callbacks->debuginfo_path : NULL) >> ?: DEFAULT_DEBUGINFO_PATH)[0] == ':') ? -1 >> : TEMP_FAILURE_RETRY (open64 (*fname, O_RDONLY))); >> >> if (fd < 0) >> { >> LINE B: >> /* look for "vmlinux" files. */ >> fd = INTUSE(dwfl_standard_find_debuginfo) (&fakemod, NULL, >> NULL, 0, >> *fname, basename >> (*fname), 0, >> &fakemod.debug.name); >> if (fd < 0 && try_debug) >> LINE C: >> /* look for "vmlinux.debug" files. */ >> fd = INTUSE(dwfl_standard_find_debuginfo) (&fakemod, NULL, >> NULL, 0, >> *fname, NULL, 0, >> &fakemod.debug.name); >> >> try_kernel_name is doing almost same thing what I have proposed. Now let's >> say we want to go ahead with your patch, than call to dwfl_standard_find_debuginfo >> in LINE C will look for both vmlinux and vmlinux.debug right? But it has >> already checked for vmlinux in LINE B. So, in this case we have to modify >> try_kernel_name as well. > Interesting find. And I think you are right. > > In our case on ubuntu ppc64le, LINE A would find the vmlinux image > already and we would never reach LINE B or C. The separate debuginfo > would then be found, through dwfl_standard_find_debuginfo, when we want > to get the DWARF for the kernel. > > But on other arches where the main image isn't called vmlinux we would > indeed hit them. When try_debug == true then we only need to do C now, > otherwise we only need to do B. > > Updated patch attached. This time with updated commit message and > ChangeLog entry. Does this look correct to you? This patch looks fine to me. I've tested it on Ubuntu. I think we can go ahead with this. Regards, Ravi ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [RFC] elfutils: Checks for debuginfo file without .debug extension as well 2016-02-22 17:13 ` Ravi Bangoria @ 2016-02-22 19:54 ` Mark Wielaard 0 siblings, 0 replies; 10+ messages in thread From: Mark Wielaard @ 2016-02-22 19:54 UTC (permalink / raw) To: Ravi Bangoria; +Cc: elfutils-devel, hemant, naveen.n.rao, systemtap On Mon, Feb 22, 2016 at 10:42:47PM +0530, Ravi Bangoria wrote: > >Updated patch attached. This time with updated commit message and > >ChangeLog entry. Does this look correct to you? > > This patch looks fine to me. I've tested it on Ubuntu. I think we can go > ahead with this. Great, thanks for testing and the various reviews and suggestions. I have pushed it to master now. Cheers, Mark ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2016-02-22 19:54 UTC | newest] Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2016-02-16 16:22 [RFC] elfutils: Checks for debuginfo file without .debug extension as well Ravi Bangoria 2016-02-16 16:45 ` Mark Wielaard 2016-02-17 8:21 ` Ravi Bangoria 2016-02-17 15:49 ` Mark Wielaard 2016-02-19 9:37 ` Ravi Bangoria 2016-02-19 15:12 ` Mark Wielaard 2016-02-20 13:41 ` Ravi Bangoria 2016-02-22 13:45 ` Mark Wielaard 2016-02-22 17:13 ` Ravi Bangoria 2016-02-22 19:54 ` Mark Wielaard
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).