Using abrt to automatically download and install debuginfo

Using abrt to automatically download and install debuginfo

[Chris Meek]

Hi all,

I recently added a feature (PR12773, commit 
e1e8b44e3293090f29262beb1b0762951b833e62)
to SystemTap that uses abrt to automatically download and install needed 
debuginfo. I will be
extending the functionality in the near future to include command line 
options to control the
use of the feature (such as whether or not to use it, or to set a 
timeout interval).

In my tests, I found that a small debuginfo file of around 0.2MB took 
only an average of
1.34 seconds to download and install, whereas a larger file of 76.5 MB 
took around 6.4
minutes to download and install. Of course these times are heavily 
dependant on the
internet connection, so they should only be taken as a rough ballpark 
estimate.

Example:

$ stap -e 'probe process("/bin/gzip").function("*") { log(pp()) }'

If the gzip debuginfo is not already installed, SystemTap will call
"abrt-action-install-debuginfo-to-abrt-cache" to first download and then 
install the "gzip.debug"
file into: "/var/cache/abrt-di/usr/lib/debug/" . SystemTap will then 
look in this directory to use the
debuginfo for the probe.

If you have any comments or suggestions, I'd like to hear them.

Cheers,
Chris

Re: Using abrt to automatically download and install debuginfo

[Chris Meek]

On 08/09/2011 02:16 PM, Chris Meek wrote:
> Hi all,
>
> I recently added a feature (PR12773, commit 
> e1e8b44e3293090f29262beb1b0762951b833e62)
> to SystemTap that uses abrt to automatically download and install 
> needed debuginfo. I will be
> extending the functionality in the near future to include command line 
> options to control the
> use of the feature (such as whether or not to use it, or to set a 
> timeout interval).
>
> In my tests, I found that a small debuginfo file of around 0.2MB took 
> only an average of
> 1.34 seconds to download and install, whereas a larger file of 76.5 MB 
> took around 6.4
> minutes to download and install. Of course these times are heavily 
> dependant on the
> internet connection, so they should only be taken as a rough ballpark 
> estimate.
>
> Example:
>
> $ stap -e 'probe process("/bin/gzip").function("*") { log(pp()) }'
>
> If the gzip debuginfo is not already installed, SystemTap will call
> "abrt-action-install-debuginfo-to-abrt-cache" to first download and 
> then install the "gzip.debug"
> file into: "/var/cache/abrt-di/usr/lib/debug/" . SystemTap will then 
> look in this directory to use the
> debuginfo for the probe.
>
> If you have any comments or suggestions, I'd like to hear them.
>
> Cheers,
> Chris
Hi all,

I am working on the command line option for this feature and I would be 
grateful for
your input. I will sketch out what I am thinking, and if you have any 
thoughts on how it
could be better, please let me know.

So by default the feature will be enabled, with a timeout of 30 seconds, 
and it will pass '-y' to abrt,
causing it to continue without user interaction.

The option "--download-debuginfo=" will have the following possible values:
no                //to disable it
<###>       //to change the timeout to the given value
ask              //to not pass '-y' to abrt, so it will ask for user 
confirmation before downloading

It could also be set up so that if no value is given (i.e. just 
"--download-debuginfo") it would continue
without a timeout, or it could have a "no-timeout" explicit value.

Please let me know what you think.

Thanks,
Chris

Re: Using abrt to automatically download and install debuginfo

[Josh Stone]

On 08/09/2011 01:32 PM, Chris Meek wrote:
> I am working on the command line option for this feature and I would be 
> grateful for
> your input. I will sketch out what I am thinking, and if you have any 
> thoughts on how it
> could be better, please let me know.
> 
> So by default the feature will be enabled, with a timeout of 30 seconds, 
> and it will pass '-y' to abrt,
> causing it to continue without user interaction.

It's a nice feature, but I'm not a fan of defaults that cause network
activity.  I rallied against the same for --use-server-on-error.
Default "ask" is not much better, as it makes us interactive and thus
burdens scripting use.  So I'd prefer default-no.

We could brainstorm various ways for users to set their own default
preferences, but that's somewhat orthogonal.


Josh

Re: Using abrt to automatically download and install debuginfo

[Mark Wielaard]

Hi Chris,

On Tue, 2011-08-09 at 14:16 -0400, Chris Meek wrote:
> $ stap -e 'probe process("/bin/gzip").function("*") { log(pp()) }'
> 
> If the gzip debuginfo is not already installed, SystemTap will call
> "abrt-action-install-debuginfo-to-abrt-cache" to first download and then 
> install the "gzip.debug"
> file into: "/var/cache/abrt-di/usr/lib/debug/" . SystemTap will then 
> look in this directory to use the
> debuginfo for the probe.
> 
> If you have any comments or suggestions, I'd like to hear them.

I tried it out on my system, but it doesn't seem to work. There is just
a small delay and then nothing... So it could provide a bit more
feedback.

$ /usr/local/install/systemtap/bin/stap -e 'probe
process("/bin/ls").function("main") { log(pp()); exit(); }'
WARNING: cannot find module /bin/ls debuginfo: No DWARF information
found
semantic error: no match while resolving probe point
process("/bin/ls").function("main")
Pass 2: analysis failed.  Try again with another '--vp 01' option.
Missing separate debuginfos, use: debuginfo-install
coreutils-8.12-3.fc17.x86_64 

Running with some -vvvvv I get:

$ /usr/local/install/systemtap/bin/stap -vvvvv -e 'probe process("/bin/ls").function("main") { log(pp()); exit(); }'
Systemtap translator/driver (version 1.7/0.152 commit release-1.6-89-g7b45b42 + changes)
Copyright (C) 2005-2011 Red Hat, Inc. and others
This is free software; see the source for copying conditions.
enabled features: LIBRPM LIBSQLITE3 NSS BOOST_SHARED_PTR TR1_UNORDERED_MAP NLS
Created temporary directory "/tmp/stapNZcnCm"
Session arch: x86_64 release: 3.1.0-0.rc1.git2.2.fc17.x86_64
Parsed kernel "/lib/modules/3.1.0-0.rc1.git2.2.fc17.x86_64/build/.config", containing 3770 tuples
Parsed kernel /lib/modules/3.1.0-0.rc1.git2.2.fc17.x86_64/build/Module.symvers, which contained 6057 vmlinux exports
Searched: " /usr/local/install/systemtap/share/systemtap/tapset/x86_64/*.stp ", found: 4, processed: 4
Searched: " /usr/local/install/systemtap/share/systemtap/tapset/*.stp ", found: 72, processed: 72
Pass 1: parsed user script and 76 library script(s) using 195488virt/22744res/2804shr kb, in 110usr/20sys/132real ms.
dwarf_builder::build for /bin/ls
parse 'main', func 'main'
pattern '/bin/ls' matches module '/bin/ls'
Extracting build ID.
Searching for debuginfo with build ID: '57f97a49fed0bc89c24b1e4917c6e862d858d6c8'.
Downloading and installing debuginfo with build ID: '57f97a49fed0bc89c24b1e4917c6e862d858d6c8' using /usr/bin/abrt-action-install-debuginfo-to-abrt-cache.
Running /bin/sh -c echo 57f97a49fed0bc89c24b1e4917c6e862d858d6c8 | /usr/bin/abrt-action-install-debuginfo-to-abrt-cache -y --ids=-
Spawn waitpid result (0x0): 0
focused on module '/bin/ls' = [0x0x400000, -0x0x81a3f8, bias 0x0 file /bin/ls ELF machine |x86_64 (code 62)
focused on module '/bin/ls'
WARNING: cannot find module /bin/ls debuginfo: No DWARF information found
Extracting build ID.
Searching for debuginfo with build ID: '57f97a49fed0bc89c24b1e4917c6e862d858d6c8'.
Downloading and installing debuginfo with build ID: '57f97a49fed0bc89c24b1e4917c6e862d858d6c8' using /usr/bin/abrt-action-install-debuginfo-to-abrt-cache.
Running /bin/sh -c echo 57f97a49fed0bc89c24b1e4917c6e862d858d6c8 | /usr/bin/abrt-action-install-debuginfo-to-abrt-cache -y --ids=-
Spawn waitpid result (0x0): 0
semantic error: no match while resolving probe point process("/bin/ls").function("main")
deleting module_cache
Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0 global(s) using 198668virt/27620res/4628shr kb, in 420usr/460sys/900real ms.
Pass 2: analysis failed.  Try again with another '--vp 01' option.
Missing separate debuginfos, use: debuginfo-install coreutils-8.12-3.fc17.x86_64 
Running rm -rf /tmp/stapNZcnCm
Spawn waitpid result (0x0): 0

So it actually tries to run twice, but fails for some reason?

Running by hand give:

$ echo 57f97a49fed0bc89c24b1e4917c6e862d858d6c8
| /usr/bin/abrt-action-install-debuginfo-to-abrt-cache -y --ids=-
Coredump references 1 debuginfo files, 1 of them are not installed
Looking for needed packages in repositories
Packages to download: 1
Downloading 3.18Mb, installed size: 19.10Mb. Continue? [y/N] Download
cancelled by user
Missing debuginfo
file: /usr/lib/debug/.build-id/57/f97a49fed0bc89c24b1e4917c6e862d858d6c8.debug

That is strange, "Download cancelled by user"?

Seems something is broken. I can only make
abrt-action-install-debuginfo-to-abrt-cache work by doing:
$ echo 57f97a49fed0bc89c24b1e4917c6e862d858d6c8 > build_ids
$ abrt-action-install-debuginfo-to-abrt-cache
With or without -y, I have to answer the question about wanting to
download.

$ /usr/bin/abrt-action-install-debuginfo-to-abrt-cache -y
Coredump references 1 debuginfo files, 1 of them are not installed
Looking for needed packages in repositories
Packages to download: 1
Downloading 3.18Mb, installed size: 19.10Mb. Continue? [y/N] y
Downloading (1 of 1) coreutils-debuginfo-8.12-3.fc17.x86_64.rpm: 100%
Extracting cpio
from /tmp/abrt-tmp-debuginfo-2011-08-12-11:53:17.27655/coreutils-debuginfo-8.12-3.fc17.x86_64.rpm
Caching files from unpacked.cpio made from
coreutils-debuginfo-8.12-3.fc17.x86_64.rpm
Removing /tmp/abrt-tmp-debuginfo-2011-08-12-11:53:17.27655

Anyway, some more (verbose) output would be helpful.

Thanks,

Mark

Re: Using abrt to automatically download and install debuginfo

[Chris Meek]

On 08/12/2011 05:55 AM, Mark Wielaard wrote:
> Hi Chris,
> 
> On Tue, 2011-08-09 at 14:16 -0400, Chris Meek wrote:
>> $ stap -e 'probe process("/bin/gzip").function("*") { log(pp()) }'
>>
>> If the gzip debuginfo is not already installed, SystemTap will call
>> "abrt-action-install-debuginfo-to-abrt-cache" to first download and then 
>> install the "gzip.debug"
>> file into: "/var/cache/abrt-di/usr/lib/debug/" . SystemTap will then 
>> look in this directory to use the
>> debuginfo for the probe.
>>
>> If you have any comments or suggestions, I'd like to hear them.
> 
> I tried it out on my system, but it doesn't seem to work. There is just
> a small delay and then nothing... So it could provide a bit more
> feedback.
> 
> $ /usr/local/install/systemtap/bin/stap -e 'probe
> process("/bin/ls").function("main") { log(pp()); exit(); }'
> WARNING: cannot find module /bin/ls debuginfo: No DWARF information
> found
> semantic error: no match while resolving probe point
> process("/bin/ls").function("main")
> Pass 2: analysis failed.  Try again with another '--vp 01' option.
> Missing separate debuginfos, use: debuginfo-install
> coreutils-8.12-3.fc17.x86_64 
> 
> Running with some -vvvvv I get:
> 
> $ /usr/local/install/systemtap/bin/stap -vvvvv -e 'probe process("/bin/ls").function("main") { log(pp()); exit(); }'
> Systemtap translator/driver (version 1.7/0.152 commit release-1.6-89-g7b45b42 + changes)
> Copyright (C) 2005-2011 Red Hat, Inc. and others
> This is free software; see the source for copying conditions.
> enabled features: LIBRPM LIBSQLITE3 NSS BOOST_SHARED_PTR TR1_UNORDERED_MAP NLS
> Created temporary directory "/tmp/stapNZcnCm"
> Session arch: x86_64 release: 3.1.0-0.rc1.git2.2.fc17.x86_64
> Parsed kernel "/lib/modules/3.1.0-0.rc1.git2.2.fc17.x86_64/build/.config", containing 3770 tuples
> Parsed kernel /lib/modules/3.1.0-0.rc1.git2.2.fc17.x86_64/build/Module.symvers, which contained 6057 vmlinux exports
> Searched: " /usr/local/install/systemtap/share/systemtap/tapset/x86_64/*.stp ", found: 4, processed: 4
> Searched: " /usr/local/install/systemtap/share/systemtap/tapset/*.stp ", found: 72, processed: 72
> Pass 1: parsed user script and 76 library script(s) using 195488virt/22744res/2804shr kb, in 110usr/20sys/132real ms.
> dwarf_builder::build for /bin/ls
> parse 'main', func 'main'
> pattern '/bin/ls' matches module '/bin/ls'
> Extracting build ID.
> Searching for debuginfo with build ID: '57f97a49fed0bc89c24b1e4917c6e862d858d6c8'.
> Downloading and installing debuginfo with build ID: '57f97a49fed0bc89c24b1e4917c6e862d858d6c8' using /usr/bin/abrt-action-install-debuginfo-to-abrt-cache.
> Running /bin/sh -c echo 57f97a49fed0bc89c24b1e4917c6e862d858d6c8 | /usr/bin/abrt-action-install-debuginfo-to-abrt-cache -y --ids=-
> Spawn waitpid result (0x0): 0
> focused on module '/bin/ls' = [0x0x400000, -0x0x81a3f8, bias 0x0 file /bin/ls ELF machine |x86_64 (code 62)
> focused on module '/bin/ls'
> WARNING: cannot find module /bin/ls debuginfo: No DWARF information found
> Extracting build ID.
> Searching for debuginfo with build ID: '57f97a49fed0bc89c24b1e4917c6e862d858d6c8'.
> Downloading and installing debuginfo with build ID: '57f97a49fed0bc89c24b1e4917c6e862d858d6c8' using /usr/bin/abrt-action-install-debuginfo-to-abrt-cache.
> Running /bin/sh -c echo 57f97a49fed0bc89c24b1e4917c6e862d858d6c8 | /usr/bin/abrt-action-install-debuginfo-to-abrt-cache -y --ids=-
> Spawn waitpid result (0x0): 0
> semantic error: no match while resolving probe point process("/bin/ls").function("main")
> deleting module_cache
> Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0 global(s) using 198668virt/27620res/4628shr kb, in 420usr/460sys/900real ms.
> Pass 2: analysis failed.  Try again with another '--vp 01' option.
> Missing separate debuginfos, use: debuginfo-install coreutils-8.12-3.fc17.x86_64 
> Running rm -rf /tmp/stapNZcnCm
> Spawn waitpid result (0x0): 0
> 
> So it actually tries to run twice, but fails for some reason?
> 
> Running by hand give:
> 
> $ echo 57f97a49fed0bc89c24b1e4917c6e862d858d6c8
> | /usr/bin/abrt-action-install-debuginfo-to-abrt-cache -y --ids=-
> Coredump references 1 debuginfo files, 1 of them are not installed
> Looking for needed packages in repositories
> Packages to download: 1
> Downloading 3.18Mb, installed size: 19.10Mb. Continue? [y/N] Download
> cancelled by user
> Missing debuginfo
> file: /usr/lib/debug/.build-id/57/f97a49fed0bc89c24b1e4917c6e862d858d6c8.debug
> 
> That is strange, "Download cancelled by user"?
> 
> Seems something is broken. I can only make
> abrt-action-install-debuginfo-to-abrt-cache work by doing:
> $ echo 57f97a49fed0bc89c24b1e4917c6e862d858d6c8 > build_ids
> $ abrt-action-install-debuginfo-to-abrt-cache
> With or without -y, I have to answer the question about wanting to
> download.
> 
> $ /usr/bin/abrt-action-install-debuginfo-to-abrt-cache -y
> Coredump references 1 debuginfo files, 1 of them are not installed
> Looking for needed packages in repositories
> Packages to download: 1
> Downloading 3.18Mb, installed size: 19.10Mb. Continue? [y/N] y
> Downloading (1 of 1) coreutils-debuginfo-8.12-3.fc17.x86_64.rpm: 100%
> Extracting cpio
> from /tmp/abrt-tmp-debuginfo-2011-08-12-11:53:17.27655/coreutils-debuginfo-8.12-3.fc17.x86_64.rpm
> Caching files from unpacked.cpio made from
> coreutils-debuginfo-8.12-3.fc17.x86_64.rpm
> Removing /tmp/abrt-tmp-debuginfo-2011-08-12-11:53:17.27655
> 
> Anyway, some more (verbose) output would be helpful.
> 
> Thanks,
> 
> Mark

Hi Mark,

There are still a few issues with the abrt-action-install-debuginfo-to-abrt-cache program itself. 
Which version of abrt do you have installed? It works for me with abrt-2.0.3-1.fc15.x86_64.
Also, make sure you have abrt-addon-ccpp-2.0.3-1.fc15.x86_64 installed.    

I am currently working on an update to PR12773 which should address a number of these issues. 
Also, it will not be enabled by default - the explicit use of --download-debuginfo will be
required for now to use the feature. The reason for this is that abrt-action-install-debuginfo-to-abrt-cache
needs to be updated to fix a few bugs that are causing problems.

I will also add some more verbose output.

Cheers,
Chris

Re: Using abrt to automatically download and install debuginfo

[Mark Wielaard]

On Fri, 2011-08-12 at 09:19 -0400, Chris Meek wrote:
> There are still a few issues with the abrt-action-install-debuginfo-to-abrt-cache program itself. 
> Which version of abrt do you have installed? It works for me with abrt-2.0.3-1.fc15.x86_64.

abrt-2.0.4-1.fc16.x86_64

> Also, make sure you have abrt-addon-ccpp-2.0.3-1.fc15.x86_64 installed.    

abrt-addon-ccpp-2.0.4-1.fc16.x86_64

> I am currently working on an update to PR12773 which should address a number of these issues. 
> Also, it will not be enabled by default - the explicit use of --download-debuginfo will be
> required for now to use the feature. The reason for this is that abrt-action-install-debuginfo-to-abrt-cache
> needs to be updated to fix a few bugs that are causing problems.

OK, I'll wait for the updates.

Thanks,

Mark

Re: Using abrt to automatically download and install debuginfo

[Chris Meek]

On 08/12/2011 09:31 AM, Mark Wielaard wrote:
> On Fri, 2011-08-12 at 09:19 -0400, Chris Meek wrote:
>> There are still a few issues with the abrt-action-install-debuginfo-to-abrt-cache program itself. 
>> Which version of abrt do you have installed? It works for me with abrt-2.0.3-1.fc15.x86_64.
> 
> abrt-2.0.4-1.fc16.x86_64
> 
>> Also, make sure you have abrt-addon-ccpp-2.0.3-1.fc15.x86_64 installed.    
> 
> abrt-addon-ccpp-2.0.4-1.fc16.x86_64
> 
>> I am currently working on an update to PR12773 which should address a number of these issues. 
>> Also, it will not be enabled by default - the explicit use of --download-debuginfo will be
>> required for now to use the feature. The reason for this is that abrt-action-install-debuginfo-to-abrt-cache
>> needs to be updated to fix a few bugs that are causing problems.
> 
> OK, I'll wait for the updates.
> 
> Thanks,
> 
> Mark

Hi Mark,

There seems to be an even greater issue with abrt-2.0.4-1.fc16.x86_64. I also couldn't get it to work with that one installed. Try it with the F15 version: abrt-2.0.3-1.fc15.x86_64.

Cheers,
Chris

Re: Using abrt to automatically download and install debuginfo

[Mark Wielaard]

On Fri, 2011-08-12 at 09:44 -0400, Chris Meek wrote:
> On 08/12/2011 09:31 AM, Mark Wielaard wrote:
> > On Fri, 2011-08-12 at 09:19 -0400, Chris Meek wrote:
> >> There are still a few issues with the abrt-action-install-debuginfo-to-abrt-cache program itself. 
> >> Which version of abrt do you have installed? It works for me with abrt-2.0.3-1.fc15.x86_64.
> > 
> > abrt-2.0.4-1.fc16.x86_64
> > 
> >> Also, make sure you have abrt-addon-ccpp-2.0.3-1.fc15.x86_64 installed.    
> > 
> > abrt-addon-ccpp-2.0.4-1.fc16.x86_64
> > 
> There seems to be an even greater issue with abrt-2.0.4-1.fc16.x86_64.
> I also couldn't get it to work with that one installed. Try it with
> the F15 version: abrt-2.0.3-1.fc15.x86_64.

You are right. That version does work. Weird there are problems with
both older and newer versions.

Thanks,

Mark