systemtap-1.0.0 runtime failure on ARM - simple stap script translation fails

systemtap-1.0.0 runtime failure on ARM - simple stap script translation fails

[Indraneel Mukherjee]

Hi,
I'm getting the following runtime error on ARM when i try to execute a
simple stap script. The error log is below.

[root@fedora-arm root]# stap -v test.stp
Pass 1: parsed user script and 56 library script(s) in
6040usr/120sys/6210real ms.
Pass 2: analyzed script: 1 probe(s), 11 function(s), 2 embed(s), 1
global(s) in 5050usr/4610sys/13504real ms.
Pass 3: translated to C into
"/tmp/stapkharFx/stap_b865062b08e6d305fd52a1f8999b8bbf_4768.c" in
12590usr/1230sys/22308real ms.
In file included from /root/stap/share/systemtap/runtime/runtime.h:124,
                 from
/tmp/stapkharFx/stap_b865062b08e6d305fd52a1f8999b8bbf_4768.c:51:
/root/stap/share/systemtap/runtime/regs.c: In function '_stp_print_regs':
/root/stap/share/systemtap/runtime/regs.c:270: error: implicit
declaration of function 'condition_codes'
make[1]: *** [/tmp/stapkharFx/stap_b865062b08e6d305fd52a1f8999b8bbf_4768.o]
Error 1
make: *** [_module_/tmp/stapkharFx] Error 2
Pass 4: compiled C into
"stap_b865062b08e6d305fd52a1f8999b8bbf_4768.ko" in
53690usr/9670sys/66343real ms.
Pass 4: compilation failed.  Try again with another '--vp 0001' option.


The Stap Script is a simple one:
#! /usr/bin/env stap
proble vfs.read {
printf("Systemtap ARM test.\n");
exit();
}

SYSTEMTAP VERSION INFO:
I'm using the Fedora ARM distro
- systemtap-1.0.0
- gcc (GCC) 4.4.2 20091019 (Red Hat 4.4.2-5)
- elfutils-0.143-1.fc12.armv5tel

The build and run m/c is ARM(no cross instrumentation). Please let me
know if more verbose logs are required.
Googled without any meaningful pointers. Any pointers will be very helpful!!

TIA,
Indro

Re: systemtap-1.0.0 runtime failure on ARM - simple stap script translation fails

[Eugeniy Meshcheryakov]

[-- Attachment #1.1: Type: text/plain, Size: 1984 bytes --]

Hello,

This bug was fixed yesterday in the git repository. You can build
systemtap from git repo, or use simple patch (attached).

Regards,
Eugeniy Meshcheryakov

14 грудня 2009 о 11:25 +0530 Indraneel Mukherjee написав(-ла):
> Hi,
> I'm getting the following runtime error on ARM when i try to execute a
> simple stap script. The error log is below.
> 
> [root@fedora-arm root]# stap -v test.stp
> Pass 1: parsed user script and 56 library script(s) in
> 6040usr/120sys/6210real ms.
> Pass 2: analyzed script: 1 probe(s), 11 function(s), 2 embed(s), 1
> global(s) in 5050usr/4610sys/13504real ms.
> Pass 3: translated to C into
> "/tmp/stapkharFx/stap_b865062b08e6d305fd52a1f8999b8bbf_4768.c" in
> 12590usr/1230sys/22308real ms.
> In file included from /root/stap/share/systemtap/runtime/runtime.h:124,
>                  from
> /tmp/stapkharFx/stap_b865062b08e6d305fd52a1f8999b8bbf_4768.c:51:
> /root/stap/share/systemtap/runtime/regs.c: In function '_stp_print_regs':
> /root/stap/share/systemtap/runtime/regs.c:270: error: implicit
> declaration of function 'condition_codes'
> make[1]: *** [/tmp/stapkharFx/stap_b865062b08e6d305fd52a1f8999b8bbf_4768.o]
> Error 1
> make: *** [_module_/tmp/stapkharFx] Error 2
> Pass 4: compiled C into
> "stap_b865062b08e6d305fd52a1f8999b8bbf_4768.ko" in
> 53690usr/9670sys/66343real ms.
> Pass 4: compilation failed.  Try again with another '--vp 0001' option.
> 
> 
> The Stap Script is a simple one:
> #! /usr/bin/env stap
> proble vfs.read {
> printf("Systemtap ARM test.\n");
> exit();
> }
> 
> SYSTEMTAP VERSION INFO:
> I'm using the Fedora ARM distro
> - systemtap-1.0.0
> - gcc (GCC) 4.4.2 20091019 (Red Hat 4.4.2-5)
> - elfutils-0.143-1.fc12.armv5tel
> 
> The build and run m/c is ARM(no cross instrumentation). Please let me
> know if more verbose logs are required.
> Googled without any meaningful pointers. Any pointers will be very helpful!!
> 
> TIA,
> Indro

[-- Attachment #1.2: arm-condition-codes.diff --]
[-- Type: text/x-diff, Size: 862 bytes --]

commit 958c58e8231563e9349e4d8ea56c04c25e1501c0
Author: Eugeniy Meshcheryakov <eugen@debian.org>
Date:   Sat Dec 12 00:56:36 2009 +0100

    Do not use condition_codes() on arm
    
    It does not exist in recent kernels.
    
    Use regs->ARM_cpsr instead, this should work with all versions
    of linux found in git repo (versions >=2.6.12-rc2). Difference
    between condition_codes() and regs->ARM_cpsr should not matter
    for systemtap.

diff --git a/runtime/regs.c b/runtime/regs.c
index e963aff..61f2f31 100644
--- a/runtime/regs.c
+++ b/runtime/regs.c
@@ -267,7 +267,7 @@ static const char *processor_modes[]=
 
 static void _stp_print_regs(struct pt_regs * regs)
 {
-	unsigned long flags = condition_codes(regs);
+	unsigned long flags = regs->ARM_cpsr;
 
 #ifdef CONFIG_SMP
 	_stp_printf(" CPU: %d", smp_processor_id());

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: systemtap-1.0.0 runtime failure on ARM - simple stap script translation fails

[Eugeniy Meshcheryakov]

Also you can apply the patch to /usr/share/systemtap/runtime/regs.c,
no need to rebuild systemtap.

14 грудня 2009 о 08:09 +0100 Eugeniy Meshcheryakov написав(-ла):
> This bug was fixed yesterday in the git repository. You can build
> systemtap from git repo, or use simple patch (attached).
> 

Re: systemtap-1.0.0 runtime failure on ARM - simple stap script translation fails

[Indraneel Mukherjee]

Hello Eugeniy,
Thanks for the fix. Also, there was another patch that i had to apply
to get Systemtap working for ARM. I checked the latest git and the fix
is not available for ARM. The fix is here:
http://sources.redhat.com/ml/systemtap/2009-q4/msg00574.html
Without the above fix, the ARM(arm-linux is non-relocatable) stap FAILS.

This is related to
http://www.mail-archive.com/debian-bugs-dist@lists.debian.org/msg690166.html

Shouldn't this fix be added to the Systemtap tree to get it right for ARM?

Regards,
Indro


On Mon, Dec 14, 2009 at 12:42 PM, Eugeniy Meshcheryakov
<eugen@debian.org> wrote:
> Also you can apply the patch to /usr/share/systemtap/runtime/regs.c,
> no need to rebuild systemtap.
>
> 14 грудня 2009 о 08:09 +0100 Eugeniy Meshcheryakov написав(-ла):
>> This bug was fixed yesterday in the git repository. You can build
>> systemtap from git repo, or use simple patch (attached).
>>
>

Re: systemtap-1.0.0 runtime failure on ARM - simple stap script translation fails

[Eugeniy Meshcheryakov]

[-- Attachment #1: Type: text/plain, Size: 873 bytes --]

Yes, I know about that problem. But I'm not sure whether proposed patch
fixes anything. Were you able to _use_ modules made with that patch applied?
Or does it fixes only compilation? My guess is that module
initialization will fail because build-id does not match.

14 грудня 2009 о 16:11 +0530 Indraneel Mukherjee написав(-ла):
> Thanks for the fix. Also, there was another patch that i had to apply
> to get Systemtap working for ARM. I checked the latest git and the fix
> is not available for ARM. The fix is here:
> http://sources.redhat.com/ml/systemtap/2009-q4/msg00574.html
> Without the above fix, the ARM(arm-linux is non-relocatable) stap FAILS.
> 
> This is related to
> http://www.mail-archive.com/debian-bugs-dist@lists.debian.org/msg690166.html
> 
> Shouldn't this fix be added to the Systemtap tree to get it right for ARM?

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: systemtap-1.0.0 runtime failure on ARM - simple stap script translation fails

[Indraneel Mukherjee]

Hi Eugeniy,

On Mon, Dec 14, 2009 at 4:30 PM, Eugeniy Meshcheryakov <eugen@debian.org> wrote:
> Yes, I know about that problem. But I'm not sure whether proposed patch
> fixes anything. Were you able to _use_ modules made with that patch applied?

Yes, i was able to _use_ SystemTap on ARM after applying that patch +
your fix. I ran the following script and it worked absolutely fine:

global reads
probe kernel.function("do_writepages") {
        reads[execname()]++
        }
probe timer.s(3)
{
        foreach (count in reads)
                printf("%s : %d \n", count, reads[count])
}

OUTPUT of script:
pdflush : 4
pdflush : 4
pdflush : 8
pdflush : 8
pdflush : 12
pdflush : 12
pdflush : 12
...
...
> Or does it fixes only compilation? My guess is that module
> initialization will fail because build-id does not match.

No it does not fail.

Regards,
Indro

>
> 14 грудня 2009 о 16:11 +0530 Indraneel Mukherjee написав(-ла):
>> Thanks for the fix. Also, there was another patch that i had to apply
>> to get Systemtap working for ARM. I checked the latest git and the fix
>> is not available for ARM. The fix is here:
>> http://sources.redhat.com/ml/systemtap/2009-q4/msg00574.html
>> Without the above fix, the ARM(arm-linux is non-relocatable) stap FAILS.
>>
>> This is related to
>> http://www.mail-archive.com/debian-bugs-dist@lists.debian.org/msg690166.html
>>
>> Shouldn't this fix be added to the Systemtap tree to get it right for ARM?
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (GNU/Linux)
>
> iEYEARECAAYFAksmGt0ACgkQKaC6+zmozOLZtwCfasFxn9Q8LgZgLQ59sKIEyJT3
> dEMAoJC1rOkL/NN+mABa5p16hDqXMMV1
> =ud4E
> -----END PGP SIGNATURE-----
>
>

Re: systemtap-1.0.0 runtime failure on ARM - simple stap script translation fails

[Eugeniy Meshcheryakov]

[-- Attachment #1: Type: text/plain, Size: 2016 bytes --]

14 грудня 2009 о 17:55 +0530 Indraneel Mukherjee написав(-ла):
> On Mon, Dec 14, 2009 at 4:30 PM, Eugeniy Meshcheryakov <eugen@debian.org> wrote:
> > Yes, I know about that problem. But I'm not sure whether proposed patch
> > fixes anything. Were you able to _use_ modules made with that patch applied?
> 
> Yes, i was able to _use_ SystemTap on ARM after applying that patch +
> your fix. I ran the following script and it worked absolutely fine:
Hmm, that's strange. But thanks for the information. I'll look what it
is possible to do.

> 
> global reads
> probe kernel.function("do_writepages") {
>         reads[execname()]++
>         }
> probe timer.s(3)
> {
>         foreach (count in reads)
>                 printf("%s : %d \n", count, reads[count])
> }
> 
> OUTPUT of script:
> pdflush : 4
> pdflush : 4
> pdflush : 8
> pdflush : 8
> pdflush : 12
> pdflush : 12
> pdflush : 12
> ...
> ...
> > Or does it fixes only compilation? My guess is that module
> > initialization will fail because build-id does not match.
> 
> No it does not fail.
> 
> Regards,
> Indro
> 
> >
> > 14 грудня 2009 о 16:11 +0530 Indraneel Mukherjee написав(-ла):
> >> Thanks for the fix. Also, there was another patch that i had to apply
> >> to get Systemtap working for ARM. I checked the latest git and the fix
> >> is not available for ARM. The fix is here:
> >> http://sources.redhat.com/ml/systemtap/2009-q4/msg00574.html
> >> Without the above fix, the ARM(arm-linux is non-relocatable) stap FAILS.
> >>
> >> This is related to
> >> http://www.mail-archive.com/debian-bugs-dist@lists.debian.org/msg690166.html
> >>
> >> Shouldn't this fix be added to the Systemtap tree to get it right for ARM?
> >
> > -----BEGIN PGP SIGNATURE-----
> > Version: GnuPG v1.4.10 (GNU/Linux)
> >
> > iEYEARECAAYFAksmGt0ACgkQKaC6+zmozOLZtwCfasFxn9Q8LgZgLQ59sKIEyJT3
> > dEMAoJC1rOkL/NN+mABa5p16hDqXMMV1
> > =ud4E
> > -----END PGP SIGNATURE-----
> >
> >

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: systemtap-1.0.0 runtime failure on ARM - simple stap script translation fails

[Eugeniy Meshcheryakov]

[-- Attachment #1.1: Type: text/plain, Size: 2464 bytes --]

Hello Indraneel,

I think I fixed this bug in git repo, see attached patch. I was only
able to test it with arm emulator and cross-compiler and probes located
in the kernel itself, not in modules, so if you could test it with
native compilation and probes in modules it will be very good.

This patch skips saving of build-id for kernels like one for ARM. I
think the previous patch only worked because build-id check was skipped
because of check in sym.c:261 (comment says "shouldn't happen").

Regards,
Eugeniy Meshcheryakov

14 грудня 2009 о 17:55 +0530 Indraneel Mukherjee написав(-ла):
> On Mon, Dec 14, 2009 at 4:30 PM, Eugeniy Meshcheryakov <eugen@debian.org> wrote:
> > Yes, I know about that problem. But I'm not sure whether proposed patch
> > fixes anything. Were you able to _use_ modules made with that patch applied?
> 
> Yes, i was able to _use_ SystemTap on ARM after applying that patch +
> your fix. I ran the following script and it worked absolutely fine:
> 
> global reads
> probe kernel.function("do_writepages") {
>         reads[execname()]++
>         }
> probe timer.s(3)
> {
>         foreach (count in reads)
>                 printf("%s : %d \n", count, reads[count])
> }
> 
> OUTPUT of script:
> pdflush : 4
> pdflush : 4
> pdflush : 8
> pdflush : 8
> pdflush : 12
> pdflush : 12
> pdflush : 12
> ...
> ...
> > Or does it fixes only compilation? My guess is that module
> > initialization will fail because build-id does not match.
> 
> No it does not fail.
> 
> Regards,
> Indro
> 
> >
> > 14 грудня 2009 о 16:11 +0530 Indraneel Mukherjee написав(-ла):
> >> Thanks for the fix. Also, there was another patch that i had to apply
> >> to get Systemtap working for ARM. I checked the latest git and the fix
> >> is not available for ARM. The fix is here:
> >> http://sources.redhat.com/ml/systemtap/2009-q4/msg00574.html
> >> Without the above fix, the ARM(arm-linux is non-relocatable) stap FAILS.
> >>
> >> This is related to
> >> http://www.mail-archive.com/debian-bugs-dist@lists.debian.org/msg690166.html
> >>
> >> Shouldn't this fix be added to the Systemtap tree to get it right for ARM?
> >
> > -----BEGIN PGP SIGNATURE-----
> > Version: GnuPG v1.4.10 (GNU/Linux)
> >
> > iEYEARECAAYFAksmGt0ACgkQKaC6+zmozOLZtwCfasFxn9Q8LgZgLQ59sKIEyJT3
> > dEMAoJC1rOkL/NN+mABa5p16hDqXMMV1
> > =ud4E
> > -----END PGP SIGNATURE-----
> >
> >

[-- Attachment #1.2: build-id-arm.diff --]
[-- Type: text/x-diff, Size: 1204 bytes --]

commit 66f65c4fb0ddcdf8a85a29662a23a546cfd5dffe
Author: Eugeniy Meshcheryakov <eugen@debian.org>
Date:   Tue Dec 15 14:17:51 2009 +0100

    Don't save build-id if it is located before _stext
    
    This probably means that build-id will not be loaded at all and
    happens for example with ARM kernel.
    
    See also: http://sources.redhat.com/ml/systemtap/2009-q4/msg00574.html

diff --git a/translate.cxx b/translate.cxx
index 4b00615..aca0d86 100644
--- a/translate.cxx
+++ b/translate.cxx
@@ -4909,7 +4909,14 @@ dump_unwindsyms (Dwfl_Module *m,
   c->output << ".num_sections = sizeof(_stp_module_" << stpmod_idx << "_sections)/"
             << "sizeof(struct _stp_section),\n";
 
-  if (build_id_len > 0) {
+  /* Don't save build-id if it is located before _stext.
+   * This probably means that build-id will not be loaded at all and
+   * happens for example with ARM kernel.
+   *
+   * See also:
+   *    http://sources.redhat.com/ml/systemtap/2009-q4/msg00574.html
+   */
+  if (build_id_len > 0 && (build_id_vaddr > base + extra_offset)) {
     c->output << ".build_id_bits = \"" ;
     for (int j=0; j<build_id_len;j++)
       c->output << "\\x" << hex

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: systemtap-1.0.0 runtime failure on ARM - simple stap script translation fails

[naresh kamboju]

>> >
>> > 14 грудня 2009 о 16:11 +0530 Indraneel Mukherjee написав(-ла):
>> >> Thanks for the fix. Also, there was another patch that i had to apply
>> >> to get Systemtap working for ARM. I checked the latest git and the fix
>> >> is not available for ARM. The fix is here:
>> >> http://sources.redhat.com/ml/systemtap/2009-q4/msg00574.html
>> >> Without the above fix, the ARM(arm-linux is non-relocatable) stap FAILS.

Thanks for considering and modifying ARM fix patch.

Best regards
Naresh Kamboju
>> >>
>> >> This is related to
>> >> http://www.mail-archive.com/debian-bugs-dist@lists.debian.org/msg690166.html
>> >>
>> >> Shouldn't this fix be added to the Systemtap tree to get it right for ARM?
>> >
>> > -----BEGIN PGP SIGNATURE-----
>> > Version: GnuPG v1.4.10 (GNU/Linux)
>> >
>> > iEYEARECAAYFAksmGt0ACgkQKaC6+zmozOLZtwCfasFxn9Q8LgZgLQ59sKIEyJT3
>> > dEMAoJC1rOkL/NN+mABa5p16hDqXMMV1
>> > =ud4E
>> > -----END PGP SIGNATURE-----
>> >
>> >
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (GNU/Linux)
>
> iEYEARECAAYFAksnlBkACgkQKaC6+zmozOKIGQCdEiap7TFQhuQiIz9e1ewA+OCG
> AhkAn3LjlL7hsjTSRbJw/7wczoNfmjtI
> =mleX
> -----END PGP SIGNATURE-----
>
>