Prelinking on ARM

Prelinking on ARM

[Torsten Polle]

Hi,

I've a problem with prelinked shared and non-prelinked debug information
for the same on ARM. The relative offset to the .text section differs
for the prelinked shared library and the non-prelinked debug
information. SystemTap only compensates for the difference in the start
addresses. I tried to debug the problem, but I've to admit that the
problem is well over my head. Could someone give me a hint on this
matter?

Thanks,
Torsten

Re: Prelinking on ARM

[Mark Wielaard]

On Mon, 2015-06-01 at 22:43 +0200, Torsten Polle wrote:
> I've a problem with prelinked shared and non-prelinked debug information
> for the same on ARM. The relative offset to the .text section differs
> for the prelinked shared library and the non-prelinked debug
> information. SystemTap only compensates for the difference in the start
> addresses. I tried to debug the problem, but I've to admit that the
> problem is well over my head. Could someone give me a hint on this
> matter?

Does the testsuite/systemtap.exelib/exelib.exp test work for you?

# Builds various variants of an executable and a shared library
# (with gcc/g++, -O0/-O3, prelinked/pie, seperate debuginfo)
# Then runs tests with a list of execs.

# Don't enable all sub-tests by default, enable them separately when
# you want to track down an issue.
# Enable just the all-in-one test for regular test runs to safe time
#set subtestlist {lib mark uname ustack cleanup}

make installcheck RUNTESTFLAGS=exelib.exp

Which for me on an x86_64 and a armv7hl give:

Snapshot: version 2.8/0.160, commit release-2.7-126-gf80d9cdb3078
GCC: 4.8.3 [gcc (GCC) 4.8.3 20140911 (Red Hat 4.8.3-9)]
Distro: Red Hat Enterprise Linux Workstation release 7.1 (Maipo)
SElinux: Enforcing


Running /home/mark/src/systemtap/testsuite/systemtap.exelib/exelib.exp ...

		=== systemtap Summary ===

# of expected passes		376

Snapshot: version 2.8/0.161, commit release-2.7-160-g1eed3df165bc +
changes
GCC: 5.1.1 [gcc (GCC) 5.1.1 20150422 (Red Hat 5.1.1-1)]
Distro: Fedora release 22 (Twenty Two)
SElinux: Enforcing


Running ./systemtap.exelib/exelib.exp ...

		=== systemtap Summary ===

# of expected passes		188

(arm has half the number of passes since x86_64 actually does twice as
many tests, for both normal 64bits binaries and for an -m32 binaries.)

If the above doesn't match what you see on your setup could you enable
the subtests as indicated in the .exp file and see which particular
subtests fail (maybe include the relevant parts of the
testsuite/systemtap.log file).

Thanks,

Mark

Re: Prelinking on ARM

[Torsten Polle]

Mark,

Am 02.06.2015 um 20:28 schrieb Mark Wielaard <mjw@redhat.com>:
> On Mon, 2015-06-01 at 22:43 +0200, Torsten Polle wrote:
>> I've a problem with prelinked shared and non-prelinked debug information
>> for the same on ARM. The relative offset to the .text section differs
>> for the prelinked shared library and the non-prelinked debug
>> information. SystemTap only compensates for the difference in the start
>> addresses. I tried to debug the problem, but I've to admit that the
>> problem is well over my head. Could someone give me a hint on this
>> matter?
> 
> Does the testsuite/systemtap.exelib/exelib.exp test work for you?

I've to check. I'm using a cross compile environment, i.e. I only run
passes 1 - 4 on my host (X86) and execute staprun on my target
(ARM). I could provide a small shared library as an example instead.

> # Builds various variants of an executable and a shared library
> # (with gcc/g++, -O0/-O3, prelinked/pie, seperate debuginfo)
> # Then runs tests with a list of execs.
> 
> # Don't enable all sub-tests by default, enable them separately when
> # you want to track down an issue.
> # Enable just the all-in-one test for regular test runs to safe time
> #set subtestlist {lib mark uname ustack cleanup}
> 
> make installcheck RUNTESTFLAGS=exelib.exp
> 
> Which for me on an x86_64 and a armv7hl give:
> 
> Snapshot: version 2.8/0.160, commit release-2.7-126-gf80d9cdb3078
> GCC: 4.8.3 [gcc (GCC) 4.8.3 20140911 (Red Hat 4.8.3-9)]
> Distro: Red Hat Enterprise Linux Workstation release 7.1 (Maipo)
> SElinux: Enforcing
> 
> 
> Running /home/mark/src/systemtap/testsuite/systemtap.exelib/exelib.exp ...
> 
> 		=== systemtap Summary ===
> 
> # of expected passes		376
> 
> Snapshot: version 2.8/0.161, commit release-2.7-160-g1eed3df165bc +
> changes
> GCC: 5.1.1 [gcc (GCC) 5.1.1 20150422 (Red Hat 5.1.1-1)]
> Distro: Fedora release 22 (Twenty Two)
> SElinux: Enforcing
> 
> 
> Running ./systemtap.exelib/exelib.exp ...
> 
> 		=== systemtap Summary ===
> 
> # of expected passes		188
> 
> (arm has half the number of passes since x86_64 actually does twice as
> many tests, for both normal 64bits binaries and for an -m32 binaries.)
> 
> If the above doesn't match what you see on your setup could you enable
> the subtests as indicated in the .exp file and see which particular
> subtests fail (maybe include the relevant parts of the
> testsuite/systemtap.log file).
> 
> Thanks,
> 
> Mark

Thanks,
Torsten

Re: Prelinking on ARM

[Mark Wielaard]

On Tue, 2015-06-02 at 22:29 +0200, Torsten Polle wrote:
> Am 02.06.2015 um 20:28 schrieb Mark Wielaard <mjw@redhat.com>:
> > On Mon, 2015-06-01 at 22:43 +0200, Torsten Polle wrote:
> >> I've a problem with prelinked shared and non-prelinked debug information
> >> for the same on ARM. The relative offset to the .text section differs
> >> for the prelinked shared library and the non-prelinked debug
> >> information. SystemTap only compensates for the difference in the start
> >> addresses. I tried to debug the problem, but I've to admit that the
> >> problem is well over my head. Could someone give me a hint on this
> >> matter?
> > 
> > Does the testsuite/systemtap.exelib/exelib.exp test work for you?
> 
> I've to check. I'm using a cross compile environment, i.e. I only run
> passes 1 - 4 on my host (X86) and execute staprun on my target
> (ARM). I could provide a small shared library as an example instead.

I must admit to not have much experience with the
cross-compiling/cross-stapping mechanism. If you could provide a small
shared library (plus separate debuginfo file?) and the stap invocation
that doesn't work correctly that would be helpful for understanding what
goes wrong.

BTW. Which versions of systemtap and elfutils does stap -V give you?

$ stap -V
Systemtap translator/driver (version 2.7/0.161, rpm 2.7-1.fc22)
                    the stap version -^   ^- the elfutils version

Thanks,

Mark

Re: Prelinking on ARM

[Torsten Polle]

Mark,

Mark Wielaard writes:
 > On Tue, 2015-06-02 at 22:29 +0200, Torsten Polle wrote:
 >> Am 02.06.2015 um 20:28 schrieb Mark Wielaard <mjw@redhat.com>:
 >>> On Mon, 2015-06-01 at 22:43 +0200, Torsten Polle wrote:
 >>>> I've a problem with prelinked shared and non-prelinked debug information
 >>>> for the same on ARM. The relative offset to the .text section differs
 >>>> for the prelinked shared library and the non-prelinked debug
 >>>> information. SystemTap only compensates for the difference in the start
 >>>> addresses. I tried to debug the problem, but I've to admit that the
 >>>> problem is well over my head. Could someone give me a hint on this
 >>>> matter?
 >>> 
 >>> Does the testsuite/systemtap.exelib/exelib.exp test work for you?
 >> 
 >> I've to check. I'm using a cross compile environment, i.e. I only run
 >> passes 1 - 4 on my host (X86) and execute staprun on my target
 >> (ARM). I could provide a small shared library as an example instead.

 > I must admit to not have much experience with the
 > cross-compiling/cross-stapping mechanism. If you could provide a
 > small shared library (plus separate debuginfo file?) and the stap
 > invocation that doesn't work correctly that would be helpful for
 > understanding what goes wrong.

I tried to pull together a small example. But so far I failed.
Therefore I please allow me to summarise my understanding of the
situation.

prelinked                 debug information
+----------------------+    +----------------------+
|         ...          |    |         ...          |
+----------------------+    +----------------------+
| .rel.dyn size = 0x20 |    | .rel.dyn size = 0x10 |
+----------------------+    +----------------------+
|         ...          |    |         ...          |
+----------------------+    +----------------------+
| .text                |    | .text                |
+----------------------+    +----------------------+

The relocations of type R_ARM_RELATIVE in section ".rel.dyn" change 
their type from REL to RELA, thereby increasing the size of the section ".rel.dyn“.

Consequently the offset of the ".text" section differs. The value of
variables (elf, module) bias in tapset.cxx only seem to take into
account that the difference between the start address of the prelinked
library and the start of the debug information is the same as the difference
between the ".text" section in both files. Alas this is not the case as
section ".rel.dyn" has changed.

If the load address of the prelinked library is 0x4000000, the address of the section „.text "might be 0x40000120, whereas the address of the section „.text“ in the debug information is 0x00000110. The bias is always calculated as 0x40000000. All probes in the .text section have a wrong offset by 0x10.

I hope this provides some insight into the problem. I'll continue my
attempts to provide a small example in the meantime.

 > BTW. Which versions of systemtap and elfutils does stap -V give you?

 > $ stap -V
 > Systemtap translator/driver (version 2.7/0.161, rpm 2.7-1.fc22)
 >                     the stap version -^   ^- the elfutils version

The versions I used for my current tests are:
Systemtap translator/driver (version 2.7/0.160, commit release-2.6-199-gd26e62cd8b30)
Systemtap translator/driver (version 2.8/0.160, commit release-2.7-166-gae0db08baa54)

 > Thanks,
 > Mark

Thanks,
Torsten

Re: Prelinking on ARM

[Torsten Polle]

[-- Attachment #1: Type: text/plain, Size: 3506 bytes --]

Mark,

Torsten Polle writes:
 > Mark,

 > Mark Wielaard writes:
 >> On Tue, 2015-06-02 at 22:29 +0200, Torsten Polle wrote:
 >>> Am 02.06.2015 um 20:28 schrieb Mark Wielaard <mjw@redhat.com>:
 >>>> On Mon, 2015-06-01 at 22:43 +0200, Torsten Polle wrote:
 >>>>> I've a problem with prelinked shared and non-prelinked debug information
 >>>>> for the same on ARM. The relative offset to the .text section differs
 >>>>> for the prelinked shared library and the non-prelinked debug
 >>>>> information. SystemTap only compensates for the difference in the start
 >>>>> addresses. I tried to debug the problem, but I've to admit that the
 >>>>> problem is well over my head. Could someone give me a hint on this
 >>>>> matter?
 >>>> 
 >>>> Does the testsuite/systemtap.exelib/exelib.exp test work for you?
 >>> 
 >>> I've to check. I'm using a cross compile environment, i.e. I only run
 >>> passes 1 - 4 on my host (X86) and execute staprun on my target
 >>> (ARM). I could provide a small shared library as an example instead.

 >> I must admit to not have much experience with the
 >> cross-compiling/cross-stapping mechanism. If you could provide a
 >> small shared library (plus separate debuginfo file?) and the stap
 >> invocation that doesn't work correctly that would be helpful for
 >> understanding what goes wrong.

please find my example below. Unfortunately, I could not reproduce the
problem with a smaller library. The prelinked library and the debug
information is a little bit larger.

I set a function call and function return probe in function
"__libc_malloc".

In the prelinked file [1], the address is 0x410ddd94. The offset from the load
address is 0x410ddd94 - 0x41068000 = 0x75D94

But the offest given in the debug information is only 0x00074954.

The debug output from SystemTap [3] shows a pc of 0x74954 and a module bias
0x41079440.

[1] prelinked:
objdump -x 
    LOAD off    0x00000000 vaddr 0x41068000 paddr 0x41068000 align 2**15
         filesz 0x0012e268 memsz 0x0012e268 flags r-x
...
  7 .rel.dyn      00003c18  4107b318  4107b318  00013318  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
...
 10 .text         000f3878  4107f080  4107f080  00017080  2**6
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
objdump -T
410ddd94 g    DF .text     00000204  GLIBC_2.4   __libc_malloc
 

[2] debug information:
objdump -x 
    LOAD off    0x00000000 vaddr 0x00000000 paddr 0x00000000 align 2**15
         filesz 0x00000194 memsz 0x0012ce28 flags r-x
 ...
  7 .rel.dyn      00002810  00013318  00013318  00000194  2**2
                  ALLOC, READONLY
...
 10 .text         000f3878  00015c40  00015c40  00000194  2**6
                  ALLOC, READONLY, CODE
...
00074954 g     F .text        00000204 __libc_malloc
 
[3] stap --vp=0800
probe __libc_malloc@/opt/codesourcery/arm-none-linux-gnueabi/src/glibc/malloc/malloc.c:2840 process=/opt/export/arm-root/e03_RFS1/lib/libc-2.18.so reloc=.dynamic pc=0x74954
die parent cache /opt/export/arm-root/e03_RFS1/lib/libc-2.18.so:malloc.c size 726
finding location for local 'bytes' near address 0x74954, module bias 0x41079440

[...]

 > The versions I used for my current tests are:
 > Systemtap translator/driver (version 2.7/0.160, commit release-2.6-199-gd26e62cd8b30)
 > Systemtap translator/driver (version 2.8/0.160, commit release-2.7-166-gae0db08baa54)

 >> Thanks,
 >> Mark

 > Thanks,
 > Torsten

Thanks,
Torsten

[-- Attachment #2: lib.tar.bz2 --]
[-- Type: application/x-bzip2, Size: 2444900 bytes --]

Re: Prelinking on ARM

[Mark Wielaard]

Hi Torsten,

On Fri, 2015-06-05 at 21:55 +0200, Torsten Polle wrote:
>  >> I must admit to not have much experience with the
>  >> cross-compiling/cross-stapping mechanism. If you could provide a
>  >> small shared library (plus separate debuginfo file?) and the stap
>  >> invocation that doesn't work correctly that would be helpful for
>  >> understanding what goes wrong.
> 
> please find my example below. Unfortunately, I could not reproduce the
> problem with a smaller library. The prelinked library and the debug
> information is a little bit larger.
> 
> I set a function call and function return probe in function
> "__libc_malloc".
> 
> In the prelinked file [1], the address is 0x410ddd94. The offset from the load
> address is 0x410ddd94 - 0x41068000 = 0x75D94
> 
> But the offest given in the debug information is only 0x00074954.
> 
> The debug output from SystemTap [3] shows a pc of 0x74954 and a module bias
> 0x41079440.

That might, or might not, be correct. systemtap uses libdwfl to create a
memory layout mimicking how the process/library may be placed in memory.
It might pick a (fictional) load address for the library that is
different than what the dynamic loader might pick (the prelinked load
address). If so, that might explain the difference of module bias.

It does look to me like the pc address that stap finds (0x74954) is
correct for the malloc function you are trying to probe.

Could you show the complete stap command line you are using?
What do you expect the output is and what does the full output actually
look like?

Thanks,

Mark

Re: Prelinking on ARM

[Torsten Polle]

Hi Mark,

Mark Wielaard writes:
 > Hi Torsten,

 > On Fri, 2015-06-05 at 21:55 +0200, Torsten Polle wrote:
 >> >> I must admit to not have much experience with the
 >> >> cross-compiling/cross-stapping mechanism. If you could provide a
 >> >> small shared library (plus separate debuginfo file?) and the stap
 >> >> invocation that doesn't work correctly that would be helpful for
 >> >> understanding what goes wrong.
 >> 
 >> please find my example below. Unfortunately, I could not reproduce the
 >> problem with a smaller library. The prelinked library and the debug
 >> information is a little bit larger.
 >> 
 >> I set a function call and function return probe in function
 >> "__libc_malloc".
 >> 
 >> In the prelinked file [1], the address is 0x410ddd94. The offset from the load
 >> address is 0x410ddd94 - 0x41068000 = 0x75D94
 >> 
 >> But the offest given in the debug information is only 0x00074954.
 >> 
 >> The debug output from SystemTap [3] shows a pc of 0x74954 and a module bias
 >> 0x41079440.

 > That might, or might not, be correct. systemtap uses libdwfl to create a
 > memory layout mimicking how the process/library may be placed in memory.
 > It might pick a (fictional) load address for the library that is
 > different than what the dynamic loader might pick (the prelinked load
 > address). If so, that might explain the difference of module bias.

I've to apologies for my late answer.

I double checked my example again and again. I took a HW debugger to
check the place where kprobes set the "breakpoint" in the library. To my
embarassement SystemTap or better libdwfl is performing the right
calculations. The probe hit again. Since I'm very sure that I had a
cases where probes were not hit, I tried various ways to reproduce the
failure. Again without success.

 > It does look to me like the pc address that stap finds (0x74954) is
 > correct for the malloc function you are trying to probe.

 > Could you show the complete stap command line you are using?
 > What do you expect the output is and what does the full output actually
 > look like?

 > Thanks,

 > Mark

Thanks for your support,
Torsten

Re: Prelinking on ARM

[Mark Wielaard]

On Thu, 2015-06-25 at 22:09 +0200, Torsten Polle wrote:
> I double checked my example again and again. I took a HW debugger to
> check the place where kprobes set the "breakpoint" in the library. To my
> embarassement SystemTap or better libdwfl is performing the right
> calculations. The probe hit again. Since I'm very sure that I had a
> cases where probes were not hit, I tried various ways to reproduce the
> failure. Again without success.

No worries. Thanks for double checking and reporting back. Knowing we
(probably) don't have a bug is as valuable as knowing we (might) have a
bug. At least I sleep a little better now :)

Cheers,

Mark