Dear Community,

Please find in-depth analyze about the issue:

It was tested with Beagle Bone Black (BBB) and QEMU (the same binary
rootfs images).
(If needed I will upload images and script to run QEMU to some server
for reproduction).
Branch: https://github.com/lmajewski/y2038_glibc/commits/y2038_edge

On working setup to trigger the core dump:
/home/root/ld-linux-armhf.so.3 /sbin/init
gdb ./ld-linux-armhf.so.3 core

(and the /home/root/ld-linux-armhf.so.3 is the "broken" one).


Not working (patch [1] not applied):
====================================

All the code is located in _dl_start in elf/rtld.c and
elf/get-dynamic-info.h files:

(gdb) p/x $r5
$46 = 0xb6fc8000
r5 is set from the elf_machine_load_address()

Then we enter the elf_get_dynamic_info()

0xb6fc95fc      99            ADJUST_DYN_INFO (DT_SYMTAB);
   0xb6fc95f8 <_dl_start+308>:  04 30 92 15     ldrne   r3, [r2, #4]
=> 0xb6fc95fc <_dl_start+312>:  05 30 83 10     addne   r3, r3, r5
   0xb6fc9600 <_dl_start+316>:  04 30 82 15     strne   r3, [r2, #4]
(gdb) p/x $r3
$63 = 0x410003f4
(gdb) p/x $r5
$64 = 0xb6fc8000
(gdb) si
0xb6fc9600      99            ADJUST_DYN_INFO (DT_SYMTAB);
   0xb6fc95f8 <_dl_start+308>:  04 30 92 15     ldrne   r3, [r2, #4]
   0xb6fc95fc <_dl_start+312>:  05 30 83 10     addne   r3, r3, r5
=> 0xb6fc9600 <_dl_start+316>:  04 30 82 15     strne   r3, [r2, #4]
(gdb) p/x $r3
$65 = 0xf7fc83f4

The above address is the kernel space one - so accessing it causes:
"Program received signal SIGSEGV, Segmentation fault."

(gdb) p/x $r3
$55 = 0xf7fc88e4
112                 DO_ELF_MACHINE_REL_RELATIVE (map, l_addr, relative);
=> 0xb6fc979c <_dl_start+728>:  08 10 13 e5     ldr     r1, [r3, #-8]


The problem is that elf_machine_load_address() returns wrong value - or
maybe to say it differently - the one used afterwards by glibc is wrong.

The correct one is, which corresponds to using 'adr' command (with
patch [1] applied):

0xb6fc9508 in _dl_start (arg=0xbefffdf0) at rtld.c:545
545       bootstrap_map.l_addr = elf_machine_load_address ();
=> 0xb6fc9508 <_dl_start+68>:   98 55 81 e5     str     r5, [r1, #1432]
; 0x598 (gdb) p/x $r5
$12 = 0x75fc8000


Another issue is the way the assembler code is generated. With the
working code one would have:

(gdb) si
0xb6fc9608      99            ADJUST_DYN_INFO (DT_SYMTAB);
   0xb6fc9600 <_dl_start+316>:  00 37 9f e5     ldr     r3, [pc, #1792]
; 0xb6fc9604 <_dl_start+320>:  03 30 8f e0     add     r3, pc, r3
=> 0xb6fc9608 <_dl_start+324>:  d0 35 93 e5     ldr     r3, [r3, #1488]
; 0x5d0 0xb6fc960c <_dl_start+328>:  00 00 53 e3     cmp     r3, #0

(gdb) p/x $r3
$13 = 0xb6fff010
(gdb) p/x $pc
$14 = 0xb6fc9608
(gdb) 

In the above example the $r3 value is relative to $pc and $r5 is not
used at all.


The code which fails now looks like:
112                 DO_ELF_MACHINE_REL_RELATIVE (map, l_addr, relative);
=> 0xb6fc9870 <_dl_start+940>:  08 10 13 e5     ldr     r1, [r3, #-8]

(gdb) p/x l_addr
$15 = 0x75fc8000

and everything works as expected.


Questions:
==========

1. I guess that different addressing assembler generated ($r3
calculation) is related to the address to be relocated (when the offset
is too large the register is used - not the $pc itself).

2. I don't understand why values provided by elf_machine_load_address()
differ? I would expect that those are equal.

And help and input appreciated.





Links:
[1] -
https://github.com/lmajewski/y2038_glibc/commit/e67e0f589b88a44be8f8b9b770b08950dd7e5bd5

readelf -e ld-linux-armhf.so.3 

[10] .plt              PROGBITS        41000994 000994 000050 04  AX  0   0  4
[11] .text             PROGBITS        41000a00 000a00 01fed0 00  AX  0   0 64
[12] .rodata           PROGBITS        410208d0 0208d0 004b24 00   A  0   0  4
[13] .ARM.extab        PROGBITS        410253f4 0253f4 000018 00   A  0   0  4
[14] .ARM.exidx        ARM_EXIDX       4102540c 02540c 0000c8 00  AL 11   0  4
[15] .data.rel.ro      PROGBITS        41036200 026200 000cf4 00  WA  0   0  8
[16] .dynamic          DYNAMIC         41036ef4 026ef4 0000c8 08  WA  5   0  4
[17] .got              PROGBITS        41036fbc 026fbc 000040 04  WA  0   0  4


> Hi Fangrui,
> 
> > On 2021-10-05, Lukasz Majewski wrote:  
> > >Hi Fangrui,
> > >    
> > >> >
> > >> >I'm also wondering if similar issue is visible with "normal" -
> > >> >i.e. not QEMU ARM run init.
> > >> >
> > >> >Maybe the "rough" environment in which /sbin/init is run is the
> > >> >culprit?
> > >> >    
> > >>
> > >> I had tested running ld.so and elf/ldconfig which covered the
> > >> usage.
> > >>
> > >> I don't know. It needs some debugging from you:)    
> > >
> > >Could you share on which hardware (ARM) you run this change?
> > >
> > >
> > >Best regards,
> > >
> > >Lukasz Majewski    
> >   
> 
> Thanks for the info.
> 
> > I ran ld.so and elf/ldconfig with qemu-arm-static (built from  
> 
> Ok, so this was only the user mode emulation, not qemu-system-arm ?
> 
> > qemu/linux-user) (via binfmt_misc) on x86-64 Linux (Debian testing).
> > I don't have real ARM hardware.  
> 
> The built image for BBB (Beagle Bone Black) also shows OOPs on the
> real HW.
> 
> I'm debugging this now and share results asap.
> 
> 
> Best regards,
> 
> Lukasz Majewski
> 
> --
> 
> DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
> Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email:
> lukma@denx.de




Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email: lukma@denx.de