Re: Debugging ld.so in gdb

[Jacob Kroon <jacob.kroon@gmail.com>]

On 2/7/22 12:55, Florian Weimer wrote:
> * Jacob Kroon:
> 
>> On 2/7/22 09:36, Jacob Kroon wrote:
>>> Hi Florian,
>>>
>>> On 2/4/22 18:15, Florian Weimer wrote:
>>>> I suspect we are writing beyond the start of the array passed to
>>>> _dl_sort_maps.
>>>>
>>>
>>> It looks like it is writing passed the beginning of the rpo[] array in
>>> _dl_sort_maps_dfs(). The output below is right before the crash happens
>>> (stepping one instruction garbles the backtrace):
>>>
>>>> (gdb) bt
>>>> #0  dfs_traversal (rpo=rpo@entry=0x7fffffffd320, map=0x7ffff7fad590, do_reldeps=do_reldeps@entry=0x0) at dl-sort-maps.c:175
>>>> #1  0x00007ffff7fd85d4 in dfs_traversal (do_reldeps=0x0, map=<optimized out>, rpo=0x7fffffffd320) at dl-sort-maps.c:143
>>>> #2  dfs_traversal (rpo=rpo@entry=0x7fffffffd320, map=0x7ffff7fadb70, do_reldeps=do_reldeps@entry=0x0) at dl-sort-maps.c:155
>>>> #3  0x00007ffff7fd89cd in dfs_traversal (do_reldeps=0x0, map=<optimized out>, rpo=0x7fffffffd320) at dl-sort-maps.c:143
>>>> #4  _dl_sort_maps_dfs (skip=<optimized out>, for_fini=<optimized out>, nmaps=15, maps=0x7ffff7953de0) at dl-sort-maps.c:233
>>>> #5  _dl_sort_maps (maps=maps@entry=0x7ffff7953de0, nmaps=nmaps@entry=15, skip=<optimized out>, for_fini=for_fini@entry=false) at dl-sort-maps.c:299
>>>> #6  0x00007ffff7fcaf0f in _dl_map_object_deps (map=<optimized out>, preloads=<optimized out>, npreloads=<optimized out>, trace_mode=<optimized out>, 
>>>>     open_mode=<optimized out>) at dl-deps.c:616
>>>> #7  0x00007ffff7fe6970 in dl_main (phdr=<optimized out>, phnum=<optimized out>, user_entry=<optimized out>, auxv=<optimized out>) at rtld.c:1968
>>>> #8  0x00007ffff7fe2c7c in _dl_sysdep_start (start_argptr=<optimized out>, dl_main=0x7ffff7fe4bb0 <dl_main>) at ../elf/dl-sysdep.c:264
>>>> #9  0x00007ffff7fe4678 in _dl_start_final (arg=0x7fffffffdec0) at rtld.c:493
>>>> #10 _dl_start (arg=0x7fffffffdec0) at rtld.c:587
>>>> #11 0x00007ffff7fe36a8 in _start ()
>>>> (gdb) f 0
>>>> #0  dfs_traversal (rpo=rpo@entry=0x7fffffffd320, map=0x7ffff7fad590, do_reldeps=do_reldeps@entry=0x0) at dl-sort-maps.c:175
>>>> 175       **rpo = map;
>>>> (gdb) print *rpo
>>>> $62 = (struct link_map **) 0x7fffffffd238
>>>> (gdb) f 4
>>>> #4  _dl_sort_maps_dfs (skip=<optimized out>, for_fini=<optimized out>, nmaps=15, maps=0x7ffff7953de0) at dl-sort-maps.c:233
>>>> 233           dfs_traversal (&rpo_head, maps[i], do_reldeps_ref);
>>>> (gdb) print &rpo[-1]
>>>> $63 = (struct link_map **) 0x7fffffffd238
>>>
>>> I inspected the "maps" vector and it containes *multiple* entries to
>>> "libjvm.so", is that allowed ? I wonder if "nmaps" is calculated
>>> correctly, since that determines the array size. Can I verify that somehow ?
> 
> Curiously I was wondering about duplicates as well as I couldn't sleep.
> 
> Do you use DT_FILTER or anything unusual?  What about LD_PRELOAD?
> 

Not sure what "DT_FILTER" is, but it is not something that shows up if I
run "readelf -a" on the library in question. I don't have LD_PRELOAD set
in my environment.

>> Actually that is was not correct. "maps[]->l_name" does not contain any
>> "libjvm.so" at all, but the resulting rpo[] does contain several
>> "libjvm.so" entries.
> 
> What I find really confusing is that this is not the result of a dlopen
> call.  I definitely would expect that the maps array contains *all*
> objects that are being loaded.  Clearly this is not the case here.
> Somehow certain objects are missing, and then they get written into the
> rpo array.
> 
> Please try to find libjvm.so among the l_initfini arrays of the objects.

I do find libjvm.so in a couple of the maps[]->l_initfini[]->l_name
arrays, yes.

> It must be present somewhere.  I assume it's also on the main list,
> which starts off at _rtld_global._dl_ns[0]._ns_loaded.

Hmm how do I iterate over that data structure ?

See below:
> (gdb) print _rtld_global._dl_ns[0]._ns_loaded[0]->l_name
> $186 = 0x7ffff7ff1d97 ""
> (gdb) print _rtld_global._dl_ns[0]._ns_loaded[1]->l_name
> $187 = 0x0
> (gdb) print _rtld_global._dl_ns[0]._ns_loaded[2]->l_name
> $188 = 0x0
> (gdb) print _rtld_global._dl_ns[0]._ns_loaded->l_name
> $189 = 0x7ffff7ff1d97 ""
> (gdb) print _rtld_global._dl_ns[1]._ns_loaded->l_name
> Cannot access memory at address 0x8
> (gdb) print _rtld_global._dl_ns[2]._ns_loaded->l_name
> Cannot access memory at address 0x8

Jacob