Re: Problem with atexit and _dl_fini

[Adhemerval Zanella <adhemerval.zanella@linaro.org>]

On 10/06/2019 17:27, Adhemerval Zanella wrote:
> 
> 
> On 10/06/2019 10:07, Nat! wrote:
>>
>> On 10.06.19 13:48, Adhemerval Zanella wrote:
>>>
>>> On 09/06/2019 17:59, Nat! wrote:
>>>> Another datapoint to support my claim that _dl-fini breaks atexit. This time its very easy to reproduce ;)
>>>>
>>>> Here 's the README.md from the Github Repo https://github.com/mulle-nat/atexit-breakage-linux
>>>>
>>>>
>>>> ```
>>>>
>>>> # Shows another breakage involving `atexit` on linux
>>>>
>>>> Here the `atexit` callback is invoked mistakenly multiple times.
>>> This 'example' does not really show the issue because ldd script issues
>>> the loader multiple times, see below. You can check exactly what ldd is
>>> doing by calling with sh -x.
>>
>> I agree it doesn't show the same issue, but it shows that something else is going very wrong. :) Or are you happy, that atexit is called multiple times ? Who's calling exit here anyway ? Check out the debugger output too (see updated README.md)
> 
> The ldd is not a program, but rather a shell script that issues the target
> binary along with system loader multiple times. What you are seeing is not 
> atexit called multiple times, but rather how the script is called.
> 
> When you set LD_PRELOAD *before* issuing ldd you will make the shell binary
> to also pre-load the library.  I instrumented the binary to also print the
> output command line from the issue binary (get either by program_invocation_name
> or /proc/self/cmdline):
> 
> $ LD_PRELOAD=./libld-preload.so ./ldd ./main
> /bin/bash: load
> /bin/bash: unload
> /bin/bash: unload
> /bin/bash: unload
> 	linux-vdso.so.1 (0x00007ffd445ef000)
> 	./libld-preload.so (0x00007fa866ac5000)
> 	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa8664b5000)
> 	/lib64/ld-linux-x86-64.so.2 (0x00007fa8668a6000)
> /bin/bash: unload
> /bin/bash: unload
> 
> The program is not load since although ldd does call the loader, it calls
> in a trace mode that does not actually load any shared library.  The first
> 'load' is issued by library when bash is first executed and later multiple
> 'unload' is due bash forks and then exits multiple times.
> 
>>
>>
>>>
>>> I will try to use your instruction to run on docker to see what exactly
>>> is happening in your environment.
>>
>> That's not necessary anymore. I managed to make it reproducible in a much simpler form just now.
>>
>> The ld-so-breakage project is basically a recreation of the original "docker" scenario written from scratch. I try to explain in the README , what is going on. But if there are questions hit me up (maybe as an issue ?) :
>>
>>     https://github.com/mulle-nat/ld-so-breakage
> 
> Thanks, it is way more useful. I now I understand what is happening and
> IMHO this behaviour is a required because on glibc we set that atexit/on_exit 
> handlers are ran when deregister a library (as for dlclose).
> 
> Using the example in your testcase:
> 
> ---
> USE_A=YES ./build/main_adbc
> -- install atexit_b
> -- install atexit_a
> -- run atexit_a
> -- run atexit_b
> ---
> 
> The behaviour of atexit handlers being called in wrong order is they are
> being registered with '__cxa_atexit' which in turn sets its internal type
> as 'ef_cxa'.  Since _dl_init is registered last (after all shared library
> loading and constructors calls), it will call _dl_fini which in turn will
> call '__cxa_finalize' (through __do_global_dtors_aux generated by compiler).
> 
> The '__cxa_finalize' will then all 'ef_cxa' function for the module passed
> by __do_global_dtors_aux and set the function as 'ef_free'. It will then
> prevent '__run_exit_handlers' to run the handlers more than once.
> 
> So the question you might ask is why not just to use 'ef_at' for atexit
> handlers, make them no to run on __cxa_finalize and thus make your example
> run as you expect? The issue is glibc does not know whether your library
> would be dlopened or not.  
> 
> If you set an atfork handler by a constructor that references to a function 
> inside the shared library and if do *not* set to *not* be ran later you might, 
> a case of dlopen -> constructor -> dlclose -> exit will try to execute and
> invalid mapping.  This is exactly what dlfcn/bug-atexit{1,2}.c.
> 
> So the question is why exactly glibc defined that atexit should be called
> by dlclose. I understand that __cxa_finalize / destructor make sense to
> make it possible the shared library to free allocated resources, but I
> can't really get why there a need to extend it to 'atexit' as well.

It seems that this requirement seems to come from LSB, although I am not
sure which one came first (the specification or the implementation). 
It also states that __cxa_atexit should register a function to be called
by exit or when a shared library is unloaded. 

And __cxa_finalize requires to call atexit registers functions as well. It 
also states __cxa_finalize should be called on dlclose.

I think it might due the fact old gcc version uses atexit to register C++
destructors for local static and global objects. However it seems to be
enabled as default for GLIBC (since it support __cxa_atexit since initial
versions).

So I think there is no impeding reason to make atexit not be called from
__cxa_finalize, although I am not sure how we would handle the LSB deviation.
I will write down a libc-alpha to check what other developer think.

[1] http://refspecs.linuxbase.org/LSB_5.0.0/LSB-Core-generic/LSB-Core-generic.pdf