Re: Problem with atexit and _dl_fini

[Adhemerval Zanella <adhemerval.zanella@linaro.org>]

On 10/06/2019 10:07, Nat! wrote:
> 
> On 10.06.19 13:48, Adhemerval Zanella wrote:
>>
>> On 09/06/2019 17:59, Nat! wrote:
>>> Another datapoint to support my claim that _dl-fini breaks atexit. This time its very easy to reproduce ;)
>>>
>>> Here 's the README.md from the Github Repo https://github.com/mulle-nat/atexit-breakage-linux
>>>
>>>
>>> ```
>>>
>>> # Shows another breakage involving `atexit` on linux
>>>
>>> Here the `atexit` callback is invoked mistakenly multiple times.
>> This 'example' does not really show the issue because ldd script issues
>> the loader multiple times, see below. You can check exactly what ldd is
>> doing by calling with sh -x.
> 
> I agree it doesn't show the same issue, but it shows that something else is going very wrong. :) Or are you happy, that atexit is called multiple times ? Who's calling exit here anyway ? Check out the debugger output too (see updated README.md)

The ldd is not a program, but rather a shell script that issues the target
binary along with system loader multiple times. What you are seeing is not 
atexit called multiple times, but rather how the script is called.

When you set LD_PRELOAD *before* issuing ldd you will make the shell binary
to also pre-load the library.  I instrumented the binary to also print the
output command line from the issue binary (get either by program_invocation_name
or /proc/self/cmdline):

$ LD_PRELOAD=./libld-preload.so ./ldd ./main
/bin/bash: load
/bin/bash: unload
/bin/bash: unload
/bin/bash: unload
	linux-vdso.so.1 (0x00007ffd445ef000)
	./libld-preload.so (0x00007fa866ac5000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa8664b5000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fa8668a6000)
/bin/bash: unload
/bin/bash: unload

The program is not load since although ldd does call the loader, it calls
in a trace mode that does not actually load any shared library.  The first
'load' is issued by library when bash is first executed and later multiple
'unload' is due bash forks and then exits multiple times.

> 
> 
>>
>> I will try to use your instruction to run on docker to see what exactly
>> is happening in your environment.
> 
> That's not necessary anymore. I managed to make it reproducible in a much simpler form just now.
> 
> The ld-so-breakage project is basically a recreation of the original "docker" scenario written from scratch. I try to explain in the README , what is going on. But if there are questions hit me up (maybe as an issue ?) :
> 
>     https://github.com/mulle-nat/ld-so-breakage

Thanks, it is way more useful. I now I understand what is happening and
IMHO this behaviour is a required because on glibc we set that atexit/on_exit 
handlers are ran when deregister a library (as for dlclose).

Using the example in your testcase:

---
USE_A=YES ./build/main_adbc
-- install atexit_b
-- install atexit_a
-- run atexit_a
-- run atexit_b
---

The behaviour of atexit handlers being called in wrong order is they are
being registered with '__cxa_atexit' which in turn sets its internal type
as 'ef_cxa'.  Since _dl_init is registered last (after all shared library
loading and constructors calls), it will call _dl_fini which in turn will
call '__cxa_finalize' (through __do_global_dtors_aux generated by compiler).

The '__cxa_finalize' will then all 'ef_cxa' function for the module passed
by __do_global_dtors_aux and set the function as 'ef_free'. It will then
prevent '__run_exit_handlers' to run the handlers more than once.

So the question you might ask is why not just to use 'ef_at' for atexit
handlers, make them no to run on __cxa_finalize and thus make your example
run as you expect? The issue is glibc does not know whether your library
would be dlopened or not.  

If you set an atfork handler by a constructor that references to a function 
inside the shared library and if do *not* set to *not* be ran later you might, 
a case of dlopen -> constructor -> dlclose -> exit will try to execute and
invalid mapping.  This is exactly what dlfcn/bug-atexit{1,2}.c.

So the question is why exactly glibc defined that atexit should be called
by dlclose. I understand that __cxa_finalize / destructor make sense to
make it possible the shared library to free allocated resources, but I
can't really get why there a need to extend it to 'atexit' as well.

> 
> 
> The "another datapoint" project shows how constructor/destructor don't pair up:
> 
>     https://github.com/mulle-nat/atexit-breakage-linux
> 
> 
> And as a random bonus this project indicates to me that LD_PRELOAD doesn't do what its supposed to either:
> 
>     https://github.com/mulle-nat/LD_PRELOAD-breakage-linux
> 
> 
> In total I think the state of affairs is pretty dismal. I didn't expect that much basic stuff not working on linux. With hindsight, I probably have wasted _weeks_ on these problems.
> 
> I still maintain that the concept to let `atexit` callbacks not run by `exit` is broken. An `atexit` callback is not the same as an `__attribute__((destructor))__`.
> 
> 
> Ciao
> 
>    Nat!
> 
>