Re: Problem with atexit and _dl_fini

[Adhemerval Zanella <adhemerval.zanella@linaro.org>]

On 13/06/2019 19:53, Nat! wrote:
> Funnily enough, if you read the Itanium C++ ABI, on which __cxa_finalize is based, then the algorithm described
> there is doing exactly the right thing.
> Beause the wording of __cxa_finalize is so shortened, it its hard to pick out the original meaning. But the description is
> actually fully compatible with how `atexit` is supposed to function.
> 
> The gist is this. For atexit, functions are stored in a unique way in the termination function table (clarifications in []):
> 
> http://refspecs.linuxbase.org/LSB_5.0.0/LSB-Core-generic/LSB-Core-generic.html#BASELIB---CXA-FINALIZE
> 
> ```
> In the latter case [atexit] the pointer to the function is the pointer passed to atexit(), while the other pointers [operand, handle] are NULL.
> ```
> 
> When dlclose hits, the handle to be closed is `d` and not NULL:
> 
> ```
> The implementation shall arrange for__cxa_finalize() to be called during early shared library unload (e.g. dlclose()) with a handle to the shared library.
> ```
> 
> And then
> 
> ```
> When __cxa_finalize(d) is called, it shall walk the termination function list, calling each in turn if d matches the handle of the termination function entry.
> ```
> 
> So `atexit`s don't match, since the handle stored is NULL. Only if `d` is NULL (the base process terminates), then will the atexits be called. Currently though at `dlclose` time all handlers are called, which breaks the `atexit` specification as well as your own LSB.
> 
> Well it's a goof up, but FreeBSD and MacOS aren't doing any better.
> 

The problem is currently for glibc atexit is implemented as __cxa_atexit as:

---
/* Register FUNC to be executed by `exit'.  */
int
#ifndef atexit
attribute_hidden
#endif
atexit (void (*func) (void))
{
  return __cxa_atexit ((void (*) (void *)) func, NULL, __dso_handle);
}
---

And linked against a glibc's provided static library (libc_nonshared.a). 
The compiler then defines the __dso_handle variable to be an unique
value for each shared-object (on libgcc for gcc case), and the static 
linking allows the atexit register to use that value.

This is due by design to make atexit work as __cxa_atexit created by
compiler itself.

What I advocate on a recent discussion on libc-alpha [1] is indeed to
follow what you described. My initial suggestion was to add atexit
handlers using a different mechanism, essentially they would be different
than __cxa_atexit handlers. This would make then not to be called
with __cxa_finalize (NULL), rather exit() will be responsible to actually
call them.

It causes a semantic change though: dlclose will need to actually remove
the atexit the shared library registers (because we can't potentially issue
a function callback where its texts has been 'unmaped'). That's why I think 
we will need to use another symbol to register atexit handler, since we will 
need to pass to libc the __dso_handler value to allow __cxa_finalize remove 
the handler on dlclose.

I have a WIP patch to fix, I will push on a user branch if you want to
check this out.

[1] https://sourceware.org/ml/libc-alpha/2019-06/msg00229.html