Re: LD_AUDIT: Not enough space in static TLS block

[Florian Weimer <fweimer@redhat.com>]

* Florian Weimer:

> * Florian Weimer:
>
>> * Jonathon Anderson:
>>
>>> Hello all,
>>>
>>> We (the HPCToolkit team) have encountered another critical LD_AUDIT
>>> bug. When LD_AUDIT is specified, the allocation of the static TLS
>>> block does not account for the TLS requirements of executable
>>> dependencies or of the auditors themselves. If:
>>>  - an executable accesses a thread-local variable in a linked library
>>> with sufficiently large TLS requirements, or
>>>  - an auditor itself uses sufficiently large TLS and optimizes access
>>> with `-ftls-model=initial-exec`,
>>>
>>> then the process or auditor will fail with the error "cannot allocate
>>> memory in static TLS block."
>>
>> We have a tunable that can be used as a workaround.  Your reproducer
>> passes for me with our 2.28 backport (glibc-2.28-164.el8) if I run it
>> like this:
>>
>>   GLIBC_TUNABLES=glibc.rtld.optional_static_tls=4000 make
>>
>> The best we can do in the short term would be an increase of the default
>> limit.  On 64-bit platforms, defaulting to a dozen or so kilobytes per
>> thread should not be a problem as far as virtual address space
>> consumption is concerned.  We can also add an additional reservation of
>> similar size for every auditor that is loaded, to compensate for the
>> lack of auto-tuning of the TLS allocation size in auditing mode.
>>
>> The fundamental issue is that there is always going to be a hard limit
>> for initial-exec TLS.  Initial-exec TLS requires a fixed offset from the
>> thread pointer, and we cannot relocate TLS variables because they are
>> ordinary C objects with an observable address.  There are some other
>> things we can try to improve auto-tuning, but in the end, there is
>> always going to be a fixed-size reserved area dedicated to initial-exec
>> TLS set up at process startup, and with dlopen, that might not be enough
>> even without any auditor use.
>
> Jonathon,
>
> does setting the environment variable work for you?

Do you have any additional feedback here?

In the meantime, we have updated Fedora rawhide with the bug fix to
enable early <dlfcn.h> usage from auditors, and the new RTLD_DI_PHDR
dlinfo is included as well.  If you could test glibc-2.35.9000-16 or
later, that would be great.

Thanks,
Florian