Re: Should we make DT_HASH dynamic section for glibc?

[Sam James <sam@gentoo.org>]

[-- Attachment #1: Type: text/plain, Size: 6411 bytes --]



> On 8 Aug 2022, at 23:59, Fangrui Song via Libc-alpha <libc-alpha@sourceware.org> wrote:
> 
> On 2022-08-08, Carlos O'Donell via Libc-alpha wrote:
>> On 8/8/22 13:31, Adhemerval Zanella Netto via Libc-alpha wrote:
>>> It seems that the recent change to remove the multiple hash schemes on
>>> 2.36 [1] broke some specific tools used on proton games [2].  So instead
>>> of explicit set the section type to use both sysv and gnu, we use the
>>> toolchain default which might exclude the sysv type.
>> 
>> Right.
>> 
>>> The last gABI states that DT_HASH is mandatory [3], but DT_GNU_HASH works
>>> a direct replacement meaning that it contains all information for symbol
>>> resolution that DT_HASH provides.
>> 
>> Correct.
>> 
>>> It was done as size optimization from perceived unused features since
>>> DT_GNU_HASH is being used as a default on most distros for a long time,
>>> meaning DT_HASH might not be set.  For instance, on a Ubuntu 22.05 system
>>> (GLIBC 2.35) only the glibc provided binaries (pldd, gencat, etc.) and some
>>> external tools (nvidia command line) do provide DT_HASH.
>> 
>> Correct.
>> 
>>> So the question is whether we should enforce at least on glibc by reverting
>>> e47de5cb2d4dbec.  It does sounds muddle to keep this solely on glibc, where
>>> rest of the system is not enforcing it, and only for compatibility with an
>>> obscure tools where there is no much information on why it requires it.
>> 
>> The tool is EAC.
>> 
>> It is EPICs "Easy Anti-Cheat" (https://www.easy.ac/en-us/) and like other non-standard
>> consumers it has to know some specific ELF details.
>> 
>> The game "Rogue Company" is known to use EAC and is likely broken by this change.
>> 
>> The Nobara Project includes "changes" to make EAC-enabled Steam games work:
>> https://nobaraproject.org/
>> 
>> I have reviewed the changes that Nobara is carrying and I would not apply them upstream.
>> 
>> The include such changes as reverting the clone3 addition because it impacts seccomp-based
>> isolation.
>> 
>>> Another option might to extend gABI to state that if DT_GNU_HASH is presents,
>>> it works as DT_HASH and it should not mark as mandated.
>> 
>> We should ask the gABI to mark DT_HASH as optional.
>> 
>> This doesn't resolve the user issue though...
>> 
>>> And if DT_HASH is require required, one possibility is to add a binutils
>>> option to emit an empty DT_HASH just for compatibility and get the code size
>>> improvement.
>> 
>> The chain array needs to be as big as the symbol table since the presence
>> of DT_HASH means it can be indexed into, therefore I don't think we can
>> have an empty DT_HASH.
>> 
>>> [1] https://sourceware.org/git/?p=glibc.git;a=commit;h=e47de5cb2d4dbecb58f569ed241e8e95c568f03c
>>> [2] https://sourceware.org/bugzilla/show_bug.cgi?id=29456
>>> [3] http://www.sco.com/developers/gabi/latest/ch5.dynamic.html
>> 
>> Using only DT_GNU_HASH is a choice we *always* wanted to allow the
>> downstream distributions to make, it was part of the binutils changes to
>> allow just DT_GNU_HASH.
>> 
>> Software that is an ELF consumer on Linux has had 16 years to be updated
>> to handle the switch from DT_HASH to DT_GNU_HASH (OS-specific).
>> 
>> While I'm sympathetic to application developers and their backwards
>> compatibility requirements, this specific case is about an ELF consumers
>> and such a consumer needs to track upstream Linux ELF developments.
>> 
>> We aren't breaking ABI when we remove the PLT, remove the old HASH, or
>> other Linux ELF changes (like the recent DT_RELR addition), but we do
>> need to allow time for these changes to be absorbed by the ecosystem
>> and ELF consumers (like debug information consumers).
>> 
>> At present I would not make any changes to glibc. I would close bug 29456
>> as RESOLVED/WONTFIX. I'm open to hearing from the EPIC EAC developers
>> if they have a case to make about DT_HASH.
>> 
>> In summary:
>> 
>> - DT_GNU_HASH was added in 2006, and for the last 16 years has been the
>> modern standard on Linux. The glibc change was made to allow the
>> distributions to choose how backwards compatible they want to be with
>> ELF consumers and the hash function and section. This is not ABI, just
>> like the PLT and RELRO are not ABI.
>> 
>> - One specific use case of "Easy Anti-Cheat" software is impacted by
>> this implementation detail change which impacts ELF consumers that
>> require DT_HASH.
>> 
>> - The choice to have DT_HASH is with the distributions. If this breaks
>> specific applications then those developers need to engage with the
>> ecosystem or adapt their software.
> 
> Thanks for analyzing the issue!  On
> https://github.com/ValveSoftware/Proton/issues/6051
> nobody says this is related to DT_HASH :(
> 

For a long time in Gentoo, we forced DT_GNU_HASH only,
but reverted it about a month ago [0] as we had a few reports
of Gentoo-only bugs as a result.

I spoke to Carlos today and regret not communicating
with him before doing that, as this might have revealed
the problem earlier, but lesson learned!

[0] https://github.com/gentoo/gentoo/commit/8afecc68b8d689dbfdbff3b16ca50be66deb3cce

At https://github.com/anyc/steam-overlay/issues/309, we saw
various reports of EAC needing DT_GNU_HASH.

> If "Easy Anti-Cheat" requires DT_HASH, I think it is a unreasonable
> request.  I left a comment on
> https://github.com/ValveSoftware/Proton/issues/6051#issuecomment-1208698263
> 

FWIW, and I don't think this is a reason to keep it, I just want to raise it,
some free software has this issue too, like libstrangle:
https://bugs.gentoo.org/863863 / https://gitlab.com/torkel104/libstrangle/-/issues/59.

But they just need to adapt to using DT_GNU_HASH.

> Many distributions have dropped DT_HASH (e.g.
> see https://github.com/llvm/llvm-project/blob/b405407/clang/lib/Driver/ToolChains/Linux.cpp#L241-L244
> for the encoded Clang knowledge), it is entirely fine for glibc to drop
> DT_HASH for its own shared objects, even if ELFOSABI_NONE is used.
> Perhaps, as Roland suggested, clarifying the optional DT_HASH thing on gnu-gabi@ may help.

The only issues we were ever aware of were libstrangle and
the EAC bug. So, all in all, having run this for several years,
the fallout wasn't that great (limited in scope).

Best,
sam

[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 358 bytes --]