* Are the pthread "compatibility" copies of symbols in libc still necessary? @ 2018-03-16 14:30 Zack Weinberg 2018-03-16 14:41 ` Florian Weimer 0 siblings, 1 reply; 9+ messages in thread From: Zack Weinberg @ 2018-03-16 14:30 UTC (permalink / raw) To: GNU C Library A number of source files that properly belong to libc.so are also compiled as part of libpthread, with a note that this is for "compatibility for old binaries". The exact set varies based on architecture, but includes basic things like read, write, and fork - I _think_ there was a difference in semantics in the distant past, having something to do with thread cancellation. It seems to me that these are no longer necessary. The overall behavior we want is for new binaries to link against the symbols in libc.so, and old binaries that were linked against a libpthread with those symbols to continue to work. ELF doesn't require undefined references to resolve to specific libraries at load time, so if we just dropped the symbols from libpthread, shouldn't the references from old binaries automatically resolve to the definitions in libc, which is what we want? Am I missing something? zw ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Are the pthread "compatibility" copies of symbols in libc still necessary? 2018-03-16 14:30 Are the pthread "compatibility" copies of symbols in libc still necessary? Zack Weinberg @ 2018-03-16 14:41 ` Florian Weimer 2018-03-16 15:35 ` Carlos O'Donell 0 siblings, 1 reply; 9+ messages in thread From: Florian Weimer @ 2018-03-16 14:41 UTC (permalink / raw) To: Zack Weinberg, GNU C Library On 03/16/2018 03:30 PM, Zack Weinberg wrote: > A number of source files that properly belong to libc.so are also > compiled as part of libpthread, with a note that this is for > "compatibility for old binaries". The exact set varies based on > architecture, but includes basic things like read, write, and fork - I > _think_ there was a difference in semantics in the distant past, > having something to do with thread cancellation. They are still needed because versioned symbols also embed a DSO name, and the dynamic linker checks that if it has not been interposed. I think it's this code in elf/dl-lookup.c: if (__glibc_unlikely (res < 0) && skip_map == NULL) { /* Oh, oh. The file named in the relocation entry does not contain the needed symbol. This code is never reached for unversioned lookups. */ assert (version != NULL); const char *reference_name = undef_map ? undef_map->l_name : ""; struct dl_exception exception; /* XXX We cannot translate the message. */ _dl_exception_create_format (&exception, DSO_FILENAME (reference_name), "symbol %s version %s not defined in file %s" " with link time reference%s", undef_name, version->name, version->filename, res == -2 ? " (no version symbols)" : ""); _dl_signal_cexception (0, &exception, N_("relocation error")); _dl_exception_free (&exception); *ref = NULL; return 0; } This is required by the GNU symbol versioning spec, but I don't know why. Even new binaries use these symbols: $ LD_DEBUG=bindings systemctl |& grep binding.*system.*libpthread.*[^_]fork 11905: binding file /usr/lib/systemd/libsystemd-shared-234.so [0] to /lib64/libpthread.so.0 [0]: normal symbol `fork' [GLIBC_2.2.5] 11905: binding file systemctl [0] to /lib64/libpthread.so.0 [0]: normal symbol `fork' [GLIBC_2.2.5] This may be a linker bug because at least fork is a compat symbol. Thanks, Florian ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Are the pthread "compatibility" copies of symbols in libc still necessary? 2018-03-16 14:41 ` Florian Weimer @ 2018-03-16 15:35 ` Carlos O'Donell 2018-03-16 15:42 ` Florian Weimer 0 siblings, 1 reply; 9+ messages in thread From: Carlos O'Donell @ 2018-03-16 15:35 UTC (permalink / raw) To: Florian Weimer, Zack Weinberg, GNU C Library On 03/16/2018 08:41 AM, Florian Weimer wrote: > On 03/16/2018 03:30 PM, Zack Weinberg wrote: >> A number of source files that properly belong to libc.so are also >> compiled as part of libpthread, with a note that this is for >> "compatibility for old binaries". The exact set varies based on >> architecture, but includes basic things like read, write, and fork - I >> _think_ there was a difference in semantics in the distant past, >> having something to do with thread cancellation. > > They are still needed because versioned symbols also embed a DSO name, and the dynamic linker checks that if it has not been interposed. > > I think it's this code in elf/dl-lookup.c: > >      if (__glibc_unlikely (res < 0) && skip_map == NULL) >     { >      /* Oh, oh. The file named in the relocation entry does not >         contain the needed symbol. This code is never reached >         for unversioned lookups. */ >      assert (version != NULL); >      const char *reference_name = undef_map ? undef_map->l_name : ""; >      struct dl_exception exception; >      /* XXX We cannot translate the message. */ >      _dl_exception_create_format >        (&exception, DSO_FILENAME (reference_name), >         "symbol %s version %s not defined in file %s" >         " with link time reference%s", >         undef_name, version->name, version->filename, >         res == -2 ? " (no version symbols)" : ""); >      _dl_signal_cexception (0, &exception, N_("relocation error")); >      _dl_exception_free (&exception); >      *ref = NULL; >      return 0; >     } > > This is required by the GNU symbol versioning spec, but I don't know why. The specification says the data must be recorded, but it doesn't say what you have to *do* with the data? I am of the opinion that this is simply to produce a reasonable error message if the versioned symbol is missing completely. We should probably reach out to Ulrich to see if has any opinion on this, as the primary author of the specification he might have an input here. My own opinion is that the check is overly restrictive, and that we could relax it to allow versioned symbols to move to other shared objects. > Even new binaries use these symbols: > > $ LD_DEBUG=bindings systemctl |& grep binding.*system.*libpthread.*[^_]fork >     11905:   binding file /usr/lib/systemd/libsystemd-shared-234.so [0] to /lib64/libpthread.so.0 [0]: normal symbol `fork' [GLIBC_2.2.5] >     11905:   binding file systemctl [0] to /lib64/libpthread.so.0 [0]: normal symbol `fork' [GLIBC_2.2.5] > > This may be a linker bug because at least fork is a compat symbol. Is it though? libc.so.6 has fork@@GLIBC_2.2.5, which causes the binary to have a reference to fork@GLIBC_2.2.5, and that's correct. However, now we have both libc.so.6 and libpthread.so.0 with definitions of fork at GLIBC_2.2.5. Also libsystemd-shared-234.so has: 000000000043f580 0000019a00000007 R_X86_64_JUMP_SLOT 0000000000000000 fork@GLIBC_2.2.5 + 0 410: 0000000000000000 0 FUNC GLOBAL DEFAULT UND fork@GLIBC_2.2.5 (2) 0x0170: Version: 1 File: libc.so.6 Cnt: 16 ... 0x0270: Name: GLIBC_2.2.5 Flags: none Version: 2 Is it a dynamic loader bug that the compat symbol was selected? systemctl has libpthread.so.0 as the *first* DT_NEEDED entry, and we process them in order. 0x0000000000000001 (NEEDED) Shared library: [libpthread.so.0] 0x0000000000000001 (NEEDED) Shared library: [libc.so.6] 0x0000000000000001 (NEEDED) Shared library: [libsystemd-shared-234.so] 0x0000000000000001 (NEEDED) Shared library: [libgcc_s.so.1] So libpthread.so.0 interposes the versioned symbol from libc.so.6. 26067: Initial object scopes 26067: object=systemctl [0] 26067: scope 0: systemctl /lib64/libpthread.so.0 /lib64/libc.so.6 /usr/lib/systemd/libsystemd-shared-234.so /lib64/libgcc_s.so.1 /lib64/ld-linux-x86-64.so.2 /lib64/librt.so.1 /lib64/libcap.so.2 /lib64/libacl.so.1 /lib64/libcryptsetup.so.4 /lib64/libgcrypt.so.20 /lib64/libip4tc.so.0 /lib64/libseccomp.so.2 /lib64/libselinux.so.1 /lib64/libidn.so.11 /lib64/liblzma.so.5 /lib64/liblz4.so.1 /lib64/libblkid.so.1 /lib64/libattr.so.1 /lib64/libuuid.so.1 /lib64/libdevmapper.so.1.02 /lib64/libdl.so.2 /lib64/libgpg-error.so.0 /lib64/libpcap.so.1 /lib64/libpcre2-8.so.0 /lib64/libsepol.so.1 /lib64/libudev.so.1 /lib64/libm.so.6 26067: So the search scope is libpthread *first*. The above error never triggers because libc.so.6 *does* have the named versioned symbol, but libpthread.so.0 also has it too. My conclusion is this: * We can remove fork@GLIBC_2.2.5 from libpthread.so, the shared object doesn't encode it as being needed and won't cause a failure per the rules that require the referenced shared object to have the versioned symbol that was bound at static link time. Cheers, Carlos. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Are the pthread "compatibility" copies of symbols in libc still necessary? 2018-03-16 15:35 ` Carlos O'Donell @ 2018-03-16 15:42 ` Florian Weimer 2018-03-16 17:39 ` Carlos O'Donell 0 siblings, 1 reply; 9+ messages in thread From: Florian Weimer @ 2018-03-16 15:42 UTC (permalink / raw) To: Carlos O'Donell, Zack Weinberg, GNU C Library On 03/16/2018 04:35 PM, Carlos O'Donell wrote: > On 03/16/2018 08:41 AM, Florian Weimer wrote: >> On 03/16/2018 03:30 PM, Zack Weinberg wrote: >>> A number of source files that properly belong to libc.so are also >>> compiled as part of libpthread, with a note that this is for >>> "compatibility for old binaries". The exact set varies based on >>> architecture, but includes basic things like read, write, and fork - I >>> _think_ there was a difference in semantics in the distant past, >>> having something to do with thread cancellation. >> >> They are still needed because versioned symbols also embed a DSO name, and the dynamic linker checks that if it has not been interposed. >> >> I think it's this code in elf/dl-lookup.c: >> >>      if (__glibc_unlikely (res < 0) && skip_map == NULL) >>     { >>      /* Oh, oh. The file named in the relocation entry does not >>         contain the needed symbol. This code is never reached >>         for unversioned lookups. */ >>      assert (version != NULL); >>      const char *reference_name = undef_map ? undef_map->l_name : ""; >>      struct dl_exception exception; >>      /* XXX We cannot translate the message. */ >>      _dl_exception_create_format >>        (&exception, DSO_FILENAME (reference_name), >>         "symbol %s version %s not defined in file %s" >>         " with link time reference%s", >>         undef_name, version->name, version->filename, >>         res == -2 ? " (no version symbols)" : ""); >>      _dl_signal_cexception (0, &exception, N_("relocation error")); >>      _dl_exception_free (&exception); >>      *ref = NULL; >>      return 0; >>     } >> >> This is required by the GNU symbol versioning spec, but I don't know why. > > The specification says the data must be recorded, but it doesn't say what you > have to *do* with the data? It does: â A fatal error shall be triggered when no matching definition can be found in the object whose name is the one referenced by the vn_name element in the Elfxx_Verneed entry. â >> Even new binaries use these symbols: >> >> $ LD_DEBUG=bindings systemctl |& grep binding.*system.*libpthread.*[^_]fork >>     11905:   binding file /usr/lib/systemd/libsystemd-shared-234.so [0] to /lib64/libpthread.so.0 [0]: normal symbol `fork' [GLIBC_2.2.5] >>     11905:   binding file systemctl [0] to /lib64/libpthread.so.0 [0]: normal symbol `fork' [GLIBC_2.2.5] >> >> This may be a linker bug because at least fork is a compat symbol. > Is it a dynamic loader bug that the compat symbol was selected? No, I don't think the dynamic linker has to make this distinction, at least not if the soname matches the one in the version. > My conclusion is this: > > * We can remove fork@GLIBC_2.2.5 from libpthread.so, the shared object doesn't > encode it as being needed and won't cause a failure per the rules that > require the referenced shared object to have the versioned symbol that was > bound at static link time. I already tried a while bug and failed because the system does not boot anymore after that, due to the check mentioned above. I don't think the code has changed since then. Thanks, Florian ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Are the pthread "compatibility" copies of symbols in libc still necessary? 2018-03-16 15:42 ` Florian Weimer @ 2018-03-16 17:39 ` Carlos O'Donell 2018-03-16 17:40 ` Florian Weimer 2018-03-16 18:13 ` Zack Weinberg 0 siblings, 2 replies; 9+ messages in thread From: Carlos O'Donell @ 2018-03-16 17:39 UTC (permalink / raw) To: Florian Weimer, Zack Weinberg, GNU C Library On 03/16/2018 09:42 AM, Florian Weimer wrote: > On 03/16/2018 04:35 PM, Carlos O'Donell wrote: >> On 03/16/2018 08:41 AM, Florian Weimer wrote: >>> On 03/16/2018 03:30 PM, Zack Weinberg wrote: >>>> A number of source files that properly belong to libc.so are also >>>> compiled as part of libpthread, with a note that this is for >>>> "compatibility for old binaries". The exact set varies based on >>>> architecture, but includes basic things like read, write, and fork - I >>>> _think_ there was a difference in semantics in the distant past, >>>> having something to do with thread cancellation. >>> >>> They are still needed because versioned symbols also embed a DSO name, and the dynamic linker checks that if it has not been interposed. >>> >>> I think it's this code in elf/dl-lookup.c: >>> >>>       if (__glibc_unlikely (res < 0) && skip_map == NULL) >>>      { >>>       /* Oh, oh. The file named in the relocation entry does not >>>          contain the needed symbol. This code is never reached >>>          for unversioned lookups. */ >>>       assert (version != NULL); >>>       const char *reference_name = undef_map ? undef_map->l_name : ""; >>>       struct dl_exception exception; >>>       /* XXX We cannot translate the message. */ >>>       _dl_exception_create_format >>>         (&exception, DSO_FILENAME (reference_name), >>>          "symbol %s version %s not defined in file %s" >>>          " with link time reference%s", >>>          undef_name, version->name, version->filename, >>>          res == -2 ? " (no version symbols)" : ""); >>>       _dl_signal_cexception (0, &exception, N_("relocation error")); >>>       _dl_exception_free (&exception); >>>       *ref = NULL; >>>       return 0; >>>      } >>> >>> This is required by the GNU symbol versioning spec, but I don't know why. >> >> The specification says the data must be recorded, but it doesn't say what you >> have to *do* with the data? > > It does: > > â > A fatal error shall be triggered when no matching definition can be > found in the object whose name is the one referenced by the vn_name > element in the Elfxx_Verneed entry. > â I don't know what drove this requirement. From a first-principles perspective is doesn't seem to derive from any foundational aspect of the linkage model. Can you rationalize the requirement yourself? Am I missing something? >>> Even new binaries use these symbols: >>> >>> $ LD_DEBUG=bindings systemctl |& grep binding.*system.*libpthread.*[^_]fork >>>      11905:   binding file /usr/lib/systemd/libsystemd-shared-234.so [0] to /lib64/libpthread.so.0 [0]: normal symbol `fork' [GLIBC_2.2.5] >>>      11905:   binding file systemctl [0] to /lib64/libpthread.so.0 [0]: normal symbol `fork' [GLIBC_2.2.5] >>> >>> This may be a linker bug because at least fork is a compat symbol. > >> Is it a dynamic loader bug that the compat symbol was selected? > > No, I don't think the dynamic linker has to make this distinction, at > least not if the soname matches the one in the version. I think the problem is that the two checks are orthogonal. You have one check to find the versioned symbol based on lookup scopes. You have another check to determine if a DSO with the appropriate symbol and version exists. Both of these are true in the fork case. >> My conclusion is this: >> >> * We can remove fork@GLIBC_2.2.5 from libpthread.so, the shared object doesn't >>   encode it as being needed and won't cause a failure per the rules that >>   require the referenced shared object to have the versioned symbol that was >>   bound at static link time. > > I already tried a while bug and failed because the system does not > boot anymore after that, due to the check mentioned above. I don't > think the code has changed since then. Do you know what failed? I should try this. Cheers, Carlos. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Are the pthread "compatibility" copies of symbols in libc still necessary? 2018-03-16 17:39 ` Carlos O'Donell @ 2018-03-16 17:40 ` Florian Weimer 2018-03-16 18:13 ` Zack Weinberg 1 sibling, 0 replies; 9+ messages in thread From: Florian Weimer @ 2018-03-16 17:40 UTC (permalink / raw) To: Carlos O'Donell, Zack Weinberg, GNU C Library On 03/16/2018 06:38 PM, Carlos O'Donell wrote: >> It does: >> >> â >> A fatal error shall be triggered when no matching definition can be >> found in the object whose name is the one referenced by the vn_name >> element in the Elfxx_Verneed entry. >> â > >>> My conclusion is this: >>> >>> * We can remove fork@GLIBC_2.2.5 from libpthread.so, the shared object doesn't >>> Â Â encode it as being needed and won't cause a failure per the rules that >>> Â Â require the referenced shared object to have the versioned symbol that was >>> Â Â bound at static link time. >> >> I already tried a while bug and failed because the system does not >> boot anymore after that, due to the check mentioned above. I don't >> think the code has changed since then. > > Do you know what failed? I should try this. I got the fatal error required by the specification. Florian ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Are the pthread "compatibility" copies of symbols in libc still necessary? 2018-03-16 17:39 ` Carlos O'Donell 2018-03-16 17:40 ` Florian Weimer @ 2018-03-16 18:13 ` Zack Weinberg 2018-03-16 20:46 ` Carlos O'Donell 1 sibling, 1 reply; 9+ messages in thread From: Zack Weinberg @ 2018-03-16 18:13 UTC (permalink / raw) To: Carlos O'Donell; +Cc: Florian Weimer, GNU C Library On Fri, Mar 16, 2018 at 1:38 PM, Carlos O'Donell <carlos@redhat.com> wrote: > On 03/16/2018 09:42 AM, Florian Weimer wrote: >> On 03/16/2018 04:35 PM, Carlos O'Donell wrote: >>> >>> The specification says the data must be recorded, but it doesn't say what you >>> have to *do* with the data? >> >> It does: >> >> “ >> A fatal error shall be triggered when no matching definition can be >> found in the object whose name is the one referenced by the vn_name >> element in the Elfxx_Verneed entry. >> ” > > I don't know what drove this requirement. > > From a first-principles perspective is doesn't seem to derive from any > foundational aspect of the linkage model. I don't claim to know what Ulrich was thinking at the time, but if you start from the premise that it would be more consistent in general if symbols always resolved at load time to the library they were found in at link time, then adding this requirement for versioned symbols can be seen as a way to introduce that consistency in a backward-compatible fashion. That said, without any way to declare that symbol X is _supposed_ to have moved from foo.so to bar.so, it puts us in a hole. This is a GNU spec, so we _could_ change it. We'd need to do a bit of formal consultation with everyone else who might be affected, but I don't think our hands should be tied by a decision that nobody remembers the original rationale for anymore. zw ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Are the pthread "compatibility" copies of symbols in libc still necessary? 2018-03-16 18:13 ` Zack Weinberg @ 2018-03-16 20:46 ` Carlos O'Donell 2018-03-16 21:12 ` Zack Weinberg 0 siblings, 1 reply; 9+ messages in thread From: Carlos O'Donell @ 2018-03-16 20:46 UTC (permalink / raw) To: Zack Weinberg; +Cc: Florian Weimer, GNU C Library On 03/16/2018 12:13 PM, Zack Weinberg wrote: > On Fri, Mar 16, 2018 at 1:38 PM, Carlos O'Donell <carlos@redhat.com> wrote: >> On 03/16/2018 09:42 AM, Florian Weimer wrote: >>> On 03/16/2018 04:35 PM, Carlos O'Donell wrote: >>>> >>>> The specification says the data must be recorded, but it doesn't say what you >>>> have to *do* with the data? >>> >>> It does: >>> >>> â >>> A fatal error shall be triggered when no matching definition can be >>> found in the object whose name is the one referenced by the vn_name >>> element in the Elfxx_Verneed entry. >>> â >> >> I don't know what drove this requirement. >> >> From a first-principles perspective is doesn't seem to derive from any >> foundational aspect of the linkage model. > > I don't claim to know what Ulrich was thinking at the time, but if you > start from the premise that it would be more consistent in general if > symbols always resolved at load time to the library they were found in > at link time, then adding this requirement for versioned symbols can > be seen as a way to introduce that consistency in a > backward-compatible fashion. That said, without any way to declare > that symbol X is _supposed_ to have moved from foo.so to bar.so, it > puts us in a hole. > > This is a GNU spec, so we _could_ change it. We'd need to do a bit of > formal consultation with everyone else who might be affected, but I > don't think our hands should be tied by a decision that nobody > remembers the original rationale for anymore. I agree. I'd like to investigate changing this, but don't see myself having the time to do so in the next 4-6 months, not at least until 2.28 is out the door (I'm the RM). Cheers, Carlos. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Are the pthread "compatibility" copies of symbols in libc still necessary? 2018-03-16 20:46 ` Carlos O'Donell @ 2018-03-16 21:12 ` Zack Weinberg 0 siblings, 0 replies; 9+ messages in thread From: Zack Weinberg @ 2018-03-16 21:12 UTC (permalink / raw) To: Carlos O'Donell; +Cc: GNU C Library On Fri, Mar 16, 2018 at 4:46 PM, Carlos O'Donell <carlos@redhat.com> wrote: >> This is a GNU spec, so we _could_ change it. We'd need to do a bit of >> formal consultation with everyone else who might be affected, but I >> don't think our hands should be tied by a decision that nobody >> remembers the original rationale for anymore. > > I agree. I'd like to investigate changing this, but don't see myself > having the time to do so in the next 4-6 months, not at least until > 2.28 is out the door (I'm the RM). My time is also extremely limited through September or so. ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2018-03-16 21:12 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-03-16 14:30 Are the pthread "compatibility" copies of symbols in libc still necessary? Zack Weinberg 2018-03-16 14:41 ` Florian Weimer 2018-03-16 15:35 ` Carlos O'Donell 2018-03-16 15:42 ` Florian Weimer 2018-03-16 17:39 ` Carlos O'Donell 2018-03-16 17:40 ` Florian Weimer 2018-03-16 18:13 ` Zack Weinberg 2018-03-16 20:46 ` Carlos O'Donell 2018-03-16 21:12 ` Zack Weinberg
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).