libstdc++ link errors in support/links-dso-program

libstdc++ link errors in support/links-dso-program

[Горбешко Богдан]

Hi.

I attempt to build glibc 2.31 on Debian GNU/Linux 12 Bookworm. It fails at:

gcc 
-Wl,-rpath-link=/home/bodqhrohro/git/glibc-2.31-build:/home/bodqhrohro/git/glibc-2.31-build/math:/home/bodqhrohro/git/glibc-2.31-build/elf:/home/bodqhrohro/git/glibc-2.31-build/dlfcn:/home/bodqhrohro/git/glibc-2.31-build/nss:/home/bodqhrohro/git/glibc-2.31-build/nis:/home/bodqhrohro/git/glibc-2.31-build/rt:/home/bodqhrohro/git/glibc-2.31-build/resolv:/home/bodqhrohro/git/glibc-2.31-build/mathvec:/home/bodqhrohro/git/glibc-2.31-build/support:/home/bodqhrohro/git/glibc-2.31-build/crypt:/home/bodqhrohro/git/glibc-2.31-build/nptl 
-pie -Wl,-O1 -nostdlib -nostartfiles -o 
/home/bodqhrohro/git/glibc-2.31-build/support/links-dso-program 
-Wl,-z,combreloc -Wl,-z,relro -Wl,--hash-style=both 
/home/bodqhrohro/git/glibc-2.31-build/csu/Scrt1.o 
/home/bodqhrohro/git/glibc-2.31-build/csu/crti.o `gcc 
--print-file-name=crtbeginS.o` 
/home/bodqhrohro/git/glibc-2.31-build/support/links-dso-program.o 
-lstdc++ -lgcc -lgcc_s 
-Wl,-dynamic-linker=/home/bodqhrohro/git/glibc-2.31-build/lib/ld-linux-x86-64.so.2 
/home/bodqhrohro/git/glibc-2.31-build/libc.so.6 
/home/bodqhrohro/git/glibc-2.31-build/libc_nonshared.a -Wl,--as-needed 
/home/bodqhrohro/git/glibc-2.31-build/elf/ld.so -Wl,--no-as-needed 
-lgcc  `gcc  --print-file-name=crtendS.o` 
/home/bodqhrohro/git/glibc-2.31-build/csu/crtn.o
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/12/libstdc++.so: undefined 
reference to `fstat64@GLIBC_2.33'
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/12/libstdc++.so: undefined 
reference to `pthread_key_create@GLIBC_2.34'
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/12/libstdc++.so: undefined 
reference to `pthread_rwlock_unlock@GLIBC_2.34'
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/12/libstdc++.so: undefined 
reference to `pthread_detach@GLIBC_2.34'
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/12/libstdc++.so: undefined 
reference to `pthread_setspecific@GLIBC_2.34'
/usr/bin/ld: 
/usr/lib/gcc/x86_64-linux-gnu/12/../../../x86_64-linux-gnu/libgcc_s.so.1: 
undefined reference to `_dl_find_object@GLIBC_2.35'
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/12/libstdc++.so: undefined 
reference to `arc4random@GLIBC_2.36'
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/12/libstdc++.so: undefined 
reference to `__libc_single_threaded@GLIBC_2.32'
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/12/libstdc++.so: undefined 
reference to `pthread_join@GLIBC_2.34'
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/12/libstdc++.so: undefined 
reference to `pthread_rwlock_wrlock@GLIBC_2.34'
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/12/libstdc++.so: undefined 
reference to `pthread_getspecific@GLIBC_2.34'
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/12/libstdc++.so: undefined 
reference to `pthread_key_delete@GLIBC_2.34'
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/12/libstdc++.so: undefined 
reference to `lstat@GLIBC_2.33'
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/12/libstdc++.so: undefined 
reference to `stat@GLIBC_2.33'
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/12/libstdc++.so: undefined 
reference to `pthread_once@GLIBC_2.34'
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/12/libstdc++.so: undefined 
reference to `pthread_rwlock_rdlock@GLIBC_2.34'
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/12/libstdc++.so: undefined 
reference to `pthread_create@GLIBC_2.34'
collect2: error: ld returned 1 exit status
make[2]: *** [../Rules:215: 
/home/bodqhrohro/git/glibc-2.31-build/support/links-dso-program] Error 1


Is that because it attempts to link against the system libstdc++, which 
is too new and requires symbols not present in this glibc version? Do I 
need to compile it myself? Or maybe even the whole GCC? I attempted to 
configure it without the SELinux support, but there's still the same 
problem.

Re: libstdc++ link errors in support/links-dso-program

[Florian Weimer]

* Горбешко Богдан via Libc-help:

> Is that because it attempts to link against the system libstdc++,
> which is too new and requires symbols not present in this glibc
> version? Do I need to compile it myself? Or maybe even the whole GCC?

Going back before 2.34 if the distribution is on 2.34 or later is not
really possible.  You need to build a whole cross-toolchain environment.
It's easier to build on an older version of the distribution?

Why do you need to go back 2.31?  What kind of problem are you trying to
solve?

Thanks,
Florian

Re: libstdc++ link errors in support/links-dso-program

[Горбешко Богдан]

I'm not going to install it system-wide, just to build it for 
cross-compiling a Golang project for a Debian Buster system, by 
specifying -rpath and --dynamic-linker.

On 15/03/2024 09:27, Florian Weimer wrote:
> * Горбешко Богдан via Libc-help:
>
>> Is that because it attempts to link against the system libstdc++,
>> which is too new and requires symbols not present in this glibc
>> version? Do I need to compile it myself? Or maybe even the whole GCC?
> Going back before 2.34 if the distribution is on 2.34 or later is not
> really possible.  You need to build a whole cross-toolchain environment.
> It's easier to build on an older version of the distribution?
>
> Why do you need to go back 2.31?  What kind of problem are you trying to
> solve?
>
> Thanks,
> Florian
>

Re: libstdc++ link errors in support/links-dso-program

[Florian Weimer]

* Горбешко Богдан:

> I'm not going to install it system-wide, just to build it for
> cross-compiling a Golang project for a Debian Buster system, by
> specifying -rpath and --dynamic-linker.

Is this because you need a newer GCC?  If not, it's easier to use
pbuilder (or one of its variants) to build the program in a chroot with
Debian buster.  Many people nowadays use podman or docker with a Debian
image for the same purpose.  The pbuilder approach is perhaps preferable
if you already build a .deb archive, and podman can be scripted more
easily to build something that isn't.

Thanks,
Florian

Re: libstdc++ link errors in support/links-dso-program

[Горбешко Богдан]

On 15/03/2024 19:07, Florian Weimer wrote:
> * Горбешко Богдан:
>
>> I'm not going to install it system-wide, just to build it for
>> cross-compiling a Golang project for a Debian Buster system, by
>> specifying -rpath and --dynamic-linker.
> Is this because you need a newer GCC?
Not really, GCC 12 is just the default on Bullseye, GCC 10 seems to be 
available there too. So I should attempt to build and link with GCC 10?
> If not, it's easier to use
> pbuilder (or one of its variants) to build the program in a chroot with
> Debian buster.  Many people nowadays use podman or docker with a Debian
> image for the same purpose.  The pbuilder approach is perhaps preferable
> if you already build a .deb archive, and podman can be scripted more
> easily to build something that isn't.

I don't really build .deb packages, Golang compiler produces one static 
binary, so that would be redundant.

We already have a solution to build this in Docker, though the target 
test VPS is tight on resources and I wouldn't really like to mess with 
Docker there.

My scenario is developing the project on one VPS, building it there and 
sending test builds (just copying one binary file, pretty easy) to 
another VPS where they are run. Previously they had the same Debian 
version, so this was a non-issue; recently I upgraded the first one to 
Bullseye, but the target VPS has some legacy and 3rd party software 
running, so an upgrade would be complicated and I postponed it for now. 
I could actually run the builds on the development VPS as well, migrate 
project data there and bind a remote port via SSH, but this would make 
it more prone to network interruptions.

>
> Thanks,
> Florian
>

Re: libstdc++ link errors in support/links-dso-program

[Florian Weimer]

* Горбешко Богдан:

> We already have a solution to build this in Docker, though the target
> test VPS is tight on resources and I wouldn't really like to mess with
> Docker there.

You don't have to build on the target system directly, you just need to
use a container/chroot environment that matches the target distribution
version.  These containers tend to be fairly lightweight, requiring less
space and resources than a full glibc build.

Thanks,
Florian

Re: libstdc++ link errors in support/links-dso-program

[Горбешко Богдан]

On 10/04/2024 15:03, Florian Weimer wrote:
> * Горбешко Богдан:
>
>> We already have a solution to build this in Docker, though the target
>> test VPS is tight on resources and I wouldn't really like to mess with
>> Docker there.
> You don't have to build on the target system directly, you just need to
> use a container/chroot environment that matches the target distribution
> version.  These containers tend to be fairly lightweight, requiring less
> space and resources than a full glibc build.

An unpacked binary Glibc package takes just 12 MB. A minimal Debian 
image is orders of magnitude larger.

And the primary problem with Docker is that continuous rebuilds occupy 
the disk space rapidly fast, as it normally preserves old builds, which 
is not a case with builds on a host system. This has to be fought in 
non-trivial ways, like thorough caching. And cleaning needs to be 
precise and accurate, otherwise it would involve a full recompilation, 
which is much longer than partial recompilation.

I attempted to go with a balanced approach, by building only the C++ 
library part in Docker, exporting it and linking the Golang part against 
it and old Glibc/libssl, but forfeited even with that, as symbols 
requiring a new Glibc still somehow leak into the build. So I still have 
to suffer with Docker BuildKit now.

>
> Thanks,
> Florian
>

Re: libstdc++ link errors in support/links-dso-program

[Konstantin Kharlamov]

On Wed, 2024-04-10 at 15:50 +0300, Горбешко Богдан via Libc-help wrote:
> On 10/04/2024 15:03, Florian Weimer wrote:
> > * Горбешко Богдан:
> > 
> > > We already have a solution to build this in Docker, though the
> > > target
> > > test VPS is tight on resources and I wouldn't really like to mess
> > > with
> > > Docker there.
> > You don't have to build on the target system directly, you just
> > need to
> > use a container/chroot environment that matches the target
> > distribution
> > version.  These containers tend to be fairly lightweight, requiring
> > less
> > space and resources than a full glibc build.
> 
> An unpacked binary Glibc package takes just 12 MB. A minimal Debian 
> image is orders of magnitude larger.
> 
> And the primary problem with Docker is that continuous rebuilds
> occupy 
> the disk space rapidly fast, as it normally preserves old builds,
> which 
> is not a case with builds on a host system. 

I am confused. Docker may or may not preserve something depending on
how you use it. You're saying compilation on the host system is not
preserved, which confuses me even further because why would it not.
Unless you run `make clean && make distclean` sure you have all
artifacts in place.

Anyway. As far as I understand what you're saying you don't want Docker
to preserve a build. I presume it's being "preserved" because you're
building on a host directory that is mount-bound via `-v` option (there
isn't many other ways you'd be getting that. Saving container state is
another one but why would you do that if you don't want the state to be
saved. So it is likely not that). You can solve that by using mount-
bind in such way so that any changes the container does to the mound-
bind will only be visible inside the container. Idk of any ways to do
that in the vanilla Docker, but in Podman it's supported by `:O`
option, i.e. `-v my_host_dir:/mnt:O`.

Re: libstdc++ link errors in support/links-dso-program

[Горбешко Богдан]

On 10/04/2024 16:22, Konstantin Kharlamov wrote:
> On Wed, 2024-04-10 at 15:50 +0300, Горбешко Богдан via Libc-help wrote:
>> On 10/04/2024 15:03, Florian Weimer wrote:
>>> * Горбешко Богдан:
>>>
>>>> We already have a solution to build this in Docker, though the
>>>> target
>>>> test VPS is tight on resources and I wouldn't really like to mess
>>>> with
>>>> Docker there.
>>> You don't have to build on the target system directly, you just
>>> need to
>>> use a container/chroot environment that matches the target
>>> distribution
>>> version.  These containers tend to be fairly lightweight, requiring
>>> less
>>> space and resources than a full glibc build.
>> An unpacked binary Glibc package takes just 12 MB. A minimal Debian
>> image is orders of magnitude larger.
>>
>> And the primary problem with Docker is that continuous rebuilds
>> occupy
>> the disk space rapidly fast, as it normally preserves old builds,
>> which
>> is not a case with builds on a host system.
> I am confused. Docker may or may not preserve something depending on
> how you use it. You're saying compilation on the host system is not
> preserved, which confuses me even further because why would it not.
> Unless you run `make clean && make distclean` sure you have all
> artifacts in place.
The primary difference is that artifacts overwrite the same files, while 
in Docker every difference involves creating new images (or new cache 
entries if BuildKit is used) for the whole chain after the layer where a 
change happened.
>
> Anyway. As far as I understand what you're saying you don't want Docker
> to preserve a build. I presume it's being "preserved" because you're
> building on a host directory that is mount-bound via `-v` option (there
> isn't many other ways you'd be getting that. Saving container state is
> another one but why would you do that if you don't want the state to be
> saved. So it is likely not that). You can solve that by using mount-
> bind in such way so that any changes the container does to the mound-
> bind will only be visible inside the container. Idk of any ways to do
> that in the vanilla Docker, but in Podman it's supported by `:O`
> option, i.e. `-v my_host_dir:/mnt:O`.

I already added a lot of bind mounts and cache mounts to minimize the 
redundancy. Cache mounts are reused between builds, at least.

Re: libstdc++ link errors in support/links-dso-program

[Konstantin Kharlamov]

On Wed, 2024-04-10 at 16:30 +0300, Горбешко Богдан wrote:
> On 10/04/2024 16:22, Konstantin Kharlamov wrote:
> > On Wed, 2024-04-10 at 15:50 +0300, Горбешко Богдан via Libc-help
> > wrote:
> > > On 10/04/2024 15:03, Florian Weimer wrote:
> > > > * Горбешко Богдан:
> > > > 
> > > > > We already have a solution to build this in Docker, though
> > > > > the
> > > > > target
> > > > > test VPS is tight on resources and I wouldn't really like to
> > > > > mess
> > > > > with
> > > > > Docker there.
> > > > You don't have to build on the target system directly, you just
> > > > need to
> > > > use a container/chroot environment that matches the target
> > > > distribution
> > > > version.  These containers tend to be fairly lightweight,
> > > > requiring
> > > > less
> > > > space and resources than a full glibc build.
> > > An unpacked binary Glibc package takes just 12 MB. A minimal
> > > Debian
> > > image is orders of magnitude larger.
> > > 
> > > And the primary problem with Docker is that continuous rebuilds
> > > occupy
> > > the disk space rapidly fast, as it normally preserves old builds,
> > > which
> > > is not a case with builds on a host system.
> > I am confused. Docker may or may not preserve something depending
> > on
> > how you use it. You're saying compilation on the host system is not
> > preserved, which confuses me even further because why would it not.
> > Unless you run `make clean && make distclean` sure you have all
> > artifacts in place.
> The primary difference is that artifacts overwrite the same files,
> while 
> in Docker every difference involves creating new images (or new cache
> entries if BuildKit is used) for the whole chain after the layer
> where a 
> change happened.
> > 
> > Anyway. As far as I understand what you're saying you don't want
> > Docker
> > to preserve a build. I presume it's being "preserved" because
> > you're
> > building on a host directory that is mount-bound via `-v` option
> > (there
> > isn't many other ways you'd be getting that. Saving container state
> > is
> > another one but why would you do that if you don't want the state
> > to be
> > saved. So it is likely not that). You can solve that by using
> > mount-
> > bind in such way so that any changes the container does to the
> > mound-
> > bind will only be visible inside the container. Idk of any ways to
> > do
> > that in the vanilla Docker, but in Podman it's supported by `:O`
> > option, i.e. `-v my_host_dir:/mnt:O`.
> 
> I already added a lot of bind mounts and cache mounts to minimize the
> redundancy. Cache mounts are reused between builds, at least.

Why are you re-building Glibc in CI in the first place? Are you doing
some glibc development, or is it the same Glibc version built over and
over?

In company I work at we have various 3rd party packages with
customizations, and to avoid rebuilding them every time we just store
them to a separate repo that is referred by the main project via git-
submodule. Works very well.

Re: libstdc++ link errors in support/links-dso-program

[Горбешко Богдан]

On 10/04/2024 16:55, Konstantin Kharlamov wrote:
> On Wed, 2024-04-10 at 16:30 +0300, Горбешко Богдан wrote:
>> On 10/04/2024 16:22, Konstantin Kharlamov wrote:
>>> On Wed, 2024-04-10 at 15:50 +0300, Горбешко Богдан via Libc-help
>>> wrote:
>>>> On 10/04/2024 15:03, Florian Weimer wrote:
>>>>> * Горбешко Богдан:
>>>>>
>>>>>> We already have a solution to build this in Docker, though
>>>>>> the
>>>>>> target
>>>>>> test VPS is tight on resources and I wouldn't really like to
>>>>>> mess
>>>>>> with
>>>>>> Docker there.
>>>>> You don't have to build on the target system directly, you just
>>>>> need to
>>>>> use a container/chroot environment that matches the target
>>>>> distribution
>>>>> version.  These containers tend to be fairly lightweight,
>>>>> requiring
>>>>> less
>>>>> space and resources than a full glibc build.
>>>> An unpacked binary Glibc package takes just 12 MB. A minimal
>>>> Debian
>>>> image is orders of magnitude larger.
>>>>
>>>> And the primary problem with Docker is that continuous rebuilds
>>>> occupy
>>>> the disk space rapidly fast, as it normally preserves old builds,
>>>> which
>>>> is not a case with builds on a host system.
>>> I am confused. Docker may or may not preserve something depending
>>> on
>>> how you use it. You're saying compilation on the host system is not
>>> preserved, which confuses me even further because why would it not.
>>> Unless you run `make clean && make distclean` sure you have all
>>> artifacts in place.
>> The primary difference is that artifacts overwrite the same files,
>> while
>> in Docker every difference involves creating new images (or new cache
>> entries if BuildKit is used) for the whole chain after the layer
>> where a
>> change happened.
>>> Anyway. As far as I understand what you're saying you don't want
>>> Docker
>>> to preserve a build. I presume it's being "preserved" because
>>> you're
>>> building on a host directory that is mount-bound via `-v` option
>>> (there
>>> isn't many other ways you'd be getting that. Saving container state
>>> is
>>> another one but why would you do that if you don't want the state
>>> to be
>>> saved. So it is likely not that). You can solve that by using
>>> mount-
>>> bind in such way so that any changes the container does to the
>>> mound-
>>> bind will only be visible inside the container. Idk of any ways to
>>> do
>>> that in the vanilla Docker, but in Podman it's supported by `:O`
>>> option, i.e. `-v my_host_dir:/mnt:O`.
>> I already added a lot of bind mounts and cache mounts to minimize the
>> redundancy. Cache mounts are reused between builds, at least.
> Why are you re-building Glibc in CI in the first place? Are you doing
> some glibc development, or is it the same Glibc version built over and
> over?

It's not a CI and I don't need a patched Glibc actually.

I mentioned already that I'm just cross-compiling a Golang project to be 
run and tested during development frequently on an older distribution.

>
> In company I work at we have various 3rd party packages with
> customizations, and to avoid rebuilding them every time we just store
> them to a separate repo that is referred by the main project via git-
> submodule. Works very well.

Re: libstdc++ link errors in support/links-dso-program

[Konstantin Kharlamov]

On Wed, 2024-04-10 at 17:02 +0300, Горбешко Богдан wrote:
> It's not a CI and I don't need a patched Glibc actually.

Ah, sorry, "continuous rebuilds" sentence threw me off, so I thought
it's about CI 😄

> I mentioned already that I'm just cross-compiling a Golang project to
> be 
> run and tested during development frequently on an older
> distribution.

Okay. I still don't get the whole picture, but as you mentioned that
you've added lots of bind-mounts, etc, to minimize the redundancy,
hopefully that part is solved 😊

Re: libstdc++ link errors in support/links-dso-program

[Горбешко Богдан]

On 10/04/2024 17:11, Konstantin Kharlamov wrote:
> On Wed, 2024-04-10 at 17:02 +0300, Горбешко Богдан wrote:
>> It's not a CI and I don't need a patched Glibc actually.
> Ah, sorry, "continuous rebuilds" sentence threw me off, so I thought
> it's about CI 😄
>
>> I mentioned already that I'm just cross-compiling a Golang project to
>> be
>> run and tested during development frequently on an older
>> distribution.
> Okay. I still don't get the whole picture, but as you mentioned that
> you've added lots of bind-mounts, etc, to minimize the redundancy,
> hopefully that part is solved 😊
Technically: yes, to some tolerable level. Though I'd still like to know 
if it's still possible to cross-compile for older Glibc with Glibc ≥2.34 
on the host system. Seems like the changes are too breaking there, so 
others forfeit too: 
https://github.com/wheybags/glibc_version_header/issues/32#issuecomment-1826368565