Build hangs on stdio-common for glibc-2.30-67-g4748829f86a458b76642f3e98b1d80f7b868e427

Build hangs on stdio-common for glibc-2.30-67-g4748829f86a458b76642f3e98b1d80f7b868e427

[Boran Car]

While using crosstools-NG and Buildroot, and even manually going into
the folder and executing make, I would end up with the build hanging
at

`make -d subdir=stdio-common -C stdio-common ..=../ subdir_lib`

I'm using GNU Make 4.4.1.

I modified the Makefile to print out tracing information when hitting
stdio-common and re-ran it and I get make itself re-executing the
target, never finishing, despite the target already being built.
Here's a very short and snipped output of the trace, because it's very noisy:
```
 Considering target file
'/home/boran/git/buildroot-2020.02.3/output/build/glibc-2.30-67-g4748829f86a458b76642f3e98b1d80f7b868e427/build/bits/stdio_lim.d'.
 File '/home/boran/git/buildroot-2020.02.3/output/build/glibc-2.30-67-g4748829f86a458b76642f3e98b1d80f7b868e427/build/bits/stdio_lim.d'
was considered already.
Re-executing[48]: make -d subdir=stdio-common -C stdio-common ..=../ subdir_lib
```

I can also modify the Makefile to skip stdio-common/subdir_lib and
will then get stuck in stdio-common/others, and modifying that to skip
it will get stuck on stdio-common/subdir_install. Those are the only 3
issues, and I usually have to let them run on the first try and skip
on subsequent tries. With those changes in place, the glibc builds
successfully.

My questions are:
- Are there any pointers where I should go looking for what's the
cause of the re-executions and how to prevent them?
- Is there anything I can use from subsequent glibc Makefiles? Was an
issue discovered with the target computation?

Re: Build hangs on stdio-common for glibc-2.30-67-g4748829f86a458b76642f3e98b1d80f7b868e427

[Boran Car]

I have some more info, from stdio-common/others. It genuinely seems
that .stmp and .st files are being created for every invocation.

```
Re-executing[30]: /usr/bin/make --debug=Makefile subdir=stdio-common
-C stdio-common ..=../ others
GNU Make 4.4.1
Built for x86_64-redhat-linux-gnu
Copyright (C) 1988-2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Reading makefiles...
Updating makefiles....
  Prerequisite '/home/boran/git/buildroot-2020.02.3/output/build/glibc-2.30-67-g4748829f86a458b76642f3e98b1d80f7b868e427/build/versions.stmp'
is newer than target '/home/boran/git/buildroot-2020.02.3/outp
ut/build/glibc-2.30-67-g4748829f86a458b76642f3e98b1d80f7b868e427/build/sysd-versions'.
     Prerequisite
'/home/boran/git/buildroot-2020.02.3/output/build/glibc-2.30-67-g4748829f86a458b76642f3e98b1d80f7b868e427/build/libc-modules.stmp'
is newer than target '/home/boran/git/buildroot-2020.02
.3/output/build/glibc-2.30-67-g4748829f86a458b76642f3e98b1d80f7b868e427/build/libc-modules.h'.
    Must remake target
'/home/boran/git/buildroot-2020.02.3/output/build/glibc-2.30-67-g4748829f86a458b76642f3e98b1d80f7b868e427/build/libc-modules.h'.
    Successfully remade target file
'/home/boran/git/buildroot-2020.02.3/output/build/glibc-2.30-67-g4748829f86a458b76642f3e98b1d80f7b868e427/build/libc-modules.h'.
     Prerequisite
'/home/boran/git/buildroot-2020.02.3/output/build/glibc-2.30-67-g4748829f86a458b76642f3e98b1d80f7b868e427/build/dl-tunable-list.stmp'
is newer than target '/home/boran/git/buildroot-2020
.02.3/output/build/glibc-2.30-67-g4748829f86a458b76642f3e98b1d80f7b868e427/build/dl-tunable-list.h'.
    Must remake target
'/home/boran/git/buildroot-2020.02.3/output/build/glibc-2.30-67-g4748829f86a458b76642f3e98b1d80f7b868e427/build/dl-tunable-list.h'.
    Successfully remade target file
'/home/boran/git/buildroot-2020.02.3/output/build/glibc-2.30-67-g4748829f86a458b76642f3e98b1d80f7b868e427/build/dl-tunable-list.h'.
     Prerequisite
'/home/boran/git/buildroot-2020.02.3/output/build/glibc-2.30-67-g4748829f86a458b76642f3e98b1d80f7b868e427/build/bits/stdio_lim.st'
is newer than target '/home/boran/git/buildroot-2020.02
.3/output/build/glibc-2.30-67-g4748829f86a458b76642f3e98b1d80f7b868e427/build/bits/stdio_lim.h'.
    Must remake target
'/home/boran/git/buildroot-2020.02.3/output/build/glibc-2.30-67-g4748829f86a458b76642f3e98b1d80f7b868e427/build/bits/stdio_lim.h'.
    Successfully remade target file
'/home/boran/git/buildroot-2020.02.3/output/build/glibc-2.30-67-g4748829f86a458b76642f3e98b1d80f7b868e427/build/bits/stdio_lim.h'.
Re-executing[31]: /usr/bin/make --debug=Makefile subdir=stdio-common
-C stdio-common ..=../ others

             ```

On Sun, Sep 17, 2023 at 8:53 PM Boran Car <boran.car@gmail.com> wrote:
>
> While using crosstools-NG and Buildroot, and even manually going into
> the folder and executing make, I would end up with the build hanging
> at
>
> `make -d subdir=stdio-common -C stdio-common ..=../ subdir_lib`
>
> I'm using GNU Make 4.4.1.
>
> I modified the Makefile to print out tracing information when hitting
> stdio-common and re-ran it and I get make itself re-executing the
> target, never finishing, despite the target already being built.
> Here's a very short and snipped output of the trace, because it's very noisy:
> ```
>  Considering target file
> '/home/boran/git/buildroot-2020.02.3/output/build/glibc-2.30-67-g4748829f86a458b76642f3e98b1d80f7b868e427/build/bits/stdio_lim.d'.
>  File '/home/boran/git/buildroot-2020.02.3/output/build/glibc-2.30-67-g4748829f86a458b76642f3e98b1d80f7b868e427/build/bits/stdio_lim.d'
> was considered already.
> Re-executing[48]: make -d subdir=stdio-common -C stdio-common ..=../ subdir_lib
> ```
>
> I can also modify the Makefile to skip stdio-common/subdir_lib and
> will then get stuck in stdio-common/others, and modifying that to skip
> it will get stuck on stdio-common/subdir_install. Those are the only 3
> issues, and I usually have to let them run on the first try and skip
> on subsequent tries. With those changes in place, the glibc builds
> successfully.
>
> My questions are:
> - Are there any pointers where I should go looking for what's the
> cause of the re-executions and how to prevent them?
> - Is there anything I can use from subsequent glibc Makefiles? Was an
> issue discovered with the target computation?

Re: Build hangs on stdio-common for glibc-2.30-67-g4748829f86a458b76642f3e98b1d80f7b868e427

[Chris Packham]

[-- Attachment #1: Type: text/plain, Size: 2022 bytes --]

Hi Boran

On Mon, 18 Sep 2023, 6:54 AM Boran Car via Libc-help, <
libc-help@sourceware.org> wrote:

> While using crosstools-NG and Buildroot, and even manually going into
> the folder and executing make, I would end up with the build hanging
> at
>
> `make -d subdir=stdio-common -C stdio-common ..=../ subdir_lib`
>
> I'm using GNU Make 4.4.1.
>
> I modified the Makefile to print out tracing information when hitting
> stdio-common and re-ran it and I get make itself re-executing the
> target, never finishing, despite the target already being built.
> Here's a very short and snipped output of the trace, because it's very
> noisy:
> ```
>  Considering target file
>
> '/home/boran/git/buildroot-2020.02.3/output/build/glibc-2.30-67-g4748829f86a458b76642f3e98b1d80f7b868e427/build/bits/stdio_lim.d'.
>  File
> '/home/boran/git/buildroot-2020.02.3/output/build/glibc-2.30-67-g4748829f86a458b76642f3e98b1d80f7b868e427/build/bits/stdio_lim.d'
> was considered already.
> Re-executing[48]: make -d subdir=stdio-common -C stdio-common ..=../
> subdir_lib
> ```
>
> I can also modify the Makefile to skip stdio-common/subdir_lib and
> will then get stuck in stdio-common/others, and modifying that to skip
> it will get stuck on stdio-common/subdir_install. Those are the only 3
> issues, and I usually have to let them run on the first try and skip
> on subsequent tries. With those changes in place, the glibc builds
> successfully.
>
> My questions are:
> - Are there any pointers where I should go looking for what's the
> cause of the re-executions and how to prevent them?
> - Is there anything I can use from subsequent glibc Makefiles? Was an
> issue discovered with the target computation?
>

There is a pretty good explanation of the problem and some workarounds at
https://github.com/crosstool-ng/crosstool-ng/issues/1932#issuecomment-1528139734

Using a newer glibc is probably the best solution. If you really need to
use that specific version then downgrading your local version of make is
another option.

>

Re: Build hangs on stdio-common for glibc-2.30-67-g4748829f86a458b76642f3e98b1d80f7b868e427

[Florian Weimer]

* Chris Packham via Libc-help:

>> My questions are:
>> - Are there any pointers where I should go looking for what's the
>> cause of the re-executions and how to prevent them?
>> - Is there anything I can use from subsequent glibc Makefiles? Was an
>> issue discovered with the target computation?
>>
>
> There is a pretty good explanation of the problem and some workarounds at
> https://github.com/crosstool-ng/crosstool-ng/issues/1932#issuecomment-1528139734
>
> Using a newer glibc is probably the best solution. If you really need to
> use that specific version then downgrading your local version of make is
> another option.

Has this reported anywhere as a make or glibc issue?  I don't
immediately recall how we fixed this in glibc 2.31.

Thanks,
Florian

Re: Build hangs on stdio-common for glibc-2.30-67-g4748829f86a458b76642f3e98b1d80f7b868e427

[Boran Car]

I don't think so, Googling has been quite difficult - lead me to a
Stack Overflow question that prompted me to reach out to the mailing
list where I found a mention of a GitHub issue. I can confirm that it
just works (tm) with an earlier Make version, v4.2, and I have a
simple Nix shell derivation to prove it (yeah, I know Nix + Buildroot,
go figure...).

shell.nix:
```
let pkgs = import  { };
in pkgs.mkShell {
    hardeningDisable = [ "all" ];
    nativeBuildInputs = [ pkgs.gcc7 pkgs.gnumake42 ];

    CC="${pkgs.gcc7}/bin/gcc";
    CXX="${pkgs.gcc7}/bin/g++";
}
```

There's definitely interest in fixing this, at least from my side, but
I do need handholding through the build-system. If a Makefile can be
back-ported, even better. There's still interest in pre 2.31 glibcs
out there, and Buildroot and crosstools-NG were definitely not made to
build their own Make as part of the process.

On Mon, Sep 18, 2023 at 1:43 PM Florian Weimer <fweimer@redhat.com> wrote:
>
> * Chris Packham via Libc-help:
>
> >> My questions are:
> >> - Are there any pointers where I should go looking for what's the
> >> cause of the re-executions and how to prevent them?
> >> - Is there anything I can use from subsequent glibc Makefiles? Was an
> >> issue discovered with the target computation?
> >>
> >
> > There is a pretty good explanation of the problem and some workarounds at
> > https://github.com/crosstool-ng/crosstool-ng/issues/1932#issuecomment-1528139734
> >
> > Using a newer glibc is probably the best solution. If you really need to
> > use that specific version then downgrading your local version of make is
> > another option.
>
> Has this reported anywhere as a make or glibc issue?  I don't
> immediately recall how we fixed this in glibc 2.31.
>
> Thanks,
> Florian
>

Re: Build hangs on stdio-common for glibc-2.30-67-g4748829f86a458b76642f3e98b1d80f7b868e427

[Chris Packham]

On Tue, Sep 19, 2023 at 4:06 AM Boran Car <boran.car@gmail.com> wrote:
>
> There's definitely interest in fixing this, at least from my side, but
> I do need handholding through the build-system. If a Makefile can be
> back-ported, even better. There's still interest in pre 2.31 glibcs
> out there, and Buildroot and crosstools-NG were definitely not made to
> build their own Make as part of the process.

Heh. That's exactly the workaround we put in for crosstool-ng
https://github.com/crosstool-ng/crosstool-ng/commit/e63c40854

I didn't stare too long at
https://sourceware.org/git/?p=glibc.git;a=commit;h=37f802f864006 which
is the change that "fixes" the issue. Fixing the make issue appears to
be more of a side effect than the actual intent of the change. Might
be work seeing how cleanly that commit cherry-picks onto older
releases.

Re: Build hangs on stdio-common for glibc-2.30-67-g4748829f86a458b76642f3e98b1d80f7b868e427

[Boran Car]

I finally had time to get back to this. That change seems to already
be in on my side, I'm at commit
4748829f86a458b76642f3e98b1d80f7b868e427. I think my best course of
action is to bisect, first on versions, then commits - not something I
was looking forward to, at all :(. I'm just not overly happy with my
Nix solution - why add another thing?

On Tue, Sep 19, 2023 at 11:09 PM Chris Packham <judge.packham@gmail.com> wrote:
>
> On Tue, Sep 19, 2023 at 4:06 AM Boran Car <boran.car@gmail.com> wrote:
> >
> > There's definitely interest in fixing this, at least from my side, but
> > I do need handholding through the build-system. If a Makefile can be
> > back-ported, even better. There's still interest in pre 2.31 glibcs
> > out there, and Buildroot and crosstools-NG were definitely not made to
> > build their own Make as part of the process.
>
> Heh. That's exactly the workaround we put in for crosstool-ng
> https://github.com/crosstool-ng/crosstool-ng/commit/e63c40854
>
> I didn't stare too long at
> https://sourceware.org/git/?p=glibc.git;a=commit;h=37f802f864006 which
> is the change that "fixes" the issue. Fixing the make issue appears to
> be more of a side effect than the actual intent of the change. Might
> be work seeing how cleanly that commit cherry-picks onto older
> releases.