[Bug other/109898] New: 'make install -j' sometimes corrupts 'dir' file for .info files due to parallel 'install-info' calls

[Bug other/109898] New: 'make install -j' sometimes corrupts 'dir' file for .info files due to parallel 'install-info' calls

[slyfox at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109898

            Bug ID: 109898
           Summary: 'make install -j' sometimes corrupts 'dir' file for
                    .info files due to parallel 'install-info' calls
           Product: gcc
           Version: 14.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: other
          Assignee: unassigned at gcc dot gnu.org
          Reporter: slyfox at gcc dot gnu.org
  Target Milestone: ---

In https://github.com/NixOS/nixpkgs/issues/229470 Arnout Engelen noticed that
'make install -j' sometimes installs incomplete '$prefix/share/info/dir' file.

This happens because 'make install-info' when running in parallel is executing
equivalent of:

if /<<NIX>>/bash-5.2-p15/bin/bash -c 'install-info --version' >/dev/null 2>&1;
then \
    install-info
--dir-file=/<<NIX>>/x86_64-unknown-linux-musl-stage-static-gcc-14.0.0/share/info/dir
/<<NIX>>/x86_64-unknown-linux-musl-stage-static-gcc-14.0.0/share/info/gcc.info;
\
if /<<NIX>>/bash-5.2-p15/bin/bash -c 'install-info --version' >/dev/null 2>&1;
then \
    install-info
--dir-file=/<<NIX>>/x86_64-unknown-linux-musl-stage-static-gcc-14.0.0/share/info/dir
/<<NIX>>/x86_64-unknown-linux-musl-stage-static-gcc-14.0.0/share/info/gccinstall.info;
\
if /<<NIX>>/bash-5.2-p15/bin/bash -c 'install-info --version' >/dev/null 2>&1;
then \
    install-info
--dir-file=/<<NIX>>/x86_64-unknown-linux-musl-stage-static-gcc-14.0.0/share/info/dir
/<<NIX>>/x86_64-unknown-linux-musl-stage-static-gcc-14.0.0/share/info/cppinternals.info;
\
if /<<NIX>>/bash-5.2-p15/bin/bash -c 'install-info --version' >/dev/null 2>&1;
then \
    install-info
--dir-file=/<<NIX>>/x86_64-unknown-linux-musl-stage-static-gcc-14.0.0/share/info/dir
/<<NIX>>/x86_64-unknown-linux-musl-stage-static-gcc-14.0.0/share/info/cpp.info;
\
if /<<NIX>>/bash-5.2-p15/bin/bash -c 'install-info --version' >/dev/null 2>&1;
then \
    install-info
--dir-file=/<<NIX>>/x86_64-unknown-linux-musl-stage-static-gcc-14.0.0/share/info/dir
/<<NIX>>/x86_64-unknown-linux-musl-stage-static-gcc-14.0.0/share/info/gccint.info;
\

On it's own 'install-info' does not lock $prefix/share/info/dir file
atomically. As a result multiple parallel executions of 'install-info' compete
and occasionally throw away work of one another.

Right now `nixpkgs` works it around as
https://github.com/NixOS/nixpkgs/pull/229898:

--- gcc-12.2.0/gcc/Makefile.in  2022-08-19 10:09:52.280658631 +0200
+++ gcc-12.2.0-new/gcc/Makefile.in      2023-05-04 14:35:44.401420184 +0200
@@ -3781,6 +3781,11 @@
          fi; \
        fi

+# We don't care about the order in which the info files are built, but
+# install-info doesn't support multiple parallel invocations writing to
+# the same `dir-file`, so we have to disable parallelism for that reason:
+.NOTPARALLEL: install-info
+
 # Install the info files.
 # $(INSTALL_DATA) might be a relative pathname, so we can't cd into srcdir
 # to do the install.

It might not be enough to handle language-specific 'install-info' targets.

What would be the best way to handle this race condition?

[Bug other/109898] 'make install -j' sometimes corrupts 'dir' file for .info files due to parallel 'install-info' calls

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109898

--- Comment #1 from Richard Biener <rguenth at gcc dot gnu.org> ---
make all of 'install' not parallel?

[Bug other/109898] 'make install -j' sometimes corrupts 'dir' file for .info files due to parallel 'install-info' calls

[redi at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109898

--- Comment #2 from Jonathan Wakely <redi at gcc dot gnu.org> ---
(In reply to Sergei Trofimovich from comment #0)
> --- gcc-12.2.0/gcc/Makefile.in	2022-08-19 10:09:52.280658631 +0200
> +++ gcc-12.2.0-new/gcc/Makefile.in	2023-05-04 14:35:44.401420184 +0200
> @@ -3781,6 +3781,11 @@
>  	  fi; \
>  	fi
>  
> +# We don't care about the order in which the info files are built, but
> +# install-info doesn't support multiple parallel invocations writing to
> +# the same `dir-file`, so we have to disable parallelism for that reason:
> +.NOTPARALLEL: install-info

Prerequisites of .NOTPARALLEL are ignored, so doesn't this un-parallelize
building the entire gcc sub-dir?

[Bug other/109898] 'make install -j' sometimes corrupts 'dir' file for .info files due to parallel 'install-info' calls

[slyfox at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109898

--- Comment #3 from Sergei Trofimovich <slyfox at gcc dot gnu.org> ---
(In reply to Jonathan Wakely from comment #2)
> (In reply to Sergei Trofimovich from comment #0)
> > --- gcc-12.2.0/gcc/Makefile.in	2022-08-19 10:09:52.280658631 +0200
> > +++ gcc-12.2.0-new/gcc/Makefile.in	2023-05-04 14:35:44.401420184 +0200
> > @@ -3781,6 +3781,11 @@
> >  	  fi; \
> >  	fi
> >  
> > +# We don't care about the order in which the info files are built, but
> > +# install-info doesn't support multiple parallel invocations writing to
> > +# the same `dir-file`, so we have to disable parallelism for that reason:
> > +.NOTPARALLEL: install-info
> 
> Prerequisites of .NOTPARALLEL are ignored, so doesn't this un-parallelize
> building the entire gcc sub-dir?

When I tested on small example '.NOTPARALLEL: target' serialized exactly
prerequisites of 'target' and nothing more. 'info make' seems to confirm the
behaviour:

"""
'.NOTPARALLEL'

     If '.NOTPARALLEL' is mentioned as a target with no prerequisites,
     all targets in this invocation of 'make' will be run serially, even
     if the '-j' option is given.  Any recursively invoked 'make'
     command will still run recipes in parallel (unless its makefile
     also contains this target).

     If '.NOTPARALLEL' has targets as prerequisites, then all the
     prerequisites of those targets will be run serially.  This
     implicitly adds a '.WAIT' between each prerequisite of the listed
     targets.  *Note Disabling Parallel Execution: Parallel Disable.
"""

Here is the test that confirms it:

$ cat Makefile
all: a b

a: 1 2
        echo a

1:
        sleep 1

2:
        sleep 1

b: 3 4
        echo b

3:
        sleep 1

4:
        sleep 1

.NOTPARALLEL: a

$ make -j | LANG=C ts
May 18 20:34:23 sleep 1
May 18 20:34:23 sleep 1
May 18 20:34:23 sleep 1
May 18 20:34:24 sleep 1
May 18 20:34:24 echo b
May 18 20:34:24 b
May 18 20:34:25 echo a
May 18 20:34:25 a

Note how it takes 'b' 1 second to finish (due to parallelism) while 'a' takes 2
seconds (targets are sequential).

[Bug other/109898] 'make install -j' sometimes corrupts 'dir' file for .info files due to parallel 'install-info' calls

[redi at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109898

--- Comment #4 from Jonathan Wakely <redi at gcc dot gnu.org> ---
That seems to be new in Make 4.4, because the manual for Make 4.3 says:

'.NOTPARALLEL'

     If '.NOTPARALLEL' is mentioned as a target, then this invocation of
     'make' will be run serially, even if the '-j' option is given.  Any
     recursively invoked 'make' command will still run recipes in
     parallel (unless its makefile also contains this target).  Any
     prerequisites on this target are ignored.

[Bug other/109898] 'make install -j' sometimes corrupts 'dir' file for .info files due to parallel 'install-info' calls

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109898

Jakub Jelinek <jakub at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jakub at gcc dot gnu.org

--- Comment #5 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
But perhaps make install could $(MAKE) NOT_PARALLEL=1 non-parallel-install
and the makefile could have
ifeq ($(NOT_PARALLEL),1)
.NOTPARALLEL:
endif
?