public inbox for gcc-bugs@sourceware.org
help / color / mirror / Atom feed
* [Bug other/109898] New: 'make install -j' sometimes corrupts 'dir' file for .info files due to parallel 'install-info' calls
@ 2023-05-17 20:40 slyfox at gcc dot gnu.org
2023-05-18 6:23 ` [Bug other/109898] " rguenth at gcc dot gnu.org
` (4 more replies)
0 siblings, 5 replies; 6+ messages in thread
From: slyfox at gcc dot gnu.org @ 2023-05-17 20:40 UTC (permalink / raw)
To: gcc-bugs
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109898
Bug ID: 109898
Summary: 'make install -j' sometimes corrupts 'dir' file for
.info files due to parallel 'install-info' calls
Product: gcc
Version: 14.0
Status: UNCONFIRMED
Severity: normal
Priority: P3
Component: other
Assignee: unassigned at gcc dot gnu.org
Reporter: slyfox at gcc dot gnu.org
Target Milestone: ---
In https://github.com/NixOS/nixpkgs/issues/229470 Arnout Engelen noticed that
'make install -j' sometimes installs incomplete '$prefix/share/info/dir' file.
This happens because 'make install-info' when running in parallel is executing
equivalent of:
if /<<NIX>>/bash-5.2-p15/bin/bash -c 'install-info --version' >/dev/null 2>&1;
then \
install-info
--dir-file=/<<NIX>>/x86_64-unknown-linux-musl-stage-static-gcc-14.0.0/share/info/dir
/<<NIX>>/x86_64-unknown-linux-musl-stage-static-gcc-14.0.0/share/info/gcc.info;
\
if /<<NIX>>/bash-5.2-p15/bin/bash -c 'install-info --version' >/dev/null 2>&1;
then \
install-info
--dir-file=/<<NIX>>/x86_64-unknown-linux-musl-stage-static-gcc-14.0.0/share/info/dir
/<<NIX>>/x86_64-unknown-linux-musl-stage-static-gcc-14.0.0/share/info/gccinstall.info;
\
if /<<NIX>>/bash-5.2-p15/bin/bash -c 'install-info --version' >/dev/null 2>&1;
then \
install-info
--dir-file=/<<NIX>>/x86_64-unknown-linux-musl-stage-static-gcc-14.0.0/share/info/dir
/<<NIX>>/x86_64-unknown-linux-musl-stage-static-gcc-14.0.0/share/info/cppinternals.info;
\
if /<<NIX>>/bash-5.2-p15/bin/bash -c 'install-info --version' >/dev/null 2>&1;
then \
install-info
--dir-file=/<<NIX>>/x86_64-unknown-linux-musl-stage-static-gcc-14.0.0/share/info/dir
/<<NIX>>/x86_64-unknown-linux-musl-stage-static-gcc-14.0.0/share/info/cpp.info;
\
if /<<NIX>>/bash-5.2-p15/bin/bash -c 'install-info --version' >/dev/null 2>&1;
then \
install-info
--dir-file=/<<NIX>>/x86_64-unknown-linux-musl-stage-static-gcc-14.0.0/share/info/dir
/<<NIX>>/x86_64-unknown-linux-musl-stage-static-gcc-14.0.0/share/info/gccint.info;
\
On it's own 'install-info' does not lock $prefix/share/info/dir file
atomically. As a result multiple parallel executions of 'install-info' compete
and occasionally throw away work of one another.
Right now `nixpkgs` works it around as
https://github.com/NixOS/nixpkgs/pull/229898:
--- gcc-12.2.0/gcc/Makefile.in 2022-08-19 10:09:52.280658631 +0200
+++ gcc-12.2.0-new/gcc/Makefile.in 2023-05-04 14:35:44.401420184 +0200
@@ -3781,6 +3781,11 @@
fi; \
fi
+# We don't care about the order in which the info files are built, but
+# install-info doesn't support multiple parallel invocations writing to
+# the same `dir-file`, so we have to disable parallelism for that reason:
+.NOTPARALLEL: install-info
+
# Install the info files.
# $(INSTALL_DATA) might be a relative pathname, so we can't cd into srcdir
# to do the install.
It might not be enough to handle language-specific 'install-info' targets.
What would be the best way to handle this race condition?
^ permalink raw reply [flat|nested] 6+ messages in thread* [Bug other/109898] 'make install -j' sometimes corrupts 'dir' file for .info files due to parallel 'install-info' calls
2023-05-17 20:40 [Bug other/109898] New: 'make install -j' sometimes corrupts 'dir' file for .info files due to parallel 'install-info' calls slyfox at gcc dot gnu.org
@ 2023-05-18 6:23 ` rguenth at gcc dot gnu.org
2023-05-18 8:36 ` redi at gcc dot gnu.org
` (3 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: rguenth at gcc dot gnu.org @ 2023-05-18 6:23 UTC (permalink / raw)
To: gcc-bugs
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109898
--- Comment #1 from Richard Biener <rguenth at gcc dot gnu.org> ---
make all of 'install' not parallel?
^ permalink raw reply [flat|nested] 6+ messages in thread* [Bug other/109898] 'make install -j' sometimes corrupts 'dir' file for .info files due to parallel 'install-info' calls
2023-05-17 20:40 [Bug other/109898] New: 'make install -j' sometimes corrupts 'dir' file for .info files due to parallel 'install-info' calls slyfox at gcc dot gnu.org
2023-05-18 6:23 ` [Bug other/109898] " rguenth at gcc dot gnu.org
@ 2023-05-18 8:36 ` redi at gcc dot gnu.org
2023-05-18 19:35 ` slyfox at gcc dot gnu.org
` (2 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: redi at gcc dot gnu.org @ 2023-05-18 8:36 UTC (permalink / raw)
To: gcc-bugs
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109898
--- Comment #2 from Jonathan Wakely <redi at gcc dot gnu.org> ---
(In reply to Sergei Trofimovich from comment #0)
> --- gcc-12.2.0/gcc/Makefile.in 2022-08-19 10:09:52.280658631 +0200
> +++ gcc-12.2.0-new/gcc/Makefile.in 2023-05-04 14:35:44.401420184 +0200
> @@ -3781,6 +3781,11 @@
> fi; \
> fi
>
> +# We don't care about the order in which the info files are built, but
> +# install-info doesn't support multiple parallel invocations writing to
> +# the same `dir-file`, so we have to disable parallelism for that reason:
> +.NOTPARALLEL: install-info
Prerequisites of .NOTPARALLEL are ignored, so doesn't this un-parallelize
building the entire gcc sub-dir?
^ permalink raw reply [flat|nested] 6+ messages in thread* [Bug other/109898] 'make install -j' sometimes corrupts 'dir' file for .info files due to parallel 'install-info' calls
2023-05-17 20:40 [Bug other/109898] New: 'make install -j' sometimes corrupts 'dir' file for .info files due to parallel 'install-info' calls slyfox at gcc dot gnu.org
2023-05-18 6:23 ` [Bug other/109898] " rguenth at gcc dot gnu.org
2023-05-18 8:36 ` redi at gcc dot gnu.org
@ 2023-05-18 19:35 ` slyfox at gcc dot gnu.org
2023-05-19 9:29 ` redi at gcc dot gnu.org
2023-05-19 9:52 ` jakub at gcc dot gnu.org
4 siblings, 0 replies; 6+ messages in thread
From: slyfox at gcc dot gnu.org @ 2023-05-18 19:35 UTC (permalink / raw)
To: gcc-bugs
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109898
--- Comment #3 from Sergei Trofimovich <slyfox at gcc dot gnu.org> ---
(In reply to Jonathan Wakely from comment #2)
> (In reply to Sergei Trofimovich from comment #0)
> > --- gcc-12.2.0/gcc/Makefile.in 2022-08-19 10:09:52.280658631 +0200
> > +++ gcc-12.2.0-new/gcc/Makefile.in 2023-05-04 14:35:44.401420184 +0200
> > @@ -3781,6 +3781,11 @@
> > fi; \
> > fi
> >
> > +# We don't care about the order in which the info files are built, but
> > +# install-info doesn't support multiple parallel invocations writing to
> > +# the same `dir-file`, so we have to disable parallelism for that reason:
> > +.NOTPARALLEL: install-info
>
> Prerequisites of .NOTPARALLEL are ignored, so doesn't this un-parallelize
> building the entire gcc sub-dir?
When I tested on small example '.NOTPARALLEL: target' serialized exactly
prerequisites of 'target' and nothing more. 'info make' seems to confirm the
behaviour:
"""
'.NOTPARALLEL'
If '.NOTPARALLEL' is mentioned as a target with no prerequisites,
all targets in this invocation of 'make' will be run serially, even
if the '-j' option is given. Any recursively invoked 'make'
command will still run recipes in parallel (unless its makefile
also contains this target).
If '.NOTPARALLEL' has targets as prerequisites, then all the
prerequisites of those targets will be run serially. This
implicitly adds a '.WAIT' between each prerequisite of the listed
targets. *Note Disabling Parallel Execution: Parallel Disable.
"""
Here is the test that confirms it:
$ cat Makefile
all: a b
a: 1 2
echo a
1:
sleep 1
2:
sleep 1
b: 3 4
echo b
3:
sleep 1
4:
sleep 1
.NOTPARALLEL: a
$ make -j | LANG=C ts
May 18 20:34:23 sleep 1
May 18 20:34:23 sleep 1
May 18 20:34:23 sleep 1
May 18 20:34:24 sleep 1
May 18 20:34:24 echo b
May 18 20:34:24 b
May 18 20:34:25 echo a
May 18 20:34:25 a
Note how it takes 'b' 1 second to finish (due to parallelism) while 'a' takes 2
seconds (targets are sequential).
^ permalink raw reply [flat|nested] 6+ messages in thread* [Bug other/109898] 'make install -j' sometimes corrupts 'dir' file for .info files due to parallel 'install-info' calls
2023-05-17 20:40 [Bug other/109898] New: 'make install -j' sometimes corrupts 'dir' file for .info files due to parallel 'install-info' calls slyfox at gcc dot gnu.org
` (2 preceding siblings ...)
2023-05-18 19:35 ` slyfox at gcc dot gnu.org
@ 2023-05-19 9:29 ` redi at gcc dot gnu.org
2023-05-19 9:52 ` jakub at gcc dot gnu.org
4 siblings, 0 replies; 6+ messages in thread
From: redi at gcc dot gnu.org @ 2023-05-19 9:29 UTC (permalink / raw)
To: gcc-bugs
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109898
--- Comment #4 from Jonathan Wakely <redi at gcc dot gnu.org> ---
That seems to be new in Make 4.4, because the manual for Make 4.3 says:
'.NOTPARALLEL'
If '.NOTPARALLEL' is mentioned as a target, then this invocation of
'make' will be run serially, even if the '-j' option is given. Any
recursively invoked 'make' command will still run recipes in
parallel (unless its makefile also contains this target). Any
prerequisites on this target are ignored.
^ permalink raw reply [flat|nested] 6+ messages in thread* [Bug other/109898] 'make install -j' sometimes corrupts 'dir' file for .info files due to parallel 'install-info' calls
2023-05-17 20:40 [Bug other/109898] New: 'make install -j' sometimes corrupts 'dir' file for .info files due to parallel 'install-info' calls slyfox at gcc dot gnu.org
` (3 preceding siblings ...)
2023-05-19 9:29 ` redi at gcc dot gnu.org
@ 2023-05-19 9:52 ` jakub at gcc dot gnu.org
4 siblings, 0 replies; 6+ messages in thread
From: jakub at gcc dot gnu.org @ 2023-05-19 9:52 UTC (permalink / raw)
To: gcc-bugs
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109898
Jakub Jelinek <jakub at gcc dot gnu.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |jakub at gcc dot gnu.org
--- Comment #5 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
But perhaps make install could $(MAKE) NOT_PARALLEL=1 non-parallel-install
and the makefile could have
ifeq ($(NOT_PARALLEL),1)
.NOTPARALLEL:
endif
?
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2023-05-19 9:52 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-05-17 20:40 [Bug other/109898] New: 'make install -j' sometimes corrupts 'dir' file for .info files due to parallel 'install-info' calls slyfox at gcc dot gnu.org
2023-05-18 6:23 ` [Bug other/109898] " rguenth at gcc dot gnu.org
2023-05-18 8:36 ` redi at gcc dot gnu.org
2023-05-18 19:35 ` slyfox at gcc dot gnu.org
2023-05-19 9:29 ` redi at gcc dot gnu.org
2023-05-19 9:52 ` jakub at gcc dot gnu.org
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).