public inbox for gcc-bugs@sourceware.org
help / color / mirror / Atom feed
* [Bug other/106328] New: Build doesn't respect -j N flag
@ 2022-07-16 20:15 yuri at tsoft dot com
2022-07-17 19:47 ` [Bug lto/106328] " pinskia at gcc dot gnu.org
` (9 more replies)
0 siblings, 10 replies; 11+ messages in thread
From: yuri at tsoft dot com @ 2022-07-16 20:15 UTC (permalink / raw)
To: gcc-bugs
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106328
Bug ID: 106328
Summary: Build doesn't respect -j N flag
Product: gcc
Version: 11.3.0
Status: UNCONFIRMED
Severity: normal
Priority: P3
Component: other
Assignee: unassigned at gcc dot gnu.org
Reporter: yuri at tsoft dot com
Target Milestone: ---
-j8 is given to it, but it creates ~130 lto1 processes.
See this downstream issues for details:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=265254
^ permalink raw reply [flat|nested] 11+ messages in thread
* [Bug lto/106328] Build doesn't respect -j N flag
2022-07-16 20:15 [Bug other/106328] New: Build doesn't respect -j N flag yuri at tsoft dot com
@ 2022-07-17 19:47 ` pinskia at gcc dot gnu.org
2022-07-18 7:55 ` rguenth at gcc dot gnu.org
` (8 subsequent siblings)
9 siblings, 0 replies; 11+ messages in thread
From: pinskia at gcc dot gnu.org @ 2022-07-17 19:47 UTC (permalink / raw)
To: gcc-bugs
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106328
--- Comment #1 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
Hmm:
https://gcc.gnu.org/onlinedocs/gcc-10.4.0/gcc/Optimize-Options.html#index-flto
Maybe the automation detection of job server is not working on freebsd
correctly.
^ permalink raw reply [flat|nested] 11+ messages in thread
* [Bug lto/106328] Build doesn't respect -j N flag
2022-07-16 20:15 [Bug other/106328] New: Build doesn't respect -j N flag yuri at tsoft dot com
2022-07-17 19:47 ` [Bug lto/106328] " pinskia at gcc dot gnu.org
@ 2022-07-18 7:55 ` rguenth at gcc dot gnu.org
2022-07-18 7:59 ` rguenth at gcc dot gnu.org
` (7 subsequent siblings)
9 siblings, 0 replies; 11+ messages in thread
From: rguenth at gcc dot gnu.org @ 2022-07-18 7:55 UTC (permalink / raw)
To: gcc-bugs
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106328
Richard Biener <rguenth at gcc dot gnu.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|UNCONFIRMED |NEW
Keywords| |lto
Last reconfirmed| |2022-07-18
Ever confirmed|0 |1
--- Comment #2 from Richard Biener <rguenth at gcc dot gnu.org> ---
That's the WPA LTRANS file generation which does not use the jobserver but
parallelizes for I/O. There is --param lto-max-streaming-parallelism you
can use to limit things but it's default is 32 (but that would be per
actual parallel invoked LTO link step, so with -j8 it's 8 * 32 when there
are 8 parallel invoked link steps).
See gcc/lto/lto.{c,cc}:stream_out_partitions which indeed says
#ifdef HAVE_WORKING_FORK
...
/* Do not run more than LTO_PARALLELISM streamings
FIXME: we ignore limits on jobserver. */
if (lto_parallelism > 0 && nruns >= lto_parallelism)
{
wait_for_child ();
...
if (!cpid)
{
setproctitle ("lto1-wpa-streaming");
so "confirmed" - it doesn't honor the jobserver. Note without using
-flto=auto or -flto=jobserver it would be all serial, note the above
also does not honor a limit placed via -flto=8 I think.
^ permalink raw reply [flat|nested] 11+ messages in thread
* [Bug lto/106328] Build doesn't respect -j N flag
2022-07-16 20:15 [Bug other/106328] New: Build doesn't respect -j N flag yuri at tsoft dot com
2022-07-17 19:47 ` [Bug lto/106328] " pinskia at gcc dot gnu.org
2022-07-18 7:55 ` rguenth at gcc dot gnu.org
@ 2022-07-18 7:59 ` rguenth at gcc dot gnu.org
2022-07-18 8:03 ` rguenth at gcc dot gnu.org
` (6 subsequent siblings)
9 siblings, 0 replies; 11+ messages in thread
From: rguenth at gcc dot gnu.org @ 2022-07-18 7:59 UTC (permalink / raw)
To: gcc-bugs
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106328
--- Comment #3 from Richard Biener <rguenth at gcc dot gnu.org> ---
Just to clarify, GCCs WPA stage fork()s to write out LTRANS IL object files in
parallel - those processes are not controlled by make but GCC could request
tokens from makes jobserver if it got a handle on a connection (which isn't
trivial because make does not provide the necessary open FDs to all sibling
processes it creates).
^ permalink raw reply [flat|nested] 11+ messages in thread
* [Bug lto/106328] Build doesn't respect -j N flag
2022-07-16 20:15 [Bug other/106328] New: Build doesn't respect -j N flag yuri at tsoft dot com
` (2 preceding siblings ...)
2022-07-18 7:59 ` rguenth at gcc dot gnu.org
@ 2022-07-18 8:03 ` rguenth at gcc dot gnu.org
2022-08-04 13:13 ` marxin at gcc dot gnu.org
` (5 subsequent siblings)
9 siblings, 0 replies; 11+ messages in thread
From: rguenth at gcc dot gnu.org @ 2022-07-18 8:03 UTC (permalink / raw)
To: gcc-bugs
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106328
--- Comment #4 from Richard Biener <rguenth at gcc dot gnu.org> ---
If it is possible to query the '-j=N' setting from make via the jobserver
connection then it might be practical to auto-tune
lto-max-streaming-parallelism
to that setting at least (that's still prone to N*N sub-processes).
^ permalink raw reply [flat|nested] 11+ messages in thread
* [Bug lto/106328] Build doesn't respect -j N flag
2022-07-16 20:15 [Bug other/106328] New: Build doesn't respect -j N flag yuri at tsoft dot com
` (3 preceding siblings ...)
2022-07-18 8:03 ` rguenth at gcc dot gnu.org
@ 2022-08-04 13:13 ` marxin at gcc dot gnu.org
2022-08-10 11:56 ` cvs-commit at gcc dot gnu.org
` (4 subsequent siblings)
9 siblings, 0 replies; 11+ messages in thread
From: marxin at gcc dot gnu.org @ 2022-08-04 13:13 UTC (permalink / raw)
To: gcc-bugs
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106328
Martin Liška <marxin at gcc dot gnu.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |ASSIGNED
Assignee|unassigned at gcc dot gnu.org |marxin at gcc dot gnu.org
--- Comment #5 from Martin Liška <marxin at gcc dot gnu.org> ---
I've got a patch candidate..
^ permalink raw reply [flat|nested] 11+ messages in thread
* [Bug lto/106328] Build doesn't respect -j N flag
2022-07-16 20:15 [Bug other/106328] New: Build doesn't respect -j N flag yuri at tsoft dot com
` (4 preceding siblings ...)
2022-08-04 13:13 ` marxin at gcc dot gnu.org
@ 2022-08-10 11:56 ` cvs-commit at gcc dot gnu.org
2022-08-10 11:56 ` marxin at gcc dot gnu.org
` (3 subsequent siblings)
9 siblings, 0 replies; 11+ messages in thread
From: cvs-commit at gcc dot gnu.org @ 2022-08-10 11:56 UTC (permalink / raw)
To: gcc-bugs
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106328
--- Comment #6 from CVS Commits <cvs-commit at gcc dot gnu.org> ---
The master branch has been updated by Martin Liska <marxin@gcc.gnu.org>:
https://gcc.gnu.org/g:fed766af32ed6cd371016cc24e931131e19b4eb1
commit r13-2012-gfed766af32ed6cd371016cc24e931131e19b4eb1
Author: Martin Liska <mliska@suse.cz>
Date: Tue Aug 9 13:59:39 2022 +0200
lto: respect jobserver in parallel WPA streaming
PR lto/106328
gcc/ChangeLog:
* opts-jobserver.h (struct jobserver_info): Add pipefd.
(jobserver_info::connect): New.
(jobserver_info::disconnect): Likewise.
(jobserver_info::get_token): Likewise.
(jobserver_info::return_token): Likewise.
* opts-common.cc: Implement the new functions.
gcc/lto/ChangeLog:
* lto.cc (wait_for_child): Decrement nruns once a process
finishes.
(stream_out_partitions): Use job server if active.
(do_whole_program_analysis): Likewise.
^ permalink raw reply [flat|nested] 11+ messages in thread
* [Bug lto/106328] Build doesn't respect -j N flag
2022-07-16 20:15 [Bug other/106328] New: Build doesn't respect -j N flag yuri at tsoft dot com
` (5 preceding siblings ...)
2022-08-10 11:56 ` cvs-commit at gcc dot gnu.org
@ 2022-08-10 11:56 ` marxin at gcc dot gnu.org
2022-08-10 12:40 ` rguenther at suse dot de
` (2 subsequent siblings)
9 siblings, 0 replies; 11+ messages in thread
From: marxin at gcc dot gnu.org @ 2022-08-10 11:56 UTC (permalink / raw)
To: gcc-bugs
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106328
Martin Liška <marxin at gcc dot gnu.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Resolution|--- |FIXED
Status|ASSIGNED |RESOLVED
--- Comment #7 from Martin Liška <marxin at gcc dot gnu.org> ---
Implemented for GCC 13.
^ permalink raw reply [flat|nested] 11+ messages in thread
* [Bug lto/106328] Build doesn't respect -j N flag
2022-07-16 20:15 [Bug other/106328] New: Build doesn't respect -j N flag yuri at tsoft dot com
` (6 preceding siblings ...)
2022-08-10 11:56 ` marxin at gcc dot gnu.org
@ 2022-08-10 12:40 ` rguenther at suse dot de
2022-08-10 12:46 ` marxin at gcc dot gnu.org
2022-08-10 12:51 ` rguenther at suse dot de
9 siblings, 0 replies; 11+ messages in thread
From: rguenther at suse dot de @ 2022-08-10 12:40 UTC (permalink / raw)
To: gcc-bugs
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106328
--- Comment #8 from rguenther at suse dot de <rguenther at suse dot de> ---
On Wed, 10 Aug 2022, marxin at gcc dot gnu.org wrote:
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106328
>
> Martin Liška <marxin at gcc dot gnu.org> changed:
>
> What |Removed |Added
> ----------------------------------------------------------------------------
> Resolution|--- |FIXED
> Status|ASSIGNED |RESOLVED
>
> --- Comment #7 from Martin Liška <marxin at gcc dot gnu.org> ---
> Implemented for GCC 13.
Magically only with recent GNU make, otherwise needs proper prefixed
rules in the lto-wrapper generated makefile which I don't think we do.
Without either it's now slow by default?
^ permalink raw reply [flat|nested] 11+ messages in thread
* [Bug lto/106328] Build doesn't respect -j N flag
2022-07-16 20:15 [Bug other/106328] New: Build doesn't respect -j N flag yuri at tsoft dot com
` (7 preceding siblings ...)
2022-08-10 12:40 ` rguenther at suse dot de
@ 2022-08-10 12:46 ` marxin at gcc dot gnu.org
2022-08-10 12:51 ` rguenther at suse dot de
9 siblings, 0 replies; 11+ messages in thread
From: marxin at gcc dot gnu.org @ 2022-08-10 12:46 UTC (permalink / raw)
To: gcc-bugs
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106328
--- Comment #9 from Martin Liška <marxin at gcc dot gnu.org> ---
> Magically only with recent GNU make, otherwise needs proper prefixed
> rules in the lto-wrapper generated makefile which I don't think we do.
Wait, the cooperation works with older GNU make if a Makefile uses prefixed (+)
rules. WPA does not email any artificial Makefile for WPA streaming. It's a
Makefile we emit for LTRANS run, e.g.:
marxin@marxinbox:/dev/shm/objdir> cat /tmp/ccuhgkQs.mk
./a.ltrans0.ltrans.o:
@g++ '-xlto' '-c' '-fno-openmp' '-fno-openacc' '-fno-pie'
'-fcf-protection=none' '-mtune=generic' '-march=x86-64' '-O2' '-save-temps'
'-v' '-shared-libgcc' '-mtune=generic' '-march=x86-64' '-dumpdir' 'a.'
'-dumpbase' './a.ltrans0.ltrans' '-fltrans' '-o' './a.ltrans0.ltrans.o'
'./a.ltrans0.o'
./a.ltrans1.ltrans.o:
@g++ '-xlto' '-c' '-fno-openmp' '-fno-openacc' '-fno-pie'
'-fcf-protection=none' '-mtune=generic' '-march=x86-64' '-O2' '-save-temps'
'-v' '-shared-libgcc' '-mtune=generic' '-march=x86-64' '-dumpdir' 'a.'
'-dumpbase' './a.ltrans1.ltrans' '-fltrans' '-o' './a.ltrans1.ltrans.o'
'./a.ltrans1.o'
...
So what can be miss is jobserver detection on BSD that can fail for some
reason, but it should work fine apart from that. Or do I miss something?
^ permalink raw reply [flat|nested] 11+ messages in thread
* [Bug lto/106328] Build doesn't respect -j N flag
2022-07-16 20:15 [Bug other/106328] New: Build doesn't respect -j N flag yuri at tsoft dot com
` (8 preceding siblings ...)
2022-08-10 12:46 ` marxin at gcc dot gnu.org
@ 2022-08-10 12:51 ` rguenther at suse dot de
9 siblings, 0 replies; 11+ messages in thread
From: rguenther at suse dot de @ 2022-08-10 12:51 UTC (permalink / raw)
To: gcc-bugs
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106328
--- Comment #10 from rguenther at suse dot de <rguenther at suse dot de> ---
On Wed, 10 Aug 2022, marxin at gcc dot gnu.org wrote:
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106328
>
> --- Comment #9 from Martin Liška <marxin at gcc dot gnu.org> ---
> > Magically only with recent GNU make, otherwise needs proper prefixed
> > rules in the lto-wrapper generated makefile which I don't think we do.
>
> Wait, the cooperation works with older GNU make if a Makefile uses prefixed (+)
> rules. WPA does not email any artificial Makefile for WPA streaming. It's a
> Makefile we emit for LTRANS run, e.g.:
>
> marxin@marxinbox:/dev/shm/objdir> cat /tmp/ccuhgkQs.mk
> ./a.ltrans0.ltrans.o:
> @g++ '-xlto' '-c' '-fno-openmp' '-fno-openacc' '-fno-pie'
> '-fcf-protection=none' '-mtune=generic' '-march=x86-64' '-O2' '-save-temps'
> '-v' '-shared-libgcc' '-mtune=generic' '-march=x86-64' '-dumpdir' 'a.'
> '-dumpbase' './a.ltrans0.ltrans' '-fltrans' '-o' './a.ltrans0.ltrans.o'
> './a.ltrans0.o'
> ./a.ltrans1.ltrans.o:
> @g++ '-xlto' '-c' '-fno-openmp' '-fno-openacc' '-fno-pie'
> '-fcf-protection=none' '-mtune=generic' '-march=x86-64' '-O2' '-save-temps'
> '-v' '-shared-libgcc' '-mtune=generic' '-march=x86-64' '-dumpdir' 'a.'
> '-dumpbase' './a.ltrans1.ltrans' '-fltrans' '-o' './a.ltrans1.ltrans.o'
> './a.ltrans1.o'
> ...
>
> So what can be miss is jobserver detection on BSD that can fail for some
> reason, but it should work fine apart from that. Or do I miss something?
Ah, indeed. That still leaves the question whether we execute the
WPA stage with the FDs open - I suppose you checked? And whether
pex_* "properly" does this for all host OSs (how does make jobserver
work on mingw/cygwin?). I wonder because Honza once said he didn't
implement jobserver support because it would require more fiddling
to get it actually work.
And IIRC BSD 'make' is not GNU make but I think gmake is available
from the ports repo. The documentation about -flto=jobserver mentions
that already.
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2022-08-10 12:51 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-07-16 20:15 [Bug other/106328] New: Build doesn't respect -j N flag yuri at tsoft dot com
2022-07-17 19:47 ` [Bug lto/106328] " pinskia at gcc dot gnu.org
2022-07-18 7:55 ` rguenth at gcc dot gnu.org
2022-07-18 7:59 ` rguenth at gcc dot gnu.org
2022-07-18 8:03 ` rguenth at gcc dot gnu.org
2022-08-04 13:13 ` marxin at gcc dot gnu.org
2022-08-10 11:56 ` cvs-commit at gcc dot gnu.org
2022-08-10 11:56 ` marxin at gcc dot gnu.org
2022-08-10 12:40 ` rguenther at suse dot de
2022-08-10 12:46 ` marxin at gcc dot gnu.org
2022-08-10 12:51 ` rguenther at suse dot de
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).