* [PATCH] LTO: fallback to -flto=N if -flto=jobserver does not work. @ 2021-04-21 11:50 Martin Liška 2021-04-21 15:11 ` Jeff Law 2021-04-22 8:04 ` Richard Biener 0 siblings, 2 replies; 10+ messages in thread From: Martin Liška @ 2021-04-21 11:50 UTC (permalink / raw) To: gcc-patches When -flto=jobserver is used and we cannot detect job server, then we can still fallbackto -flto=N mode. Patch can bootstrap on x86_64-linux-gnu and survives regression tests. Ready to be installed? Thanks, Martin gcc/ChangeLog: * lto-wrapper.c (run_gcc): When -flto=jobserver is used, but the makeserver cannot be detected, then use -flto=N fallback. --- gcc/lto-wrapper.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/gcc/lto-wrapper.c b/gcc/lto-wrapper.c index 03a5922f8ea..0b626d7c811 100644 --- a/gcc/lto-wrapper.c +++ b/gcc/lto-wrapper.c @@ -1585,8 +1585,9 @@ run_gcc (unsigned argc, char *argv[]) if (jobserver && jobserver_error != NULL) { warning (0, jobserver_error); - parallel = 0; + /* Fall back to auto parallelism. */ jobserver = 0; + auto_parallel = 1; } else if (!jobserver && jobserver_error == NULL) { -- 2.31.1 ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH] LTO: fallback to -flto=N if -flto=jobserver does not work. 2021-04-21 11:50 [PATCH] LTO: fallback to -flto=N if -flto=jobserver does not work Martin Liška @ 2021-04-21 15:11 ` Jeff Law 2021-04-22 8:04 ` Richard Biener 1 sibling, 0 replies; 10+ messages in thread From: Jeff Law @ 2021-04-21 15:11 UTC (permalink / raw) To: Martin Liška, gcc-patches On 4/21/2021 5:50 AM, Martin Liška wrote: > When -flto=jobserver is used and we cannot detect job server, then we can > still fallbackto -flto=N mode. > > Patch can bootstrap on x86_64-linux-gnu and survives regression tests. > > Ready to be installed? > Thanks, > Martin > > gcc/ChangeLog: > > * lto-wrapper.c (run_gcc): When -flto=jobserver is used, but the > makeserver cannot be detected, then use -flto=N fallback. OK jeff ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH] LTO: fallback to -flto=N if -flto=jobserver does not work. 2021-04-21 11:50 [PATCH] LTO: fallback to -flto=N if -flto=jobserver does not work Martin Liška 2021-04-21 15:11 ` Jeff Law @ 2021-04-22 8:04 ` Richard Biener 2021-04-22 9:02 ` Martin Liška 1 sibling, 1 reply; 10+ messages in thread From: Richard Biener @ 2021-04-22 8:04 UTC (permalink / raw) To: Martin Liška; +Cc: GCC Patches On Wed, Apr 21, 2021 at 3:08 PM Martin Liška <mliska@suse.cz> wrote: > > When -flto=jobserver is used and we cannot detect job server, then we can > still fallbackto -flto=N mode. > > Patch can bootstrap on x86_64-linux-gnu and survives regression tests. > > Ready to be installed? I think this behavior needs to be documented - it falls back to a less conservative (possibly system overloading) mode - which IMHO is non-obvious and IMHO we shouldn't do. Richard. > Thanks, > Martin > > gcc/ChangeLog: > > * lto-wrapper.c (run_gcc): When -flto=jobserver is used, but the > makeserver cannot be detected, then use -flto=N fallback. > --- > gcc/lto-wrapper.c | 3 ++- > 1 file changed, 2 insertions(+), 1 deletion(-) > > diff --git a/gcc/lto-wrapper.c b/gcc/lto-wrapper.c > index 03a5922f8ea..0b626d7c811 100644 > --- a/gcc/lto-wrapper.c > +++ b/gcc/lto-wrapper.c > @@ -1585,8 +1585,9 @@ run_gcc (unsigned argc, char *argv[]) > if (jobserver && jobserver_error != NULL) > { > warning (0, jobserver_error); > - parallel = 0; > + /* Fall back to auto parallelism. */ > jobserver = 0; > + auto_parallel = 1; > } > else if (!jobserver && jobserver_error == NULL) > { > -- > 2.31.1 > ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH] LTO: fallback to -flto=N if -flto=jobserver does not work. 2021-04-22 8:04 ` Richard Biener @ 2021-04-22 9:02 ` Martin Liška 2021-04-22 11:19 ` Richard Biener 0 siblings, 1 reply; 10+ messages in thread From: Martin Liška @ 2021-04-22 9:02 UTC (permalink / raw) To: Richard Biener; +Cc: GCC Patches [-- Attachment #1: Type: text/plain, Size: 1535 bytes --] On 4/22/21 10:04 AM, Richard Biener wrote: > On Wed, Apr 21, 2021 at 3:08 PM Martin Liška <mliska@suse.cz> wrote: >> >> When -flto=jobserver is used and we cannot detect job server, then we can >> still fallbackto -flto=N mode. >> >> Patch can bootstrap on x86_64-linux-gnu and survives regression tests. >> >> Ready to be installed? > > I think this behavior needs to be documented - it falls back to a less > conservative (possibly system overloading) mode - which IMHO is > non-obvious and IMHO we shouldn't do. Sure, I'm sending corresponding patch. Note that it's quite common mistake that '+' is missing in Makefile rule. That was motivation for my change. Martin > > Richard. > >> Thanks, >> Martin >> >> gcc/ChangeLog: >> >> * lto-wrapper.c (run_gcc): When -flto=jobserver is used, but the >> makeserver cannot be detected, then use -flto=N fallback. >> --- >> gcc/lto-wrapper.c | 3 ++- >> 1 file changed, 2 insertions(+), 1 deletion(-) >> >> diff --git a/gcc/lto-wrapper.c b/gcc/lto-wrapper.c >> index 03a5922f8ea..0b626d7c811 100644 >> --- a/gcc/lto-wrapper.c >> +++ b/gcc/lto-wrapper.c >> @@ -1585,8 +1585,9 @@ run_gcc (unsigned argc, char *argv[]) >> if (jobserver && jobserver_error != NULL) >> { >> warning (0, jobserver_error); >> - parallel = 0; >> + /* Fall back to auto parallelism. */ >> jobserver = 0; >> + auto_parallel = 1; >> } >> else if (!jobserver && jobserver_error == NULL) >> { >> -- >> 2.31.1 >> [-- Attachment #2: 0001-Document-flto-jobserver-fallback.patch --] [-- Type: text/x-patch, Size: 1017 bytes --] From b81f5288fc81256e63d44e00793e2732eb2a2d88 Mon Sep 17 00:00:00 2001 From: Martin Liska <mliska@suse.cz> Date: Thu, 22 Apr 2021 10:59:51 +0200 Subject: [PATCH] Document -flto=jobserver fallback. gcc/ChangeLog: * doc/invoke.texi: Document what happens when GCC can't detect make's job server with -flto=jobserver. --- gcc/doc/invoke.texi | 3 +++ 1 file changed, 3 insertions(+) diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi index e98b0962b9f..1cfff4c14a9 100644 --- a/gcc/doc/invoke.texi +++ b/gcc/doc/invoke.texi @@ -12340,6 +12340,9 @@ Use @option{-flto=auto} to use GNU make's job server, if available, or otherwise fall back to autodetection of the number of CPU threads present in your system. +If you specify @option{-flto=jobserver} and make's job server can't be detected, +then well fall back to the number of CPU threads present in your system. + @item -flto-partition=@var{alg} @opindex flto-partition Specify the partitioning algorithm used by the link-time optimizer. -- 2.31.1 ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH] LTO: fallback to -flto=N if -flto=jobserver does not work. 2021-04-22 9:02 ` Martin Liška @ 2021-04-22 11:19 ` Richard Biener 2021-04-22 12:21 ` Martin Liška 0 siblings, 1 reply; 10+ messages in thread From: Richard Biener @ 2021-04-22 11:19 UTC (permalink / raw) To: Martin Liška; +Cc: GCC Patches On Thu, Apr 22, 2021 at 11:02 AM Martin Liška <mliska@suse.cz> wrote: > > On 4/22/21 10:04 AM, Richard Biener wrote: > > On Wed, Apr 21, 2021 at 3:08 PM Martin Liška <mliska@suse.cz> wrote: > >> > >> When -flto=jobserver is used and we cannot detect job server, then we can > >> still fallbackto -flto=N mode. > >> > >> Patch can bootstrap on x86_64-linux-gnu and survives regression tests. > >> > >> Ready to be installed? > > > > I think this behavior needs to be documented - it falls back to a less > > conservative (possibly system overloading) mode - which IMHO is > > non-obvious and IMHO we shouldn't do. > > Sure, I'm sending corresponding patch. Note that it's quite common mistake > that '+' is missing in Makefile rule. That was motivation for my change. Sure, but that change won't get this fixed. IMHO we should eventually emit diagnostic like warning: could not find jobserver, compiling N jobs serially once N > 1 (or 2?). Likewise if people just use -flto and auto-detection finds nothing: warning: using serial compilation of N LTRANS jobs note: refer to http://.... for how to use parallel compile using the URL diagnostics to point to -flto=... documentation. That is, teach users rather than second-guessing and eventually blowing things up. IMHO only the jobserver mode is safe to automatically use. Richard. > Martin > > > > > Richard. > > > >> Thanks, > >> Martin > >> > >> gcc/ChangeLog: > >> > >> * lto-wrapper.c (run_gcc): When -flto=jobserver is used, but the > >> makeserver cannot be detected, then use -flto=N fallback. > >> --- > >> gcc/lto-wrapper.c | 3 ++- > >> 1 file changed, 2 insertions(+), 1 deletion(-) > >> > >> diff --git a/gcc/lto-wrapper.c b/gcc/lto-wrapper.c > >> index 03a5922f8ea..0b626d7c811 100644 > >> --- a/gcc/lto-wrapper.c > >> +++ b/gcc/lto-wrapper.c > >> @@ -1585,8 +1585,9 @@ run_gcc (unsigned argc, char *argv[]) > >> if (jobserver && jobserver_error != NULL) > >> { > >> warning (0, jobserver_error); > >> - parallel = 0; > >> + /* Fall back to auto parallelism. */ > >> jobserver = 0; > >> + auto_parallel = 1; > >> } > >> else if (!jobserver && jobserver_error == NULL) > >> { > >> -- > >> 2.31.1 > >> > ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH] LTO: fallback to -flto=N if -flto=jobserver does not work. 2021-04-22 11:19 ` Richard Biener @ 2021-04-22 12:21 ` Martin Liška 2021-04-22 12:47 ` Richard Biener 0 siblings, 1 reply; 10+ messages in thread From: Martin Liška @ 2021-04-22 12:21 UTC (permalink / raw) To: Richard Biener; +Cc: GCC Patches On 4/22/21 1:19 PM, Richard Biener wrote: > On Thu, Apr 22, 2021 at 11:02 AM Martin Liška <mliska@suse.cz> wrote: >> >> On 4/22/21 10:04 AM, Richard Biener wrote: >>> On Wed, Apr 21, 2021 at 3:08 PM Martin Liška <mliska@suse.cz> wrote: >>>> >>>> When -flto=jobserver is used and we cannot detect job server, then we can >>>> still fallbackto -flto=N mode. >>>> >>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests. >>>> >>>> Ready to be installed? >>> >>> I think this behavior needs to be documented - it falls back to a less >>> conservative (possibly system overloading) mode - which IMHO is >>> non-obvious and IMHO we shouldn't do. >> >> Sure, I'm sending corresponding patch. Note that it's quite common mistake >> that '+' is missing in Makefile rule. That was motivation for my change. > > Sure, but that change won't get this fixed. It will as linker command line will contain (-flto=jobserver) and LTO will fallback to -flto=N. > IMHO we should eventually > emit diagnostic like > > warning: could not find jobserver, compiling N jobs serially > > once N > 1 (or 2?). We do that now (for all N): lto-wrapper: warning: jobserver is not available: ‘MAKEFLAGS’ environment variable is unset > Likewise if people just use -flto and auto-detection > finds nothing: -flto != -flto=auto Yes, -flto is a serial linking and we can emit a warning. > > warning: using serial compilation of N LTRANS jobs > note: refer to http://.... for how to use parallel compile > > using the URL diagnostics to point to -flto=... documentation. What about making that a proper warning (-Wlto)? We have diagnostics infrastructure that prints URL links. > > That is, teach users rather than second-guessing and eventually > blowing things up. IMHO only the jobserver mode is safe to > automatically use. Well, -flto=auto is also fine and document. I think there is no possibility auto CPU deduction can fail. So -flto=jobserver (with missing make job server) and -flto (equal to -flto=1) worth emitting a warning. What do you think? Martin > > Richard. > >> Martin >> >>> >>> Richard. >>> >>>> Thanks, >>>> Martin >>>> >>>> gcc/ChangeLog: >>>> >>>> * lto-wrapper.c (run_gcc): When -flto=jobserver is used, but the >>>> makeserver cannot be detected, then use -flto=N fallback. >>>> --- >>>> gcc/lto-wrapper.c | 3 ++- >>>> 1 file changed, 2 insertions(+), 1 deletion(-) >>>> >>>> diff --git a/gcc/lto-wrapper.c b/gcc/lto-wrapper.c >>>> index 03a5922f8ea..0b626d7c811 100644 >>>> --- a/gcc/lto-wrapper.c >>>> +++ b/gcc/lto-wrapper.c >>>> @@ -1585,8 +1585,9 @@ run_gcc (unsigned argc, char *argv[]) >>>> if (jobserver && jobserver_error != NULL) >>>> { >>>> warning (0, jobserver_error); >>>> - parallel = 0; >>>> + /* Fall back to auto parallelism. */ >>>> jobserver = 0; >>>> + auto_parallel = 1; >>>> } >>>> else if (!jobserver && jobserver_error == NULL) >>>> { >>>> -- >>>> 2.31.1 >>>> >> ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH] LTO: fallback to -flto=N if -flto=jobserver does not work. 2021-04-22 12:21 ` Martin Liška @ 2021-04-22 12:47 ` Richard Biener 2021-04-22 14:30 ` Martin Liška 0 siblings, 1 reply; 10+ messages in thread From: Richard Biener @ 2021-04-22 12:47 UTC (permalink / raw) To: Martin Liška; +Cc: GCC Patches On Thu, Apr 22, 2021 at 2:21 PM Martin Liška <mliska@suse.cz> wrote: > > On 4/22/21 1:19 PM, Richard Biener wrote: > > On Thu, Apr 22, 2021 at 11:02 AM Martin Liška <mliska@suse.cz> wrote: > >> > >> On 4/22/21 10:04 AM, Richard Biener wrote: > >>> On Wed, Apr 21, 2021 at 3:08 PM Martin Liška <mliska@suse.cz> wrote: > >>>> > >>>> When -flto=jobserver is used and we cannot detect job server, then we can > >>>> still fallbackto -flto=N mode. > >>>> > >>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests. > >>>> > >>>> Ready to be installed? > >>> > >>> I think this behavior needs to be documented - it falls back to a less > >>> conservative (possibly system overloading) mode - which IMHO is > >>> non-obvious and IMHO we shouldn't do. > >> > >> Sure, I'm sending corresponding patch. Note that it's quite common mistake > >> that '+' is missing in Makefile rule. That was motivation for my change. > > > > Sure, but that change won't get this fixed. > > It will as linker command line will contain (-flto=jobserver) and LTO will fallback to -flto=N. > > > IMHO we should eventually > > emit diagnostic like > > > > warning: could not find jobserver, compiling N jobs serially > > > > once N > 1 (or 2?). > > We do that now (for all N): > lto-wrapper: warning: jobserver is not available: ‘MAKEFLAGS’ environment variable is unset > > > > Likewise if people just use -flto and auto-detection > > finds nothing: > > -flto != -flto=auto > > Yes, -flto is a serial linking and we can emit a warning. I'd avoid warning if there's just a single ltrans unit. > > warning: using serial compilation of N LTRANS jobs > > note: refer to http://.... for how to use parallel compile > > > > using the URL diagnostics to point to -flto=... documentation. > > What about making that a proper warning (-Wlto)? We have diagnostics infrastructure > that prints URL links. Note that drivers like lto-wrapper do not have fully initialized diagnostic machinery and use a "different" set of overloads (likewise gen* programs). > > > > That is, teach users rather than second-guessing and eventually > > blowing things up. IMHO only the jobserver mode is safe to > > automatically use. > > Well, -flto=auto is also fine and document. I think there is no possibility > auto CPU deduction can fail. So -flto=jobserver (with missing make job server) > and -flto (equal to -flto=1) worth emitting a warning. > > What do you think? Yes, that sounds reasonable. I suspect that people might want to see -flto default to -flto=auto but then I don't think that's good because there's no system wide job scheduler limiting things (systemd-jobserver anyone?) Richard. > Martin > > > > > Richard. > > > >> Martin > >> > >>> > >>> Richard. > >>> > >>>> Thanks, > >>>> Martin > >>>> > >>>> gcc/ChangeLog: > >>>> > >>>> * lto-wrapper.c (run_gcc): When -flto=jobserver is used, but the > >>>> makeserver cannot be detected, then use -flto=N fallback. > >>>> --- > >>>> gcc/lto-wrapper.c | 3 ++- > >>>> 1 file changed, 2 insertions(+), 1 deletion(-) > >>>> > >>>> diff --git a/gcc/lto-wrapper.c b/gcc/lto-wrapper.c > >>>> index 03a5922f8ea..0b626d7c811 100644 > >>>> --- a/gcc/lto-wrapper.c > >>>> +++ b/gcc/lto-wrapper.c > >>>> @@ -1585,8 +1585,9 @@ run_gcc (unsigned argc, char *argv[]) > >>>> if (jobserver && jobserver_error != NULL) > >>>> { > >>>> warning (0, jobserver_error); > >>>> - parallel = 0; > >>>> + /* Fall back to auto parallelism. */ > >>>> jobserver = 0; > >>>> + auto_parallel = 1; > >>>> } > >>>> else if (!jobserver && jobserver_error == NULL) > >>>> { > >>>> -- > >>>> 2.31.1 > >>>> > >> > ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH] LTO: fallback to -flto=N if -flto=jobserver does not work. 2021-04-22 12:47 ` Richard Biener @ 2021-04-22 14:30 ` Martin Liška 2021-05-12 9:10 ` Martin Liška 0 siblings, 1 reply; 10+ messages in thread From: Martin Liška @ 2021-04-22 14:30 UTC (permalink / raw) To: Richard Biener; +Cc: GCC Patches [-- Attachment #1: Type: text/plain, Size: 4397 bytes --] On 4/22/21 2:47 PM, Richard Biener wrote: > On Thu, Apr 22, 2021 at 2:21 PM Martin Liška <mliska@suse.cz> wrote: >> >> On 4/22/21 1:19 PM, Richard Biener wrote: >>> On Thu, Apr 22, 2021 at 11:02 AM Martin Liška <mliska@suse.cz> wrote: >>>> >>>> On 4/22/21 10:04 AM, Richard Biener wrote: >>>>> On Wed, Apr 21, 2021 at 3:08 PM Martin Liška <mliska@suse.cz> wrote: >>>>>> >>>>>> When -flto=jobserver is used and we cannot detect job server, then we can >>>>>> still fallbackto -flto=N mode. >>>>>> >>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests. >>>>>> >>>>>> Ready to be installed? >>>>> >>>>> I think this behavior needs to be documented - it falls back to a less >>>>> conservative (possibly system overloading) mode - which IMHO is >>>>> non-obvious and IMHO we shouldn't do. >>>> >>>> Sure, I'm sending corresponding patch. Note that it's quite common mistake >>>> that '+' is missing in Makefile rule. That was motivation for my change. >>> >>> Sure, but that change won't get this fixed. >> >> It will as linker command line will contain (-flto=jobserver) and LTO will fallback to -flto=N. >> >>> IMHO we should eventually >>> emit diagnostic like >>> >>> warning: could not find jobserver, compiling N jobs serially >>> >>> once N > 1 (or 2?). >> >> We do that now (for all N): >> lto-wrapper: warning: jobserver is not available: ‘MAKEFLAGS’ environment variable is unset >> >> >>> Likewise if people just use -flto and auto-detection >>> finds nothing: >> >> -flto != -flto=auto >> >> Yes, -flto is a serial linking and we can emit a warning. > > I'd avoid warning if there's just a single ltrans unit. That's doable and I've just done that in the attached patch. Two disadvantages: - one needs waiting for the warning after WPA - source change (>1 LTRANS) can trigger the warning > >>> warning: using serial compilation of N LTRANS jobs >>> note: refer to http://.... for how to use parallel compile >>> >>> using the URL diagnostics to point to -flto=... documentation. >> >> What about making that a proper warning (-Wlto)? We have diagnostics infrastructure >> that prints URL links. > > Note that drivers like lto-wrapper do not have fully initialized diagnostic > machinery and use a "different" set of overloads (likewise gen* programs). I managed printing the warning: lto-wrapper: warning: jobserver is not available: ‘MAKEFLAGS’ environment variable is unset lto-wrapper: note: see the ‘-flto’ option documentation for more information and lto-wrapper: warning: using serial compilation of 128 LTRANS jobs lto-wrapper: note: see the ‘-flto’ option documentation for more information > >>> >>> That is, teach users rather than second-guessing and eventually >>> blowing things up. IMHO only the jobserver mode is safe to >>> automatically use. >> >> Well, -flto=auto is also fine and document. I think there is no possibility >> auto CPU deduction can fail. So -flto=jobserver (with missing make job server) >> and -flto (equal to -flto=1) worth emitting a warning. >> >> What do you think? > > Yes, that sounds reasonable. I suspect that people might want to see > -flto default to -flto=auto but then I don't think that's good because there's > no system wide job scheduler limiting things (systemd-jobserver anyone?) Done that. Thoughts? Martin > > Richard. > >> Martin >> >>> >>> Richard. >>> >>>> Martin >>>> >>>>> >>>>> Richard. >>>>> >>>>>> Thanks, >>>>>> Martin >>>>>> >>>>>> gcc/ChangeLog: >>>>>> >>>>>> * lto-wrapper.c (run_gcc): When -flto=jobserver is used, but the >>>>>> makeserver cannot be detected, then use -flto=N fallback. >>>>>> --- >>>>>> gcc/lto-wrapper.c | 3 ++- >>>>>> 1 file changed, 2 insertions(+), 1 deletion(-) >>>>>> >>>>>> diff --git a/gcc/lto-wrapper.c b/gcc/lto-wrapper.c >>>>>> index 03a5922f8ea..0b626d7c811 100644 >>>>>> --- a/gcc/lto-wrapper.c >>>>>> +++ b/gcc/lto-wrapper.c >>>>>> @@ -1585,8 +1585,9 @@ run_gcc (unsigned argc, char *argv[]) >>>>>> if (jobserver && jobserver_error != NULL) >>>>>> { >>>>>> warning (0, jobserver_error); >>>>>> - parallel = 0; >>>>>> + /* Fall back to auto parallelism. */ >>>>>> jobserver = 0; >>>>>> + auto_parallel = 1; >>>>>> } >>>>>> else if (!jobserver && jobserver_error == NULL) >>>>>> { >>>>>> -- >>>>>> 2.31.1 >>>>>> >>>> >> [-- Attachment #2: 0001-Print-warning-diagnostics-for-flto-issues.patch --] [-- Type: text/x-patch, Size: 4042 bytes --] From 7cd89e17c49c027027e2aed1722c0758331ac7ac Mon Sep 17 00:00:00 2001 From: Martin Liska <mliska@suse.cz> Date: Thu, 22 Apr 2021 16:27:19 +0200 Subject: [PATCH] Print warning diagnostics for -flto issues. gcc/ChangeLog: * lto-wrapper.c (print_lto_docs_link): New function. (run_gcc): Print warning about missing job server detection after we know NR of partitions. Do the same for -flto{,=1}. * opts.c (get_option_html_page): Support -flto option. --- gcc/lto-wrapper.c | 39 +++++++++++++++++++++++++++++++++++++-- gcc/opts.c | 6 +++++- 2 files changed, 42 insertions(+), 3 deletions(-) diff --git a/gcc/lto-wrapper.c b/gcc/lto-wrapper.c index 0b626d7c811..f7ca084ef6c 100644 --- a/gcc/lto-wrapper.c +++ b/gcc/lto-wrapper.c @@ -48,6 +48,7 @@ along with GCC; see the file COPYING3. If not see #include "simple-object.h" #include "lto-section-names.h" #include "collect-utils.h" +#include "opts-diagnostic.h" /* Environment variable, used for passing the names of offload targets from GCC driver to lto-wrapper. */ @@ -1335,6 +1336,23 @@ jobserver_active_p (void) return JS_PREFIX "cannot access %<" JS_NEEDLE "%> file descriptors"; } +/* Print link to -flto documentation with a hint message. */ + +void +print_lto_docs_link () +{ + const char *url = get_option_url (NULL, OPT_flto); + + pretty_printer pp; + pp.url_format = URL_FORMAT_DEFAULT; + pp_string (&pp, "see the "); + pp_begin_url (&pp, url); + pp_string (&pp, "%<-flto%> option documentation"); + pp_end_url (&pp); + pp_string (&pp, " for more information"); + inform (UNKNOWN_LOCATION, pp_formatted_text (&pp)); +} + /* Test that a make command is present and working, return true if so. */ static bool @@ -1369,8 +1387,10 @@ run_gcc (unsigned argc, char *argv[]) char *collect_gcc_options; int parallel = 0; int jobserver = 0; + bool jobserver_requested = false; int auto_parallel = 0; bool no_partition = false; + const char *jobserver_error = NULL; struct cl_decoded_option *fdecoded_options = NULL; struct cl_decoded_option *offload_fdecoded_options = NULL; unsigned int fdecoded_options_count = 0; @@ -1522,6 +1542,7 @@ run_gcc (unsigned argc, char *argv[]) { parallel = 1; jobserver = 1; + jobserver_requested = true; } else if (strcmp (option->arg, "auto") == 0) { @@ -1576,15 +1597,15 @@ run_gcc (unsigned argc, char *argv[]) { lto_mode = LTO_MODE_LTO; jobserver = 0; + jobserver_requested = false; auto_parallel = 0; parallel = 0; } else { - const char *jobserver_error = jobserver_active_p (); + jobserver_error = jobserver_active_p (); if (jobserver && jobserver_error != NULL) { - warning (0, jobserver_error); /* Fall back to auto parallelism. */ jobserver = 0; auto_parallel = 1; @@ -1904,6 +1925,20 @@ cont: maybe_unlink (ltrans_output_file); ltrans_output_file = NULL; + if (nr > 1) + { + if (jobserver_requested && jobserver_error != NULL) + { + warning (0, jobserver_error); + print_lto_docs_link (); + } + else if (parallel == 0) + { + warning (0, "using serial compilation of %d LTRANS jobs", nr); + print_lto_docs_link (); + } + } + if (parallel) { makefile = make_temp_file (".mk"); diff --git a/gcc/opts.c b/gcc/opts.c index 24bb64198c8..fe6fddbf095 100644 --- a/gcc/opts.c +++ b/gcc/opts.c @@ -3243,9 +3243,13 @@ get_option_html_page (int option_index) const cl_option *cl_opt = &cl_options[option_index]; /* Analyzer options are on their own page. */ - if (strstr(cl_opt->opt_text, "analyzer-")) + if (strstr (cl_opt->opt_text, "analyzer-")) return "gcc/Static-Analyzer-Options.html"; + /* Handle -flto= option. */ + if (strstr (cl_opt->opt_text, "flto")) + return "gcc/Optimize-Options.html"; + #ifdef CL_Fortran if ((cl_opt->flags & CL_Fortran) != 0 /* If it is option common to both C/C++ and Fortran, it is documented -- 2.31.1 ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH] LTO: fallback to -flto=N if -flto=jobserver does not work. 2021-04-22 14:30 ` Martin Liška @ 2021-05-12 9:10 ` Martin Liška 2021-05-12 9:21 ` Richard Biener 0 siblings, 1 reply; 10+ messages in thread From: Martin Liška @ 2021-05-12 9:10 UTC (permalink / raw) To: Richard Biener; +Cc: GCC Patches May I please ping this Richi? Thanks, Martin On 4/22/21 4:30 PM, Martin Liška wrote: > On 4/22/21 2:47 PM, Richard Biener wrote: >> On Thu, Apr 22, 2021 at 2:21 PM Martin Liška <mliska@suse.cz> wrote: >>> >>> On 4/22/21 1:19 PM, Richard Biener wrote: >>>> On Thu, Apr 22, 2021 at 11:02 AM Martin Liška <mliska@suse.cz> wrote: >>>>> >>>>> On 4/22/21 10:04 AM, Richard Biener wrote: >>>>>> On Wed, Apr 21, 2021 at 3:08 PM Martin Liška <mliska@suse.cz> wrote: >>>>>>> >>>>>>> When -flto=jobserver is used and we cannot detect job server, then we can >>>>>>> still fallbackto -flto=N mode. >>>>>>> >>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests. >>>>>>> >>>>>>> Ready to be installed? >>>>>> >>>>>> I think this behavior needs to be documented - it falls back to a less >>>>>> conservative (possibly system overloading) mode - which IMHO is >>>>>> non-obvious and IMHO we shouldn't do. >>>>> >>>>> Sure, I'm sending corresponding patch. Note that it's quite common mistake >>>>> that '+' is missing in Makefile rule. That was motivation for my change. >>>> >>>> Sure, but that change won't get this fixed. >>> >>> It will as linker command line will contain (-flto=jobserver) and LTO will fallback to -flto=N. >>> >>>> IMHO we should eventually >>>> emit diagnostic like >>>> >>>> warning: could not find jobserver, compiling N jobs serially >>>> >>>> once N > 1 (or 2?). >>> >>> We do that now (for all N): >>> lto-wrapper: warning: jobserver is not available: ‘MAKEFLAGS’ environment variable is unset >>> >>> >>>> Likewise if people just use -flto and auto-detection >>>> finds nothing: >>> >>> -flto != -flto=auto >>> >>> Yes, -flto is a serial linking and we can emit a warning. >> >> I'd avoid warning if there's just a single ltrans unit. > > That's doable and I've just done that in the attached patch. > Two disadvantages: > - one needs waiting for the warning after WPA > - source change (>1 LTRANS) can trigger the warning > >> >>>> warning: using serial compilation of N LTRANS jobs >>>> note: refer to http://.... for how to use parallel compile >>>> >>>> using the URL diagnostics to point to -flto=... documentation. >>> >>> What about making that a proper warning (-Wlto)? We have diagnostics infrastructure >>> that prints URL links. >> >> Note that drivers like lto-wrapper do not have fully initialized diagnostic >> machinery and use a "different" set of overloads (likewise gen* programs). > > I managed printing the warning: > > lto-wrapper: warning: jobserver is not available: ‘MAKEFLAGS’ environment variable is unset > > lto-wrapper: note: see the ‘-flto’ option documentation for more information > > > and > > lto-wrapper: warning: using serial compilation of 128 LTRANS jobs > > lto-wrapper: note: see the ‘-flto’ option documentation for more information > > >> >>>> >>>> That is, teach users rather than second-guessing and eventually >>>> blowing things up. IMHO only the jobserver mode is safe to >>>> automatically use. >>> >>> Well, -flto=auto is also fine and document. I think there is no possibility >>> auto CPU deduction can fail. So -flto=jobserver (with missing make job server) >>> and -flto (equal to -flto=1) worth emitting a warning. >>> >>> What do you think? >> >> Yes, that sounds reasonable. I suspect that people might want to see >> -flto default to -flto=auto but then I don't think that's good because there's >> no system wide job scheduler limiting things (systemd-jobserver anyone?) > > Done that. > > Thoughts? > Martin > >> >> Richard. >> >>> Martin >>> >>>> >>>> Richard. >>>> >>>>> Martin >>>>> >>>>>> >>>>>> Richard. >>>>>> >>>>>>> Thanks, >>>>>>> Martin >>>>>>> >>>>>>> gcc/ChangeLog: >>>>>>> >>>>>>> * lto-wrapper.c (run_gcc): When -flto=jobserver is used, but the >>>>>>> makeserver cannot be detected, then use -flto=N fallback. >>>>>>> --- >>>>>>> gcc/lto-wrapper.c | 3 ++- >>>>>>> 1 file changed, 2 insertions(+), 1 deletion(-) >>>>>>> >>>>>>> diff --git a/gcc/lto-wrapper.c b/gcc/lto-wrapper.c >>>>>>> index 03a5922f8ea..0b626d7c811 100644 >>>>>>> --- a/gcc/lto-wrapper.c >>>>>>> +++ b/gcc/lto-wrapper.c >>>>>>> @@ -1585,8 +1585,9 @@ run_gcc (unsigned argc, char *argv[]) >>>>>>> if (jobserver && jobserver_error != NULL) >>>>>>> { >>>>>>> warning (0, jobserver_error); >>>>>>> - parallel = 0; >>>>>>> + /* Fall back to auto parallelism. */ >>>>>>> jobserver = 0; >>>>>>> + auto_parallel = 1; >>>>>>> } >>>>>>> else if (!jobserver && jobserver_error == NULL) >>>>>>> { >>>>>>> -- >>>>>>> 2.31.1 >>>>>>> >>>>> >>> > ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH] LTO: fallback to -flto=N if -flto=jobserver does not work. 2021-05-12 9:10 ` Martin Liška @ 2021-05-12 9:21 ` Richard Biener 0 siblings, 0 replies; 10+ messages in thread From: Richard Biener @ 2021-05-12 9:21 UTC (permalink / raw) To: Martin Liška; +Cc: GCC Patches On Wed, May 12, 2021 at 11:10 AM Martin Liška <mliska@suse.cz> wrote: > > May I please ping this Richi? OK. Thanks, Richard. > Thanks, > Martin > > On 4/22/21 4:30 PM, Martin Liška wrote: > > On 4/22/21 2:47 PM, Richard Biener wrote: > >> On Thu, Apr 22, 2021 at 2:21 PM Martin Liška <mliska@suse.cz> wrote: > >>> > >>> On 4/22/21 1:19 PM, Richard Biener wrote: > >>>> On Thu, Apr 22, 2021 at 11:02 AM Martin Liška <mliska@suse.cz> wrote: > >>>>> > >>>>> On 4/22/21 10:04 AM, Richard Biener wrote: > >>>>>> On Wed, Apr 21, 2021 at 3:08 PM Martin Liška <mliska@suse.cz> wrote: > >>>>>>> > >>>>>>> When -flto=jobserver is used and we cannot detect job server, then we can > >>>>>>> still fallbackto -flto=N mode. > >>>>>>> > >>>>>>> Patch can bootstrap on x86_64-linux-gnu and survives regression tests. > >>>>>>> > >>>>>>> Ready to be installed? > >>>>>> > >>>>>> I think this behavior needs to be documented - it falls back to a less > >>>>>> conservative (possibly system overloading) mode - which IMHO is > >>>>>> non-obvious and IMHO we shouldn't do. > >>>>> > >>>>> Sure, I'm sending corresponding patch. Note that it's quite common mistake > >>>>> that '+' is missing in Makefile rule. That was motivation for my change. > >>>> > >>>> Sure, but that change won't get this fixed. > >>> > >>> It will as linker command line will contain (-flto=jobserver) and LTO will fallback to -flto=N. > >>> > >>>> IMHO we should eventually > >>>> emit diagnostic like > >>>> > >>>> warning: could not find jobserver, compiling N jobs serially > >>>> > >>>> once N > 1 (or 2?). > >>> > >>> We do that now (for all N): > >>> lto-wrapper: warning: jobserver is not available: ‘MAKEFLAGS’ environment variable is unset > >>> > >>> > >>>> Likewise if people just use -flto and auto-detection > >>>> finds nothing: > >>> > >>> -flto != -flto=auto > >>> > >>> Yes, -flto is a serial linking and we can emit a warning. > >> > >> I'd avoid warning if there's just a single ltrans unit. > > > > That's doable and I've just done that in the attached patch. > > Two disadvantages: > > - one needs waiting for the warning after WPA > > - source change (>1 LTRANS) can trigger the warning > > > >> > >>>> warning: using serial compilation of N LTRANS jobs > >>>> note: refer to http://.... for how to use parallel compile > >>>> > >>>> using the URL diagnostics to point to -flto=... documentation. > >>> > >>> What about making that a proper warning (-Wlto)? We have diagnostics infrastructure > >>> that prints URL links. > >> > >> Note that drivers like lto-wrapper do not have fully initialized diagnostic > >> machinery and use a "different" set of overloads (likewise gen* programs). > > > > I managed printing the warning: > > > > lto-wrapper: warning: jobserver is not available: ‘MAKEFLAGS’ environment variable is unset > > > > lto-wrapper: note: see the ‘-flto’ option documentation for more information > > > > > > and > > > > lto-wrapper: warning: using serial compilation of 128 LTRANS jobs > > > > lto-wrapper: note: see the ‘-flto’ option documentation for more information > > > > > >> > >>>> > >>>> That is, teach users rather than second-guessing and eventually > >>>> blowing things up. IMHO only the jobserver mode is safe to > >>>> automatically use. > >>> > >>> Well, -flto=auto is also fine and document. I think there is no possibility > >>> auto CPU deduction can fail. So -flto=jobserver (with missing make job server) > >>> and -flto (equal to -flto=1) worth emitting a warning. > >>> > >>> What do you think? > >> > >> Yes, that sounds reasonable. I suspect that people might want to see > >> -flto default to -flto=auto but then I don't think that's good because there's > >> no system wide job scheduler limiting things (systemd-jobserver anyone?) > > > > Done that. > > > > Thoughts? > > Martin > > > >> > >> Richard. > >> > >>> Martin > >>> > >>>> > >>>> Richard. > >>>> > >>>>> Martin > >>>>> > >>>>>> > >>>>>> Richard. > >>>>>> > >>>>>>> Thanks, > >>>>>>> Martin > >>>>>>> > >>>>>>> gcc/ChangeLog: > >>>>>>> > >>>>>>> * lto-wrapper.c (run_gcc): When -flto=jobserver is used, but the > >>>>>>> makeserver cannot be detected, then use -flto=N fallback. > >>>>>>> --- > >>>>>>> gcc/lto-wrapper.c | 3 ++- > >>>>>>> 1 file changed, 2 insertions(+), 1 deletion(-) > >>>>>>> > >>>>>>> diff --git a/gcc/lto-wrapper.c b/gcc/lto-wrapper.c > >>>>>>> index 03a5922f8ea..0b626d7c811 100644 > >>>>>>> --- a/gcc/lto-wrapper.c > >>>>>>> +++ b/gcc/lto-wrapper.c > >>>>>>> @@ -1585,8 +1585,9 @@ run_gcc (unsigned argc, char *argv[]) > >>>>>>> if (jobserver && jobserver_error != NULL) > >>>>>>> { > >>>>>>> warning (0, jobserver_error); > >>>>>>> - parallel = 0; > >>>>>>> + /* Fall back to auto parallelism. */ > >>>>>>> jobserver = 0; > >>>>>>> + auto_parallel = 1; > >>>>>>> } > >>>>>>> else if (!jobserver && jobserver_error == NULL) > >>>>>>> { > >>>>>>> -- > >>>>>>> 2.31.1 > >>>>>>> > >>>>> > >>> > > > ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2021-05-12 9:21 UTC | newest] Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-04-21 11:50 [PATCH] LTO: fallback to -flto=N if -flto=jobserver does not work Martin Liška 2021-04-21 15:11 ` Jeff Law 2021-04-22 8:04 ` Richard Biener 2021-04-22 9:02 ` Martin Liška 2021-04-22 11:19 ` Richard Biener 2021-04-22 12:21 ` Martin Liška 2021-04-22 12:47 ` Richard Biener 2021-04-22 14:30 ` Martin Liška 2021-05-12 9:10 ` Martin Liška 2021-05-12 9:21 ` Richard Biener
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).