-flto=jobserver does not use all CPU cores

-flto=jobserver does not use all CPU cores

[R. Diez via gcc-help]

Hi all:

I have an AMD FX-4300 Quad-Core Processor with no hyperthreading.

If I run GCC with -flto, only core is used, as expected.

If I run GCC with -flto=4, all 4 cores are used, as expected.

If I run GCC with -flto=jobserver, only 2 cores are apparently used, even if the top-level makefile runs with -j 40. The increased linking 
time also corroborates this.

I am using a self-built GCC 8.3.0 cross-compiler for ARM Cortex-M4F on Ubuntu 18.04.4 LTS.

I am also using a self-built GNU Make 4.2.1. Everything else is building in parallel with all cores.

Is this problem with "-flto=jobserver" already known? I couldn't find anything on this.

How can I debug what is going on? Is there a way to trace jobserver-related information?

Thanks in advance,
   rdiez

Re: -flto=jobserver does not use all CPU cores

[Florian Weimer]

* R. Diez via gcc-help:

> If I run GCC with -flto=jobserver, only 2 cores are apparently used,
> even if the top-level makefile runs with -j 40. The increased linking
> time also corroborates this.

Can you show us the relevant parts of your makefile?

Thanks,
Florian

Re: -flto=jobserver does not use all CPU cores

[R. Diez via gcc-help]

>> If I run GCC with -flto=jobserver, only 2 cores are apparently used,
>> even if the top-level makefile runs with -j 40. The increased linking
>> time also corroborates this.
> 
> Can you show us the relevant parts of your makefile?

I think I know what is going on. It is easy to forget that you need to add a '+' character in front of the makefile recipe, in order for GNU 
Make to let the child processes inherit the jobserver file descriptors.

The trouble is, there does not seem to be a way to prepend that '+' special character with Autoconf/Automake. I posted a question about it here:

   Prepending '+' to the recipe line when linking with GCC's -flto=jobserve
   https://lists.gnu.org/archive/html/automake/2020-02/msg00012.html

But GCC's behaviour is probably still not quite right. If it is not getting those file descriptors, how come it uses 2 cores instead of 1, 
and how come it does not complain about not being able to open those file descriptors or properly contact the jobserver?

GNU Make does try to help in this scenario with the following warning:

make[1]: warning: jobserver unavailable: using -j1.  Add `+' to parent make rule.

Regards,
   rdiez

Re: -flto=jobserver does not use all CPU cores

[R. Diez via gcc-help]

>> If I run GCC with -flto=jobserver, only 2 cores are apparently used,
>> even if the top-level makefile runs with -j 40. The increased linking
>> time also corroborates this.
> 
> Can you show us the relevant parts of your makefile?

I think I know what is going on. It is easy to forget that you need to add a '+' character in front of the makefile recipe, in order for GNU 
Make to let the child processes inherit the jobserver file descriptors.

The trouble is, there does not seem to be a way to prepend that '+' special character with Autoconf/Automake. I posted a question about it here:

   Prepending '+' to the recipe line when linking with GCC's -flto=jobserve
   https://lists.gnu.org/archive/html/automake/2020-02/msg00012.html

But GCC's behaviour is probably still not quite right. If it is not getting those file descriptors, how come it uses 2 cores instead of 1, 
and how come it does not complain about not being able to open those file descriptors or properly contact the jobserver?

GNU Make does try to help in this scenario with the following warning:

make[1]: warning: jobserver unavailable: using -j1.  Add `+' to parent make rule.

Regards,
   rdiez