[Bug bootstrap/96794] New: --with-build-config=bootstrap-lto-lean with --enable-link-mutex leads to poor LTRANS utilization

[Bug bootstrap/96794] New: --with-build-config=bootstrap-lto-lean with --enable-link-mutex leads to poor LTRANS utilization

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96794

            Bug ID: 96794
           Summary: --with-build-config=bootstrap-lto-lean with
                    --enable-link-mutex leads to poor LTRANS utilization
           Product: gcc
           Version: 10.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: bootstrap
          Assignee: unassigned at gcc dot gnu.org
          Reporter: marxin at gcc dot gnu.org
  Target Milestone: ---

As seen
here:https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/7e0363da60dcddbfde4ab68fa3be755515166297/gcc-10-with-zstd.svg

each blocking linking of a GCC front-end leads to a wasted jobserver worker.

[Bug bootstrap/96794] --with-build-config=bootstrap-lto-lean with --enable-link-mutex leads to poor LTRANS utilization

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96794

Martin Liška <marxin at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
     Ever confirmed|0                           |1
                 CC|                            |hubicka at gcc dot gnu.org,
                   |                            |matz at gcc dot gnu.org,
                   |                            |rguenth at gcc dot gnu.org
             Status|UNCONFIRMED                 |NEW
   Last reconfirmed|                            |2020-08-26

Re: [Bug bootstrap/96794] New: --with-build-config=bootstrap-lto-lean with --enable-link-mutex leads to poor LTRANS utilization

[Jan Hubicka]

> As seen
> here:https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/7e0363da60dcddbfde4ab68fa3be755515166297/gcc-10-with-zstd.svg
> 
> each blocking linking of a GCC front-end leads to a wasted jobserver worker.
Hmm, I am not sure how to interpret the graph. I can see that there is a
funny staircase of ltranses but how that relates to jobserver workers?
We limit makefile to link a binary at a time to avoid Richi's box getting
out of memory, right?

NUmber of partitions is currently 128 what is 100% of CPU usage for you?

Honza

[Bug bootstrap/96794] --with-build-config=bootstrap-lto-lean with --enable-link-mutex leads to poor LTRANS utilization

[hubicka at ucw dot cz]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96794

--- Comment #1 from Jan Hubicka <hubicka at ucw dot cz> ---
> As seen
> here:https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/7e0363da60dcddbfde4ab68fa3be755515166297/gcc-10-with-zstd.svg
> 
> each blocking linking of a GCC front-end leads to a wasted jobserver worker.
Hmm, I am not sure how to interpret the graph. I can see that there is a
funny staircase of ltranses but how that relates to jobserver workers?
We limit makefile to link a binary at a time to avoid Richi's box getting
out of memory, right?

NUmber of partitions is currently 128 what is 100% of CPU usage for you?

Honza

[Bug bootstrap/96794] --with-build-config=bootstrap-lto-lean with --enable-link-mutex leads to poor LTRANS utilization

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96794

--- Comment #2 from Martin Liška <marxin at gcc dot gnu.org> ---
(In reply to Jan Hubicka from comment #1)
> > As seen
> > here:https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/7e0363da60dcddbfde4ab68fa3be755515166297/gcc-10-with-zstd.svg
> > 
> > each blocking linking of a GCC front-end leads to a wasted jobserver worker.
> Hmm, I am not sure how to interpret the graph. I can see that there is a
> funny staircase of ltranses but how that relates to jobserver workers?

Yes, I mean the staircase of LTRANS because at the beginning N-1 links are
waiting for lock:

[  299s] lock-and-run.sh: (PID 7351) waiting 0 sec to acquire linkfe.lck from
PID 7347
...

For jobserver they are still running even though they sleep.


> We limit makefile to link a binary at a time to avoid Richi's box getting
> out of memory, right?

No. It's because we want to have a reasonable contrains which is right now 8GB.
Without --enable-link-mutex, we would consume ~ 10 x 1.3 GB (plus WPA parallel
streaming peak), which is probably not desired.

> 
> NUmber of partitions is currently 128 what is 100% of CPU usage for you?

It's written in the graph footnote "CPU count: 16".

Re: [Bug bootstrap/96794] --with-build-config=bootstrap-lto-lean with --enable-link-mutex leads to poor LTRANS utilization

[Jan Hubicka]

> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96794
> 
> --- Comment #2 from Martin Liška <marxin at gcc dot gnu.org> ---
> (In reply to Jan Hubicka from comment #1)
> > > As seen
> > > here:https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/7e0363da60dcddbfde4ab68fa3be755515166297/gcc-10-with-zstd.svg
> > > 
> > > each blocking linking of a GCC front-end leads to a wasted jobserver worker.
> > Hmm, I am not sure how to interpret the graph. I can see that there is a
> > funny staircase of ltranses but how that relates to jobserver workers?
> 
> Yes, I mean the staircase of LTRANS because at the beginning N-1 links are
> waiting for lock:
> 
> [  299s] lock-and-run.sh: (PID 7351) waiting 0 sec to acquire linkfe.lck from
> PID 7347
> ...
> 
> For jobserver they are still running even though they sleep.
Aha, so it is extra locking mechanizm we add without jobserver
knowledge.
> 
> 
> > We limit makefile to link a binary at a time to avoid Richi's box getting
> > out of memory, right?
> 
> No. It's because we want to have a reasonable contrains which is right now 8GB.
> Without --enable-link-mutex, we would consume ~ 10 x 1.3 GB (plus WPA parallel
> streaming peak), which is probably not desired.

10x1.3GB will get consumed only if the building machine has 10 threads.
I wonder if the jobserver WPA streaming integration will happen this
year, with that snd some patches to WPA memory use we could fit in 8GB
unless very large parallelism is configured.

I suppose only really effective solution would to teach the jobserver
that some jobs are "big" and consume multiple tokens, that is WPA, while
other jobs like ltranses and streaming are small.

This is of course still not very pretty, but it is impossible to tell in
advance what job is big and what job is small.

Honza

[Bug bootstrap/96794] --with-build-config=bootstrap-lto-lean with --enable-link-mutex leads to poor LTRANS utilization

[hubicka at ucw dot cz]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96794

--- Comment #3 from Jan Hubicka <hubicka at ucw dot cz> ---
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96794
> 
> --- Comment #2 from Martin Liška <marxin at gcc dot gnu.org> ---
> (In reply to Jan Hubicka from comment #1)
> > > As seen
> > > here:https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/7e0363da60dcddbfde4ab68fa3be755515166297/gcc-10-with-zstd.svg
> > > 
> > > each blocking linking of a GCC front-end leads to a wasted jobserver worker.
> > Hmm, I am not sure how to interpret the graph. I can see that there is a
> > funny staircase of ltranses but how that relates to jobserver workers?
> 
> Yes, I mean the staircase of LTRANS because at the beginning N-1 links are
> waiting for lock:
> 
> [  299s] lock-and-run.sh: (PID 7351) waiting 0 sec to acquire linkfe.lck from
> PID 7347
> ...
> 
> For jobserver they are still running even though they sleep.
Aha, so it is extra locking mechanizm we add without jobserver
knowledge.
> 
> 
> > We limit makefile to link a binary at a time to avoid Richi's box getting
> > out of memory, right?
> 
> No. It's because we want to have a reasonable contrains which is right now 8GB.
> Without --enable-link-mutex, we would consume ~ 10 x 1.3 GB (plus WPA parallel
> streaming peak), which is probably not desired.

10x1.3GB will get consumed only if the building machine has 10 threads.
I wonder if the jobserver WPA streaming integration will happen this
year, with that snd some patches to WPA memory use we could fit in 8GB
unless very large parallelism is configured.

I suppose only really effective solution would to teach the jobserver
that some jobs are "big" and consume multiple tokens, that is WPA, while
other jobs like ltranses and streaming are small.

This is of course still not very pretty, but it is impossible to tell in
advance what job is big and what job is small.

Honza

[Bug bootstrap/96794] --with-build-config=bootstrap-lto-lean with --enable-link-mutex leads to poor LTRANS utilization

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96794

--- Comment #4 from Martin Liška <marxin at gcc dot gnu.org> ---
> > For jobserver they are still running even though they sleep.
> Aha, so it is extra locking mechanizm we add without jobserver
> knowledge.

It's unrelated to jobserver, one can enable it with configure option mentioned
in the title.

> > 
> > 
> > > We limit makefile to link a binary at a time to avoid Richi's box getting
> > > out of memory, right?
> > 
> > No. It's because we want to have a reasonable contrains which is right now 8GB.
> > Without --enable-link-mutex, we would consume ~ 10 x 1.3 GB (plus WPA parallel
> > streaming peak), which is probably not desired.
> 
> 10x1.3GB will get consumed only if the building machine has 10 threads.

Typical OBS machine used for package build is either 8 (required minimum) or
16. 

> I wonder if the jobserver WPA streaming integration will happen this
> year, with that snd some patches to WPA memory use we could fit in 8GB
> unless very large parallelism is configured.
> 
> I suppose only really effective solution would to teach the jobserver
> that some jobs are "big" and consume multiple tokens, that is WPA, while
> other jobs like ltranses and streaming are small.
> 
> This is of course still not very pretty, but it is impossible to tell in
> advance what job is big and what job is small.

Sure, it's all quite compilicated. One needs to negotiate with jobserver :)

I'm going to collect graph w/o --enable-link-mutex on my machine.

> 
> Honza

Re: [Bug bootstrap/96794] --with-build-config=bootstrap-lto-lean with --enable-link-mutex leads to poor LTRANS utilization

[Jan Hubicka]

> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96794
> 
> --- Comment #4 from Martin Liška <marxin at gcc dot gnu.org> ---
> > > For jobserver they are still running even though they sleep.
> > Aha, so it is extra locking mechanizm we add without jobserver
> > knowledge.
> 
> It's unrelated to jobserver, one can enable it with configure option mentioned
> in the title.

Yep, but that is where I see the problem - if we simply wait for lock
and jobserver does not know that, he counts it as a job..
> > This is of course still not very pretty, but it is impossible to tell in
> > advance what job is big and what job is small.
> 
> Sure, it's all quite compilicated. One needs to negotiate with jobserver :)

Yep...
Honza
> 
> I'm going to collect graph w/o --enable-link-mutex on my machine.
> 
> > 
> > Honza

[Bug bootstrap/96794] --with-build-config=bootstrap-lto-lean with --enable-link-mutex leads to poor LTRANS utilization

[hubicka at ucw dot cz]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96794

--- Comment #5 from Jan Hubicka <hubicka at ucw dot cz> ---
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96794
> 
> --- Comment #4 from Martin Liška <marxin at gcc dot gnu.org> ---
> > > For jobserver they are still running even though they sleep.
> > Aha, so it is extra locking mechanizm we add without jobserver
> > knowledge.
> 
> It's unrelated to jobserver, one can enable it with configure option mentioned
> in the title.

Yep, but that is where I see the problem - if we simply wait for lock
and jobserver does not know that, he counts it as a job..
> > This is of course still not very pretty, but it is impossible to tell in
> > advance what job is big and what job is small.
> 
> Sure, it's all quite compilicated. One needs to negotiate with jobserver :)

Yep...
Honza
> 
> I'm going to collect graph w/o --enable-link-mutex on my machine.
> 
> > 
> > Honza

[Bug bootstrap/96794] --with-build-config=bootstrap-lto-lean with --enable-link-mutex leads to poor LTRANS utilization

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96794

--- Comment #6 from Richard Biener <rguenth at gcc dot gnu.org> ---
(In reply to Jan Hubicka from comment #1)
> > As seen
> > here:https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/7e0363da60dcddbfde4ab68fa3be755515166297/gcc-10-with-zstd.svg
> > 
> > each blocking linking of a GCC front-end leads to a wasted jobserver worker.
> Hmm, I am not sure how to interpret the graph. I can see that there is a
> funny staircase of ltranses but how that relates to jobserver workers?
> We limit makefile to link a binary at a time to avoid Richi's box getting
> out of memory, right?

Yes.  With LTO there's also the parallel stream-out which hard-locks the box
since it is/was not integrated with jobserver.

> NUmber of partitions is currently 128 what is 100% of CPU usage for you?
> 
> Honza

[Bug bootstrap/96794] --with-build-config=bootstrap-lto-lean with --enable-link-mutex leads to poor LTRANS utilization

[rguenther at suse dot de]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96794

--- Comment #7 from rguenther at suse dot de <rguenther at suse dot de> ---
On Wed, 26 Aug 2020, hubicka at ucw dot cz wrote:

> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96794
> 
> --- Comment #3 from Jan Hubicka <hubicka at ucw dot cz> ---
> > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96794
> > 
> > --- Comment #2 from Martin Liška <marxin at gcc dot gnu.org> ---
> > (In reply to Jan Hubicka from comment #1)
> > > > As seen
> > > > here:https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/7e0363da60dcddbfde4ab68fa3be755515166297/gcc-10-with-zstd.svg
> > > > 
> > > > each blocking linking of a GCC front-end leads to a wasted jobserver worker.
> > > Hmm, I am not sure how to interpret the graph. I can see that there is a
> > > funny staircase of ltranses but how that relates to jobserver workers?
> > 
> > Yes, I mean the staircase of LTRANS because at the beginning N-1 links are
> > waiting for lock:
> > 
> > [  299s] lock-and-run.sh: (PID 7351) waiting 0 sec to acquire linkfe.lck from
> > PID 7347
> > ...
> > 
> > For jobserver they are still running even though they sleep.
> Aha, so it is extra locking mechanizm we add without jobserver
> knowledge.
> > 
> > 
> > > We limit makefile to link a binary at a time to avoid Richi's box getting
> > > out of memory, right?
> > 
> > No. It's because we want to have a reasonable contrains which is right now 8GB.
> > Without --enable-link-mutex, we would consume ~ 10 x 1.3 GB (plus WPA parallel
> > streaming peak), which is probably not desired.
> 
> 10x1.3GB will get consumed only if the building machine has 10 threads.
> I wonder if the jobserver WPA streaming integration will happen this
> year, with that snd some patches to WPA memory use we could fit in 8GB
> unless very large parallelism is configured.

Note even without LTO the link step will consume about 1GB for each FE,
this is enough to make my box with 6GB ram swap and die miserably
when bootstrapping with all languages enabled.  Yes, without LTO
bootstrap - ld.bfd (also gold) really are that memory hungry.

> I suppose only really effective solution would to teach the jobserver
> that some jobs are "big" and consume multiple tokens, that is WPA, while
> other jobs like ltranses and streaming are small.

The only effective solution would be to make the glue with the waiting
mechanism pass on its token?  Hmm, I guess since lto-wrapper invokes
make again and also waits lto-wrapper itself still sits on the original
token from the parent jobserver and thus this isn't really dependent on
the waiting mechanism but an inherent "bug" in the way we execute
the LTRANS compiles in jobserver mode?

[Bug bootstrap/96794] --with-build-config=bootstrap-lto-lean with --enable-link-mutex leads to poor LTRANS utilization

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96794

--- Comment #8 from Martin Liška <marxin at gcc dot gnu.org> ---
There's graph when omitting link mutex:
https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/24409fcf099230dbc19ef50e193655c578d6d527/gcc-10-without-mutex-lock.svg

As seen, memory peak is slightly more than 12GB.

[Bug bootstrap/96794] --with-build-config=bootstrap-lto-lean with --enable-link-mutex leads to poor LTRANS utilization

[matz at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96794

--- Comment #9 from Michael Matz <matz at gcc dot gnu.org> ---
We could also punt: when enable-link-mutex we could artificially up the job
number for make to account for the waiting link steps.  I.e. when normally -jN
was given, the link step could be done under -j(N + nr-language - 1).  That
would lead to the
nr-of-languages-1 job server tokens taken up by the sleeping link steps to be
accounted for, and hence the (single running) link step still be able to use N
threads/processes in parallel.

[Bug bootstrap/96794] --with-build-config=bootstrap-lto-lean with --enable-link-mutex leads to poor LTRANS utilization

[hubicka at ucw dot cz]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96794

--- Comment #10 from Jan Hubicka <hubicka at ucw dot cz> ---
> We could also punt: when enable-link-mutex we could artificially up the job
> number for make to account for the waiting link steps.  I.e. when normally -jN
> was given, the link step could be done under -j(N + nr-language - 1).  That
> would lead to the
> nr-of-languages-1 job server tokens taken up by the sleeping link steps to be
> accounted for, and hence the (single running) link step still be able to use N
> threads/processes in parallel.

I do not think it is easy to do with current make jobserver.  The
toplevel jobserver opens the pipi with -jN many tokens and all other
makes connect to it.  It does not provide interface to return a token
and re-take it unless we implement it of course.

Honza
> 
> -- 
> You are receiving this mail because:
> You are on the CC list for the bug.

[Bug bootstrap/96794] --with-build-config=bootstrap-lto-lean with --enable-link-mutex leads to poor LTRANS utilization

[matz at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96794

--- Comment #11 from Michael Matz <matz at gcc dot gnu.org> ---
(In reply to Jan Hubicka from comment #10)
> > We could also punt: when enable-link-mutex we could artificially up the job
> > number for make to account for the waiting link steps.  I.e. when normally -jN
> > was given, the link step could be done under -j(N + nr-language - 1).  That
> > would lead to the
> > nr-of-languages-1 job server tokens taken up by the sleeping link steps to be
> > accounted for, and hence the (single running) link step still be able to use N
> > threads/processes in parallel.
> 
> I do not think it is easy to do with current make jobserver.  The
> toplevel jobserver opens the pipi with -jN many tokens and all other
> makes connect to it.  It does not provide interface to return a token
> and re-take it unless we implement it of course.

Sure, but that's not the only way the above punting could be done.  The
parallel
phase that currently communicates with the jobserver essentially tries to get N
tokens.  It could be changed to assume that there are implicitely
nr-of-languages-1 more tokens available.  I.e. the number of threads/processes
would always be enlarged by nr-of-languages-1, even without any tokens from the
jobserver.

[Bug bootstrap/96794] --with-build-config=bootstrap-lto-lean with --enable-link-mutex leads to poor LTRANS utilization

[rguenther at suse dot de]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96794

--- Comment #12 from rguenther at suse dot de <rguenther at suse dot de> ---
On Wed, 26 Aug 2020, matz at gcc dot gnu.org wrote:

> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96794
> 
> --- Comment #11 from Michael Matz <matz at gcc dot gnu.org> ---
> (In reply to Jan Hubicka from comment #10)
> > > We could also punt: when enable-link-mutex we could artificially up the job
> > > number for make to account for the waiting link steps.  I.e. when normally -jN
> > > was given, the link step could be done under -j(N + nr-language - 1).  That
> > > would lead to the
> > > nr-of-languages-1 job server tokens taken up by the sleeping link steps to be
> > > accounted for, and hence the (single running) link step still be able to use N
> > > threads/processes in parallel.
> > 
> > I do not think it is easy to do with current make jobserver.  The
> > toplevel jobserver opens the pipi with -jN many tokens and all other
> > makes connect to it.  It does not provide interface to return a token
> > and re-take it unless we implement it of course.
> 
> Sure, but that's not the only way the above punting could be done.  The
> parallel
> phase that currently communicates with the jobserver essentially tries to get N
> tokens.  It could be changed to assume that there are implicitely
> nr-of-languages-1 more tokens available.  I.e. the number of threads/processes
> would always be enlarged by nr-of-languages-1, even without any tokens from the
> jobserver.

I wonder whether the link-mutex is better be implemented with
a set of artificial dependences.  If we want to serialize
building cc1, cc1plus, etc. then just have

cc1: cc1plus
cc1plus: ...
...

and only conditionally enable those somehow.  But IIRC nowadays
that "waiting" also prints those progress markers ...