[Bug lto/60295] [4.9 Regression] Too many lto1-wpa-stream processes are forked

[Bug lto/60295] [4.9 Regression] Too many lto1-wpa-stream processes are forked

[hjl.tools at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=60295

--- Comment #2 from H.J. Lu <hjl.tools at gmail dot com> ---
-flto=jobserver is added by bootstrap-lto.mk.  There are

static void 
stream_out (char *temp_filename, lto_symtab_encoder_t encoder, bool last)
{
#ifdef HAVE_WORKING_FORK
  static int nruns;

  if (!lto_parallelism || lto_parallelism == 1)
    {    
      do_stream_out (temp_filename, encoder);
      return;
    }    

  /* Do not run more than LTO_PARALLELISM streamings
     FIXME: we ignore limits on jobserver.  */
  if (lto_parallelism > 0 && nruns >= lto_parallelism)
    {    
      wait_for_child ();
      nruns --;
    }
...
  /* TODO: jobserver communicatoin is not supported, yet.  */
  if (!strcmp (flag_wpa, "jobserver"))
    lto_parallelism = -1;
  else
    {
      lto_parallelism = atoi (flag_wpa);
      if (lto_parallelism <= 0)
        lto_parallelism = 0;
    }

I would assume lto_parallelism == -1.

[Bug lto/60295] [4.9 Regression] Too many lto1-wpa-stream processes are forked

[rguenth at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=60295

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|---                         |4.9.0

[Bug lto/60295] [4.9 Regression] Too many lto1-wpa-stream processes are forked

[hjl.tools at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=60295

--- Comment #3 from H.J. Lu <hjl.tools at gmail dot com> ---
lto_wpa_write_files has

for (i = 0; i < n_sets; i++)
{
...
stream_out (temp_filename, part->encoder, i == n_sets - 1);
...
}

n_sets is 32 when bootstrapping GCC.  With parallel build, we
may build cc1, cc1plus, f951, cc1obj at the same time.  If machine
is under heavy load, we may start 4*32 == 128 lto1-wpa-stream
processes at the same time with -flto=jobserver.  This patch:

diff --git a/gcc/lto/lto.c b/gcc/lto/lto.c
index c676d79..4023036 100644
--- a/gcc/lto/lto.c
+++ b/gcc/lto/lto.c
@@ -3219,9 +3219,7 @@ do_whole_program_analysis (void)
   lto_parallelism = 1;

   /* TODO: jobserver communicatoin is not supported, yet.  */
-  if (!strcmp (flag_wpa, "jobserver"))
-    lto_parallelism = -1;
-  else
+  if (strcmp (flag_wpa, "jobserver"))
     {
       lto_parallelism = atoi (flag_wpa);
       if (lto_parallelism <= 0)

limits lto1-wpa-stream process to 1 if -flto=jobserver is used.

[Bug lto/60295] [4.9 Regression] Too many lto1-wpa-stream processes are forked

[hubicka at ucw dot cz]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=60295

--- Comment #4 from Jan Hubicka <hubicka at ucw dot cz> ---
> http://gcc.gnu.org/bugzilla/show_bug.cgi?id=60295
> 
> --- Comment #3 from H.J. Lu <hjl.tools at gmail dot com> ---
> lto_wpa_write_files has
> 
> for (i = 0; i < n_sets; i++)
> {
> ...
> stream_out (temp_filename, part->encoder, i == n_sets - 1);
> ...
> }
> 
> n_sets is 32 when bootstrapping GCC.  With parallel build, we
> may build cc1, cc1plus, f951, cc1obj at the same time.  If machine
> is under heavy load, we may start 4*32 == 128 lto1-wpa-stream
> processes at the same time with -flto=jobserver.  This patch:
> 
> diff --git a/gcc/lto/lto.c b/gcc/lto/lto.c
> index c676d79..4023036 100644
> --- a/gcc/lto/lto.c
> +++ b/gcc/lto/lto.c
> @@ -3219,9 +3219,7 @@ do_whole_program_analysis (void)
>    lto_parallelism = 1;
> 
>    /* TODO: jobserver communicatoin is not supported, yet.  */
> -  if (!strcmp (flag_wpa, "jobserver"))
> -    lto_parallelism = -1;
> -  else
> +  if (strcmp (flag_wpa, "jobserver"))
>      {
>        lto_parallelism = atoi (flag_wpa);
>        if (lto_parallelism <= 0)
> 
> limits lto1-wpa-stream process to 1 if -flto=jobserver is used.

I think this is better variant
$ svn diff ~/trunk/gcc/lto/lto.c
Index: /aux/hubicka/trunk/gcc/lto/lto.c
===================================================================
--- /aux/hubicka/trunk/gcc/lto/lto.c    (revision 207702)
+++ /aux/hubicka/trunk/gcc/lto/lto.c    (working copy)
@@ -2491,7 +2491,7 @@ stream_out (char *temp_filename, lto_sym
 #ifdef HAVE_WORKING_FORK
   static int nruns;

-  if (!lto_parallelism || lto_parallelism == 1)
+  if (lto_parallelism <= 1)
     {
       do_stream_out (temp_filename, encoder);
       return;

I basically wanted to have simple jobserver client with lto_parallelism=-1, but
I did not have time to implement it.  (after glancing over GNU Make's
implementation
it seems actually bit non-trivial)

I am adding Paul D. Smith into CC, since he wrote article on the
implementation.
Paul, we would like GCC to actually use jobserver to limit number of processes
it forks internally
for streaming.  Is there an elegant and simple solution to get this
implemented?

I was thinking that if connecting to jobserver is hard, we can just produce
makefile
with rules waiting for read to finish and use it to get tokens into GCC
streamer, but
that seems somewhat kludgy.

Jan

[Bug lto/60295] [4.9 Regression] Too many lto1-wpa-stream processes are forked

[hjl.tools at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=60295

--- Comment #5 from H.J. Lu <hjl.tools at gmail dot com> ---
(In reply to Jan Hubicka from comment #4)

> I think this is better variant
> $ svn diff ~/trunk/gcc/lto/lto.c
> Index: /aux/hubicka/trunk/gcc/lto/lto.c
> ===================================================================
> --- /aux/hubicka/trunk/gcc/lto/lto.c    (revision 207702)
> +++ /aux/hubicka/trunk/gcc/lto/lto.c    (working copy)
> @@ -2491,7 +2491,7 @@ stream_out (char *temp_filename, lto_sym
>  #ifdef HAVE_WORKING_FORK
>    static int nruns;
>  
> -  if (!lto_parallelism || lto_parallelism == 1)
> +  if (lto_parallelism <= 1)
>      {
>        do_stream_out (temp_filename, encoder);
>        return;
> 
> I basically wanted to have simple jobserver client with lto_parallelism=-1,
> but
> I did not have time to implement it.  (after glancing over GNU Make's
> implementation
> it seems actually bit non-trivial)
> 

This works for me.  Can you check it in?

[Bug lto/60295] [4.9 Regression] Too many lto1-wpa-stream processes are forked

[hubicka at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=60295

--- Comment #6 from Jan Hubicka <hubicka at gcc dot gnu.org> ---
Author: hubicka
Date: Mon Feb 24 22:58:44 2014
New Revision: 208097

URL: http://gcc.gnu.org/viewcvs?rev=208097&root=gcc&view=rev
Log:

    PR lto/60295
    * lto.c (stream_out): Avoid parallel streaming with
    -flto=jobserver until we are able to throttle it down
    resonably.

Modified:
    trunk/gcc/lto/ChangeLog
    trunk/gcc/lto/lto.c

[Bug lto/60295] [4.9 Regression] Too many lto1-wpa-stream processes are forked

[rguenth at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=60295

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |RESOLVED
         Resolution|---                         |FIXED

--- Comment #7 from Richard Biener <rguenth at gcc dot gnu.org> ---
Fixed.