public inbox for gcc-patches@gcc.gnu.org
 help / color / mirror / Atom feed
From: Jan Hubicka <hubicka@ucw.cz>
To: Jakub Jelinek <jakub@redhat.com>
Cc: "Martin Liška" <mliska@suse.cz>, "Jeff Law" <law@redhat.com>,
	gcc-patches@gcc.gnu.org, "Michael Matz" <matz@suse.de>,
	"Richard Biener" <richard.guenther@gmail.com>
Subject: Re: [PATCH] Deduce automatically number of cores for -flto option.
Date: Wed, 31 Jul 2019 07:49:00 -0000	[thread overview]
Message-ID: <20190731074019.b4l6pwcpeh2q2zai@kam.mff.cuni.cz> (raw)
In-Reply-To: <20190731073407.GU15878@tucnak>

> Neither will work very well, we have thousands of -flto tests outside of
> lto.exp, e.g. dg-torture.exp (or libgomp and others) cycle through various
> options including -flto etc.
> 
> Some env var would be useful I guess, though shouldn't it have GCC in the
> name?  I mean, if we run into these fork-bomb problems in gcc, won't other
> projects run into those as well?
> 
> Why doesn't the jobserver work in the tests?  Is that because of missing +
> somewhere in the Makefiles or is something unsetting MFLAGS or MAKEFLAGS
> env vars?

Main trouble with make's jobserver is that it works by 
1) defining environment variable saying which file descriptior to
connect to
2) keeping the file descriptor open upon invoking "+" prefixed lines

Adding "+" to GCC invocation is wrong since it breaks dry run (we do not
want to link at that time) but it is only way to access the jobserver.
If "+" is not present, make will keep the environment vairable but will
close file descriptors prior exec.

Make developers said that this is because some old prorams misbehave
when you exec them with more than 3 file descriptors open. I tried to
negotiate for named pipe which would solve this and it would make it
easy to connect to outermost jobserver from anything invoked form
toplevel make, but they was worried about systems w/o named pipes.

I wonder why we do not detect jobserv as unavailable in this case and do
not default to -flto=<numthreads>?
Is it because dejagnu machinery actually opens some other file
descriptor that gets same ID and executes us with it?

Or does LTO wrapper open something prior accessing jobserver?

> 
> Though, as I said on IRC, I think we might run out of filedescriptors when
> using jobserver too, say if on 64 thread machine one does make -j64 -k check
> and each test simultaneously tries to create 64 partitions, that would be
> 4096 connections to the jobserver, right?

Only WPA process connects to jobserver (which is 1 per linker
invokation), so I think this should be safe.

Honza
> 
> 	Jakub

  reply	other threads:[~2019-07-31  7:40 UTC|newest]

Thread overview: 54+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-07-23  8:55 [PATCH] Come up with -flto=auto option Martin Liška
2019-07-23  9:29 ` Jan Hubicka
2019-07-23 10:34   ` [PATCH] Deduce automatically number of cores for -flto option Martin Liška
2019-07-24 15:47     ` Jeff Law
2019-07-29 13:37       ` Martin Liška
2019-07-30 13:47         ` Martin Liška
2019-07-31  1:23         ` Jakub Jelinek
2019-07-31  7:24           ` Martin Liška
2019-07-31  7:40             ` Jakub Jelinek
2019-07-31  7:49               ` Jan Hubicka [this message]
2019-07-31  7:50                 ` Martin Liška
2019-07-31  7:54               ` Martin Liška
2019-07-31  8:08                 ` Jakub Jelinek
2019-07-31  8:21                 ` Jan Hubicka
2019-07-31  8:37                   ` Martin Liška
2019-07-31  9:12                     ` Jakub Jelinek
2019-07-31  9:15                       ` Jan Hubicka
2019-07-31  9:17                         ` Jakub Jelinek
2019-07-31  9:22                           ` Jan Hubicka
2019-07-31 10:02                         ` Martin Liška
2019-07-31 12:02                           ` Jan Hubicka
2019-07-31 15:42                           ` Martin Liška
2019-08-01 13:19                             ` Jakub Jelinek
2019-08-01 14:34                               ` [PATCH] Properly detect working jobserver in gcc driver Martin Liška
2019-08-01 14:41                                 ` Jakub Jelinek
2019-08-02  6:30                                   ` Martin Liška
2019-08-02  7:45                                     ` Jakub Jelinek
2019-08-02  8:47                                       ` Martin Liška
2019-08-02  8:50                                         ` Jakub Jelinek
2019-08-02  9:04                                           ` Richard Biener
2019-08-02  9:08                                             ` Jan Hubicka
2019-08-02  9:15                                             ` Jan Hubicka
2019-08-02  9:19                                               ` Martin Liška
2019-08-02  9:55                                                 ` Richard Biener
2019-08-05  6:41                                                   ` Martin Liška
2019-08-09  8:14                                                     ` Martin Liška
2019-08-09  8:22                                                       ` Richard Biener
2019-08-09 12:51                                                         ` Martin Liška
2019-08-09 13:56                                                           ` Martin Liška
2019-08-12 15:18                                                             ` Jeff Law
2019-07-23 13:13   ` [PATCH] Come up with -flto=auto option Jeff Law
2019-07-23 13:22     ` Richard Biener
2019-07-23 13:57     ` Michael Matz
2019-07-23 14:00       ` Jeff Law
2019-07-23 14:27         ` Martin Liška
2019-07-23 22:56           ` Jeff Law
2019-07-24  6:59             ` Martin Liška
2019-07-24 15:16               ` Jeff Law
2019-07-23 22:32 ` Allan Sandfeld Jensen
2019-07-24  6:47   ` Martin Liška
2019-07-24  7:12     ` Allan Sandfeld Jensen
2019-07-24  7:15       ` Martin Liška
2019-07-24 11:09         ` Nathan Sidwell
2019-07-24 15:46 ` Jeff Law

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20190731074019.b4l6pwcpeh2q2zai@kam.mff.cuni.cz \
    --to=hubicka@ucw.cz \
    --cc=gcc-patches@gcc.gnu.org \
    --cc=jakub@redhat.com \
    --cc=law@redhat.com \
    --cc=matz@suse.de \
    --cc=mliska@suse.cz \
    --cc=richard.guenther@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).