Re: [PATCH] Deduce automatically number of cores for -flto option.

["Martin Liška" <mliska@suse.cz>]

On 7/31/19 9:40 AM, Jan Hubicka wrote:
>> Neither will work very well, we have thousands of -flto tests outside of
>> lto.exp, e.g. dg-torture.exp (or libgomp and others) cycle through various
>> options including -flto etc.
>>
>> Some env var would be useful I guess, though shouldn't it have GCC in the
>> name?  I mean, if we run into these fork-bomb problems in gcc, won't other
>> projects run into those as well?
>>
>> Why doesn't the jobserver work in the tests?  Is that because of missing +
>> somewhere in the Makefiles or is something unsetting MFLAGS or MAKEFLAGS
>> env vars?
> 
> Main trouble with make's jobserver is that it works by 
> 1) defining environment variable saying which file descriptior to
> connect to
> 2) keeping the file descriptor open upon invoking "+" prefixed lines
> 
> Adding "+" to GCC invocation is wrong since it breaks dry run (we do not
> want to link at that time) but it is only way to access the jobserver.
> If "+" is not present, make will keep the environment vairable but will
> close file descriptors prior exec.
> 
> Make developers said that this is because some old prorams misbehave
> when you exec them with more than 3 file descriptors open. I tried to
> negotiate for named pipe which would solve this and it would make it
> easy to connect to outermost jobserver from anything invoked form
> toplevel make, but they was worried about systems w/o named pipes.

Yes a more generic approach would be welcome as other build systems
would also be able to utilize it.

> 
> I wonder why we do not detect jobserv as unavailable in this case and do
> not default to -flto=<numthreads>?

We do not detect jobserver because of Dejagnu is not using it.
And yes, we default to -flto=<numthreads> in LTO tests now. That's
what is causing issues right now.

> Is it because dejagnu machinery actually opens some other file
> descriptor that gets same ID and executes us with it?
> 
> Or does LTO wrapper open something prior accessing jobserver?
> 
>>
>> Though, as I said on IRC, I think we might run out of filedescriptors when
>> using jobserver too, say if on 64 thread machine one does make -j64 -k check
>> and each test simultaneously tries to create 64 partitions, that would be
>> 4096 connections to the jobserver, right?
> 
> Only WPA process connects to jobserver (which is 1 per linker
> invokation), so I think this should be safe.

Yes and it's only about a quick fcntl. But as mentioned, Dejagnu is not passing
us jobserver, so we don't do it right now.

Martin

> 
> Honza
>>
>> 	Jakub