Optional machine prefix for programs in for -B dirs, matching Clang

Optional machine prefix for programs in for -B dirs, matching Clang

[John Ericson]

Problem:

It's somewhat annoying to have to tell GCC --with-as=... --with-ld=... just to prefix those commands the same way GCC is prefixed.

In particular, when doing host-only build (skipping all target libraries), one otherwise doesn't need the target-specific binutils to be yet built, but --with-as and --with-ld will complain if the referenced exes cannot be found. This might sound esoteric, but as someone that spends a lot of time optimizing bootstrap dependency graphs for incrementality / parallelism, it is quite a real-world annoyance.

Solution:

I think the solution is to stop making cross compilers rely on these --with-flags to do the obvious things. Executables like `collect2` hidden within a libexesubdir (libexec/gcc/<machine>/<version>) have no need for prefixing, but the assembler and linker are very much public-facing executables in their own right, and usually are prefixed.

Per [1], Clang does in fact look up prefixed exes against -B across the board. Making GCC look up exes that same way seems like a fine solution too.

What do you all think?

John 

[1]: https://clang.llvm.org/docs/ClangCommandLineReference.html#cmdoption-clang-b-prefix

Re: Optional machine prefix for programs in for -B dirs, matching Clang

[Jonathan Wakely]

On Wed, 4 Aug 2021, 08:26 John Ericson wrote:

> Problem:
>
> It's somewhat annoying to have to tell GCC --with-as=... --with-ld=...
> just to prefix those commands the same way GCC is prefixed.
>

Doesn't GCC automatically look for those commands in the --prefix directory
that you configure GCC with? Or is that only for native compilers?

Re: Optional machine prefix for programs in for -B dirs, matching Clang

[John Ericson]

On Wed, Aug 4, 2021, at 3:32 AM, Jonathan Wakely via Gcc wrote:
> 
> Doesn't GCC automatically look for those commands in the --prefix directory
> that you configure GCC with? Or is that only for native compilers?
> 

It will search only if --with-*=... was not passed, and it will never prefix the query. So yes in practice for cross compilers people do the --with-* and no searching happens, and for native compilers no one bothers and searching does happen. But to be a pedant strictly speaking the behavior is independent of whether the compiler is host == target or not.

Re: Optional machine prefix for programs in for -B dirs, matching Clang

[Jonathan Wakely]

On Wed, 4 Aug 2021 at 08:41, John Ericson wrote:
>
> On Wed, Aug 4, 2021, at 3:32 AM, Jonathan Wakely via Gcc wrote:
> >
> > Doesn't GCC automatically look for those commands in the --prefix directory
> > that you configure GCC with? Or is that only for native compilers?
> >
>
> It will search only if --with-*=... was not passed, and it will never prefix the query. So yes in practice for cross compilers people do the --with-* and no searching happens, and for native compilers no one bothers and searching does happen. But to be a pedant strictly speaking the behavior is independent of whether the compiler is host == target or not.

I've just checked, and I have a few cross-compilers (for testing
purposes) where I configure binutils and gcc with the same --prefix,
and don't use --with-as or --with-ld, and GCC finds the right
assembler and linker.

So do you need to change anything in GCC? Can't you just not use
--with-as and --with-ld if they are causing problems?

Re: Optional machine prefix for programs in for -B dirs, matching Clang

[Paul Koning]

> On Aug 4, 2021, at 3:32 AM, Jonathan Wakely via Gcc <gcc@gcc.gnu.org> wrote:
> 
> On Wed, 4 Aug 2021, 08:26 John Ericson wrote:
> 
>> Problem:
>> 
>> It's somewhat annoying to have to tell GCC --with-as=... --with-ld=...
>> just to prefix those commands the same way GCC is prefixed.
>> 
> 
> Doesn't GCC automatically look for those commands in the --prefix directory
> that you configure GCC with? Or is that only for native compilers?

It does.  That's how I configure my cross-builds.

	paul

Re: Re: Optional machine prefix for programs in for -B dirs, match ing Clang

[Michael Matz]

Hello,

On Wed, 4 Aug 2021, John Ericson wrote:

> > Doesn't GCC automatically look for those commands in the --prefix 
> > directory that you configure GCC with? Or is that only for native 
> > compilers?
> 
> It will search only if --with-*=... was not passed, and it will never 
> prefix the query. So yes in practice for cross compilers people do the 
> --with-* 

Hmm, no?  Because as you said ...

> I think the solution is to stop making cross compilers rely on these 
> --with-flags to do the obvious things. Executables like `collect2` 
> hidden within a libexesubdir (libexec/gcc/<machine>/<version>) have no 
> need for prefixing, but the assembler and linker are very much 
> public-facing executables in their own right, and usually are prefixed.

... the 'as' and 'ld' executables should be simply found within the 
version and target specific GCC libexecsubdir, possibly by being symlinks 
to whatever you want.  That's at least how my crosss are configured and 
installed, without any --with-{as,ld} options.

> and no searching happens, and for native compilers no one 
> bothers and searching does happen. But to be a pedant strictly speaking 
> the behavior is independent of whether the compiler is host == target or 
> not.


Ciao,
Michael.

Re: Optional machine prefix for programs in for -B dirs, match ing Clang

[John Ericson]

On Wed, Aug 4, 2021, at 10:48 AM, Michael Matz wrote:
> ... the 'as' and 'ld' executables should be simply found within the 
> version and target specific GCC libexecsubdir, possibly by being symlinks 
> to whatever you want.  That's at least how my crosss are configured and 
> installed, without any --with-{as,ld} options.

Yes that does work, and that's probably the best option today. I'm just a little wary of unprefixing things programmatically. 

For some context, this is NixOS where we assemble a ton of cross compilers automatically and each package gets its own isolated many FHS. For that reason I would like to eventually avoid the target-specific subdirs entirely, as I have the separate package trees to disambiguate things. Now, I know that exact same argument could also be used to say target prefixing is also superfluous, but eventually things on the PATH need to be disambiguated. There is no requirement that the libexec things be named like the bin things, but I sort of feel it's one less thing to remember and makes debugging easier.

I am sympathetic to the issue that if GCC accepts everything Clang does and vice-versa, we'll Postel's-law ourselves ourselves over time into madness as mistakes are accumulated rather than weeded out. But this one thing feels pretty innocuous to me, and it could also be made stricter by ensuring that we never *both* add the machine subdir *and* prefix the path --- i.e. at most 1 target-disambiguating method is used at a time.

I now have some patches for this change I suppose I could also submit.

Cheers,

John

Re: Optional machine prefix for programs in for -B dirs, match ing Clang

[Michael Matz]

Hello,

On Wed, 4 Aug 2021, John Ericson wrote:

> On Wed, Aug 4, 2021, at 10:48 AM, Michael Matz wrote:
> > ... the 'as' and 'ld' executables should be simply found within the 
> > version and target specific GCC libexecsubdir, possibly by being symlinks 
> > to whatever you want.  That's at least how my crosss are configured and 
> > installed, without any --with-{as,ld} options.
> 
> Yes that does work, and that's probably the best option today. I'm just 
> a little wary of unprefixing things programmatically.

The libexecsubdir _is_ the prefix in above case :)

> For some context, this is NixOS where we assemble a ton of cross 
> compilers automatically and each package gets its own isolated many FHS. 
> For that reason I would like to eventually avoid the target-specific 
> subdirs entirely, as I have the separate package trees to disambiguate 
> things. Now, I know that exact same argument could also be used to say 
> target prefixing is also superfluous, but eventually things on the PATH 
> need to be disambiguated.

Sure, which is why (e.g.) cross binutils do install with an arch prefix 
into ${bindir}.  But as GCC has the capability to look into libexecsubdir 
for binaries as well (which quite surely should never be in $PATH on any 
system), I don't see the conflict.

> There is no requirement that the libexec things be named like the bin 
> things, but I sort of feel it's one less thing to remember and makes 
> debugging easier.

Well, the naming scheme of binaries in libexecsubdir reflects the scheme 
that the compilers are using: cc1, cc1plus etc.  Not 
aarch64-unknown-linux-cc1.

> I am sympathetic to the issue that if GCC accepts everything Clang does 
> and vice-versa, we'll Postel's-law ourselves ourselves over time into 
> madness as mistakes are accumulated rather than weeded out.

Right.  I supposed it wouldn't hurt to also look for "${targettriple}-as" 
in $PATH before looking for 'as' (in $PATH).  But I don't think we can (or 
should) switch off looking for 'as' in libexecsubdir.  I don't even see 
why that behaviour should depend on an option, it could just be added by 
default.

> I now have some patches for this change I suppose I could also submit.

Even better :)


Ciao,
Michael.

Re: Optional machine prefix for programs in for -B dirs, match ing Clang

[John Ericson]

On Thu, Aug 5, 2021, at 8:30 AM, Michael Matz wrote:
> Hello,
> 
> On Wed, 4 Aug 2021, John Ericson wrote:
> 
> > On Wed, Aug 4, 2021, at 10:48 AM, Michael Matz wrote:
> > > ... the 'as' and 'ld' executables should be simply found within the 
> > > version and target specific GCC libexecsubdir, possibly by being symlinks 
> > > to whatever you want.  That's at least how my crosss are configured and 
> > > installed, without any --with-{as,ld} options.
> > 
> > Yes that does work, and that's probably the best option today. I'm just 
> > a little wary of unprefixing things programmatically.
> 
> The libexecsubdir _is_ the prefix in above case :)

Right. I meant stripping off the `cpu-vendor-os-` (conventionally) that ld and as are prefixed with. stripping off leading directories is easier.

> > For some context, this is NixOS where we assemble a ton of cross 
> > compilers automatically and each package gets its own isolated many FHS. 
> > For that reason I would like to eventually avoid the target-specific 
> > subdirs entirely, as I have the separate package trees to disambiguate 
> > things. Now, I know that exact same argument could also be used to say 
> > target prefixing is also superfluous, but eventually things on the PATH 
> > need to be disambiguated.
> 
> Sure, which is why (e.g.) cross binutils do install with an arch prefix 
> into ${bindir}.  But as GCC has the capability to look into libexecsubdir 
> for binaries as well (which quite surely should never be in $PATH on any 
> system), I don't see the conflict.

Yes there is no actual conflict. Our originally wrapper scripts may have been confused about this at some point but that's on us.

> 
> > There is no requirement that the libexec things be named like the bin 
> > things, but I sort of feel it's one less thing to remember and makes 
> > debugging easier.
> 
> Well, the naming scheme of binaries in libexecsubdir reflects the scheme 
> that the compilers are using: cc1, cc1plus etc.  Not 
> aarch64-unknown-linux-cc1.

Right.

> 
> > I am sympathetic to the issue that if GCC accepts everything Clang does 
> > and vice-versa, we'll Postel's-law ourselves ourselves over time into 
> > madness as mistakes are accumulated rather than weeded out.
> 
> Right.  I supposed it wouldn't hurt to also look for "${targettriple}-as" 
> in $PATH before looking for 'as' (in $PATH).  But I don't think we can (or 
> should) switch off looking for 'as' in libexecsubdir.  I don't even see 
> why that behaviour should depend on an option, it could just be added by 
> default.

OK I agree with that. so if someone passes -B$x, how about looking for

- $x/$machine/$version/$prog
- $x/$machine/$prog
- $x/$machine-prog
- $x/prog

so no prefixing in the subdir, only in the main dir?

($libexecsubdir is morally $libexec being a search dir + subdir IIRC)

> > I now have some patches for this change I suppose I could also submit.
> 
> Even better :)

Great!

I will continue improving my patch based on the above. In the meantime, I posted https://gcc.gnu.org/pipermail/gcc-patches/2021-August/576725.html which is a small cleanup that, while helping with my changes, doesn't change the behavior and I hope is good in any event.