[Bug driver/86030] specs file processing does not create response files for input directories

[Bug driver/86030] specs file processing does not create response files for input directories

[redi at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=86030

Jonathan Wakely <redi at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Last reconfirmed|2018-07-02 00:00:00         |2021-5-19

--- Comment #4 from Jonathan Wakely <redi at gcc dot gnu.org> ---
This is still a problem today: https://github.com/envoyproxy/envoy/issues/16196

[Bug driver/86030] specs file processing does not create response files for input directories

[costas.argyris at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=86030

Costas Argyris <costas.argyris at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |costas.argyris at gmail dot com

--- Comment #5 from Costas Argyris <costas.argyris at gmail dot com> ---
Very likely fixed by

https://gcc.gnu.org/git/?p=gcc.git;a=commitdiff;h=180ebb8a24d24fc5b105f2257d6216f6dfde62df

[Bug driver/86030] specs file processing does not create response files for input directories

[john.soo+gcc-bugzilla at arista dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=86030

John Soo <john.soo+gcc-bugzilla at arista dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |john.soo+gcc-bugzilla@arist
                   |                            |a.com

--- Comment #6 from John Soo <john.soo+gcc-bugzilla at arista dot com> ---
This is not a Windows-only bug, so I don't think it is fixed.

[Bug driver/86030] specs file processing does not create response files for input directories

[costas.argyris at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=86030

--- Comment #7 from Costas Argyris <costas.argyris at gmail dot com> ---
(In reply to John Soo from comment #6)
> This is not a Windows-only bug, so I don't think it is fixed.

Althought it is not mentioned explicitly in the title of this PR, the original
reporter did describe it as Windows-specific problem:

"GCC does not use response files for %D and %I handling (see do_spec_1), so if
you have a project which contrains a lot of Includes or library directories cc1
or collect2 respectively will overflow the commandline limit on Windows and
error out."

Also, it is typically Windows that suffers from this limitation of command line
length.

[Bug driver/86030] specs file processing does not create response files for input directories

[john.soo+gcc-bugzilla at arista dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=86030

--- Comment #8 from John Soo <john.soo+gcc-bugzilla at arista dot com> ---
> Also, it is typically Windows that suffers from this limitation of command line length.

Ok that may be true but I am effected by this on linux as are quite a few
others in this issue https://github.com/NixOS/nixpkgs/issues/41340

[Bug driver/86030] specs file processing does not create response files for input directories

[john.soo+gcc-bugzilla at arista dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=86030

--- Comment #9 from John Soo <john.soo+gcc-bugzilla at arista dot com> ---
Would a patch for unix doing something similar to
https://gcc.gnu.org/git/?p=gcc.git;a=commitdiff;h=180ebb8a24d24fc5b105f2257d6216f6dfde62df
be accepted? I am happy to start working on something like it but I have no gcc
contributions yet and would like to know ahead of time if it is desired.

[Bug driver/86030] specs file processing does not create response files for input directories

[john.soo+gcc-bugzilla at arista dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=86030

--- Comment #10 from John Soo <john.soo+gcc-bugzilla at arista dot com> ---
I'm also not sure
https://gcc.gnu.org/git/?p=gcc.git;a=commitdiff;h=180ebb8a24d24fc5b105f2257d6216f6dfde62df
fixes the collect bug because collect uses collect_execute instead of the pex_*
exec functions.

[Bug driver/86030] specs file processing does not create response files for input directories

[costas.argyris at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=86030

--- Comment #11 from Costas Argyris <costas.argyris at gmail dot com> ---
(In reply to John Soo from comment #10)
> I'm also not sure
> https://gcc.gnu.org/git/?p=gcc.git;a=commitdiff;
> h=180ebb8a24d24fc5b105f2257d6216f6dfde62df fixes the collect bug because
> collect uses collect_execute instead of the pex_* exec functions.

That patch was only meant to deal with the 32K command line length limitation
on Windows hosts, for programs spawned with the pex functions (to my knowledge,
this is how the driver spawns its sub-programs like the compiler, assembler and
linker).

In this PR though the problem appears to be something else which I am not
familiar with, so I don't know if its going to be helpful to see that patch as
a guide.   I only put the reference in the hopes that it solved this as well
because it looked similar, but it doesn't look like it.    You may need to come
up with a completely new solution for all I know.

[Bug driver/86030] specs file processing does not create response files for input directories

[john.soo+gcc-bugzilla at arista dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=86030

--- Comment #12 from John Soo <john.soo+gcc-bugzilla at arista dot com> ---
I think the general problem in that issue is that ARG_MAX is not respected when
the driver (or any subprocess) execs things on linux. I think that it is not
the same as the original issue here, though.

> I don't know if its going to be helpful to see that patch as a guide

Do you think using response files like in pex-win32.c in pex-unix.c could help?
I tried this out and it seems like this may not solve all the ARG_MAX problems.

[Bug driver/86030] specs file processing does not create response files for input directories

[costas.argyris at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=86030

--- Comment #13 from Costas Argyris <costas.argyris at gmail dot com> ---
(In reply to John Soo from comment #12)
> I think the general problem in that issue is that ARG_MAX is not respected
> when the driver (or any subprocess) execs things on linux. I think that it
> is not the same as the original issue here, though.
> 
> > I don't know if its going to be helpful to see that patch as a guide
> 
> Do you think using response files like in pex-win32.c in pex-unix.c could
> help? I tried this out and it seems like this may not solve all the ARG_MAX
> problems.

I really don't know because the problem might be happening even before it gets
to the point where the new subprocess is spawned.    Then, passing the command
line through a response file wouldn't help with anything, if the problem has
already happened by that point.

The Windows patch above just checks if the subprocess is about to be spawned
with a command line > 32K and if that is true it simply puts it in a temporary
response file and calls the subprocess with @rsp as the only arg, instead of
the large command line which would break it.    But obviously this makes the
assumption that the large command line is correct (nothing has been dropped or
otherwise messed up).

Here though it seems that you are dealing with another sort of limit which is
much larger (I have seen 128K being mentioned on the GH page).    If this
somehow corrupts the command line, it wouldn't help if that command line went
into a response file because it would still be wrong.    To my knowledge,
Linux-based systems don't have a command line length limitation, so I can't see
how a response file approach would be useful at the point where the subprocess
is spawned.    Whether something similar can be used at an earlier point to
save it from the 128K limit, whatever it is, is unknown to me.

[Bug driver/86030] specs file processing does not create response files for input directories

[john.soo+gcc-bugzilla at arista dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=86030

--- Comment #14 from John Soo <john.soo+gcc-bugzilla at arista dot com> ---
> Here though it seems that you are dealing with another sort of limit which is much larger (I have seen 128K being mentioned on the GH page).    If this somehow corrupts the command line, it wouldn't help if that command line went into a response file because it would still be wrong.    To my knowledge, Linux-based systems don't have a command line length limitation, so I can't see how a response file approach would be useful at the point where the subprocess is spawned.    Whether something similar can be used at an earlier point to save it from the 128K limit, whatever it is, is unknown to me.

It is a much larger limit (ARG_MAX resulting in E2BIG), but it is fundamentally
the same problem. I think we should assume that the command line is correct and
still respect ARG_MAX on linux/unix systems, too. It seems to me that the
temporary response file is the best way to do this.

[Bug driver/86030] specs file processing does not create response files for input directories

[john.soo+gcc-bugzilla at arista dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=86030

--- Comment #15 from John Soo <john.soo+gcc-bugzilla at arista dot com> ---
Just for some context on what limit is hit: under man sysconf you will find a
description of ARG_MAX
(https://www.man7.org/linux/man-pages/man3/sysconf.3.html)

> ARG_MAX - _SC_ARG_MAX
>     The maximum length of the arguments to the exec(3) family
>     of functions.  Must not be less than _POSIX_ARG_MAX
>     (4096).

Exceeding ARG_MAX will result in E2BIG
(https://man7.org/linux/man-pages/man3/errno.3.html).

> E2BIG  Argument list too long (POSIX.1-2001).

[Bug driver/86030] specs file processing does not create response files for input directories

[john.soo+gcc-bugzilla at arista dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=86030

--- Comment #16 from John Soo <john.soo+gcc-bugzilla at arista dot com> ---
It is actually somewhat likely that ARG_MAX will be hit when running on linux
because it is hit when the stack can't contain enough pointers to contain argv
and environ (see exec.c in the kernel
https://github.com/torvalds/linux/blob/master/fs/exec.c#L509).

The bad news is that response files can only mitigate the problem since environ
must also be small enough to fit into an execv* call.

Is there a reasonable way to keep only the env vars required to exec
subprocesses?

[Bug driver/86030] specs file processing does not create response files for input directories

[john.soo+gcc-bugzilla at arista dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=86030

--- Comment #17 from John Soo <john.soo+gcc-bugzilla at arista dot com> ---
Created attachment 55910
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=55910&action=edit
libiberty, Unix: pass argv over ARG_MAX through an @file

This does not handle environ being too large, but it is an adaptation of the
argv fix in pex-win32.c.

[Bug driver/86030] specs file processing does not create response files for input directories

[john.soo+gcc-bugzilla at arista dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=86030

--- Comment #18 from John Soo <john.soo+gcc-bugzilla at arista dot com> ---
And actually the proposed patch is not conservative enough, because the size of
the strings in argv/env also matter.

[Bug driver/86030] specs file processing does not create response files for input directories

[john.soo+gcc-bugzilla at arista dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=86030

--- Comment #19 from John Soo <john.soo+gcc-bugzilla at arista dot com> ---
I verified the proposed patch sent argv through @file, but COLLECT_GCC_OPTIONS
still caused E2BIG. In the failing execve, COLLECT_GCC_OPTIONS was 134227
characters long.

[Bug driver/86030] specs file processing does not create response files for input directories

[john.soo+gcc-bugzilla at arista dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=86030

--- Comment #20 from John Soo <john.soo+gcc-bugzilla at arista dot com> ---
I think that in order to really rid gcc of the E2BIG problem on linux,
COLLECT_*_OPTIONS will have to be deprecated and removed. This is particularly
a problem when executing spec files, it seems.