[PATCH v2 0/2] Fix run-time handling of `libgcc_s' in testing

[PATCH v2 0/2] Fix run-time handling of `libgcc_s' in testing

[Maciej W. Rozycki]

Hi,

 This is a follow-up to the original proposal of a change posted here:

<https://gcc.gnu.org/ml/gcc-patches/2019-11/msg00756.html>

to address a catastrophic libgo testsuite failure in cross-compilation 
where shared `libgcc_s' library cannot be found by the loader at run time 
in build-tree testing and consequently no test case executes.

 In the course of discussion it has turned out that the culprit is in a 
generic GCC test framework helper and in an attempt to handle this I have 
come across a nonsensical warning message produced by the GCC driver 
interfering with the solution I have come up with.

 Consequently I have prepared this small patch series where in 1/2 I 
propose to drop the warning message in the affected case and then in 2/2 I 
address the actual problem.  See individual change descriptions for 
details.

 These two changes have been regression-tested with `make check' using the 
`x86_64-linux-gnu' native system as well as the `x86_64-linux-gnu' host 
and the `riscv64-linux-gnu' target, with RISC-V QEMU in the Linux user 
emulation mode as the target board.

 OK to apply?

  Maciej

[PATCH v2 1/2] driver: Do not warn about ineffective `-x' option if no inputs were given

[Maciej W. Rozycki]

Fix an issue with the GCC driver and the `-x' option where a warning is 
issued in an invocation like:

$ riscv64-linux-gnu-gcc -print-multi-directory -x c++
riscv64-linux-gnu-gcc: warning: '-x c++' after last input file has no effect
lib64/lp64d
$ 

where no inputs were given and hence the use of `-x' is irrelevant.  
The statement printed is also untrue as the `-x' does not come after the 
last input file given that none was given.  Do not print it then if no 
inputs were supplied.

	gcc/
	* gcc.c (process_command): Only warn about an ineffective `-x' 
	option if any input files have actually been supplied.
---
Hi,

 This warning interferes with 2/2 and libgomp testing where `-x c++' is 
included with compiler invocation for a subset of tests.  This could 
probably be dealt with in the test suite, but I have concluded that the 
warning makes no sense in the first place and would better be simply 
removed instead.

  Maciej

New change in v2.
---
 gcc/gcc.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

gcc-spec-lang-warn-no-input.diff
Index: gcc/gcc/gcc.c
===================================================================
--- gcc.orig/gcc/gcc.c
+++ gcc/gcc/gcc.c
@@ -4741,7 +4741,7 @@ process_command (unsigned int decoded_op
   /* More prefixes are enabled in main, after we read the specs file
      and determine whether this is cross-compilation or not.  */
 
-  if (n_infiles == last_language_n_infiles && spec_lang != 0)
+  if (n_infiles != 0 && n_infiles == last_language_n_infiles && spec_lang != 0)
     warning (0, "%<-x %s%> after last input file has no effect", spec_lang);
 
   /* Synthesize -fcompare-debug flag from the GCC_COMPARE_DEBUG

Re: [PATCH v2 1/2] driver: Do not warn about ineffective `-x' option if no inputs were given

[Joseph Myers]

On Fri, 29 Nov 2019, Maciej W. Rozycki wrote:

> Fix an issue with the GCC driver and the `-x' option where a warning is 
> issued in an invocation like:
> 
> $ riscv64-linux-gnu-gcc -print-multi-directory -x c++
> riscv64-linux-gnu-gcc: warning: '-x c++' after last input file has no effect
> lib64/lp64d
> $ 
> 
> where no inputs were given and hence the use of `-x' is irrelevant.  
> The statement printed is also untrue as the `-x' does not come after the 
> last input file given that none was given.  Do not print it then if no 
> inputs were supplied.
> 
> 	gcc/
> 	* gcc.c (process_command): Only warn about an ineffective `-x' 
> 	option if any input files have actually been supplied.

OK.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: [PATCH v2 1/2] driver: Do not warn about ineffective `-x' option if no inputs were given

[Maciej W. Rozycki]

On Fri, 29 Nov 2019, Joseph Myers wrote:

> > 	* gcc.c (process_command): Only warn about an ineffective `-x' 
> > 	option if any input files have actually been supplied.
> 
> OK.

 Applied, thanks for your review.

  Maciej

[PATCH v2 2/2] testsuite: Fix run-time tracking down of `libgcc_s'

[Maciej W. Rozycki]

Fix a catastrophic libgo testsuite failure in cross-compilation where 
the shared `libgcc_s' library cannot be found by the loader at run time 
in build-tree testing and consequently all test cases fail the execution 
stage, giving output (here with the `x86_64-linux-gnu' host and the 
`riscv64-linux-gnu' target, with RISC-V QEMU in the Linux user emulation 
mode as the target board) like:

spawn qemu-riscv64 -E LD_LIBRARY_PATH=.:.../riscv64-linux-gnu/lib64/lp64d/libgo/.libs ./a.exe
./a.exe: error while loading shared libraries: libgcc_s.so.1: cannot open shared object file: No such file or directory
FAIL: archive/tar

To do so rework `gcc-set-multilib-library-path' so as not to rely on the 
`rootme' TCL variable to have been preset in testsuite invocation, which 
only works for the GCC test suites and not for library test suites, and 
also use `remote_exec host' rather than `exec' to invoke the compiler in 
determination of `libgcc_s' locations, so that the solution works in 
remote testing as well while also avoiding the hardcoded limit of the 
executable's path length imposed by `exec'.

This is based on an observation that the multilib root directory can be 
determined by stripping out the multilib directory in effect as printed 
with the `-print-multi-directory' option from the path produced by the 
`-print-file-name=' option.  And then individual full multilib paths can 
be assembled for the other multilibs by appending their respective 
multilib directories to the multilib root directory.

Unlike with the old solution the full multilib paths are not checked for 
the presence of the shared `libgcc_s' library there, but that is 
supposed to be harmless.  Also the full multilib path for the multilib 
used with the compiler used for testing will now come first, which 
should reduce run-time processing in the usual case.

With this change in place test output instead looks like:

spawn qemu-riscv64 -E LD_LIBRARY_PATH=.:.../riscv64-linux-gnu/lib64/lp64d/libgo/.libs:..././gcc/lib64/lp64d:..././gcc/.:..././gcc/lib32/ilp32:..././gcc/lib32/ilp32d:..././gcc/lib64/lp64 ./a.exe
PASS
PASS: archive/tar

No summary comparison, because the libgo testsuite does not provide one 
in this configuration for some reason, however this change improves 
overall results from 0 PASSes and 159 FAILs to 133 PASSes and 26 FAILs.

	gcc/testsuite/
	* lib/gcc-defs.exp (gcc-set-multilib-library-path): Use 
	`-print-file-name=' to determine the multilib root directory.  
	Use `remote_exec host' rather than `exec' to invoke the 
	compiler.
---
Hi,

 As PR testsuite/40699, PR testsuite/40707 and PR testsuite/40709 and 
resulting r149508, a revert of r149113 ("Tidy up testsuite handling of 
LD_LIBRARY_PATH"), 
<https://gcc.gnu.org/ml/gcc-patches/2009-06/msg00151.html> and some other 
commits, have indicated non-selected multilib directories have to be 
included in the dynamic loader's library path for some targets for some 
reason.  So I have decided to preserve the approach, even though it 
appears odd to me.

  Maciej

Changes from v1:

- Resolve the issue globally in `gcc-set-multilib-library-path' in 
  gcc-defs.exp rather than for libgo only in `go_link_flags' in go.exp.
---
 gcc/testsuite/lib/gcc-defs.exp |   43 +++++++++++++++++++++++++++--------------
 1 file changed, 29 insertions(+), 14 deletions(-)

Index: gcc/gcc/testsuite/lib/gcc-defs.exp
===================================================================
--- gcc.orig/gcc/testsuite/lib/gcc-defs.exp
+++ gcc/gcc/testsuite/lib/gcc-defs.exp
@@ -324,29 +324,44 @@ proc dg-additional-files-options { optio
 # for COMPILER, including multilib directories.
 
 proc gcc-set-multilib-library-path { compiler } {
-    global rootme
+    set shlib_ext [get_shlib_extension]
+    set options [lrange $compiler 1 end]
+    set compiler [lindex $compiler 0]
 
-    # ??? rootme will not be set when testing an installed compiler.
-    # In that case, we should perhaps use some other method to find
-    # libraries.
-    if {![info exists rootme]} {
+    set libgcc_s_x [remote_exec host "$compiler" \
+		    "$options -print-file-name=libgcc_s.${shlib_ext}"]
+    if { [lindex $libgcc_s_x 0] == 0 \
+	 && [set libgcc_s_dir [file dirname [lindex $libgcc_s_x 1]]] != "" } {
+	set libpath ":${libgcc_s_dir}"
+    } else {
 	return ""
     }
 
-    set libpath ":${rootme}"
-    set options [lrange $compiler 1 end]
-    set compiler [lindex $compiler 0]
-    if { [is_remote host] == 0 && [which $compiler] != 0 } {
-	foreach i "[eval exec $compiler $options --print-multi-lib]" {
+    set multi_dir_x [remote_exec host "$compiler" \
+		     "$options -print-multi-directory"]
+    set multi_lib_x [remote_exec host "$compiler" \
+		     "$options -print-multi-lib"]
+    if { [lindex $multi_dir_x 0] == 0 && [lindex $multi_lib_x 0] == 0 } {
+	set multi_dir [string trim [lindex $multi_dir_x 1]]
+	set multi_lib [string trim [lindex $multi_lib_x 1]]
+	if { "$multi_dir" == "." } {
+	    set multi_root "$libgcc_s_dir"
+	} else {
+	    set multi_match [string last "/$multi_dir" "$libgcc_s_dir"]
+	    if { "$multi_match" < 0 } {
+		return $libpath
+	    }
+	    set multi_root [string range "$libgcc_s_dir" \
+			    0 [expr $multi_match - 1]]
+	}
+	foreach i "$multi_lib" {
 	    set mldir ""
 	    regexp -- "\[a-z0-9=_/\.-\]*;" $i mldir
 	    set mldir [string trimright $mldir "\;@"]
-	    if { "$mldir" == "." } {
+	    if { "$mldir" == "$multi_dir" } {
 		continue
 	    }
-	    if { [llength [glob -nocomplain ${rootme}/${mldir}/libgcc_s*.so.*]] >= 1 } {
-		append libpath ":${rootme}/${mldir}"
-	    }
+	    append libpath ":${multi_root}/${mldir}"
 	}
     }
 

Re: [PATCH v2 2/2] testsuite: Fix run-time tracking down of `libgcc_s'

[Ian Lance Taylor]

On Fri, Nov 29, 2019 at 1:09 AM Maciej W. Rozycki <macro@wdc.com> wrote:
>
> Fix a catastrophic libgo testsuite failure in cross-compilation where
> the shared `libgcc_s' library cannot be found by the loader at run time
> in build-tree testing and consequently all test cases fail the execution
> stage, giving output (here with the `x86_64-linux-gnu' host and the
> `riscv64-linux-gnu' target, with RISC-V QEMU in the Linux user emulation
> mode as the target board) like:
>
> spawn qemu-riscv64 -E LD_LIBRARY_PATH=.:.../riscv64-linux-gnu/lib64/lp64d/libgo/.libs ./a.exe
> ./a.exe: error while loading shared libraries: libgcc_s.so.1: cannot open shared object file: No such file or directory
> FAIL: archive/tar
>
> To do so rework `gcc-set-multilib-library-path' so as not to rely on the
> `rootme' TCL variable to have been preset in testsuite invocation, which
> only works for the GCC test suites and not for library test suites, and
> also use `remote_exec host' rather than `exec' to invoke the compiler in
> determination of `libgcc_s' locations, so that the solution works in
> remote testing as well while also avoiding the hardcoded limit of the
> executable's path length imposed by `exec'.
>
> This is based on an observation that the multilib root directory can be
> determined by stripping out the multilib directory in effect as printed
> with the `-print-multi-directory' option from the path produced by the
> `-print-file-name=' option.  And then individual full multilib paths can
> be assembled for the other multilibs by appending their respective
> multilib directories to the multilib root directory.
>
> Unlike with the old solution the full multilib paths are not checked for
> the presence of the shared `libgcc_s' library there, but that is
> supposed to be harmless.  Also the full multilib path for the multilib
> used with the compiler used for testing will now come first, which
> should reduce run-time processing in the usual case.
>
> With this change in place test output instead looks like:
>
> spawn qemu-riscv64 -E LD_LIBRARY_PATH=.:.../riscv64-linux-gnu/lib64/lp64d/libgo/.libs:..././gcc/lib64/lp64d:..././gcc/.:..././gcc/lib32/ilp32:..././gcc/lib32/ilp32d:..././gcc/lib64/lp64 ./a.exe
> PASS
> PASS: archive/tar
>
> No summary comparison, because the libgo testsuite does not provide one
> in this configuration for some reason, however this change improves
> overall results from 0 PASSes and 159 FAILs to 133 PASSes and 26 FAILs.
>
>         gcc/testsuite/
>         * lib/gcc-defs.exp (gcc-set-multilib-library-path): Use
>         `-print-file-name=' to determine the multilib root directory.
>         Use `remote_exec host' rather than `exec' to invoke the
>         compiler.

This patch is for better Go support but the patch is to generic code.
The patch is fine with me but should ideally be reviewed by a
testsuite maintainer.

Ian

Re: [PATCH v2 2/2] testsuite: Fix run-time tracking down of `libgcc_s'

[Maciej W. Rozycki]

On Mon, 2 Dec 2019, Ian Lance Taylor wrote:

> >         gcc/testsuite/
> >         * lib/gcc-defs.exp (gcc-set-multilib-library-path): Use
> >         `-print-file-name=' to determine the multilib root directory.
> >         Use `remote_exec host' rather than `exec' to invoke the
> >         compiler.
> 
> This patch is for better Go support but the patch is to generic code.
> The patch is fine with me but should ideally be reviewed by a
> testsuite maintainer.

 I agree.  I have cc-ed you to keep you in the circle and I have added the 
testsuite people now too, but ultimately I think it's Richard who might be 
the best person to look into it as he's been highly skilled in this area 
and he has also dealt with this particular piece in the past.

 Richard, would you be able to find a few cycles and be kind enough to 
look into it?

 I'm somewhat surprised it's only libgo that suffers from the lack of 
run-time library paths in my setup where a random copy of `libgcc_s' is 
not already present in the environment, but maybe no other target library 
test suite pulls `libgcc_s' with any of its tests.

  Maciej

[PING][PATCH v2 2/2] testsuite: Fix run-time tracking down of `libgcc_s'

[Maciej W. Rozycki]

On Fri, 29 Nov 2019, Maciej W. Rozycki wrote:

> Fix a catastrophic libgo testsuite failure in cross-compilation where 
> the shared `libgcc_s' library cannot be found by the loader at run time 
> in build-tree testing and consequently all test cases fail the execution 
> stage, giving output (here with the `x86_64-linux-gnu' host and the 
> `riscv64-linux-gnu' target, with RISC-V QEMU in the Linux user emulation 
> mode as the target board) like:
> 
> spawn qemu-riscv64 -E LD_LIBRARY_PATH=.:.../riscv64-linux-gnu/lib64/lp64d/libgo/.libs ./a.exe
> ./a.exe: error while loading shared libraries: libgcc_s.so.1: cannot open shared object file: No such file or directory
> FAIL: archive/tar

 Ping for:

<https://gcc.gnu.org/ml/gcc-patches/2019-11/msg02610.html>

  Maciej

Re: [PING][PATCH v2 2/2] testsuite: Fix run-time tracking down of `libgcc_s'

[Mike Stump]

On Dec 9, 2019, at 1:30 PM, Maciej W. Rozycki <macro@wdc.com> wrote:
> 
> On Fri, 29 Nov 2019, Maciej W. Rozycki wrote:
> 
>> Fix a catastrophic libgo testsuite failure in cross-compilation where 
>> the shared `libgcc_s' library cannot be found by the loader at run time 
>> in build-tree testing and consequently all test cases fail the execution 
>> stage, giving output (here with the `x86_64-linux-gnu' host and the 
>> `riscv64-linux-gnu' target, with RISC-V QEMU in the Linux user emulation 
>> mode as the target board) like:
>> 
>> spawn qemu-riscv64 -E LD_LIBRARY_PATH=.:.../riscv64-linux-gnu/lib64/lp64d/libgo/.libs ./a.exe
>> ./a.exe: error while loading shared libraries: libgcc_s.so.1: cannot open shared object file: No such file or directory
>> FAIL: archive/tar
> 
> Ping for:
> 
> <https://gcc.gnu.org/ml/gcc-patches/2019-11/msg02610.html>

Ok.

Watch for any breakage.

Re: [PING][PATCH v2 2/2] testsuite: Fix run-time tracking down of `libgcc_s'

[Maciej W. Rozycki]

On Fri, 20 Dec 2019, Mike Stump wrote:

> >> Fix a catastrophic libgo testsuite failure in cross-compilation where 
> >> the shared `libgcc_s' library cannot be found by the loader at run time 
> >> in build-tree testing and consequently all test cases fail the execution 
> >> stage, giving output (here with the `x86_64-linux-gnu' host and the 
> >> `riscv64-linux-gnu' target, with RISC-V QEMU in the Linux user emulation 
> >> mode as the target board) like:
> >> 
> >> spawn qemu-riscv64 -E LD_LIBRARY_PATH=.:.../riscv64-linux-gnu/lib64/lp64d/libgo/.libs ./a.exe
> >> ./a.exe: error while loading shared libraries: libgcc_s.so.1: cannot open shared object file: No such file or directory
> >> FAIL: archive/tar
> > 
> > Ping for:
> > 
> > <https://gcc.gnu.org/ml/gcc-patches/2019-11/msg02610.html>
> 
> Ok.

 Applied now, thanks for your review.

> Watch for any breakage.

 Will do.

  Maciej

[PING^2][PATCH v2 2/2] testsuite: Fix run-time tracking down of `libgcc_s'

[Maciej W. Rozycki]

On Fri, 29 Nov 2019, Maciej W. Rozycki wrote:

> Fix a catastrophic libgo testsuite failure in cross-compilation where 
> the shared `libgcc_s' library cannot be found by the loader at run time 
> in build-tree testing and consequently all test cases fail the execution 
> stage, giving output (here with the `x86_64-linux-gnu' host and the 
> `riscv64-linux-gnu' target, with RISC-V QEMU in the Linux user emulation 
> mode as the target board) like:
> 
> spawn qemu-riscv64 -E LD_LIBRARY_PATH=.:.../riscv64-linux-gnu/lib64/lp64d/libgo/.libs ./a.exe
> ./a.exe: error while loading shared libraries: libgcc_s.so.1: cannot open shared object file: No such file or directory
> FAIL: archive/tar

 Ping for:

<https://gcc.gnu.org/ml/gcc-patches/2019-11/msg02610.html>

  Maciej