From: Thomas Schwinge <thomas@codesourcery.com>
To: Jakub Jelinek <jakub@redhat.com>,
Rainer Orth <ro@CeBiTec.Uni-Bielefeld.DE>,
<gcc-patches@gcc.gnu.org>
Cc: Tobias Burnus <tobias@codesourcery.com>,
Mike Stump <mikestump@comcast.net>,
Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>,
Jan Hubicka <hubicka@ucw.cz>,
Segher Boessenkool <segher@kernel.crashing.org>
Subject: Support parallel testing in libgomp: fallback Perl 'flock' [PR66005]
Date: Tue, 16 May 2023 16:32:03 +0200 [thread overview]
Message-ID: <87o7mk8hzw.fsf@euler.schwinge.homeip.net> (raw)
In-Reply-To: <87ednvb1cc.fsf@euler.schwinge.homeip.net>
[-- Attachment #1: Type: text/plain, Size: 1873 bytes --]
Hi!
On 2023-05-05T10:59:31+0200, I wrote:
> On 2023-05-05T10:55:41+0200, I wrote:
>> [Putting Bernhard, Honza, Segher in CC, as they are eager to test this,
>> based on recent comments on IRC.] ;-P
>> First, establish the parallel testing infrastructure -- while still
>> hard-coding the number of parallel slots to one.
>> "Support parallel testing in libgomp, part I [PR66005]"
> On top of that, second, enable parallel testing.
> implemented what I'd described in
> <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=66005#c2>:
>
> | [...] parallelize *all* compilation, while just allowing for *one*
> | execution test job slot. That will require some GCC DejaGnu test
> | harness hackery which I've [now] gotten to look into. That is, enable
> | the usual GCC/DejaGnu parallel testing, but also have some kind of
> | mutex for the execution test invocation. This has to play nicely with
> | DejaGnu timeout handling, etc.
> Subject: [PATCH] Support parallel testing in libgomp, part II [PR66005]
>
> ..., and enable if 'flock' is available for serializing execution testing.
OK to push the attached
"Support parallel testing in libgomp: fallback Perl 'flock' [PR66005]"?
Per the PR66005 discussion, if 'flock' is not available, having a
fallback Perl 'flock' for parallelizing 'check-target-libgomp' wasn't met
with the greatest of all enthusiasm -- but in my opinion it's still
better than continued all-serial 'check-target-libgomp'?
We may then proceed working on a more integrated solution, using TCL or
shell features.
Grüße
Thomas
-----------------
Siemens Electronic Design Automation GmbH; Anschrift: Arnulfstraße 201, 80634 München; Gesellschaft mit beschränkter Haftung; Geschäftsführer: Thomas Heurung, Frank Thürauf; Sitz der Gesellschaft: München; Registergericht München, HRB 106955
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Support-parallel-testing-in-libgomp-fallback-Perl-fl.patch --]
[-- Type: text/x-diff, Size: 6458 bytes --]
From c62858bf888fec2f61febafcd6afe2dc8c3f679b Mon Sep 17 00:00:00 2001
From: Thomas Schwinge <thomas@codesourcery.com>
Date: Mon, 15 May 2023 20:00:07 +0200
Subject: [PATCH] Support parallel testing in libgomp: fallback Perl 'flock'
[PR66005]
Follow-up to commit 6c3b30ef9e0578509bdaf59c13da4a212fe6c2ba
"Support parallel testing in libgomp, part II [PR66005]"
("..., and enable if 'flock' is available for serializing execution testing"),
where we saw:
> On my Dell Precision 7530 laptop:
>
> $ uname -srvi
> Linux 5.15.0-71-generic #78-Ubuntu SMP Tue Apr 18 09:00:29 UTC 2023 x86_64
> $ grep '^model name' < /proc/cpuinfo | uniq -c
> 12 model name : Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz
> $ nvidia-smi -L
> GPU 0: Quadro P1000 (UUID: GPU-e043973b-b52a-d02b-c066-a8fdbf64e8ea)
>
> ... [...]: case (c) standard configuration, no offloading
> configured, [...]
> $ \time make check-target-libgomp
>
> Case (c), baseline; [...]:
>
> 1180.98user 110.80system 19:36.40elapsed 109%CPU (0avgtext+0avgdata 505148maxresident)k
> 1133.22user 111.08system 19:35.75elapsed 105%CPU (0avgtext+0avgdata 505212maxresident)k
>
> Case (c), parallelized [using 'flock']:
>
> [...]
> -j12 GCC_TEST_PARALLEL_SLOTS=12
> 2591.04user 192.64system 4:44.98elapsed 976%CPU (0avgtext+0avgdata 505216maxresident)k
> 2581.23user 195.21system 4:47.51elapsed 965%CPU (0avgtext+0avgdata 505212maxresident)k
Quite the same when instead of 'flock' using this fallback Perl 'flock':
2565.23user 194.35system 4:46.77elapsed 962%CPU (0avgtext+0avgdata 505216maxresident)k
2549.38user 200.20system 4:46.08elapsed 961%CPU (0avgtext+0avgdata 505216maxresident)k
PR testsuite/66005
gcc/
* doc/install.texi: Document (optional) Perl usage for parallel
testing of libgomp.
libgomp/
* testsuite/lib/libgomp.exp: 'flock' through stdout.
* testsuite/flock: New.
* configure.ac (FLOCK): Point to that if no 'flock' available, but
'perl' is.
* configure: Regenerate.
---
gcc/doc/install.texi | 3 +++
libgomp/configure | 42 +++++++++++++++++++++++++++++++
libgomp/configure.ac | 5 ++++
libgomp/testsuite/flock | 17 +++++++++++++
libgomp/testsuite/lib/libgomp.exp | 4 ++-
5 files changed, 70 insertions(+), 1 deletion(-)
create mode 100755 libgomp/testsuite/flock
diff --git a/gcc/doc/install.texi b/gcc/doc/install.texi
index dfab47dac96..fe4a972980f 100644
--- a/gcc/doc/install.texi
+++ b/gcc/doc/install.texi
@@ -382,6 +382,9 @@ tables.
Used by @command{automake}.
+If available, enables parallel testing of @samp{libgomp} in case that
+@command{flock} is not available.
+
@end table
Several support libraries are necessary to build GCC, some are required,
diff --git a/libgomp/configure b/libgomp/configure
index 2b45acd08c6..a280ca9238a 100755
--- a/libgomp/configure
+++ b/libgomp/configure
@@ -16457,6 +16457,8 @@ $as_echo "unable to detect (assuming 1)" >&6; }
fi
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for flock implementation" >&5
+$as_echo "$as_me: checking for flock implementation" >&6;}
for ac_prog in flock
do
# Extract the first word of "$ac_prog", so it can be a program name with args.
@@ -16499,6 +16501,46 @@ fi
test -n "$FLOCK" && break
done
+# Fallback if 'perl' is available.
+if test -z "$FLOCK"; then
+ # Extract the first word of "perl", so it can be a program name with args.
+set dummy perl; ac_word=$2
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
+$as_echo_n "checking for $ac_word... " >&6; }
+if ${ac_cv_prog_FLOCK+:} false; then :
+ $as_echo_n "(cached) " >&6
+else
+ if test -n "$FLOCK"; then
+ ac_cv_prog_FLOCK="$FLOCK" # Let the user override the test.
+else
+as_save_IFS=$IFS; IFS=$PATH_SEPARATOR
+for as_dir in $PATH
+do
+ IFS=$as_save_IFS
+ test -z "$as_dir" && as_dir=.
+ for ac_exec_ext in '' $ac_executable_extensions; do
+ if as_fn_executable_p "$as_dir/$ac_word$ac_exec_ext"; then
+ ac_cv_prog_FLOCK="$srcdir/testsuite/flock"
+ $as_echo "$as_me:${as_lineno-$LINENO}: found $as_dir/$ac_word$ac_exec_ext" >&5
+ break 2
+ fi
+done
+ done
+IFS=$as_save_IFS
+
+fi
+fi
+FLOCK=$ac_cv_prog_FLOCK
+if test -n "$FLOCK"; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: $FLOCK" >&5
+$as_echo "$FLOCK" >&6; }
+else
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+fi
+
+
+fi
# Get target configury.
. ${srcdir}/configure.tgt
diff --git a/libgomp/configure.ac b/libgomp/configure.ac
index 701573e3d99..2a7bf7c174a 100644
--- a/libgomp/configure.ac
+++ b/libgomp/configure.ac
@@ -349,7 +349,12 @@ fi
AX_COUNT_CPUS
AC_SUBST(CPU_COUNT)
+AC_MSG_NOTICE([checking for flock implementation])
AC_CHECK_PROGS(FLOCK, flock)
+# Fallback if 'perl' is available.
+if test -z "$FLOCK"; then
+ AC_CHECK_PROG(FLOCK, perl, $srcdir/testsuite/flock)
+fi
# Get target configury.
. ${srcdir}/configure.tgt
diff --git a/libgomp/testsuite/flock b/libgomp/testsuite/flock
new file mode 100755
index 00000000000..71878b104f1
--- /dev/null
+++ b/libgomp/testsuite/flock
@@ -0,0 +1,17 @@
+#!/usr/bin/env perl
+
+use strict;
+use warnings;
+
+# Only arguments '--exclusive 1' exactly are supported.
+(@ARGV == 2) or die;
+my $mode = shift;
+($mode eq "--exclusive") or die;
+my $fd = shift;
+($fd eq "1") or die;
+
+use Fcntl ':flock';
+
+open(my $fh, '>&=', 1) or die "open: $!";
+
+flock($fh, LOCK_EX) or die "flock: $!";
diff --git a/libgomp/testsuite/lib/libgomp.exp b/libgomp/testsuite/lib/libgomp.exp
index 3ad5503e287..2f9e538278f 100644
--- a/libgomp/testsuite/lib/libgomp.exp
+++ b/libgomp/testsuite/lib/libgomp.exp
@@ -338,12 +338,14 @@ if ![info exists ::env(GCC_RUNTEST_PARALLELIZE_DIR)] {
rename libgomp_load standard_libgomp_load
proc libgomp_load { program args } {
# ... in order to serialize execution testing via an exclusive lock.
+ # We use stdout, as per <https://perldoc.perl.org/functions/flock>
+ # "[...] FILEHANDLE [...] be open with write intent to use LOCK_EX".
set lock_file ../lock
set lock_kind --exclusive
set lock_fd [open $lock_file a+]
set lock_clock_begin [clock seconds]
global FLOCK
- exec $FLOCK $lock_kind 0 <@ $lock_fd
+ exec $FLOCK $lock_kind 1 >@ $lock_fd
set lock_clock_end [clock seconds]
verbose -log "Got ${FLOCK}('$lock_file', '$lock_kind') at [clock format $lock_clock_end] after [expr $lock_clock_end - $lock_clock_begin] s" 2
--
2.34.1
next prev parent reply other threads:[~2023-05-16 14:32 UTC|newest]
Thread overview: 12+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-05-07 11:27 [libgomp, testsuite] Support parallel testing in libgomp (PR libgomp/66005) Rainer Orth
2015-05-07 11:39 ` Jakub Jelinek
2015-05-07 18:07 ` Mike Stump
2015-05-08 8:40 ` Thomas Schwinge
2018-08-14 8:37 ` Martin Liška
2023-05-05 8:55 ` Support parallel testing in libgomp, part I [PR66005] Thomas Schwinge
2023-05-05 8:59 ` Support parallel testing in libgomp, part II [PR66005] Thomas Schwinge
2023-05-09 17:27 ` Bernhard Reutner-Fischer
2023-05-16 14:32 ` Thomas Schwinge [this message]
2023-05-06 14:15 ` Support parallel testing in libgomp, part I [PR66005] Bernhard Reutner-Fischer
2023-05-08 10:42 ` Thomas Schwinge
2023-05-08 17:52 ` Bernhard Reutner-Fischer
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87o7mk8hzw.fsf@euler.schwinge.homeip.net \
--to=thomas@codesourcery.com \
--cc=gcc-patches@gcc.gnu.org \
--cc=hubicka@ucw.cz \
--cc=jakub@redhat.com \
--cc=mikestump@comcast.net \
--cc=rep.dot.nop@gmail.com \
--cc=ro@CeBiTec.Uni-Bielefeld.DE \
--cc=segher@kernel.crashing.org \
--cc=tobias@codesourcery.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).