From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from esa1.mentor.iphmx.com (esa1.mentor.iphmx.com [68.232.129.153]) by sourceware.org (Postfix) with ESMTPS id 0A74A3858C83 for ; Tue, 16 May 2023 14:32:11 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 0A74A3858C83 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=codesourcery.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=mentor.com X-IronPort-AV: E=Sophos;i="5.99,278,1677571200"; d="scan'208,223";a="6192739" Received: from orw-gwy-02-in.mentorg.com ([192.94.38.167]) by esa1.mentor.iphmx.com with ESMTP; 16 May 2023 06:32:10 -0800 IronPort-SDR: k6oJ5Qft8FBguNFUro7bj/T/bh9a5GRP6uotF4QW86MDxw4lj7yaLEWs8m/E6SYZ/xDV1baDxK fzxjCNJ8irVhP1Sd+pdub/KjjKoJFNV4i92V6DAWcIx3JEcleC6mC3NVvK4GMhziDhgjRsLym9 K42EK77jG9vxyJfiuB+XtCIaeUxW9xaq7jSPkgs9FlMKpY2W6johYYcbsfL6czEOV6DbVlqK5X zNNUO6qZh0M8QRLV0e+IXAfmNBAD+ISsM6jq+SCA6AM/mSHickgx4rKfQwGnbBG+2u4oy/nsYq aJ8= From: Thomas Schwinge To: Jakub Jelinek , Rainer Orth , CC: Tobias Burnus , Mike Stump , Bernhard Reutner-Fischer , Jan Hubicka , Segher Boessenkool Subject: Support parallel testing in libgomp: fallback Perl 'flock' [PR66005] In-Reply-To: <87ednvb1cc.fsf@euler.schwinge.homeip.net> References: <20150507113940.GI1751@tucnak.redhat.com> <87mw1fmpi5.fsf@schwinge.name> <87h6srb1iq.fsf@euler.schwinge.homeip.net> <87ednvb1cc.fsf@euler.schwinge.homeip.net> User-Agent: Notmuch/0.29.3+94~g74c3f1b (https://notmuchmail.org) Emacs/28.2 (x86_64-pc-linux-gnu) Date: Tue, 16 May 2023 16:32:03 +0200 Message-ID: <87o7mk8hzw.fsf@euler.schwinge.homeip.net> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Originating-IP: [137.202.0.90] X-ClientProxiedBy: svr-ies-mbx-12.mgc.mentorg.com (139.181.222.12) To svr-ies-mbx-10.mgc.mentorg.com (139.181.222.10) X-Spam-Status: No, score=-11.9 required=5.0 tests=BAYES_00,GIT_PATCH_0,HEADER_FROM_DIFFERENT_DOMAINS,KAM_DMARC_STATUS,KAM_SHORT,SPF_HELO_PASS,SPF_PASS,TXREP,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: --=-=-= Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hi! On 2023-05-05T10:59:31+0200, I wrote: > On 2023-05-05T10:55:41+0200, I wrote: >> [Putting Bernhard, Honza, Segher in CC, as they are eager to test this, >> based on recent comments on IRC.] ;-P >> First, establish the parallel testing infrastructure -- while still >> hard-coding the number of parallel slots to one. >> "Support parallel testing in libgomp, part I [PR66005]" > On top of that, second, enable parallel testing. > implemented what I'd described in > : > > | [...] parallelize *all* compilation, while just allowing for *one* > | execution test job slot. That will require some GCC DejaGnu test > | harness hackery which I've [now] gotten to look into. That is, enable > | the usual GCC/DejaGnu parallel testing, but also have some kind of > | mutex for the execution test invocation. This has to play nicely with > | DejaGnu timeout handling, etc. > Subject: [PATCH] Support parallel testing in libgomp, part II [PR66005] > > ..., and enable if 'flock' is available for serializing execution testing= . OK to push the attached "Support parallel testing in libgomp: fallback Perl 'flock' [PR66005]"? Per the PR66005 discussion, if 'flock' is not available, having a fallback Perl 'flock' for parallelizing 'check-target-libgomp' wasn't met with the greatest of all enthusiasm -- but in my opinion it's still better than continued all-serial 'check-target-libgomp'? We may then proceed working on a more integrated solution, using TCL or shell features. Gr=C3=BC=C3=9Fe Thomas ----------------- Siemens Electronic Design Automation GmbH; Anschrift: Arnulfstra=C3=9Fe 201= , 80634 M=C3=BCnchen; Gesellschaft mit beschr=C3=A4nkter Haftung; Gesch=C3= =A4ftsf=C3=BChrer: Thomas Heurung, Frank Th=C3=BCrauf; Sitz der Gesellschaf= t: M=C3=BCnchen; Registergericht M=C3=BCnchen, HRB 106955 --=-=-= Content-Type: text/x-diff Content-Disposition: inline; filename="0001-Support-parallel-testing-in-libgomp-fallback-Perl-fl.patch" >From c62858bf888fec2f61febafcd6afe2dc8c3f679b Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Mon, 15 May 2023 20:00:07 +0200 Subject: [PATCH] Support parallel testing in libgomp: fallback Perl 'flock' [PR66005] Follow-up to commit 6c3b30ef9e0578509bdaf59c13da4a212fe6c2ba "Support parallel testing in libgomp, part II [PR66005]" ("..., and enable if 'flock' is available for serializing execution testing"), where we saw: > On my Dell Precision 7530 laptop: > > $ uname -srvi > Linux 5.15.0-71-generic #78-Ubuntu SMP Tue Apr 18 09:00:29 UTC 2023 x86_64 > $ grep '^model name' < /proc/cpuinfo | uniq -c > 12 model name : Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz > $ nvidia-smi -L > GPU 0: Quadro P1000 (UUID: GPU-e043973b-b52a-d02b-c066-a8fdbf64e8ea) > > ... [...]: case (c) standard configuration, no offloading > configured, [...] > $ \time make check-target-libgomp > > Case (c), baseline; [...]: > > 1180.98user 110.80system 19:36.40elapsed 109%CPU (0avgtext+0avgdata 505148maxresident)k > 1133.22user 111.08system 19:35.75elapsed 105%CPU (0avgtext+0avgdata 505212maxresident)k > > Case (c), parallelized [using 'flock']: > > [...] > -j12 GCC_TEST_PARALLEL_SLOTS=12 > 2591.04user 192.64system 4:44.98elapsed 976%CPU (0avgtext+0avgdata 505216maxresident)k > 2581.23user 195.21system 4:47.51elapsed 965%CPU (0avgtext+0avgdata 505212maxresident)k Quite the same when instead of 'flock' using this fallback Perl 'flock': 2565.23user 194.35system 4:46.77elapsed 962%CPU (0avgtext+0avgdata 505216maxresident)k 2549.38user 200.20system 4:46.08elapsed 961%CPU (0avgtext+0avgdata 505216maxresident)k PR testsuite/66005 gcc/ * doc/install.texi: Document (optional) Perl usage for parallel testing of libgomp. libgomp/ * testsuite/lib/libgomp.exp: 'flock' through stdout. * testsuite/flock: New. * configure.ac (FLOCK): Point to that if no 'flock' available, but 'perl' is. * configure: Regenerate. --- gcc/doc/install.texi | 3 +++ libgomp/configure | 42 +++++++++++++++++++++++++++++++ libgomp/configure.ac | 5 ++++ libgomp/testsuite/flock | 17 +++++++++++++ libgomp/testsuite/lib/libgomp.exp | 4 ++- 5 files changed, 70 insertions(+), 1 deletion(-) create mode 100755 libgomp/testsuite/flock diff --git a/gcc/doc/install.texi b/gcc/doc/install.texi index dfab47dac96..fe4a972980f 100644 --- a/gcc/doc/install.texi +++ b/gcc/doc/install.texi @@ -382,6 +382,9 @@ tables. Used by @command{automake}. +If available, enables parallel testing of @samp{libgomp} in case that +@command{flock} is not available. + @end table Several support libraries are necessary to build GCC, some are required, diff --git a/libgomp/configure b/libgomp/configure index 2b45acd08c6..a280ca9238a 100755 --- a/libgomp/configure +++ b/libgomp/configure @@ -16457,6 +16457,8 @@ $as_echo "unable to detect (assuming 1)" >&6; } fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for flock implementation" >&5 +$as_echo "$as_me: checking for flock implementation" >&6;} for ac_prog in flock do # Extract the first word of "$ac_prog", so it can be a program name with args. @@ -16499,6 +16501,46 @@ fi test -n "$FLOCK" && break done +# Fallback if 'perl' is available. +if test -z "$FLOCK"; then + # Extract the first word of "perl", so it can be a program name with args. +set dummy perl; ac_word=$2 +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5 +$as_echo_n "checking for $ac_word... " >&6; } +if ${ac_cv_prog_FLOCK+:} false; then : + $as_echo_n "(cached) " >&6 +else + if test -n "$FLOCK"; then + ac_cv_prog_FLOCK="$FLOCK" # Let the user override the test. +else +as_save_IFS=$IFS; IFS=$PATH_SEPARATOR +for as_dir in $PATH +do + IFS=$as_save_IFS + test -z "$as_dir" && as_dir=. + for ac_exec_ext in '' $ac_executable_extensions; do + if as_fn_executable_p "$as_dir/$ac_word$ac_exec_ext"; then + ac_cv_prog_FLOCK="$srcdir/testsuite/flock" + $as_echo "$as_me:${as_lineno-$LINENO}: found $as_dir/$ac_word$ac_exec_ext" >&5 + break 2 + fi +done + done +IFS=$as_save_IFS + +fi +fi +FLOCK=$ac_cv_prog_FLOCK +if test -n "$FLOCK"; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: $FLOCK" >&5 +$as_echo "$FLOCK" >&6; } +else + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } +fi + + +fi # Get target configury. . ${srcdir}/configure.tgt diff --git a/libgomp/configure.ac b/libgomp/configure.ac index 701573e3d99..2a7bf7c174a 100644 --- a/libgomp/configure.ac +++ b/libgomp/configure.ac @@ -349,7 +349,12 @@ fi AX_COUNT_CPUS AC_SUBST(CPU_COUNT) +AC_MSG_NOTICE([checking for flock implementation]) AC_CHECK_PROGS(FLOCK, flock) +# Fallback if 'perl' is available. +if test -z "$FLOCK"; then + AC_CHECK_PROG(FLOCK, perl, $srcdir/testsuite/flock) +fi # Get target configury. . ${srcdir}/configure.tgt diff --git a/libgomp/testsuite/flock b/libgomp/testsuite/flock new file mode 100755 index 00000000000..71878b104f1 --- /dev/null +++ b/libgomp/testsuite/flock @@ -0,0 +1,17 @@ +#!/usr/bin/env perl + +use strict; +use warnings; + +# Only arguments '--exclusive 1' exactly are supported. +(@ARGV == 2) or die; +my $mode = shift; +($mode eq "--exclusive") or die; +my $fd = shift; +($fd eq "1") or die; + +use Fcntl ':flock'; + +open(my $fh, '>&=', 1) or die "open: $!"; + +flock($fh, LOCK_EX) or die "flock: $!"; diff --git a/libgomp/testsuite/lib/libgomp.exp b/libgomp/testsuite/lib/libgomp.exp index 3ad5503e287..2f9e538278f 100644 --- a/libgomp/testsuite/lib/libgomp.exp +++ b/libgomp/testsuite/lib/libgomp.exp @@ -338,12 +338,14 @@ if ![info exists ::env(GCC_RUNTEST_PARALLELIZE_DIR)] { rename libgomp_load standard_libgomp_load proc libgomp_load { program args } { # ... in order to serialize execution testing via an exclusive lock. + # We use stdout, as per + # "[...] FILEHANDLE [...] be open with write intent to use LOCK_EX". set lock_file ../lock set lock_kind --exclusive set lock_fd [open $lock_file a+] set lock_clock_begin [clock seconds] global FLOCK - exec $FLOCK $lock_kind 0 <@ $lock_fd + exec $FLOCK $lock_kind 1 >@ $lock_fd set lock_clock_end [clock seconds] verbose -log "Got ${FLOCK}('$lock_file', '$lock_kind') at [clock format $lock_clock_end] after [expr $lock_clock_end - $lock_clock_begin] s" 2 -- 2.34.1 --=-=-=--