From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 14295 invoked by alias); 25 Jul 2012 08:08:12 -0000 Received: (qmail 14275 invoked by uid 22791); 25 Jul 2012 08:08:09 -0000 X-SWARE-Spam-Status: No, hits=-3.3 required=5.0 tests=AWL,BAYES_00,KHOP_THREADED,T_RP_MATCHES_RCVD X-Spam-Check-By: sourceware.org Received: from mx.meyering.net (HELO mx.meyering.net) (88.168.87.75) by sourceware.org (qpsmtpd/0.43rc1) with ESMTP; Wed, 25 Jul 2012 08:07:56 +0000 Received: from rho.meyering.net (rho.meyering.net [127.0.0.1]) by rho.meyering.net (Acme Bit-Twister) with ESMTP id 067A1600BB; Wed, 25 Jul 2012 10:07:54 +0200 (CEST) From: Jim Meyering To: Jason Merrill Cc: overseers@gcc.gnu.org Subject: Re: Multiple cvs/git mirror tasks? In-Reply-To: <500EE750.705@redhat.com> (Jason Merrill's message of "Tue, 24 Jul 2012 14:20:00 -0400") References: <50003CD7.8080401@redhat.com> <87bojjsm47.fsf@rho.meyering.net> <50045455.5090807@redhat.com> <87pq7q8kwp.fsf@rho.meyering.net> <500EE750.705@redhat.com> Date: Wed, 25 Jul 2012 08:08:00 -0000 Message-ID: <87zk6ofelh.fsf@rho.meyering.net> MIME-Version: 1.0 Content-Type: text/plain Mailing-List: contact overseers-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Archive: List-Post: List-Help: , Sender: overseers-owner@sourceware.org X-SW-Source: 2012-q3/txt/msg00038.txt.bz2 Jason Merrill wrote: > On 07/20/2012 12:19 PM, Jim Meyering wrote: >> At first glance, you may think there are three invocations of the script, >> but as recently discussed here, that's merely due to a subshelled >> pipeline in the script: each element of the pipeline ends up looking >> (in ps output) like another invocation of the parent script. It really >> was invoked only once. > > Right, but in the ps output I sent before, there were two gdb runs > (each with 3 processes) running at the same time. > >> I'm leery about adding naive cross-project locks, because that could >> let a busy project starve one that needs only a tiny update. >> >> Using lockfile(1) and a timeout might help. >> >> I could add a per-project locking mechanism to prevent e.g., two gdb >> syncs from running in parallel when the first one takes more than the >> 30m between cron invocations. That could save some system resources, >> but would not improve correctness or integrity. > > That makes sense to me. The times when it would run long are likely > to correspond with times when system resources are tightest, as with > the time I noticed. I've just adjusted the script (~meyering/bin/mirror-sw) to do that. [I did have qualms about doing this locking business in sh, and even implemented it first in Perl, mainly because using a diagnostic to distinguish the EEXIST case from other mkdir failure is so ugly. Finally I opted for this in-script sh code rather than a separate perl script.] Now it starts like this: ... repo=$1 # ============================================================= # Require a lock per repository, in case a sync operation # takes longer than the interval between cron job invocations. # It doesn't affect correctness when two of these jobs run concurrently, # but it is definitely wasted work and can contribute to performance problems. lock_dir=$HOME/mirror-git-to-cvs/$repo.lock export LC_ALL=C # we require an English diagnostic from mkdir err=$(mkdir $lock_dir 2>&1) if test $? = 0; then echo $$ > $lock_dir/pid else case $err in *exists) test -d $lock_dir || die "$lock_dir is not a directory?!?" pid=$(cat $lock_dir/pid) || die "no $lock_dir/pid file?!?" # See if PID is still valid. kill -1 $pid 2>/dev/null \ && die process $pid remains \ || { # process $pid is dead; steal lock echo $$ > $lock_dir/pid } ;; *) # mkdir failed; print the diagnostic we recorded die "$err" ;; esac fi # ============================================================= and cleans up at the end with "rm -r $lock_dir" FYI, since I've begun collecting sync-job duration data, the vast majority of them have finished in under 10 minutes. Here are the few (duration in seconds) that did not: 1132 803 797 704 684 671 658 657 656 646 631 609 607 602