From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gnu.wildebeest.org (gnu.wildebeest.org [45.83.234.184]) by sourceware.org (Postfix) with ESMTPS id 536063858430 for ; Wed, 13 Apr 2022 18:51:16 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 536063858430 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=klomp.org Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=klomp.org Received: from reform (deer0x0d.wildebeest.org [172.31.17.143]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by gnu.wildebeest.org (Postfix) with ESMTPSA id 273CC30003C6; Wed, 13 Apr 2022 20:51:15 +0200 (CEST) Received: by reform (Postfix, from userid 1000) id DD93F2E80899; Wed, 13 Apr 2022 20:51:14 +0200 (CEST) Date: Wed, 13 Apr 2022 20:51:14 +0200 From: Mark Wielaard To: buildbot@builder.wildebeest.org Cc: elfutils-devel@sourceware.org Subject: Re: =?utf-8?B?4pigIEJ1aWxkYm90IChXaWxkZWJl?= =?utf-8?Q?est?= Builder): elfutils - failed test (failure) (master) Message-ID: References: <20220413172103.6380581A9D2@builder.wildebeest.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20220413172103.6380581A9D2@builder.wildebeest.org> X-Spam-Status: No, score=-1.5 required=5.0 tests=BAYES_00, KAM_BADIPHTTP, KAM_DMARC_STATUS, KAM_SHORT, NUMERIC_HTTP_ADDR, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE, WEIRD_PORT autolearn=no autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: elfutils-devel@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Elfutils-devel mailing list List-Unsubscribe: , List-Archive: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Apr 2022 18:51:18 -0000 Hi, On Wed, Apr 13, 2022 at 05:21:03PM +0000, buildbot@builder.wildebeest.org wrote: > A new failure has been detected on builder elfutils-centos-x86_64 while building elfutils. > > Full details are available at: > https://builder.wildebeest.org/buildbot/#builders/1/builds/932 > > Build state: failed test (failure) > Revision: 399b55a75830f1854c8da9f29282810e82f270b6 > Worker: centos-x86_64 > Build Reason: (unknown) > Blamelist: Mark Wielaard > > Steps: > [...] > - 8: make check ( failure ) > Logs: > - stdio: https://builder.wildebeest.org/buildbot/#builders/1/builds/932/steps/8/logs/stdio > - test-suite.log: https://builder.wildebeest.org/buildbot/#builders/1/builds/932/steps/8/logs/test-suite_log Hmmm, this seems a little random. The change was just adding some (unused) constants to dwarf.h. The log says: command timed out: 1200 seconds without output running ['make', 'check', '-j4'], attempting to kill process killed by signal 9 program finished with exit code -1 elapsedTime=1925.591587 It looks like run-debuginfod-federation-sqlite.sh is missing. Looking at the buildbot worker the end of run-debuginfod-federation-sqlite.sh.log is: + mvalue=2 + '[' -z 2 ']' + echo 'metric thread_work_total{role="groom"}: 2' metric thread_work_total{role="groom"}: 2 + '[' 2 -eq 2 ']' + break + '[' 18 -eq 0 ']' + curl -s http://127.0.0.1:9112/buildid/beefbeefbeefd00dd00d/debuginfo + curl -s http://127.0.0.1:9112/metrics + grep 'error_count.*sqlite' error_count{sqlite3="database disk image is malformed"} 6 error_count{sqlite3="file is encrypted or is not a database"} 1 + kill -INT 28184 28371 + wait 28184 28371 Which seems to correspond to this part in run-debuginfod-federation-sqlite.sh ######################################################################## # Corrupt the sqlite database and get debuginfod to trip across its errors curl -s http://127.0.0.1:$PORT1/metrics | grep 'sqlite3.*reset' dd if=/dev/zero of=$DB bs=1 count=1 # trigger some random activity that's Sure to get sqlite3 upset kill -USR1 $PID1 wait_ready $PORT1 'thread_work_total{role="traverse"}' 2 wait_ready $PORT1 'thread_work_pending{role="scan"}' 0 wait_ready $PORT1 'thread_busy{role="scan"}' 0 kill -USR2 $PID1 wait_ready $PORT1 'thread_work_total{role="groom"}' 2 curl -s http://127.0.0.1:$PORT1/buildid/beefbeefbeefd00dd00d/debuginfo > /dev/null || true curl -s http://127.0.0.1:$PORT1/metrics | grep 'error_count.*sqlite' # Run the tests again without the servers running. The target file should # be found in the cache. kill -INT $PID1 $PID2 wait $PID1 $PID2 So maybe corruptin the sqlite database prevents a proper shutdown of the debuginfod process? Cheers, Mark