From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by sourceware.org (Postfix) with ESMTPS id 1588D3858D28 for ; Tue, 16 Jan 2024 16:15:23 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 1588D3858D28 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=redhat.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 1588D3858D28 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1705421725; cv=none; b=kEMRdGagxA4w+wkjj7DaOcuKl0i7eCeCo4+vywIhXIbwgUTKeKZWToRVvqcilTv54tdbCa3EoGKSNlbtZJKq9bLgE4geK+sdgDh++ogjhOqo1I4HwLdRULOXPzcCapVO8kUbn+2loh9JkoafqCqcEpZhkZv6v7cUEbx3dgSICrU= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1705421725; c=relaxed/simple; bh=biLw5v36kIjS7FXDV7cl5LR70IbQEtd3XXq8MIKWWrY=; h=DKIM-Signature:Date:From:To:Subject:Message-ID:MIME-Version; b=cPZgnmyA3w22CYtnj+bVWjdWWjTFoHXKZSsF4c9LvrFN2BwCERSXeG6ial7REmDpygttPE+L/xfkTA1Fxe6MVzOHZrzHPjykd7HsU4tQkQ47tdKYiCxNdBCGkX2Crq9Hgt2NKfccLuxcnFgV3qiqytD2DFqCa1m1s7W5ej7u8M4= ARC-Authentication-Results: i=1; server2.sourceware.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1705421723; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=wxYucHwW39qxz+WMmYtnJL3ITZwIagfutLs0gJ/RB8w=; b=PbfxfAcWhvDxC5mB9Q5SwHrfKm4PB4kASUgz3PXivpGK5b7zlmREJp9cWGAMdIQs4Ca8jo 1VVSMz8ZRY/TBFn3zZ0uGGqtZ96bHzK1VtA1qjQwAg4Y7C2xeluS66Wopi9MTjMzzJ87rz hEO1BVoZf0zFQ+oeVoQI8gzhhdpBUdY= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-44-xxI_arV-No6erEgv9jYjyg-1; Tue, 16 Jan 2024 11:15:22 -0500 X-MC-Unique: xxI_arV-No6erEgv9jYjyg-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id AE34C882320; Tue, 16 Jan 2024 16:15:21 +0000 (UTC) Received: from redhat.com (unknown [10.22.8.6]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 9F2961C060AF; Tue, 16 Jan 2024 16:15:21 +0000 (UTC) Received: from fche by redhat.com with local (Exim 4.94.2) (envelope-from ) id 1rPm5w-0002VF-Qu; Tue, 16 Jan 2024 11:15:20 -0500 Date: Tue, 16 Jan 2024 11:15:20 -0500 From: "Frank Ch. Eigler" To: Maxim Kuvyrkov Cc: bunsen@sourceware.org, Christophe Lyon Subject: Re: bunsen to assist ci/cd testsuite historical analysis needs Message-ID: <20240116161520.GF26700@redhat.com> References: <20231212185006.GA21248@redhat.com> <86711582-4DD8-46F4-824C-583CB19EA58D@linaro.org> <87wmsrjsdd.fsf@redhat.com> <20240104023400.GA17157@redhat.com> <20240111175140.GI28684@redhat.com> <7AE27BD9-5332-496D-9A32-C49383A039B3@linaro.org> MIME-Version: 1.0 In-Reply-To: <7AE27BD9-5332-496D-9A32-C49383A039B3@linaro.org> User-Agent: Mutt/1.12.0 (2019-05-25) X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Spam-Status: No, score=-7.9 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE,TXREP,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: Hi - > > - separating the .log and .sum files into different directories stops > > bunsen's crossmatching of the two; bunsen can accept the build tree > > raw locations of these files, without the "sumfiles" / "00-sumfiles" > > type of reorg > We partially re-run the testsuites when results change unexpectedly. > This allows us to detect and ignore flaky tests. 00-sumfiles/ is an > auxiliary directory that contains all run and re-run logs and .sum > files; sumfiles/ directory contains the final [synthetic] .sum files > that are a merge of all re-run .sum files. All this is to say that > we don't necessarily have the matching logs for the final > [synthetic] .sum files. Interesting. In the bunsen model, that would be better represented as individual testrun commits for each of the run attempts, including the raw .log/.sum files from each run. To tell them apart, one could just identify each rerun with a "/NNN" suffix in the bunsen git tag. Flakeyness would be inferred by statistical analysis from those runs (and among all the other "similar" runs in the database), so no need for the final synthetic results really, but that too could be committed separately as a new testrun ("/summary"?). > What information do you extract from the .log files? Nothing specifically extracted, but stored and parsed lightly, so that when a user wants to discover why a test failed, the appropriate segment of the appropriate log file can be easily displayed. Availability of that info can be key for diagnosis. > > - for buildbot builds on sourceware, we collect & parse the autoconf > > config.log files as first class content, rather than just informal > > console logs > > We can gather config.log files. > [...] Sweet. - FChE