From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lf1-x12d.google.com (mail-lf1-x12d.google.com [IPv6:2a00:1450:4864:20::12d]) by sourceware.org (Postfix) with ESMTPS id CBB193858D37 for ; Tue, 16 Jan 2024 16:49:18 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org CBB193858D37 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=linaro.org Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=linaro.org ARC-Filter: OpenARC Filter v1.0.0 sourceware.org CBB193858D37 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=2a00:1450:4864:20::12d ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1705423762; cv=none; b=OZpOVh7Z1RYBqE+akManyRPIYXWbUiHhzRH0qGFBrjS9xLjf3Bj7ATHGv4bnkZFhpFbbc0ncNXSD2A2bWWw21NiZ+o4+TXx+g9LoMsiItWSWv8Q6QSxs+V1d6r98eGfkheJ7lzR4U/F90YxCuQIZ4LhdT17pOfFIktey0twntfw= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1705423762; c=relaxed/simple; bh=3mLvsEP/OEWvkTtpNCQlqodEFk9Jdlu7Od4O8tWfiM8=; h=DKIM-Signature:MIME-Version:From:Date:Message-ID:Subject:To; b=IytDZKlP6Ihjz7qXQHzeQvpggQG44wITr2C/3tKsvLPIRoOVAcwC3f/Mqnk1OivRXceWYOLGWAMMYOjcwgTMiCtNPhCG9a9MfOiqgsuOc5mzJgBZJNEhKJNJl0H4Te1QqGrGpbTlpMaxhB669acOI6A8uE7nKz9h0OBYke2a4d0= ARC-Authentication-Results: i=1; server2.sourceware.org Received: by mail-lf1-x12d.google.com with SMTP id 2adb3069b0e04-50ea9daac4cso10418095e87.3 for ; Tue, 16 Jan 2024 08:49:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1705423757; x=1706028557; darn=sourceware.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=THSbz4fL9EsaadMy+g40PvEZVHrBXG9meO6fQUy24Kw=; b=oNGAu6PC+2WYB4jc7gtpBstw4daCu2++JM1WXdOGYdZq6zWZM1gmay2p/GumXVuNEr HjKFHG7bBE0h39aA9OrfzK/mP6vh3rHUTMz7xCOslOc8gt7rUQwNnOpnttikNV7/95E5 mJ1iN63ICorbBWspRmTWZAMq18rw7XBt5LEzk8YtSVkxwcd04El4rooVHqf0rj0g7fef fS4PpbtLiVP3xPxtGsu1V2IdwsR8sniFkkYh/03My8s2JC7mVnBJAnA1dWsdrXKRaeb0 6Cc+7Pm67ULV1XnR1T6HZgfxmrwR+BNFO81a0Bjff8BL8u+Jm5Fm8AOTDfATzEs1RuYu afUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705423757; x=1706028557; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=THSbz4fL9EsaadMy+g40PvEZVHrBXG9meO6fQUy24Kw=; b=FoUYw7QoxylX6wPdb6RROPY9+MmteSFABwtuLhYpR0C0UP0O8rbCNduBCkMfOpkGuA JSj6WNZHEyrrf9fmMOEJ7dJbmd0q4I2Xf75kcOAWQaFl+nw4uNig6fjIuNksX8KAHMtz NtS6qTW7olq5iJSGRiu1r8YtHbw5HBWvI4+1ajZFLYKu3phPeYsHSSfmU+JNYilgJ8/B ETa6gfl+8FPETKpApAuqhRAJ0BdK5nf8jb9UI0pKxOr866je+B0VxV4+3bDqQyOvQkAW a1oTXU/YeWf6Y7Z81drZG3LlAv+0sja8iKvZK6OxKl5d37Oyk7Bjxy/b0OvdABRboKIm P0Uw== X-Gm-Message-State: AOJu0YzWmWU0Tg4lWbAFwp7LzLxhH0DECHOYLsQiV2rBGh1FXYyWcCfH +Wt1ddJxMsYklOxDHj1WThVaTsopt/+j07C2Lze3gpcqVxKkpA== X-Google-Smtp-Source: AGHT+IFVXXQpi8Z0h7PyuelmVNYsk4MgNlGYVULhcPkzB0RP567Zf4PfzA612UqVtGEzlkixpPD5FQKgTTh/ACj0p7Y= X-Received: by 2002:a05:6512:3b90:b0:50e:935a:ca83 with SMTP id g16-20020a0565123b9000b0050e935aca83mr4222469lfv.91.1705423757016; Tue, 16 Jan 2024 08:49:17 -0800 (PST) MIME-Version: 1.0 References: <20231212185006.GA21248@redhat.com> <86711582-4DD8-46F4-824C-583CB19EA58D@linaro.org> <87wmsrjsdd.fsf@redhat.com> <20240104023400.GA17157@redhat.com> <20240111175140.GI28684@redhat.com> <7AE27BD9-5332-496D-9A32-C49383A039B3@linaro.org> <20240116161520.GF26700@redhat.com> In-Reply-To: <20240116161520.GF26700@redhat.com> From: Christophe Lyon Date: Tue, 16 Jan 2024 17:49:07 +0100 Message-ID: Subject: Re: bunsen to assist ci/cd testsuite historical analysis needs To: "Frank Ch. Eigler" Cc: Maxim Kuvyrkov , bunsen@sourceware.org Content-Type: text/plain; charset="UTF-8" X-Spam-Status: No, score=-3.9 required=5.0 tests=BAYES_00,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS,TXREP,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: On Tue, 16 Jan 2024 at 17:15, Frank Ch. Eigler wrote: > > Hi - > > > > > - separating the .log and .sum files into different directories stops > > > bunsen's crossmatching of the two; bunsen can accept the build tree > > > raw locations of these files, without the "sumfiles" / "00-sumfiles" > > > type of reorg > > > We partially re-run the testsuites when results change unexpectedly. > > This allows us to detect and ignore flaky tests. 00-sumfiles/ is an > > auxiliary directory that contains all run and re-run logs and .sum > > files; sumfiles/ directory contains the final [synthetic] .sum files > > that are a merge of all re-run .sum files. All this is to say that > > we don't necessarily have the matching logs for the final > > [synthetic] .sum files. > > Interesting. In the bunsen model, that would be better represented as > individual testrun commits for each of the run attempts, including the Do you mean each testrun includes the whole testsuite? In our case, only the first run includes the whole testsuite, the subsequent ones include only the .exp files for which we detected failures, to keep testing time reasonably low. > raw .log/.sum files from each run. To tell them apart, one could just > identify each rerun with a "/NNN" suffix in the bunsen git tag. > Flakeyness would be inferred by statistical analysis from those runs > (and among all the other "similar" runs in the database), so no need > for the final synthetic results really, but that too could be committed > separately as a new testrun ("/summary"?). > > > What information do you extract from the .log files? > > Nothing specifically extracted, but stored and parsed lightly, so that > when a user wants to discover why a test failed, the appropriate > segment of the appropriate log file can be easily displayed. > Availability of that info can be key for diagnosis. > > > > - for buildbot builds on sourceware, we collect & parse the autoconf > > > config.log files as first class content, rather than just informal > > > console logs > > > > We can gather config.log files. > > [...] > > Sweet. > > > - FChE >