Re: bunsen to assist ci/cd testsuite historical analysis needs

[Christophe Lyon <christophe.lyon@linaro.org>]

On Tue, 16 Jan 2024 at 17:15, Frank Ch. Eigler <fche@redhat.com> wrote:
>
> Hi -
>
>
> > > - separating the .log and .sum files into different directories stops
> > >  bunsen's crossmatching of the two; bunsen can accept the build tree
> > >  raw locations of these files, without the "sumfiles" / "00-sumfiles"
> > >  type of reorg
>
> > We partially re-run the testsuites when results change unexpectedly.
> > This allows us to detect and ignore flaky tests.  00-sumfiles/ is an
> > auxiliary directory that contains all run and re-run logs and .sum
> > files; sumfiles/ directory contains the final [synthetic] .sum files
> > that are a merge of all re-run .sum files.  All this is to say that
> > we don't necessarily have the matching logs for the final
> > [synthetic] .sum files.
>
> Interesting.  In the bunsen model, that would be better represented as
> individual testrun commits for each of the run attempts, including the

Do you mean each testrun includes the whole testsuite?
In our case, only the first run includes the whole testsuite, the
subsequent ones include only the .exp files for which we detected
failures, to keep testing time reasonably low.

> raw .log/.sum files from each run.  To tell them apart, one could just
> identify each rerun with a "/NNN" suffix in the bunsen git tag.
> Flakeyness would be inferred by statistical analysis from those runs
> (and among all the other "similar" runs in the database), so no need
> for the final synthetic results really, but that too could be committed
> separately as a new testrun ("/summary"?).
>
> > What information do you extract from the .log files?
>
> Nothing specifically extracted, but stored and parsed lightly, so that
> when a user wants to discover why a test failed, the appropriate
> segment of the appropriate log file can be easily displayed.
> Availability of that info can be key for diagnosis.
>
> > > - for buildbot builds on sourceware, we collect & parse the autoconf
> > >  config.log files as first class content, rather than just informal
> > >  console logs
> >
> > We can gather config.log files.
> > [...]
>
> Sweet.
>
>
> - FChE
>