bunsen to assist ci/cd testsuite historical analysis needs

bunsen to assist ci/cd testsuite historical analysis needs

[Frank Ch. Eigler]

Hi -

I write to investigate any possible linkage between your work at
linaro betweween the toolchain ci/cd work and our work with upstream
testing on sourceware.  Part of it is via buildbot, that I'm sure you
know; testsuite log archiving/analysis is done with a gadget called
bunsen.  I believe you met Mark Wielaard over at Cauldron and talked
briefly about this stuff.  Is there any material or time I could offer
to explain what we have, so as to investigate any opportunities to
interoperate?

https://sourceware.org/bunsen/

- FChE

Re: bunsen to assist ci/cd testsuite historical analysis needs

[Maxim Kuvyrkov]

> On Dec 12, 2023, at 21:50, Frank Ch. Eigler <fche@redhat.com> wrote:
> 
> Hi -
> 
> I write to investigate any possible linkage between your work at
> linaro betweween the toolchain ci/cd work and our work with upstream
> testing on sourceware.  Part of it is via buildbot, that I'm sure you
> know; testsuite log archiving/analysis is done with a gadget called
> bunsen.  I believe you met Mark Wielaard over at Cauldron and talked
> briefly about this stuff.  Is there any material or time I could offer
> to explain what we have, so as to investigate any opportunities to
> interoperate?
> 
> https://sourceware.org/bunsen/

Hi Frank,

How about a zoom/hangout/etc. call next week?  [Christophe and I are in Europe timezone.]

One obvious opportunity is to make Linaro Toolchain CI submit results to bunsen.  Are there other things we can improve?

Kind regards,

--
Maxim Kuvyrkov
https://www.linaro.org

Re: bunsen to assist ci/cd testsuite historical analysis needs

[Frank Ch. Eigler]

Hi, Maxim -

> How about a zoom/hangout/etc. call next week?  [Christophe and I are
> in Europe timezone.]

I'm so sorry, I didn't see this in time.

> One obvious opportunity is to make Linaro Toolchain CI submit results
> to bunsen.  Are there other things we can improve?

To make this more concrete, your CI system could contribute to the
sourceware bunsen pool of test results by:

- creating a temporary git repo in your CI system

- committing your local testsuite log files, and .bunsen.FOO build
  metadata files into your git repo using the bunsen t-upload-git-push
  shell script (or equivalent); see the sourceware buildbot master.cfg
  for the metadata specifics we use:
  https://sourceware.org/git/?p=builder.git;a=blob;f=builder/master.cfg;h=d2cb2d0ff1d68196ef1e8a8bcf9c9d7ceeb23211;hb=HEAD#l1267

- pushing each resulting branch/commit to the sourceware bunsen.git repo
  (we can get you ssh credentials sufficient for this)

- deleting your local git repo if you like

In this scenario, you wouldn't have to run any real bunsen code locally,
nor publish any data or run a server of your own.  The sourceware
installation of bunsen would analyze and serve the data shortly after it
hits git.

If at some point, you became interested in more, like local long term
analytics, or compressed log storage, or whatever, we can do something
more sophisticated.

- FChE

Re: bunsen to assist ci/cd testsuite historical analysis needs

[Maxim Kuvyrkov]

Hi Frank,

Thanks for the pointers!

This approach matches well with our CI system.  We already store results in a git repo, and post-process them for publishing to LNT dashboard (e.g., [1]).

Re. ssh access -- could you create an account for linaro-tcwg-bot with ssh key [2]?

[1] http://llvm.validation.linaro.org:38000/db_default/v4/tcwg_gdb_check/graph?plot.14.0=1.14.0&plot.11.2=1.11.2
[2] https://git.linaro.org/ci/dockerfiles.git/tree/tcwg-base/home-data/tcwg-buildslave/.ssh/authorized_keys#n2

Thanks!

--
Maxim Kuvyrkov
https://www.linaro.org

> On Jan 3, 2024, at 02:13, Frank Ch. Eigler <fche@redhat.com> wrote:
> 
> 
> Hi, Maxim -
> 
>> How about a zoom/hangout/etc. call next week?  [Christophe and I are
>> in Europe timezone.]
> 
> I'm so sorry, I didn't see this in time.
> 
>> One obvious opportunity is to make Linaro Toolchain CI submit results
>> to bunsen.  Are there other things we can improve?
> 
> To make this more concrete, your CI system could contribute to the
> sourceware bunsen pool of test results by:
> 
> - creating a temporary git repo in your CI system
> 
> - committing your local testsuite log files, and .bunsen.FOO build
>  metadata files into your git repo using the bunsen t-upload-git-push
>  shell script (or equivalent); see the sourceware buildbot master.cfg
>  for the metadata specifics we use:
>  https://sourceware.org/git/?p=builder.git;a=blob;f=builder/master.cfg;h=d2cb2d0ff1d68196ef1e8a8bcf9c9d7ceeb23211;hb=HEAD#l1267
> 
> - pushing each resulting branch/commit to the sourceware bunsen.git repo
>  (we can get you ssh credentials sufficient for this)
> 
> - deleting your local git repo if you like
> 
> In this scenario, you wouldn't have to run any real bunsen code locally,
> nor publish any data or run a server of your own.  The sourceware
> installation of bunsen would analyze and serve the data shortly after it
> hits git.
> 
> If at some point, you became interested in more, like local long term
> analytics, or compressed log storage, or whatever, we can do something
> more sophisticated.
> 
> - FChE
> 

Re: bunsen to assist ci/cd testsuite historical analysis needs

[Frank Ch. Eigler]

Hi -

> This approach matches well with our CI system.  We already store results in a git repo, and post-process them for publishing to LNT dashboard (e.g., [1]).

OK, will check that out.  Is the git repo with results also visible somewhere?

> Re. ssh access -- could you create an account for linaro-tcwg-bot with ssh key [2]?

Done.  For namespace disambiguation, you're welcome to use a tag
prefix such as "linaro-tcwg/" and push bunsen style commits with log
files & .bunsen metadata into ssh://sourceware.org/git/bunsendb.git .
You can of course download the current repo to see the style of
testsuite runs, so you can match that as much as practical.

> [1] http://llvm.validation.linaro.org:38000/db_default/v4/tcwg_gdb_check/graph?plot.14.0=1.14.0&plot.11.2=1.11.2

Neat.


- FChE

Re: bunsen to assist ci/cd testsuite historical analysis needs

[Maxim Kuvyrkov]

> On Jan 4, 2024, at 06:34, Frank Ch. Eigler <fche@redhat.com> wrote:
> 
> Hi -
> 
>> This approach matches well with our CI system.  We already store results in a git repo, and post-process them for publishing to LNT dashboard (e.g., [1]).
> 
> OK, will check that out.  Is the git repo with results also visible somewhere?

We have recently migrated them to an internal machine (so that results don't eat up all disk space on git.linaro.org), but backup branches are still visible on https://git.linaro.org/toolchain/ci/base-artifacts .  E.g., [2] are GCC results for aarch64.

[2] https://git.linaro.org/toolchain/ci/base-artifacts/tcwg_gcc_check/master-aarch64.git/

> 
>> Re. ssh access -- could you create an account for linaro-tcwg-bot with ssh key [2]?
> 
> Done.  For namespace disambiguation, you're welcome to use a tag
> prefix such as "linaro-tcwg/" and push bunsen style commits with log
> files & .bunsen metadata into ssh://sourceware.org/git/bunsendb.git .
> You can of course download the current repo to see the style of
> testsuite runs, so you can match that as much as practical.

Ack.

> 
>> [1] http://llvm.validation.linaro.org:38000/db_default/v4/tcwg_gdb_check/graph?plot.14.0=1.14.0&plot.11.2=1.11.2
> 
> Neat.
> 

Thanks,

--
Maxim Kuvyrkov
https://www.linaro.org

Re: bunsen to assist ci/cd testsuite historical analysis needs

[Frank Ch. Eigler]

Hi -

> > OK, will check that out.  Is the git repo with results also visible somewhere?
> [...]
> [2] https://git.linaro.org/toolchain/ci/base-artifacts/tcwg_gcc_check/master-aarch64.git/

Just for reference, after a quick glance at this, it's close to but
not quite verbatim usable by bunsen:

- the .xz compression/extension would interfere with extension based
  matching

- separating the .log and .sum files into different directories stops
  bunsen's crossmatching of the two; bunsen can accept the build tree
  raw locations of these files, without the "sumfiles" / "00-sumfiles"
  type of reorg

- for buildbot builds on sourceware, we collect & parse the autoconf
  config.log files as first class content, rather than just informal
  console logs

- bunsen would love to receive metadata about the build in little
  .bunsen.FOO text files, e.g. see:
  https://sourceware.org/git/?p=bunsendb.git;a=commit;h=613dd3b381b0910ebd5036fcf42761e2cead12ee


I assume you had good reasons for .xz compressing these files inside
the git repo ... but for what it's worth, we find that letting normal
git-repack transparently manage all the compression of plain text
payloads gives us excellent results.


- FChE

Re: bunsen to assist ci/cd testsuite historical analysis needs

[Maxim Kuvyrkov]

> On Jan 11, 2024, at 21:51, Frank Ch. Eigler <fche@redhat.com> wrote:
> 
> Hi -
> 
>>> OK, will check that out.  Is the git repo with results also visible somewhere?
>> [...]
>> [2] https://git.linaro.org/toolchain/ci/base-artifacts/tcwg_gcc_check/master-aarch64.git/
> 
> Just for reference, after a quick glance at this, it's close to but
> not quite verbatim usable by bunsen:
> 
> - the .xz compression/extension would interfere with extension based
>  matching

We can fix that.

> 
> - separating the .log and .sum files into different directories stops
>  bunsen's crossmatching of the two; bunsen can accept the build tree
>  raw locations of these files, without the "sumfiles" / "00-sumfiles"
>  type of reorg

We partially re-run the testsuites when results change unexpectedly.  This allows us to detect and ignore flaky tests.  00-sumfiles/ is an auxiliary directory that contains all run and re-run logs and .sum files; sumfiles/ directory contains the final [synthetic] .sum files that are a merge of all re-run .sum files.  All this is to say that we don't necessarily have the matching logs for the final [synthetic] .sum files.

What information do you extract from the .log files?

> 
> - for buildbot builds on sourceware, we collect & parse the autoconf
>  config.log files as first class content, rather than just informal
>  console logs

We can gather config.log files.

> 
> - bunsen would love to receive metadata about the build in little
>  .bunsen.FOO text files, e.g. see:
>  https://sourceware.org/git/?p=bunsendb.git;a=commit;h=613dd3b381b0910ebd5036fcf42761e2cead12ee
> 

Ack.

> 
> I assume you had good reasons for .xz compressing these files inside
> the git repo ... but for what it's worth, we find that letting normal
> git-repack transparently manage all the compression of plain text
> payloads gives us excellent results.
> 

These are also uploaded outside of git repos, but it's not a problem to push uncompressed files into git.

Kind regards,

--
Maxim Kuvyrkov
https://www.linaro.org

Re: bunsen to assist ci/cd testsuite historical analysis needs

[Frank Ch. Eigler]

Hi -


> > - separating the .log and .sum files into different directories stops
> >  bunsen's crossmatching of the two; bunsen can accept the build tree
> >  raw locations of these files, without the "sumfiles" / "00-sumfiles"
> >  type of reorg

> We partially re-run the testsuites when results change unexpectedly.
> This allows us to detect and ignore flaky tests.  00-sumfiles/ is an
> auxiliary directory that contains all run and re-run logs and .sum
> files; sumfiles/ directory contains the final [synthetic] .sum files
> that are a merge of all re-run .sum files.  All this is to say that
> we don't necessarily have the matching logs for the final
> [synthetic] .sum files.

Interesting.  In the bunsen model, that would be better represented as
individual testrun commits for each of the run attempts, including the
raw .log/.sum files from each run.  To tell them apart, one could just
identify each rerun with a "/NNN" suffix in the bunsen git tag.
Flakeyness would be inferred by statistical analysis from those runs
(and among all the other "similar" runs in the database), so no need
for the final synthetic results really, but that too could be committed
separately as a new testrun ("/summary"?).

> What information do you extract from the .log files?

Nothing specifically extracted, but stored and parsed lightly, so that
when a user wants to discover why a test failed, the appropriate
segment of the appropriate log file can be easily displayed.
Availability of that info can be key for diagnosis.

> > - for buildbot builds on sourceware, we collect & parse the autoconf
> >  config.log files as first class content, rather than just informal
> >  console logs
> 
> We can gather config.log files.
> [...]

Sweet.


- FChE

Re: bunsen to assist ci/cd testsuite historical analysis needs

[Christophe Lyon]

On Tue, 16 Jan 2024 at 17:15, Frank Ch. Eigler <fche@redhat.com> wrote:
>
> Hi -
>
>
> > > - separating the .log and .sum files into different directories stops
> > >  bunsen's crossmatching of the two; bunsen can accept the build tree
> > >  raw locations of these files, without the "sumfiles" / "00-sumfiles"
> > >  type of reorg
>
> > We partially re-run the testsuites when results change unexpectedly.
> > This allows us to detect and ignore flaky tests.  00-sumfiles/ is an
> > auxiliary directory that contains all run and re-run logs and .sum
> > files; sumfiles/ directory contains the final [synthetic] .sum files
> > that are a merge of all re-run .sum files.  All this is to say that
> > we don't necessarily have the matching logs for the final
> > [synthetic] .sum files.
>
> Interesting.  In the bunsen model, that would be better represented as
> individual testrun commits for each of the run attempts, including the

Do you mean each testrun includes the whole testsuite?
In our case, only the first run includes the whole testsuite, the
subsequent ones include only the .exp files for which we detected
failures, to keep testing time reasonably low.

> raw .log/.sum files from each run.  To tell them apart, one could just
> identify each rerun with a "/NNN" suffix in the bunsen git tag.
> Flakeyness would be inferred by statistical analysis from those runs
> (and among all the other "similar" runs in the database), so no need
> for the final synthetic results really, but that too could be committed
> separately as a new testrun ("/summary"?).
>
> > What information do you extract from the .log files?
>
> Nothing specifically extracted, but stored and parsed lightly, so that
> when a user wants to discover why a test failed, the appropriate
> segment of the appropriate log file can be easily displayed.
> Availability of that info can be key for diagnosis.
>
> > > - for buildbot builds on sourceware, we collect & parse the autoconf
> > >  config.log files as first class content, rather than just informal
> > >  console logs
> >
> > We can gather config.log files.
> > [...]
>
> Sweet.
>
>
> - FChE
>

Re: bunsen to assist ci/cd testsuite historical analysis needs

[Frank Ch. Eigler]

Hi -

> > Interesting.  In the bunsen model, that would be better represented as
> > individual testrun commits for each of the run attempts, including the
> 
> Do you mean each testrun includes the whole testsuite?

Each testrun includes just as much of the testsuite as you wish.

> In our case, only the first run includes the whole testsuite, the
> subsequent ones include only the .exp files for which we detected
> failures, to keep testing time reasonably low.

That makes sense (unless you get a false positive PASS in the first
run, don't bother rerun, and miss the flake!).  In any case, bunsen
would gladly accept the logs generated from each of the partial
reruns.

- FChE

Re: bunsen to assist ci/cd testsuite historical analysis needs

[Frank Ch. Eigler]

Hi -

Have y'all had time to try to start feeding some of your ci/cd logs into
sourceware's bunsen repo?  Is there any obstacle at all from our side?

- FChE

Re: bunsen to assist ci/cd testsuite historical analysis needs

[Christophe Lyon]

Hi Frank,

Sorry for not coming back to you earlier. We've been busy with
improving our framework, and did not have time yet to look at bunsen.
I hope we can start to have a look after the Linaro Connect conference
(mid-May).

At this stage, I don't think we need anything from you.

IIUC, we just need to figure out how to replicate the steps from
bunsen_logfile_upload_cpio_steps or bunsen_logfile_upload_steps in our
scripts?

Thanks,

Christophe

On Mon, 15 Apr 2024 at 17:06, Frank Ch. Eigler <fche@redhat.com> wrote:
>
> Hi -
>
> Have y'all had time to try to start feeding some of your ci/cd logs into
> sourceware's bunsen repo?  Is there any obstacle at all from our side?
>
> - FChE
>

Re: bunsen to assist ci/cd testsuite historical analysis needs

[Frank Ch. Eigler]

Hi -

> [...]  IIUC, we just need to figure out how to replicate the steps
> from bunsen_logfile_upload_cpio_steps or bunsen_logfile_upload_steps
> in our scripts?

Well, even less than that.  All the system takes as input are tagged
git commits that contain all the noteworthy log files from a
particular testsuite run.  We also offer this little dinky script that
is one way of automating it.

https://sourceware.org/git/?p=bunsen.git;a=blob;f=bin/t-upload-git-push;h=a7977aafffe08ffb0c57261a950c78d1776c265d;hb=HEAD

- FChE