* bunsen to assist ci/cd testsuite historical analysis needs @ 2023-12-12 18:50 Frank Ch. Eigler 2023-12-14 15:45 ` Maxim Kuvyrkov 0 siblings, 1 reply; 14+ messages in thread From: Frank Ch. Eigler @ 2023-12-12 18:50 UTC (permalink / raw) To: maxim.kuvyrkov; +Cc: bunsen Hi - I write to investigate any possible linkage between your work at linaro betweween the toolchain ci/cd work and our work with upstream testing on sourceware. Part of it is via buildbot, that I'm sure you know; testsuite log archiving/analysis is done with a gadget called bunsen. I believe you met Mark Wielaard over at Cauldron and talked briefly about this stuff. Is there any material or time I could offer to explain what we have, so as to investigate any opportunities to interoperate? https://sourceware.org/bunsen/ - FChE ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: bunsen to assist ci/cd testsuite historical analysis needs 2023-12-12 18:50 bunsen to assist ci/cd testsuite historical analysis needs Frank Ch. Eigler @ 2023-12-14 15:45 ` Maxim Kuvyrkov 2024-01-02 22:13 ` Frank Ch. Eigler 0 siblings, 1 reply; 14+ messages in thread From: Maxim Kuvyrkov @ 2023-12-14 15:45 UTC (permalink / raw) To: Frank Ch. Eigler; +Cc: bunsen, Christophe Lyon > On Dec 12, 2023, at 21:50, Frank Ch. Eigler <fche@redhat.com> wrote: > > Hi - > > I write to investigate any possible linkage between your work at > linaro betweween the toolchain ci/cd work and our work with upstream > testing on sourceware. Part of it is via buildbot, that I'm sure you > know; testsuite log archiving/analysis is done with a gadget called > bunsen. I believe you met Mark Wielaard over at Cauldron and talked > briefly about this stuff. Is there any material or time I could offer > to explain what we have, so as to investigate any opportunities to > interoperate? > > https://sourceware.org/bunsen/ Hi Frank, How about a zoom/hangout/etc. call next week? [Christophe and I are in Europe timezone.] One obvious opportunity is to make Linaro Toolchain CI submit results to bunsen. Are there other things we can improve? Kind regards, -- Maxim Kuvyrkov https://www.linaro.org ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: bunsen to assist ci/cd testsuite historical analysis needs 2023-12-14 15:45 ` Maxim Kuvyrkov @ 2024-01-02 22:13 ` Frank Ch. Eigler 2024-01-03 12:59 ` Maxim Kuvyrkov 0 siblings, 1 reply; 14+ messages in thread From: Frank Ch. Eigler @ 2024-01-02 22:13 UTC (permalink / raw) To: Maxim Kuvyrkov; +Cc: bunsen, Christophe Lyon Hi, Maxim - > How about a zoom/hangout/etc. call next week? [Christophe and I are > in Europe timezone.] I'm so sorry, I didn't see this in time. > One obvious opportunity is to make Linaro Toolchain CI submit results > to bunsen. Are there other things we can improve? To make this more concrete, your CI system could contribute to the sourceware bunsen pool of test results by: - creating a temporary git repo in your CI system - committing your local testsuite log files, and .bunsen.FOO build metadata files into your git repo using the bunsen t-upload-git-push shell script (or equivalent); see the sourceware buildbot master.cfg for the metadata specifics we use: https://sourceware.org/git/?p=builder.git;a=blob;f=builder/master.cfg;h=d2cb2d0ff1d68196ef1e8a8bcf9c9d7ceeb23211;hb=HEAD#l1267 - pushing each resulting branch/commit to the sourceware bunsen.git repo (we can get you ssh credentials sufficient for this) - deleting your local git repo if you like In this scenario, you wouldn't have to run any real bunsen code locally, nor publish any data or run a server of your own. The sourceware installation of bunsen would analyze and serve the data shortly after it hits git. If at some point, you became interested in more, like local long term analytics, or compressed log storage, or whatever, we can do something more sophisticated. - FChE ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: bunsen to assist ci/cd testsuite historical analysis needs 2024-01-02 22:13 ` Frank Ch. Eigler @ 2024-01-03 12:59 ` Maxim Kuvyrkov 2024-01-04 2:34 ` Frank Ch. Eigler 0 siblings, 1 reply; 14+ messages in thread From: Maxim Kuvyrkov @ 2024-01-03 12:59 UTC (permalink / raw) To: Frank Ch. Eigler; +Cc: bunsen, Christophe Lyon Hi Frank, Thanks for the pointers! This approach matches well with our CI system. We already store results in a git repo, and post-process them for publishing to LNT dashboard (e.g., [1]). Re. ssh access -- could you create an account for linaro-tcwg-bot with ssh key [2]? [1] http://llvm.validation.linaro.org:38000/db_default/v4/tcwg_gdb_check/graph?plot.14.0=1.14.0&plot.11.2=1.11.2 [2] https://git.linaro.org/ci/dockerfiles.git/tree/tcwg-base/home-data/tcwg-buildslave/.ssh/authorized_keys#n2 Thanks! -- Maxim Kuvyrkov https://www.linaro.org > On Jan 3, 2024, at 02:13, Frank Ch. Eigler <fche@redhat.com> wrote: > > > Hi, Maxim - > >> How about a zoom/hangout/etc. call next week? [Christophe and I are >> in Europe timezone.] > > I'm so sorry, I didn't see this in time. > >> One obvious opportunity is to make Linaro Toolchain CI submit results >> to bunsen. Are there other things we can improve? > > To make this more concrete, your CI system could contribute to the > sourceware bunsen pool of test results by: > > - creating a temporary git repo in your CI system > > - committing your local testsuite log files, and .bunsen.FOO build > metadata files into your git repo using the bunsen t-upload-git-push > shell script (or equivalent); see the sourceware buildbot master.cfg > for the metadata specifics we use: > https://sourceware.org/git/?p=builder.git;a=blob;f=builder/master.cfg;h=d2cb2d0ff1d68196ef1e8a8bcf9c9d7ceeb23211;hb=HEAD#l1267 > > - pushing each resulting branch/commit to the sourceware bunsen.git repo > (we can get you ssh credentials sufficient for this) > > - deleting your local git repo if you like > > In this scenario, you wouldn't have to run any real bunsen code locally, > nor publish any data or run a server of your own. The sourceware > installation of bunsen would analyze and serve the data shortly after it > hits git. > > If at some point, you became interested in more, like local long term > analytics, or compressed log storage, or whatever, we can do something > more sophisticated. > > - FChE > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: bunsen to assist ci/cd testsuite historical analysis needs 2024-01-03 12:59 ` Maxim Kuvyrkov @ 2024-01-04 2:34 ` Frank Ch. Eigler 2024-01-11 13:42 ` Maxim Kuvyrkov 0 siblings, 1 reply; 14+ messages in thread From: Frank Ch. Eigler @ 2024-01-04 2:34 UTC (permalink / raw) To: Maxim Kuvyrkov; +Cc: bunsen, Christophe Lyon Hi - > This approach matches well with our CI system. We already store results in a git repo, and post-process them for publishing to LNT dashboard (e.g., [1]). OK, will check that out. Is the git repo with results also visible somewhere? > Re. ssh access -- could you create an account for linaro-tcwg-bot with ssh key [2]? Done. For namespace disambiguation, you're welcome to use a tag prefix such as "linaro-tcwg/" and push bunsen style commits with log files & .bunsen metadata into ssh://sourceware.org/git/bunsendb.git . You can of course download the current repo to see the style of testsuite runs, so you can match that as much as practical. > [1] http://llvm.validation.linaro.org:38000/db_default/v4/tcwg_gdb_check/graph?plot.14.0=1.14.0&plot.11.2=1.11.2 Neat. - FChE ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: bunsen to assist ci/cd testsuite historical analysis needs 2024-01-04 2:34 ` Frank Ch. Eigler @ 2024-01-11 13:42 ` Maxim Kuvyrkov 2024-01-11 17:51 ` Frank Ch. Eigler 0 siblings, 1 reply; 14+ messages in thread From: Maxim Kuvyrkov @ 2024-01-11 13:42 UTC (permalink / raw) To: Frank Ch. Eigler; +Cc: bunsen, Christophe Lyon > On Jan 4, 2024, at 06:34, Frank Ch. Eigler <fche@redhat.com> wrote: > > Hi - > >> This approach matches well with our CI system. We already store results in a git repo, and post-process them for publishing to LNT dashboard (e.g., [1]). > > OK, will check that out. Is the git repo with results also visible somewhere? We have recently migrated them to an internal machine (so that results don't eat up all disk space on git.linaro.org), but backup branches are still visible on https://git.linaro.org/toolchain/ci/base-artifacts . E.g., [2] are GCC results for aarch64. [2] https://git.linaro.org/toolchain/ci/base-artifacts/tcwg_gcc_check/master-aarch64.git/ > >> Re. ssh access -- could you create an account for linaro-tcwg-bot with ssh key [2]? > > Done. For namespace disambiguation, you're welcome to use a tag > prefix such as "linaro-tcwg/" and push bunsen style commits with log > files & .bunsen metadata into ssh://sourceware.org/git/bunsendb.git . > You can of course download the current repo to see the style of > testsuite runs, so you can match that as much as practical. Ack. > >> [1] http://llvm.validation.linaro.org:38000/db_default/v4/tcwg_gdb_check/graph?plot.14.0=1.14.0&plot.11.2=1.11.2 > > Neat. > Thanks, -- Maxim Kuvyrkov https://www.linaro.org ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: bunsen to assist ci/cd testsuite historical analysis needs 2024-01-11 13:42 ` Maxim Kuvyrkov @ 2024-01-11 17:51 ` Frank Ch. Eigler 2024-01-16 10:28 ` Maxim Kuvyrkov 0 siblings, 1 reply; 14+ messages in thread From: Frank Ch. Eigler @ 2024-01-11 17:51 UTC (permalink / raw) To: Maxim Kuvyrkov; +Cc: bunsen, Christophe Lyon Hi - > > OK, will check that out. Is the git repo with results also visible somewhere? > [...] > [2] https://git.linaro.org/toolchain/ci/base-artifacts/tcwg_gcc_check/master-aarch64.git/ Just for reference, after a quick glance at this, it's close to but not quite verbatim usable by bunsen: - the .xz compression/extension would interfere with extension based matching - separating the .log and .sum files into different directories stops bunsen's crossmatching of the two; bunsen can accept the build tree raw locations of these files, without the "sumfiles" / "00-sumfiles" type of reorg - for buildbot builds on sourceware, we collect & parse the autoconf config.log files as first class content, rather than just informal console logs - bunsen would love to receive metadata about the build in little .bunsen.FOO text files, e.g. see: https://sourceware.org/git/?p=bunsendb.git;a=commit;h=613dd3b381b0910ebd5036fcf42761e2cead12ee I assume you had good reasons for .xz compressing these files inside the git repo ... but for what it's worth, we find that letting normal git-repack transparently manage all the compression of plain text payloads gives us excellent results. - FChE ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: bunsen to assist ci/cd testsuite historical analysis needs 2024-01-11 17:51 ` Frank Ch. Eigler @ 2024-01-16 10:28 ` Maxim Kuvyrkov 2024-01-16 16:15 ` Frank Ch. Eigler 0 siblings, 1 reply; 14+ messages in thread From: Maxim Kuvyrkov @ 2024-01-16 10:28 UTC (permalink / raw) To: Frank Ch. Eigler; +Cc: bunsen, Christophe Lyon > On Jan 11, 2024, at 21:51, Frank Ch. Eigler <fche@redhat.com> wrote: > > Hi - > >>> OK, will check that out. Is the git repo with results also visible somewhere? >> [...] >> [2] https://git.linaro.org/toolchain/ci/base-artifacts/tcwg_gcc_check/master-aarch64.git/ > > Just for reference, after a quick glance at this, it's close to but > not quite verbatim usable by bunsen: > > - the .xz compression/extension would interfere with extension based > matching We can fix that. > > - separating the .log and .sum files into different directories stops > bunsen's crossmatching of the two; bunsen can accept the build tree > raw locations of these files, without the "sumfiles" / "00-sumfiles" > type of reorg We partially re-run the testsuites when results change unexpectedly. This allows us to detect and ignore flaky tests. 00-sumfiles/ is an auxiliary directory that contains all run and re-run logs and .sum files; sumfiles/ directory contains the final [synthetic] .sum files that are a merge of all re-run .sum files. All this is to say that we don't necessarily have the matching logs for the final [synthetic] .sum files. What information do you extract from the .log files? > > - for buildbot builds on sourceware, we collect & parse the autoconf > config.log files as first class content, rather than just informal > console logs We can gather config.log files. > > - bunsen would love to receive metadata about the build in little > .bunsen.FOO text files, e.g. see: > https://sourceware.org/git/?p=bunsendb.git;a=commit;h=613dd3b381b0910ebd5036fcf42761e2cead12ee > Ack. > > I assume you had good reasons for .xz compressing these files inside > the git repo ... but for what it's worth, we find that letting normal > git-repack transparently manage all the compression of plain text > payloads gives us excellent results. > These are also uploaded outside of git repos, but it's not a problem to push uncompressed files into git. Kind regards, -- Maxim Kuvyrkov https://www.linaro.org ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: bunsen to assist ci/cd testsuite historical analysis needs 2024-01-16 10:28 ` Maxim Kuvyrkov @ 2024-01-16 16:15 ` Frank Ch. Eigler 2024-01-16 16:49 ` Christophe Lyon 0 siblings, 1 reply; 14+ messages in thread From: Frank Ch. Eigler @ 2024-01-16 16:15 UTC (permalink / raw) To: Maxim Kuvyrkov; +Cc: bunsen, Christophe Lyon Hi - > > - separating the .log and .sum files into different directories stops > > bunsen's crossmatching of the two; bunsen can accept the build tree > > raw locations of these files, without the "sumfiles" / "00-sumfiles" > > type of reorg > We partially re-run the testsuites when results change unexpectedly. > This allows us to detect and ignore flaky tests. 00-sumfiles/ is an > auxiliary directory that contains all run and re-run logs and .sum > files; sumfiles/ directory contains the final [synthetic] .sum files > that are a merge of all re-run .sum files. All this is to say that > we don't necessarily have the matching logs for the final > [synthetic] .sum files. Interesting. In the bunsen model, that would be better represented as individual testrun commits for each of the run attempts, including the raw .log/.sum files from each run. To tell them apart, one could just identify each rerun with a "/NNN" suffix in the bunsen git tag. Flakeyness would be inferred by statistical analysis from those runs (and among all the other "similar" runs in the database), so no need for the final synthetic results really, but that too could be committed separately as a new testrun ("/summary"?). > What information do you extract from the .log files? Nothing specifically extracted, but stored and parsed lightly, so that when a user wants to discover why a test failed, the appropriate segment of the appropriate log file can be easily displayed. Availability of that info can be key for diagnosis. > > - for buildbot builds on sourceware, we collect & parse the autoconf > > config.log files as first class content, rather than just informal > > console logs > > We can gather config.log files. > [...] Sweet. - FChE ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: bunsen to assist ci/cd testsuite historical analysis needs 2024-01-16 16:15 ` Frank Ch. Eigler @ 2024-01-16 16:49 ` Christophe Lyon 2024-01-16 17:10 ` Frank Ch. Eigler 0 siblings, 1 reply; 14+ messages in thread From: Christophe Lyon @ 2024-01-16 16:49 UTC (permalink / raw) To: Frank Ch. Eigler; +Cc: Maxim Kuvyrkov, bunsen On Tue, 16 Jan 2024 at 17:15, Frank Ch. Eigler <fche@redhat.com> wrote: > > Hi - > > > > > - separating the .log and .sum files into different directories stops > > > bunsen's crossmatching of the two; bunsen can accept the build tree > > > raw locations of these files, without the "sumfiles" / "00-sumfiles" > > > type of reorg > > > We partially re-run the testsuites when results change unexpectedly. > > This allows us to detect and ignore flaky tests. 00-sumfiles/ is an > > auxiliary directory that contains all run and re-run logs and .sum > > files; sumfiles/ directory contains the final [synthetic] .sum files > > that are a merge of all re-run .sum files. All this is to say that > > we don't necessarily have the matching logs for the final > > [synthetic] .sum files. > > Interesting. In the bunsen model, that would be better represented as > individual testrun commits for each of the run attempts, including the Do you mean each testrun includes the whole testsuite? In our case, only the first run includes the whole testsuite, the subsequent ones include only the .exp files for which we detected failures, to keep testing time reasonably low. > raw .log/.sum files from each run. To tell them apart, one could just > identify each rerun with a "/NNN" suffix in the bunsen git tag. > Flakeyness would be inferred by statistical analysis from those runs > (and among all the other "similar" runs in the database), so no need > for the final synthetic results really, but that too could be committed > separately as a new testrun ("/summary"?). > > > What information do you extract from the .log files? > > Nothing specifically extracted, but stored and parsed lightly, so that > when a user wants to discover why a test failed, the appropriate > segment of the appropriate log file can be easily displayed. > Availability of that info can be key for diagnosis. > > > > - for buildbot builds on sourceware, we collect & parse the autoconf > > > config.log files as first class content, rather than just informal > > > console logs > > > > We can gather config.log files. > > [...] > > Sweet. > > > - FChE > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: bunsen to assist ci/cd testsuite historical analysis needs 2024-01-16 16:49 ` Christophe Lyon @ 2024-01-16 17:10 ` Frank Ch. Eigler 2024-04-15 15:06 ` Frank Ch. Eigler 0 siblings, 1 reply; 14+ messages in thread From: Frank Ch. Eigler @ 2024-01-16 17:10 UTC (permalink / raw) To: Christophe Lyon; +Cc: Maxim Kuvyrkov, bunsen Hi - > > Interesting. In the bunsen model, that would be better represented as > > individual testrun commits for each of the run attempts, including the > > Do you mean each testrun includes the whole testsuite? Each testrun includes just as much of the testsuite as you wish. > In our case, only the first run includes the whole testsuite, the > subsequent ones include only the .exp files for which we detected > failures, to keep testing time reasonably low. That makes sense (unless you get a false positive PASS in the first run, don't bother rerun, and miss the flake!). In any case, bunsen would gladly accept the logs generated from each of the partial reruns. - FChE ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: bunsen to assist ci/cd testsuite historical analysis needs 2024-01-16 17:10 ` Frank Ch. Eigler @ 2024-04-15 15:06 ` Frank Ch. Eigler 2024-04-18 14:44 ` Christophe Lyon 0 siblings, 1 reply; 14+ messages in thread From: Frank Ch. Eigler @ 2024-04-15 15:06 UTC (permalink / raw) To: Christophe Lyon, Maxim Kuvyrkov; +Cc: bunsen Hi - Have y'all had time to try to start feeding some of your ci/cd logs into sourceware's bunsen repo? Is there any obstacle at all from our side? - FChE ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: bunsen to assist ci/cd testsuite historical analysis needs 2024-04-15 15:06 ` Frank Ch. Eigler @ 2024-04-18 14:44 ` Christophe Lyon 2024-04-18 14:50 ` Frank Ch. Eigler 0 siblings, 1 reply; 14+ messages in thread From: Christophe Lyon @ 2024-04-18 14:44 UTC (permalink / raw) To: Frank Ch. Eigler; +Cc: Maxim Kuvyrkov, bunsen Hi Frank, Sorry for not coming back to you earlier. We've been busy with improving our framework, and did not have time yet to look at bunsen. I hope we can start to have a look after the Linaro Connect conference (mid-May). At this stage, I don't think we need anything from you. IIUC, we just need to figure out how to replicate the steps from bunsen_logfile_upload_cpio_steps or bunsen_logfile_upload_steps in our scripts? Thanks, Christophe On Mon, 15 Apr 2024 at 17:06, Frank Ch. Eigler <fche@redhat.com> wrote: > > Hi - > > Have y'all had time to try to start feeding some of your ci/cd logs into > sourceware's bunsen repo? Is there any obstacle at all from our side? > > - FChE > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: bunsen to assist ci/cd testsuite historical analysis needs 2024-04-18 14:44 ` Christophe Lyon @ 2024-04-18 14:50 ` Frank Ch. Eigler 0 siblings, 0 replies; 14+ messages in thread From: Frank Ch. Eigler @ 2024-04-18 14:50 UTC (permalink / raw) To: Christophe Lyon; +Cc: Maxim Kuvyrkov, bunsen Hi - > [...] IIUC, we just need to figure out how to replicate the steps > from bunsen_logfile_upload_cpio_steps or bunsen_logfile_upload_steps > in our scripts? Well, even less than that. All the system takes as input are tagged git commits that contain all the noteworthy log files from a particular testsuite run. We also offer this little dinky script that is one way of automating it. https://sourceware.org/git/?p=bunsen.git;a=blob;f=bin/t-upload-git-push;h=a7977aafffe08ffb0c57261a950c78d1776c265d;hb=HEAD - FChE ^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2024-04-18 14:50 UTC | newest] Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2023-12-12 18:50 bunsen to assist ci/cd testsuite historical analysis needs Frank Ch. Eigler 2023-12-14 15:45 ` Maxim Kuvyrkov 2024-01-02 22:13 ` Frank Ch. Eigler 2024-01-03 12:59 ` Maxim Kuvyrkov 2024-01-04 2:34 ` Frank Ch. Eigler 2024-01-11 13:42 ` Maxim Kuvyrkov 2024-01-11 17:51 ` Frank Ch. Eigler 2024-01-16 10:28 ` Maxim Kuvyrkov 2024-01-16 16:15 ` Frank Ch. Eigler 2024-01-16 16:49 ` Christophe Lyon 2024-01-16 17:10 ` Frank Ch. Eigler 2024-04-15 15:06 ` Frank Ch. Eigler 2024-04-18 14:44 ` Christophe Lyon 2024-04-18 14:50 ` Frank Ch. Eigler
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).