public inbox for bunsen@sourceware.org
 help / color / mirror / Atom feed
* bunsen (re)design discussion #0: list of topics, review of terminology
@ 2022-03-07 18:59 Serhei Makarov
  2022-03-07 19:03 ` Serhei Makarov
  0 siblings, 1 reply; 7+ messages in thread
From: Serhei Makarov @ 2022-03-07 18:59 UTC (permalink / raw)
  To: Bunsen

I'm in the process of re-evaluating and re-working some core aspects of the Bunsen code, with input from Frank Ch. Eigler. This will result in several public emails like this one as I sort through various concerns.

Initial list of topics I need to cover is below. Incidentally, it approximately matches the sequence in which Bunsen concepts would need to be introduced in a Bunsen man page or tutorial.

(0) Review of Terminology
(1) Testrun Representation
  - version identifiers (source_commit, package_nvr) & sorting
  - configuration & whether to allow 'arbitrary' metadata
(2) Repository Layout
  - splitting testruns across branches, fche's suggestion
  - speculation: should we adopt SQLite? or borrow ideas from Django ORM?
(3) Analysis Pipeline
  - splitting querying from analysis from formatting
  - Index class represents the result of a query
  - Table class represents data for formatting (as plaintext, HTML, etc.)
  - configuration file format: borrowing from the 'Makefile' metaphor to describe items of data and reports Bunsen produces, and how to refresh each item when its dependencies change

===

(0) Review of Terminology

Bunsen repo: collection of test result data managed by Bunsen

source repo: Git repo with the source code of the project being tested

project: identifier of the project being tested, e.g. 'gdb', 'systemtap'
- One Bunsen repo can include data from multiple projects.
   - By default, Bunsen analysis is run on data from one project at a time. This can be overridden by providing a list of projects e.g. "--project=systemtap,systemtap-incomplete".
   - This is useful for a setup such as gcc-jenkins http://gcc.gnu.org/jenkins / http://3.14.90.209:8080/ which produces results for several projects such as gcc,ld,... in a single testsuite run.
   - This is also useful when we want to split out a category of data for separate analysis (e.g. projectX-incomplete for projectX testsuite runs that crashed partway through and therefore don't include full results for all testcases).

testlog: test result logs (a file tree of plaintext files produced by a testsuite run)
- For example, testlogs from a SystemTap testsuite run might include systemtap.{sum,log} from the DejaGNU testsuite, plus additional diagnostics such as systemtap.dmesg, stap-report, sysinfo.

testrun: the parsed representation of a testsuite run, currently stored in JSON format
- More details on what this includes in the next email.

(May have further clarifications / definitions of terms in replies to this thread.)

All the best,
    Serhei
    http://serhei.io

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: bunsen (re)design discussion #0: list of topics, review of terminology
  2022-03-07 18:59 bunsen (re)design discussion #0: list of topics, review of terminology Serhei Makarov
@ 2022-03-07 19:03 ` Serhei Makarov
  2022-04-07 16:57   ` Keith Seitz
  0 siblings, 1 reply; 7+ messages in thread
From: Serhei Makarov @ 2022-03-07 19:03 UTC (permalink / raw)
  To: Bunsen

On Mon, Mar 7, 2022, at 1:59 PM, Serhei Makarov wrote:
> ...
(once more, with proper word-wrapping for plaintext mailers; apologies)

I'm in the process of re-evaluating and re-working some core aspects
of the Bunsen code, with input from Frank Ch. Eigler. This will result
in several public emails like this one as I sort through various
concerns.

Initial list of topics I need to cover is below. Incidentally, it
approximately matches the sequence in which Bunsen concepts would need
to be introduced in a Bunsen man page or tutorial.

(0) Review of Terminology
(1) Testrun Representation
  - version identifiers (source_commit, package_nvr) & sorting
  - configuration & whether to allow 'arbitrary' metadata
(2) Repository Layout
  - splitting testruns across branches, fche's suggestion
  - speculation: should we adopt SQLite? or borrow ideas from Django ORM?
(3) Analysis Pipeline
  - splitting querying from analysis from formatting
  - Index class represents the result of a query
  - Table class represents data for formatting (as plaintext, HTML, etc.)
  - configuration file format: borrowing from the 'Makefile' metaphor
  to describe items of data and reports Bunsen produces, and how to
  refresh each item when its dependencies change

===

(0) Review of Terminology

Bunsen repo: collection of test result data managed by Bunsen

source repo: Git repo with the source code of the project being tested

project: identifier of the project being tested, e.g. 'gdb', 'systemtap'
- One Bunsen repo can include data from multiple projects.
   - By default, Bunsen analysis is run on data from one project at a
     time. This can be overridden by providing a list of projects
     e.g. "--project=systemtap,systemtap-incomplete".
   - This is useful for a setup such as gcc-jenkins
     http://gcc.gnu.org/jenkins / http://3.14.90.209:8080/ which
     produces results for several projects such as gcc,ld,... in a
     single testsuite run.
   - This is also useful when we want to split out a category of data
     for separate analysis (e.g. projectX-incomplete for projectX
     testsuite runs that crashed partway through and therefore don't
     include full results for all testcases).

testlog: test result logs (a file tree of plaintext files produced by a testsuite run)
- For example, testlogs from a SystemTap testsuite run might include
  systemtap.{sum,log} from the DejaGNU testsuite, plus additional
  diagnostics such as systemtap.dmesg, stap-report, sysinfo.

testrun: the parsed representation of a testsuite run, currently stored in JSON format
- More details on what this includes in the next email.

(May have further clarifications / definitions of terms in replies to
this thread.)

> All the best,
>     Serhei
>     http://serhei.io

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: bunsen (re)design discussion #0: list of topics, review of terminology
  2022-03-07 19:03 ` Serhei Makarov
@ 2022-04-07 16:57   ` Keith Seitz
  2022-04-08 23:13     ` Serhei Makarov
  0 siblings, 1 reply; 7+ messages in thread
From: Keith Seitz @ 2022-04-07 16:57 UTC (permalink / raw)
  To: Serhei Makarov, Bunsen

On 3/7/22 11:03, Serhei Makarov wrote:
> On Mon, Mar 7, 2022, at 1:59 PM, Serhei Makarov wrote:
>> ...
> (once more, with proper word-wrapping for plaintext mailers; apologies)
> 
> I'm in the process of re-evaluating and re-working some core aspects
> of the Bunsen code, with input from Frank Ch. Eigler. This will result
> in several public emails like this one as I sort through various
> concerns.

This is excellent, and while the timing might be uncomfortably "late"
for my tastes, I am very glad to see this formalization happening. I
am hoping to put Bunsen into production use later this year.

> Initial list of topics I need to cover is below. Incidentally, it
> approximately matches the sequence in which Bunsen concepts would need
> to be introduced in a Bunsen man page or tutorial.
> 
> (0) Review of Terminology
> (1) Testrun Representation
>    - version identifiers (source_commit, package_nvr) & sorting
>    - configuration & whether to allow 'arbitrary' metadata
> (2) Repository Layout
>    - splitting testruns across branches, fche's suggestion
>    - speculation: should we adopt SQLite? or borrow ideas from Django ORM?
> (3) Analysis Pipeline
>    - splitting querying from analysis from formatting
>    - Index class represents the result of a query
>    - Table class represents data for formatting (as plaintext, HTML, etc.)
>    - configuration file format: borrowing from the 'Makefile' metaphor
>    to describe items of data and reports Bunsen produces, and how to
>    refresh each item when its dependencies change

Missing:

Test suite. We've touched on this before, and I just want to reiterate:
THIS IS VITAL to prevent someone (realistically *me*) from breaking
your work. IIUC SystemTap and GDB are the only two projects currently
using (or attempting to use) Bunsen. I'd like to (minimally) see some
end-to-end test suite available.

Last year my main goal was to improve GDB's DejaGNU parser so that it could
output identical results to DejaGNU's summary. [This is the +summarize script.]
That was the first testing feature that I wanted. The next is to verify
log output/cursor positions. This is another fundamental feature of the
service I want to implement, comparing test .log output between arbitrary
testruns.

I have a list of wish-list items/cleanups that I'd like to see addressed, but
I'll save those for a more appropriate time.

> (0) Review of Terminology

Nice! I'm good with this.

Keithp


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: bunsen (re)design discussion #0: list of topics, review of terminology
  2022-04-07 16:57   ` Keith Seitz
@ 2022-04-08 23:13     ` Serhei Makarov
  2022-04-09  1:56       ` Frank Ch. Eigler
  0 siblings, 1 reply; 7+ messages in thread
From: Serhei Makarov @ 2022-04-08 23:13 UTC (permalink / raw)
  To: Keith Seitz, Bunsen



On Thu, Apr 7, 2022, at 12:57 PM, Keith Seitz wrote:
> Missing:
>
> Test suite. We've touched on this before, and I just want to reiterate:
> THIS IS VITAL to prevent someone (realistically *me*) from breaking
> your work. IIUC SystemTap and GDB are the only two projects currently
> using (or attempting to use) Bunsen. I'd like to (minimally) see some
> end-to-end test suite available.
Agree that this should be a priority right now.
The breakage could easily go in the other direction too if I take some
parsing code that currently works for you and try to tidy or generalize it.

Ok, I think my task #1 in connection with that
is to get some test input data committed to the repo.
For SystemTap, since we were considering the raw logs from our Buildbot
setup to be 'private' (they run on internal VMs on internal hosts), 
that was always an obstacle to putting examples in the public repo.
What I'm going to do is set up a couple of VMs on my 'hobby' system
and have them generate sample test results.

For end-to-end test actions, I believe we should do some README
driven development. I already wrote some notes for Martin Cermak
on how to adapt Bunsen for his needs. Since they tell the story of
a complete Bunsen setup (create, import data, verify the import,
generate some reports) they would be a good starting point
for a set of test cases. A similar document for GDB could produce
an additional set of end-to-end test cases.

> Last year my main goal was to improve GDB's DejaGNU parser so that it could
> output identical results to DejaGNU's summary. [This is the +summarize script.]
> That was the first testing feature that I wanted. The next is to verify
> log output/cursor positions. This is another fundamental feature of the
> service I want to implement, comparing test .log output between arbitrary
> testruns.
Aha. If I understand you correctly, that gives me a couple of
actionable items to implement, by the way:
- show (a subset of) testcases, with .log output inline
- show (a subset of) differing testcases between two testruns, with .log output inline

> I have a list of wish-list items/cleanups that I'd like to see addressed, but
> I'll save those for a more appropriate time.
Okay, sure. In that case, I'll keep you updated on what I'm planning to implement
right now in connection with your needs, and you can let me know if one of
your back-of-mind wishlist items logically takes priority.

Right now, first item on that list is testsuite,
second is more options for .log output viewing.

>
>> (0) Review of Terminology
>
> Nice! I'm good with this.
Excellent, though I'll need to write an updated version of these
notes to accommodate some of the concerns that came up
in more recent discussions with Frank. The terminology part
should not be changing wildly, though. The major new concept is:
- testlogs repo: a Git repo containing only test result logs, without the index.

All the best,
      Serhei

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: bunsen (re)design discussion #0: list of topics, review of terminology
  2022-04-08 23:13     ` Serhei Makarov
@ 2022-04-09  1:56       ` Frank Ch. Eigler
  2022-04-09  2:22         ` Serhei Makarov
  0 siblings, 1 reply; 7+ messages in thread
From: Frank Ch. Eigler @ 2022-04-09  1:56 UTC (permalink / raw)
  To: Serhei Makarov; +Cc: Keith Seitz, Bunsen

Hi -

> > Test suite. We've touched on this before, and I just want to reiterate:
> > THIS IS VITAL to prevent someone (realistically *me*) from breaking
> > your work. [...]

That should be straightforward with the new git/sqlite scheme too.
One can freeze a git repo for fixed testing, or one can edit one
under testsuite control.  And one can easily query the resulting
datasets for expected values.  Should be no problem at all for at
least the anlysis side of the house.

- FChE


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: bunsen (re)design discussion #0: list of topics, review of terminology
  2022-04-09  1:56       ` Frank Ch. Eigler
@ 2022-04-09  2:22         ` Serhei Makarov
  2022-04-09  2:26           ` Frank Ch. Eigler
  0 siblings, 1 reply; 7+ messages in thread
From: Serhei Makarov @ 2022-04-09  2:22 UTC (permalink / raw)
  To: Frank Ch. Eigler; +Cc: Keith Seitz, Bunsen



On Fri, Apr 8, 2022, at 9:56 PM, Frank Ch. Eigler wrote:
> Hi -
>
>> > Test suite. We've touched on this before, and I just want to reiterate:
>> > THIS IS VITAL to prevent someone (realistically *me*) from breaking
>> > your work. [...]
>
> That should be straightforward with the new git/sqlite scheme too.
> One can freeze a git repo for fixed testing, or one can edit one
> under testsuite control.  And one can easily query the resulting
> datasets for expected values.  Should be no problem at all for at
> least the analysis side of the house.
Sure, indeed sticking a frozen sample db into the testsuite might
be essential if we end up designing a schema that is an API,
then end up evolving the schema in a way that requires migrations,
then an existing copy of the db will be an important data source
to test against.

On the other hand, if the db is a purely transient structure
(we can always re-create it from either pure or indexed testlog data,
and 'migration' amounts to re-creating any analysis structures that
are obsolete) then need to test against a frozen db is somewhat less.
Step 1 of the testing is then to create a db, then run various tests
against it. That is, if the implied design promise that db-regeneration is
indeed quick and easy pans out on larger datasets :-P

In any case, after digesting Frank's sample code for sqlite migration,
I have a clearer idea of what pieces to adopt in a way that keeps
continuity with the existing functionality.
I'll do my own quick-experiment-in-a-separate branch to clarify
what I want from the structure.

-- 
All the best,
    Serhei

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: bunsen (re)design discussion #0: list of topics, review of terminology
  2022-04-09  2:22         ` Serhei Makarov
@ 2022-04-09  2:26           ` Frank Ch. Eigler
  0 siblings, 0 replies; 7+ messages in thread
From: Frank Ch. Eigler @ 2022-04-09  2:26 UTC (permalink / raw)
  To: Serhei Makarov; +Cc: Keith Seitz, Bunsen

Hi -

> > That should be straightforward with the new git/sqlite scheme too.
> > One can freeze a git repo for fixed testing, or one can edit one
> > under testsuite control.  And one can easily query the resulting
> > datasets for expected values.  Should be no problem at all for at
> > least the analysis side of the house.
>
> Sure, indeed sticking a frozen sample db into the testsuite might
> be essential if we end up designing a schema that is an API,
> then end up evolving the schema in a way that requires migrations,
> then an existing copy of the db will be an important data source
> to test against.

(I was not talking about including a frozen version of the sqlite
database, only of the git testlog repo ... but whatever, all is
possible.)


> [...]
> In any case, after digesting Frank's sample code for sqlite migration,
> I have a clearer idea of what pieces to adopt in a way that keeps
> continuity with the existing functionality.
> I'll do my own quick-experiment-in-a-separate branch to clarify
> what I want from the structure.

Looking forward to it!


- FChE


^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2022-04-09  2:26 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-03-07 18:59 bunsen (re)design discussion #0: list of topics, review of terminology Serhei Makarov
2022-03-07 19:03 ` Serhei Makarov
2022-04-07 16:57   ` Keith Seitz
2022-04-08 23:13     ` Serhei Makarov
2022-04-09  1:56       ` Frank Ch. Eigler
2022-04-09  2:22         ` Serhei Makarov
2022-04-09  2:26           ` Frank Ch. Eigler

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).