Re: bunsen (re)design discussion #0: list of topics, review of terminology

["Serhei Makarov" <me@serhei.io>]

On Thu, Apr 7, 2022, at 12:57 PM, Keith Seitz wrote:
> Missing:
>
> Test suite. We've touched on this before, and I just want to reiterate:
> THIS IS VITAL to prevent someone (realistically *me*) from breaking
> your work. IIUC SystemTap and GDB are the only two projects currently
> using (or attempting to use) Bunsen. I'd like to (minimally) see some
> end-to-end test suite available.
Agree that this should be a priority right now.
The breakage could easily go in the other direction too if I take some
parsing code that currently works for you and try to tidy or generalize it.

Ok, I think my task #1 in connection with that
is to get some test input data committed to the repo.
For SystemTap, since we were considering the raw logs from our Buildbot
setup to be 'private' (they run on internal VMs on internal hosts), 
that was always an obstacle to putting examples in the public repo.
What I'm going to do is set up a couple of VMs on my 'hobby' system
and have them generate sample test results.

For end-to-end test actions, I believe we should do some README
driven development. I already wrote some notes for Martin Cermak
on how to adapt Bunsen for his needs. Since they tell the story of
a complete Bunsen setup (create, import data, verify the import,
generate some reports) they would be a good starting point
for a set of test cases. A similar document for GDB could produce
an additional set of end-to-end test cases.

> Last year my main goal was to improve GDB's DejaGNU parser so that it could
> output identical results to DejaGNU's summary. [This is the +summarize script.]
> That was the first testing feature that I wanted. The next is to verify
> log output/cursor positions. This is another fundamental feature of the
> service I want to implement, comparing test .log output between arbitrary
> testruns.
Aha. If I understand you correctly, that gives me a couple of
actionable items to implement, by the way:
- show (a subset of) testcases, with .log output inline
- show (a subset of) differing testcases between two testruns, with .log output inline

> I have a list of wish-list items/cleanups that I'd like to see addressed, but
> I'll save those for a more appropriate time.
Okay, sure. In that case, I'll keep you updated on what I'm planning to implement
right now in connection with your needs, and you can let me know if one of
your back-of-mind wishlist items logically takes priority.

Right now, first item on that list is testsuite,
second is more options for .log output viewing.

>
>> (0) Review of Terminology
>
> Nice! I'm good with this.
Excellent, though I'll need to write an updated version of these
notes to accommodate some of the concerns that came up
in more recent discussions with Frank. The terminology part
should not be changing wildly, though. The major new concept is:
- testlogs repo: a Git repo containing only test result logs, without the index.

All the best,
      Serhei