Re: Initial findings of bunsen performance and questions

["Serhei Makarov" <me@serhei.io>]

Hello Keith,

Thanks for the extensive work and experimentation.
I'm not sure your show_test script should be that slow and I hope to get to the bottom of why that's happening for you.
At the same time, your SQLite database work should be very useful and necessary for other reasons.

On Wed, Sep 16, 2020, at 6:18 PM, Keith Seitz via Bunsen wrote:
> The first question to ask is: Is this an oversimplified/naive implementation
> of this script?
It looks right to me. I briefly suspected the lambda might do some additional
object copying, but that doesn't seem to be the case.

> Continuing: This script takes over twenty seconds to complete on my computer.
> IMO that is a show-stopping problem.
Hmm. Very odd.

A script would take very long to complete if it was a 'column query'
(e.g. looking at one testcase across *all* versions would require parsing the JSON
for every testrun -- essentially the entire repo -- and throwing away most of that data).
There's good reasons to run queries like that, which makes your concerns legitimate.

This doesn't look like it. It reads just one testrun. My brief tests of your script yielded
~3sec to run against a SystemTap testrun and ~1.5s against a GDB buildbot run.
Since it reads one particular testrun, the scale of the repo is immaterial
(my SystemTap repo has ~2000-3000 runs in it).

I suspect my code for building the repo has some bug when using consolidate_pass=False.
Could you place the Git/JSON repo you built somewhere I have access to?

(Also, you could try the +diff_runs script on your repo. If the JSON parsing is the source
of the slowdown and reading one run took 20s,
logically reading two runs would take you 40s.)

> There are two quantitative measures that I concentrated on: speed and size.
> 
> First up: Speed. The "show_test.py" script above now completes in less
> than one second (down from 20 seconds). Even as an unoptimized proof-of-concept,
> that is, IMO, adequate performance. [Additional gains could be realized by
> bypassing serialization altogether and using only sql queries.]
> 
> I took two measurements (with consolidate_pass=False to get all test results
> stored):
I'll need to look further into whether that option is appropriate.

> 2) With eight additional, different gdb.{log,sum} files imported on top of
>    #1. Bunsen used 46340kB of storage. The database approach used 49400kB.
>    That's a 6.6% difference in just ten runs.
IMO the comparison has to be done with 100s to 1000s of similar test runs
since Git's de-duplication must be compared to whatever SQLite does,
at that scale of data.
I doubt it's important though, for this use case we have disk space to burn
and the query speedup even justifies keeping both forms of storage.
 
> Is this an approach that seems viable as a supplement (replacement?)
> to JSON output? Is this approach something worth pursuing?
Definitely worth pursuing due to the aforementioned possibility of 'column queries'
which I don't see any way of handling well with the design I currently have.

I'm not sure if SQLite is better used as a replacement for the JSON/Git storage
or as a supplemental cache built from it and used to speed up queries.
(Also, the original log files must be retained in any case.)

All the best,
     Serhei