[PATCH] Add support for multiple data points to perf tests

[PATCH] Add support for multiple data points to perf tests

[Doug Evans]

Hi.

This patch does two things.
1) Add support for multiple data points.
2) Move the "report" output from perftest.log to perftest.sum.

I want to record the raw data somewhere, and a bit of statistical analysis
(standard deviation left for another day), but I also don't want
it to clutter up the basic report.
This patch takes a cue from gdb.{sum,log} and does the same thing
with perftest.{sum,log}.

Regression tested on amd64-linux.

2015-07-26  Doug Evans  <xdje42@gmail.com>

	* lib/perftest/reporter.py (SUM_FILE_NAME): New global.
	(LOG_FILE_NAME): New global.
	(TextReporter.__init__): Initialize self.txt_sum.
	(TextReporter.report): Add support for multiple data-points.
	Move report to perftest.sum, put raw data in perftest.log.
	(TextReporter.start): Open sum and log files.
	(TextReporter.end): Close sum and log files.
	* lib/perftest/testresult.py (SingleStatisticTestResult.record): Handle
	multiple data-points.

diff --git a/gdb/testsuite/gdb.perf/lib/perftest/reporter.py b/gdb/testsuite/gdb.perf/lib/perftest/reporter.py
index 1f9358c..646051e 100644
--- a/gdb/testsuite/gdb.perf/lib/perftest/reporter.py
+++ b/gdb/testsuite/gdb.perf/lib/perftest/reporter.py
@@ -13,6 +13,15 @@
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
+# Text reports are written here.
+# This is the perftest counterpart to gdb.sum.
+SUM_FILE_NAME = "perftest.sum"
+
+# Raw data that went into the report is written here.
+# This is the perftest counterpart to gdb.log.
+LOG_FILE_NAME = "perftest.log"
+
+
 class Reporter(object):
     """Base class of reporter to report test results in a certain format.
 
@@ -43,22 +52,34 @@ class Reporter(object):
         """
         raise NotImplementedError("Abstract Method:end.")
 
+
 class TextReporter(Reporter):
     """Report results in a plain text file 'perftest.log'."""
 
     def __init__(self, append):
         super (TextReporter, self).__init__(Reporter(append))
+        self.txt_sum = None
         self.txt_log = None
 
-    def report(self, *args):
-        self.txt_log.write(' '.join(str(arg) for arg in args))
-        self.txt_log.write('\n')
+    def report(self, test_name, measurement_name, data_points):
+        if len(data_points) == 0:
+            self.txt_sum.write("%s %s *no data recorded*\n" % (
+                test_name, measurement_name))
+            return
+        average = sum(data_points) / len(data_points)
+        data_min = min(data_points)
+        data_max = max(data_points)
+        self.txt_sum.write("%s %s %s\n" % (
+            test_name, measurement_name, average))
+        self.txt_log.write("%s %s %s, min %s, max %s, data %s\n" % (
+            test_name, measurement_name, average, data_min, data_max,
+            data_points))
 
     def start(self):
-        if self.append:
-            self.txt_log = open ("perftest.log", 'a+');
-        else:
-            self.txt_log = open ("perftest.log", 'w');
+        mode = "a+" if self.append else "w"
+        self.txt_sum = open (SUM_FILE_NAME, mode);
+        self.txt_log = open (LOG_FILE_NAME, mode);
 
     def end(self):
+        self.txt_sum.close ()
         self.txt_log.close ()
diff --git a/gdb/testsuite/gdb.perf/lib/perftest/testresult.py b/gdb/testsuite/gdb.perf/lib/perftest/testresult.py
index cf39808..c799d38 100644
--- a/gdb/testsuite/gdb.perf/lib/perftest/testresult.py
+++ b/gdb/testsuite/gdb.perf/lib/perftest/testresult.py
@@ -35,7 +35,10 @@ class SingleStatisticTestResult(TestResult):
         self.results = dict ()
 
     def record(self, parameter, result):
-        self.results[parameter] = result
+        if parameter in self.results:
+            self.results[parameter].append(result)
+        else:
+            self.results[parameter] = [result]
 
     def report(self, reporter, name):
         reporter.start()

Re: [PATCH] Add support for multiple data points to perf tests

[Yao Qi]

Doug Evans <xdje42@gmail.com> writes:

> This patch does two things.
> 1) Add support for multiple data points.
> 2) Move the "report" output from perftest.log to perftest.sum.
>

What does perftest.sum look like if we write test results by a different
reporter? say xml reporter or jason reporter?

> I want to record the raw data somewhere, and a bit of statistical analysis
> (standard deviation left for another day), but I also don't want
> it to clutter up the basic report.
> This patch takes a cue from gdb.{sum,log} and does the same thing
> with perftest.{sum,log}.

I'd like pertest framework only writes raw data down in some formats
(such as text, xml or jason) without any analysis.  Statistical analysis
can be performed in a post-mortem way by other scripts.

-- 
Yao (齐尧)

Re: [PATCH] Add support for multiple data points to perf tests

[Doug Evans]

On Mon, Jul 27, 2015 at 7:20 AM, Yao Qi <qiyaoltc@gmail.com> wrote:
> Doug Evans <xdje42@gmail.com> writes:
>
>> This patch does two things.
>> 1) Add support for multiple data points.
>> 2) Move the "report" output from perftest.log to perftest.sum.
>>
>
> What does perftest.sum look like if we write test results by a different
> reporter? say xml reporter or jason reporter?

Hi.
This question feels misguided, perhaps I haven't
sufficiently explained what I'm doing.
These same questions can be asked of perftest.log in the current tree.
All I've done is rename perftest.log to perftest.sum,
and then put more test results in text form in perftest.log.
I *could* put it all in one file, but I like what's in perftest.log today,
(though there are a few tweaks I'd like to make),
I'm just moving it to perftest.sum.
Following the pattern of file naming already in use
will be helpful to everyone using the perf testsuite.

If/when we have other forms of output I would expect them
to not go to .log or .sum files.

>> I want to record the raw data somewhere, and a bit of statistical analysis
>> (standard deviation left for another day), but I also don't want
>> it to clutter up the basic report.
>> This patch takes a cue from gdb.{sum,log} and does the same thing
>> with perftest.{sum,log}.
>
> I'd like pertest framework only writes raw data down in some formats
> (such as text, xml or jason) without any analysis.  Statistical analysis
> can be performed in a post-mortem way by other scripts.

We can generate a .csv file with the raw data if/when the need arises.
[or whatever is easiest for post-processing scripts to read]

A human readable form with basic statistical analysis
is useful today, and doable in 5 lines of code.
Let's add the complexity when there's a demonstrated
need for it.

Re: [PATCH] Add support for multiple data points to perf tests

[Yao Qi]

Doug Evans <xdje42@gmail.com> writes:

> All I've done is rename perftest.log to perftest.sum,
> and then put more test results in text form in perftest.log.

Oh, I see.

> I *could* put it all in one file, but I like what's in perftest.log today,
> (though there are a few tweaks I'd like to make),
> I'm just moving it to perftest.sum.
> Following the pattern of file naming already in use
> will be helpful to everyone using the perf testsuite.

Right.  To be clear, with your patch applied, raw data is saved in
perftest.sum and some basic statistical analysis is output to
perftest.log.

>
> If/when we have other forms of output I would expect them
> to not go to .log or .sum files.
>

I agree.

>>> I want to record the raw data somewhere, and a bit of statistical analysis
>>> (standard deviation left for another day), but I also don't want
>>> it to clutter up the basic report.
>>> This patch takes a cue from gdb.{sum,log} and does the same thing
>>> with perftest.{sum,log}.
>>
>> I'd like pertest framework only writes raw data down in some formats
>> (such as text, xml or jason) without any analysis.  Statistical analysis
>> can be performed in a post-mortem way by other scripts.
>
> We can generate a .csv file with the raw data if/when the need arises.
> [or whatever is easiest for post-processing scripts to read]
>
> A human readable form with basic statistical analysis
> is useful today, and doable in 5 lines of code.
> Let's add the complexity when there's a demonstrated
> need for it.

OK, no problem.  I don't have other questions, and your patch looks good
to me.

-- 
Yao (齐尧)

Re: [PATCH] Add support for multiple data points to perf tests

[Doug Evans]

On Wed, Jul 29, 2015 at 1:45 AM, Yao Qi <qiyaoltc@gmail.com> wrote:
> Doug Evans <xdje42@gmail.com> writes:
>
>> All I've done is rename perftest.log to perftest.sum,
>> and then put more test results in text form in perftest.log.
>
> Oh, I see.
>
>> I *could* put it all in one file, but I like what's in perftest.log today,
>> (though there are a few tweaks I'd like to make),
>> I'm just moving it to perftest.sum.
>> Following the pattern of file naming already in use
>> will be helpful to everyone using the perf testsuite.
>
> Right.  To be clear, with your patch applied, raw data is saved in
> perftest.sum and some basic statistical analysis is output to
> perftest.log.
>
>>
>> If/when we have other forms of output I would expect them
>> to not go to .log or .sum files.
>>
>
> I agree.
>
>>>> I want to record the raw data somewhere, and a bit of statistical analysis
>>>> (standard deviation left for another day), but I also don't want
>>>> it to clutter up the basic report.
>>>> This patch takes a cue from gdb.{sum,log} and does the same thing
>>>> with perftest.{sum,log}.
>>>
>>> I'd like pertest framework only writes raw data down in some formats
>>> (such as text, xml or jason) without any analysis.  Statistical analysis
>>> can be performed in a post-mortem way by other scripts.
>>
>> We can generate a .csv file with the raw data if/when the need arises.
>> [or whatever is easiest for post-processing scripts to read]
>>
>> A human readable form with basic statistical analysis
>> is useful today, and doable in 5 lines of code.
>> Let's add the complexity when there's a demonstrated
>> need for it.
>
> OK, no problem.  I don't have other questions, and your patch looks good
> to me.

Thanks.  Committed.