Test coverage data

Test coverage data

[Kris Van Hees]

So, after some struggling with lcov (giving up, and rolling my own gcov
wrapper for Frysk) I got the following preliminary (but pretty much
accurate) results on x86_64 FC6:

Module          % files used      % lines used (in used files)
frysk-imports	   91.34%            56.78%
frysk-gui          84.92%            12.43%
frysk-gtk          47.06%            79.57%
frysk-core         59.78%            42.99%

Re: Test coverage data

[Phil Muldoon]

Kris Van Hees wrote:
> So, after some struggling with lcov (giving up, and rolling my own gcov
> wrapper for Frysk) I got the following preliminary (but pretty much
> accurate) results on x86_64 FC6:
>   

Can you provide the results fo x86 too please?


Regards

Phil

Re: Test coverage data

[Kris Van Hees]

On Thu, Apr 12, 2007 at 02:33:06PM -0700, Roland McGrath wrote:
> What's the trouble with lcov?  I've used it (not on frysk or any java), and
> I think I'm the Fedora Extras maintainer for it.

I can write up a more detailed description of the problems later, but
the summary is that the combination of Java and C++ source code (rather
than C), together with a source/object directory split, and execution of
test executables using different current working directories is making
lcov extremely confused.  Add to that the fact that lcov has only
limited support for dealing with symbolic links.  E.g. if a data file is
a symbolic link that has a relative path as target, things go miserably
wrong.

	Kris

Re: Test coverage data

[Kris Van Hees]

On Thu, Apr 12, 2007 at 04:29:52PM -0500, Phil Muldoon wrote:
> Kris Van Hees wrote:
> >So, after some struggling with lcov (giving up, and rolling my own gcov
> >wrapper for Frysk) I got the following preliminary (but pretty much
> >accurate) results on x86_64 FC6:
> 
> Can you provide the results fo x86 too please?

Those are forthcoming, as soon as I have completed a few tweaks to make
building Frysk with --coverage enabled easier.  Right now it is still a
bit messy to do the build, so I'm holding off on trying to keep two
trees in sync with my changes :)  But I hope to have results for x86 by
tomorrow.

	Kris

Re: Test coverage data

[Kris Van Hees]

After some fine tuning and more tweaks to convince gcov to deal with
files that are not actually used (read: linked in) by the tests, I now
can report on the coverage across all files, rather than just those that
are used:

		executed / total       percentage
frysk-imports     5828   / 11579        50.3325%
frysk-gui         1464   / 12675        11.5503%
frysk-gtk           74   /  2791         2.6514%
frysk-core        8223   / 22556        36.4559%

So, most noteworthy is the fact that the unused portion of code in
frysk-gtk covers the vast majority of code in that module.  Note that
dogtail tests were not executed (only make -k check).  I will do that
manually, and get more results after that for comparison.

Again, these results are for x86_64.  Stand by for x86 results tomorrow
:)

	Kris

On Thu, Apr 12, 2007 at 02:27:53PM -0700, Kris Van Hees wrote:
> So, after some struggling with lcov (giving up, and rolling my own gcov
> wrapper for Frysk) I got the following preliminary (but pretty much
> accurate) results on x86_64 FC6:
> 
> Module          % files used      % lines used (in used files)
> frysk-imports	     91.34%            56.78%
> frysk-gui          84.92%            12.43%
> frysk-gtk          47.06%            79.57%
> frysk-core         59.78%            42.99%

Re: Test coverage data

[Phil Muldoon]

Kris Van Hees wrote:
> On Thu, Apr 12, 2007 at 04:29:52PM -0500, Phil Muldoon wrote:
>   
>> Kris Van Hees wrote:
>>     
>>> So, after some struggling with lcov (giving up, and rolling my own gcov
>>> wrapper for Frysk) I got the following preliminary (but pretty much
>>> accurate) results on x86_64 FC6:
>>>       
>> Can you provide the results fo x86 too please?
>>     
>
> Those are forthcoming, as soon as I have completed a few tweaks to make
> building Frysk with --coverage enabled easier.  Right now it is still a
> bit messy to do the build, so I'm holding off on trying to keep two
> trees in sync with my changes :)  But I hope to have results for x86 by
> tomorrow.
>   
Great, I'd love to see  how coverage changes on different host platforms 
(different platforms = different kernels = different bugs) and also 
different versions of Fedora Core.

Is it also possible to have the data for both x86 an x86_64 for Fedora 
Core 5 as well? There are some tests enabled on FC5, that are not 
enabled on FC6 due to above.

Regards

Phil

Re: Test coverage data

[Kris Van Hees]

On Thu, Apr 12, 2007 at 11:49:42PM -0500, Phil Muldoon wrote:
> Great, I'd love to see  how coverage changes on different host platforms 
> (different platforms = different kernels = different bugs) and also 
> different versions of Fedora Core.
> 
> Is it also possible to have the data for both x86 an x86_64 for Fedora 
> Core 5 as well? There are some tests enabled on FC5, that are not 
> enabled on FC6 due to above.

Sure, provided people who have such systems available are willing to
donate some cycles each night to do an automated build-and-test run.
Now that I have coverage data generation pretty much automated, and
am able to generate meaningful metrics from it easily, I'm going to
integrate that with the automated build-and-test runs, so we get a
nice return on investment with every new system added (both test
results *and* code coverage metrics for those tests).

The frysk build now supports an --enable-coverage option to configure to
trigger compilation of all code with the --coverage option, so coverage
data can be captured.  I also have a script to analyze the data and
report basic metrics (as emailed out earlier today).  That will be added
to the automated runs.  I can also put it in Frysk CVS if that would be
helpful (though I am not sure what the most appropriate location for
something like this would be within the tree).  Either way, the real
value from this will indeed come from being able to see results from a
variety of systems.

But again, it all depends on people being willing to donate cycles.

	Cheers,
	Kris

Re: Test coverage data

[Stan Cox]

On Thu, 2007-04-12 at 18:08 -0700, Kris Van Hees wrote:
> building Frysk with --coverage enabled easier.  

Did you try building with -O0?  I got better coverage results with -O0.
I'll volunteer doing a nightly run on the Intel Xeon in my office.

Re: Test coverage data

[Kris Van Hees]

On Fri, Apr 13, 2007 at 09:56:43AM -0400, Stan Cox wrote:
> On Thu, 2007-04-12 at 18:08 -0700, Kris Van Hees wrote:
> > building Frysk with --coverage enabled easier.  
> 
> Did you try building with -O0?  I got better coverage results with -O0.
> I'll volunteer doing a nightly run on the Intel Xeon in my office.
> 

Ah, my mistake for summarizing my change in a less verbose way.  Yes,
the --enable-coverage option does use '--coverage -O0' rather than just
'--coverage'.

Thanks for volunteering performing nightly runs.  If you can send me a
direct email with the 'uname -a' output on that box, I'll enable uploads
for that box on my end, and email you the password.

    Cheers,
    Kris