Are we fast yet?

Are we fast yet?

[Dimitrios Apostolou]

Hello list!

you may have thought I'd disappeared but here I am, after weeks of working 
on a small project of mine. What started as a very small hack for 
visualising callgrind's profiles is now an -experimental still- website, a 
GCC version of mozilla's "arewefastyet.com". For now you can see it at:

http://teras-ics.mooo.com:8003/

Of course it has lots of problems so I'd appreciate all comments. Since 
I didn't have a clue about web development, let's say that the UI is not 
/that/ intuitive:

* The dots on the first plot are clickable and each open a different 
profile breakdown.

* In each profile breakdown, functions are selectable and the blue thing 
that shows at the bottom is actually a link (I know it doesn't show...) 
that loads the same profile, but with the selected function as baseline.

* The profile breakdown CGI has also a hidden query string argument that I 
couldn't find a nice way to expose in the SVG: &f=funcname selects the 
function by name, e.g. if you want to find out the hottest callers for 
malloc you can go to:

http://teras-ics.mooo.com:8003/cgi-bin/callgrind_svg.cgi?r=4.8-20120506&f=xmalloc

Many more small SVG issues took most of my time but I won't start here, 
I'll save my wrath for the SVG-developers list. :-)


Besides the UI issues I still have some other open issues to think of, 
ideas are welcome here too:

* Some function names are not demangled correctly, I'll first try 
valgrind's trunk and maybe demangle anything that remains manually.

* Compile all GCC versions against specific glibc (and gmp/mpc/mpfr 
secondly) to avoid fluctuations when the system upgrades. I've tried 
linking statically and works well (as a bonus it also resolves all the 
name mangling issues) but I'm really short on space now... so what's the 
recommended way to link GCC against specific glibc (I was expecting some 
--with-libc configure option)?

* Expose more of the data valgrind gives (like full source annotation)

* Unfortunately Callgrind doesn't save the full stack trace so what you 
see is a statistical breakdown for callees. It doesn't necessarily mean 
that a call path displayed actually exists deeper than its first level.
But the numbers add-up so this is minor.


So I started this thinking that there was not something similar for GCC. 
Until I heard that http://gcc.opensuse.org actually measures compilation 
time and memory usage a long time now. Hopefully this project of mine adds 
some extra value by focusing cleanly on GCC's performance and by allowing 
to actually spot the regressions easily.

To demonstrate this I annotated the frontpage plot manually, that is I 
wrote the comments after diving into the callgraphs. It actually took only 
a few minutes of switching tabs back and forth to notice the differences, 
so I created the annotations as an example. (On another note this led me 
to notice that a few patches of mine actually made it in trunk, cool. :-)


Last but not least credits to the websites that inspired me, namely 
Mozilla's "arewefastyet.com" [1] and the original flamegraph at [2].

[1] http://arewefastyet.com/
[2] http://dtrace.org/blogs/brendan/2012/03/17/linux-kernel-performance-flame-graphs/


Thanks,
Dimitris


P.S. For anyone interested, I've put the main source code -parsing of 
Callgrind output and SVG generation, but /not/ the testing scripts/cron 
jobs- in my private directory at Launchpad, but it's still very ugly and 
hacky so beware: 
http://bazaar.launchpad.net/~jimis/+junk/callgrind_parser/files

Re: Are we fast yet?

[Hans-Peter Nilsson]

On Thu, 28 Jun 2012, Dimitrios Apostolou wrote:
> you may have thought I'd disappeared but here I am, after weeks of working on
> a small project of mine. What started as a very small hack for visualising
> callgrind's profiles is now an -experimental still- website, a GCC version of
> mozilla's "arewefastyet.com". For now you can see it at:
>
> http://teras-ics.mooo.com:8003/

Love the brief what-was-in-that-making/breaking-revision
analysis!  Probably won't use it for anything deeper or more
detailed.  I guess people who can change the figures would
similarly follow-up by their own analysis, but I may be biased.
I also appreciate that you put up the callgrind analysis and svg
code, not that I'm considering using it, but anyway.

brgds, H-P

Re: Are we fast yet?

[Sergei Trofimovich]

[-- Attachment #1: Type: text/plain, Size: 351 bytes --]

> * Unfortunately Callgrind doesn't save the full stack trace so what you 
> see is a statistical breakdown for callees. It doesn't necessarily mean 
> that a call path displayed actually exists deeper than its first level.
> But the numbers add-up so this is minor.

You might give a try to --num-callers= valgrind's option.

-- 

  Sergei

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Are we fast yet?

[Dimitrios Apostolou]

On Sun, 1 Jul 2012, Sergei Trofimovich wrote:

>> * Unfortunately Callgrind doesn't save the full stack trace so what you
>> see is a statistical breakdown for callees. It doesn't necessarily mean
>> that a call path displayed actually exists deeper than its first level.
>> But the numbers add-up so this is minor.
>
> You might give a try to --num-callers= valgrind's option.

Thanks for the tip, but it doesn't seem to help. The reason is that 
Callgrind's output format only tells you that you call a function X number 
of times from function Y. So you may only assume that you've reached 
function Y from any of its callers.


Dimitris

Re: Are we fast yet?

[Dimitrios Apostolou]

Hello,

On Thu, 28 Jun 2012, Dimitrios Apostolou wrote:
>
> http://teras-ics.mooo.com:8003/
>

I've updated the page with some more measurements, please let me know what 
you think.

The primary issue I've not been able to resolve is with function name
demangling: Valgrind uses libiberty's demangler, which gives 
non-consistent names to functions. For example some functions are 
name-only-no-parentheses (glibc ones), others are 
name-with-empty-parentheses. Others are name-parens-with-args and others 
also have a [constprop.7] like thing in brackets. (I do all compilations 
with -O2 -fno-inline, I believe this has something to do with it).

You can see a visible effect of this issue on the last graph that shows 
the ten hottest functions, in particular record_reg_classes: before 
gcc-4.8.20120513 the function name was being demangled to a different name 
than afterwards, so it is plotted as it was zero before then. Demangled 
names:

before 20120513: record_reg_classes(int, int, rtx_def**, machine_mode*, char const**, rtx_def*, reg_class*) [clone .constprop.7]
after 20120513 : record_reg_classes(int, int, rtx_def**, machine_mode*, char const**, rtx_def*, reg_class*) [clone .constprop.6]

What do you think?


I've got some more ideas about information to add. Primarily I want to add 
a C++ testcase. In the past I've got some feedback about Boost.Spirit 
being very template-heavy and stressful for GCC, but I'd prefer to add a 
testcase from within the GCC tree, do you have something in mind?

Finally what do you think on hosting this page on gcc.gnu.org, after 
improving the details left? Maybe we can give it a dedicated subdomain: 
arewefastyet.gcc.gnu.org. From the technical point of view I need CGI, 
python3 and restricted SFTP access to upload new data regularly.


Thanks,
Dimitris

Re: Are we fast yet?

[Dimitrios Apostolou]

Hello,

I've finished some final updates, according to recommendations you 
proposed. Please let me know what you think.

On Wed, 4 Jul 2012, Dimitrios Apostolou wrote:
> On Thu, 28 Jun 2012, Dimitrios Apostolou wrote:
>> 
>> http://teras-ics.mooo.com:8003/
>>


* 4 tests are now benchmarked: empty.c, reload.c with -g3 -O0, reload.c 
with -g -O2, XPath.cpp with -g -O2 from xalanc (someone mentioned xalanc 
benchmark of SPEC2006 as good way to stress cc1plus, unfortunately I 
couldn't find SPEC source so I used one of the most complex files of 
xalanc).

* Instruction count on the left, time on the right.

* All points on the first graph are clickable so you can view the 
callgraph and click through it to the function you are interested. You can 
also edit directly the URL to achieve this, e.g. by adding 
&f=df_install_refs to load that function.

* Memory usage is on the second graph



Thanks,
Dimitris

P.S. I'll refrain from doing any more web development until the end of 
GSOC, but all my cron jobs are running regularly so this content is being 
updated automatically. Next step on this front is to actually bring this 
online at gcc.gnu.org, I'll contact overseers@gcc.gnu.org for this.