Comparison of GCC-4.6.1 and LLVM-2.9 on x86/x86-64 targets

Comparison of GCC-4.6.1 and LLVM-2.9 on x86/x86-64 targets

[Vladimir Makarov]

   Some people asked me to do comparison of  GCC-4.6 and LLVM-2.9 (both
released this spring) as I did GCC-LLVM comparison in previous year.

   You can find it on http://vmakarov.fedorapeople.org/spec under
2011 GCC-LLVM comparison tab entry.


   This year the comparison is done on GCC 4.6 and LLVM 2.9 which were
released in spring 2011.

   As usually I am focused mostly on the compiler comparison
as *optimizing* compilers on major platform x86/x86-64.  I don't
consider other aspects of the compilers as quality of debug
information, supported languages, standards and extensions (e.g. OMP),
supported targets and ABI, support of just-in-time compilation etc.

   Different to the 2010 comparison, the SPEC2000 benchmarks were run on
a recent *Sandy Bridge processor* which will be a mainstream
processor at least for the next year.

   This year I tried to decrease the number of graphs which are still too
many with my point of view.  Some graphs are bigger than for 2010
comparison and oriented to screens with a larger resolution.  If you
need exact numbers you should look at the tables from which the graphs
were generated.

   I added GCC run with -O1 which helps to understand
that *LLVM with -O2 or -O3 is analog of GCC 4.1 with -O1
with the point of view of generated code performance and
compilation speed*.  People are frequently saying that LLVM is a much
faster compiler than GCC.  That is probably not true.  If you need the same
generated code quality and compilation speed as LLVM -O2/-O3
you should use GCC with -O1.  If you want 10%-40% faster
generated code, you should use GCC with -O2/-O3 and you need
20%-40% (150%-200% if you use GCC LTO) more time for compilation.  I
believe that LLVM code performance is far away from GCC because
it is sufficiently easy to get first percents of code improvement, it
becomes much harder to get subsequent percents, and IMHO starting with
some point of the development the relation of the code improvement to
the spent efforts might become exponential.  So there is no magic --
GCC has a better performance because much more efforts of experienced
compiler developers have been spent and are being spent for GCC
development than for LLVM.

   The above said about compilation speed is true when GCC front-end is
used for LLVM.  LLVM has another C-language family front-end called
CLANG which can speed up compilation in optimization mode
(-O2/-O3) upto 20%-25%.  So even as LLVM optimizations
are not faster than GCC optimizations, CLANG front-end is really
faster than GCC-frontend.  I think GCC community should pay more attention
to this fact.  Fortunately, a few new GCC projects address to this problem
and I hope this problem will be solved or alleviated.

   This year I used -Ofast -flto -fwhole-program instead of
-O3 for GCC and -O3 -ffast-math for LLVM for comparison of peak
performance.  I could improve GCC performance even more by using
other GCC possibilities (like support of AVX insns, Graphite optimizations
and even some experimental stuff like LIPO) but I wanted to give LLVM
some chances too.  Probably an experienced user in LLVM could improve
LLVM performance too.  So I think it is a fair comparison.

Re: Comparison of GCC-4.6.1 and LLVM-2.9 on x86/x86-64 targets

[Duncan Sands]

Hi Vladimir, thanks for doing this.

> The above said about compilation speed is true when GCC front-end is
> used for LLVM.

It's not clear to me which GCC front-end you mean.  There is llvm-gcc
(based on gcc-4.2) and the dragonegg plugin (the 2.9 version works with
gcc-4.5; the development version works also with gcc-4.6).  Can you
please clarify.  By the way, some highly unscientific experiments I did
suggest that the GCC tree optimizers are (almost) as fast as the LLVM IR
optimizers while doing a better job; while at -O3 the LLVM code generators
are significantly faster than the GCC code generators and do a comparable
and sometimes better job.  Unfortunately I haven't had time to do a serious
study, so this might just be an accident of the benchmarks I looked at and
the options I happened to use rather than anything meaningful.

Ciao, Duncan.

Re: Comparison of GCC-4.6.1 and LLVM-2.9 on x86/x86-64 targets

[Vladimir Makarov]

On 09/07/2011 11:28 AM, Duncan Sands wrote:
> Hi Vladimir, thanks for doing this.
>
>> The above said about compilation speed is true when GCC front-end is
>> used for LLVM.
>
> It's not clear to me which GCC front-end you mean.  There is llvm-gcc
> (based on gcc-4.2) and the dragonegg plugin (the 2.9 version works with
> gcc-4.5; the development version works also with gcc-4.6).  Can you
> please clarify.

This is not dragonegg.  It was llvm-gcc4.2-2.9-x86_64-linux.
>   By the way, some highly unscientific experiments I did
> suggest that the GCC tree optimizers are (almost) as fast as the LLVM IR
> optimizers while doing a better job;
That is probably not only tree optimizers.  RTL optimizers seem not slow 
as well.

> while at -O3 the LLVM code generators
> are significantly faster than the GCC code generators and do a comparable
> and sometimes better job.  Unfortunately I haven't had time to do a 
> serious
> study, so this might just be an accident of the benchmarks I looked at 
> and
> the options I happened to use rather than anything meaningful.
I frequently see that on small benchmarks some optimizations (I mostly 
work on RA) look better than on bigger benchmarks.  It would be 
interesting to run SPEC2006 for comparison because it contains even 
bigger benchmarks.  Unfortunately, it will take 2 weeks on my machine to 
run SPEC2006 for the same # of option sets.

Re: Comparison of GCC-4.6.1 and LLVM-2.9 on x86/x86-64 targets

[Xinliang David Li]

Why is lto/whole program mode not used in LLVM for peak performance
comparison? (of course, peak performance should really use FDO..)

thanks,

David

On Wed, Sep 7, 2011 at 8:15 AM, Vladimir Makarov <vmakarov@redhat.com> wrote:
>  Some people asked me to do comparison of  GCC-4.6 and LLVM-2.9 (both
> released this spring) as I did GCC-LLVM comparison in previous year.
>
>  You can find it on http://vmakarov.fedorapeople.org/spec under
> 2011 GCC-LLVM comparison tab entry.
>
>
>  This year the comparison is done on GCC 4.6 and LLVM 2.9 which were
> released in spring 2011.
>
>  As usually I am focused mostly on the compiler comparison
> as *optimizing* compilers on major platform x86/x86-64.  I don't
> consider other aspects of the compilers as quality of debug
> information, supported languages, standards and extensions (e.g. OMP),
> supported targets and ABI, support of just-in-time compilation etc.
>
>  Different to the 2010 comparison, the SPEC2000 benchmarks were run on
> a recent *Sandy Bridge processor* which will be a mainstream
> processor at least for the next year.
>
>  This year I tried to decrease the number of graphs which are still too
> many with my point of view.  Some graphs are bigger than for 2010
> comparison and oriented to screens with a larger resolution.  If you
> need exact numbers you should look at the tables from which the graphs
> were generated.
>
>  I added GCC run with -O1 which helps to understand
> that *LLVM with -O2 or -O3 is analog of GCC 4.1 with -O1
> with the point of view of generated code performance and
> compilation speed*.  People are frequently saying that LLVM is a much
> faster compiler than GCC.  That is probably not true.  If you need the same
> generated code quality and compilation speed as LLVM -O2/-O3
> you should use GCC with -O1.  If you want 10%-40% faster
> generated code, you should use GCC with -O2/-O3 and you need
> 20%-40% (150%-200% if you use GCC LTO) more time for compilation.  I
> believe that LLVM code performance is far away from GCC because
> it is sufficiently easy to get first percents of code improvement, it
> becomes much harder to get subsequent percents, and IMHO starting with
> some point of the development the relation of the code improvement to
> the spent efforts might become exponential.  So there is no magic --
> GCC has a better performance because much more efforts of experienced
> compiler developers have been spent and are being spent for GCC
> development than for LLVM.
>
>  The above said about compilation speed is true when GCC front-end is
> used for LLVM.  LLVM has another C-language family front-end called
> CLANG which can speed up compilation in optimization mode
> (-O2/-O3) upto 20%-25%.  So even as LLVM optimizations
> are not faster than GCC optimizations, CLANG front-end is really
> faster than GCC-frontend.  I think GCC community should pay more attention
> to this fact.  Fortunately, a few new GCC projects address to this problem
> and I hope this problem will be solved or alleviated.
>
>  This year I used -Ofast -flto -fwhole-program instead of
> -O3 for GCC and -O3 -ffast-math for LLVM for comparison of peak
> performance.  I could improve GCC performance even more by using
> other GCC possibilities (like support of AVX insns, Graphite optimizations
> and even some experimental stuff like LIPO) but I wanted to give LLVM
> some chances too.  Probably an experienced user in LLVM could improve
> LLVM performance too.  So I think it is a fair comparison.
>

Re: Comparison of GCC-4.6.1 and LLVM-2.9 on x86/x86-64 targets

[Vladimir Makarov]

On 09/07/2011 11:55 AM, Xinliang David Li wrote:
> Why is lto/whole program mode not used in LLVM for peak performance
> comparison? (of course, peak performance should really use FDO..)
>
Thanks for the feedback.  I did not manage to use LTO for LLVM as it 
described on

http://llvm.org/docs/LinkTimeOptimization.html#lto

I am getting 'file not recognized: File format not recognized'  during 
the linkage pass.

You probably right that I should use -Ofast without -flto for gcc then.  
Although I don't think that it significantly change GCC peak 
performance.  Still I am going to run SPEC2000 without -flto and post 
the data (probably on the next week).

As for FDO, unfortunately for some tests SPEC uses different training 
sets and it gives sometimes wrong info for the further optimizations.

I do not look at this comparison as finished work and am going to run 
more SPEC2000 tests and change the results if I have serious reasonable 
objections for the current comparison.

Re: Comparison of GCC-4.6.1 and LLVM-2.9 on x86/x86-64 targets

[Richard Guenther]

On Wed, Sep 7, 2011 at 6:23 PM, Vladimir Makarov <vmakarov@redhat.com> wrote:
> On 09/07/2011 11:55 AM, Xinliang David Li wrote:
>>
>> Why is lto/whole program mode not used in LLVM for peak performance
>> comparison? (of course, peak performance should really use FDO..)
>>
> Thanks for the feedback.  I did not manage to use LTO for LLVM as it
> described on
>
> http://llvm.org/docs/LinkTimeOptimization.html#lto
>
> I am getting 'file not recognized: File format not recognized'  during the
> linkage pass.
>
> You probably right that I should use -Ofast without -flto for gcc then.
>  Although I don't think that it significantly change GCC peak performance.
>  Still I am going to run SPEC2000 without -flto and post the data (probably
> on the next week).

Note that due to a bug in 4.6.x -Ofast is not equivalent to -O3 -ffast-math
(it doesn't use crtfastmath.o).  I'll backport the fix.

> As for FDO, unfortunately for some tests SPEC uses different training sets
> and it gives sometimes wrong info for the further optimizations.
>
> I do not look at this comparison as finished work and am going to run more
> SPEC2000 tests and change the results if I have serious reasonable
> objections for the current comparison.
>

Re: Comparison of GCC-4.6.1 and LLVM-2.9 on x86/x86-64 targets

[Duncan Sands]

>>> Why is lto/whole program mode not used in LLVM for peak performance
>>> comparison? (of course, peak performance should really use FDO..)
>>>
>> Thanks for the feedback.  I did not manage to use LTO for LLVM as it
>> described on
>>
>> http://llvm.org/docs/LinkTimeOptimization.html#lto
>>
>> I am getting 'file not recognized: File format not recognized'  during the
>> linkage pass.

Note that these are the instructions to follow on linux for LTO with llvm-gcc:
   http://llvm.org/docs/GoldPlugin.html

Ciao, Duncan.

>>
>> You probably right that I should use -Ofast without -flto for gcc then.
>>   Although I don't think that it significantly change GCC peak performance.
>>   Still I am going to run SPEC2000 without -flto and post the data (probably
>> on the next week).
>
> Note that due to a bug in 4.6.x -Ofast is not equivalent to -O3 -ffast-math
> (it doesn't use crtfastmath.o).  I'll backport the fix.
>
>> As for FDO, unfortunately for some tests SPEC uses different training sets
>> and it gives sometimes wrong info for the further optimizations.
>>
>> I do not look at this comparison as finished work and am going to run more
>> SPEC2000 tests and change the results if I have serious reasonable
>> objections for the current comparison.
>>

Re: Comparison of GCC-4.6.1 and LLVM-2.9 on x86/x86-64 targets

[Vladimir Makarov]

On 09/07/2011 12:23 PM, Vladimir Makarov wrote:
> On 09/07/2011 11:55 AM, Xinliang David Li wrote:
>> Why is lto/whole program mode not used in LLVM for peak performance
>> comparison? (of course, peak performance should really use FDO..)
>>
> Thanks for the feedback.  I did not manage to use LTO for LLVM as it 
> described on
>
> http://llvm.org/docs/LinkTimeOptimization.html#lto
>
> I am getting 'file not recognized: File format not recognized'  during 
> the linkage pass.
>
> You probably right that I should use -Ofast without -flto for gcc 
> then.  Although I don't think that it significantly change GCC peak 
> performance.  Still I am going to run SPEC2000 without -flto and post 
> the data (probably on the next week).
>
> As for FDO, unfortunately for some tests SPEC uses different training 
> sets and it gives sometimes wrong info for the further optimizations.
>
> I do not look at this comparison as finished work and am going to run 
> more SPEC2000 tests and change the results if I have serious 
> reasonable objections for the current comparison.
I've add -Ofast without -flto -fwhole-program for GCC as well and 
updated the graphs:

http://vmakarov.fedorapeople.org/spec/

Re: Comparison of GCC-4.6.1 and LLVM-2.9 on x86/x86-64 targets

[Duncan Sands]

On 07/09/11 17:55, Xinliang David Li wrote:
> Why is lto/whole program mode not used in LLVM for peak performance
> comparison? (of course, peak performance should really use FDO..)

Assuming Vladimir was using the dragonegg plugin: presumably because it's
a pain: you have to compile everything to assembler (-S) rather than to an
object file (-c).  That's because -flto outputs LLVM IR when used with this
plugin, and the system assembler doesn't understand it (and GCC insists on
sending output to the system assembler if you pass -c).  You then have to
convert each .s into a .o (or .bc) using llvm-as.  At that point if you have
the gold linker and the LLVM linker plugin you can just link using them and
you are done.

Ciao, Duncan.

>
> thanks,
>
> David
>
> On Wed, Sep 7, 2011 at 8:15 AM, Vladimir Makarov<vmakarov@redhat.com>  wrote:
>>   Some people asked me to do comparison of  GCC-4.6 and LLVM-2.9 (both
>> released this spring) as I did GCC-LLVM comparison in previous year.
>>
>>   You can find it on http://vmakarov.fedorapeople.org/spec under
>> 2011 GCC-LLVM comparison tab entry.
>>
>>
>>   This year the comparison is done on GCC 4.6 and LLVM 2.9 which were
>> released in spring 2011.
>>
>>   As usually I am focused mostly on the compiler comparison
>> as *optimizing* compilers on major platform x86/x86-64.  I don't
>> consider other aspects of the compilers as quality of debug
>> information, supported languages, standards and extensions (e.g. OMP),
>> supported targets and ABI, support of just-in-time compilation etc.
>>
>>   Different to the 2010 comparison, the SPEC2000 benchmarks were run on
>> a recent *Sandy Bridge processor* which will be a mainstream
>> processor at least for the next year.
>>
>>   This year I tried to decrease the number of graphs which are still too
>> many with my point of view.  Some graphs are bigger than for 2010
>> comparison and oriented to screens with a larger resolution.  If you
>> need exact numbers you should look at the tables from which the graphs
>> were generated.
>>
>>   I added GCC run with -O1 which helps to understand
>> that *LLVM with -O2 or -O3 is analog of GCC 4.1 with -O1
>> with the point of view of generated code performance and
>> compilation speed*.  People are frequently saying that LLVM is a much
>> faster compiler than GCC.  That is probably not true.  If you need the same
>> generated code quality and compilation speed as LLVM -O2/-O3
>> you should use GCC with -O1.  If you want 10%-40% faster
>> generated code, you should use GCC with -O2/-O3 and you need
>> 20%-40% (150%-200% if you use GCC LTO) more time for compilation.  I
>> believe that LLVM code performance is far away from GCC because
>> it is sufficiently easy to get first percents of code improvement, it
>> becomes much harder to get subsequent percents, and IMHO starting with
>> some point of the development the relation of the code improvement to
>> the spent efforts might become exponential.  So there is no magic --
>> GCC has a better performance because much more efforts of experienced
>> compiler developers have been spent and are being spent for GCC
>> development than for LLVM.
>>
>>   The above said about compilation speed is true when GCC front-end is
>> used for LLVM.  LLVM has another C-language family front-end called
>> CLANG which can speed up compilation in optimization mode
>> (-O2/-O3) upto 20%-25%.  So even as LLVM optimizations
>> are not faster than GCC optimizations, CLANG front-end is really
>> faster than GCC-frontend.  I think GCC community should pay more attention
>> to this fact.  Fortunately, a few new GCC projects address to this problem
>> and I hope this problem will be solved or alleviated.
>>
>>   This year I used -Ofast -flto -fwhole-program instead of
>> -O3 for GCC and -O3 -ffast-math for LLVM for comparison of peak
>> performance.  I could improve GCC performance even more by using
>> other GCC possibilities (like support of AVX insns, Graphite optimizations
>> and even some experimental stuff like LIPO) but I wanted to give LLVM
>> some chances too.  Probably an experienced user in LLVM could improve
>> LLVM performance too.  So I think it is a fair comparison.
>>

Re: Comparison of GCC-4.6.1 and LLVM-2.9 on x86/x86-64 targets

[Jakub Jelinek]

On Wed, Sep 07, 2011 at 11:15:39AM -0400, Vladimir Makarov wrote:
>   This year I used -Ofast -flto -fwhole-program instead of
> -O3 for GCC and -O3 -ffast-math for LLVM for comparison of peak
> performance.  I could improve GCC performance even more by using
> other GCC possibilities (like support of AVX insns, Graphite optimizations
> and even some experimental stuff like LIPO) but I wanted to give LLVM
> some chances too.  Probably an experienced user in LLVM could improve
> LLVM performance too.  So I think it is a fair comparison.

-march=native in addition would be nice to see, that can make significant
difference, especially on AVX capable CPUs.  I guess LLVM equivalent would
be -march=corei7 -mtune=corei7 and, if it works, -mavx too (though, the only
time I've tried LLVM 2.9 it crashed on almost anything with -mavx).

	Jakub

Re: Comparison of GCC-4.6.1 and LLVM-2.9 on x86/x86-64 targets

[Vladimir Makarov]

On 09/08/2011 04:47 AM, Jakub Jelinek wrote:
> On Wed, Sep 07, 2011 at 11:15:39AM -0400, Vladimir Makarov wrote:
>>    This year I used -Ofast -flto -fwhole-program instead of
>> -O3 for GCC and -O3 -ffast-math for LLVM for comparison of peak
>> performance.  I could improve GCC performance even more by using
>> other GCC possibilities (like support of AVX insns, Graphite optimizations
>> and even some experimental stuff like LIPO) but I wanted to give LLVM
>> some chances too.  Probably an experienced user in LLVM could improve
>> LLVM performance too.  So I think it is a fair comparison.
> -march=native in addition would be nice to see, that can make significant
> difference, especially on AVX capable CPUs.  I guess LLVM equivalent would
> be -march=corei7 -mtune=corei7 and, if it works, -mavx too (though, the only
> time I've tried LLVM 2.9 it crashed on almost anything with -mavx).
>
Yes, Jakub.  It would be better to use corei7 with avx for GCC.  
Unfortunately, the last tuning which llvm 2.9 supports is core2 
therefore I used -march=core2 for comparison on x86-64.  So I think it 
would be unfair to use corei7 and avx for GCC without using it for LLVM.

I mostly tried to compare state of general optimizations of GCC and 
LLVM.  There are a lot of other aspects of the compilers which we could 
compare and I did not do it.  But for me it is obvious that Apple loses 
a lot not using modern versions of GCC.

Re: Comparison of GCC-4.6.1 and LLVM-2.9 on x86/x86-64 targets

[Jakub Jelinek]

On Fri, Sep 09, 2011 at 10:26:22AM -0400, Vladimir Makarov wrote:
> Yes, Jakub.  It would be better to use corei7 with avx for GCC.
> Unfortunately, the last tuning which llvm 2.9 supports is core2
> therefore I used -march=core2 for comparison on x86-64.  So I think
> it would be unfair to use corei7 and avx for GCC without using it
> for LLVM.

LLVM 2.9 seems to accept -march=corei7 (though, maybe it just accepts it
and tunes fore core2 anyway, haven't checked), doesn't accept
-march=corei7-avx.

I wonder for which CPUs LLVM actually tunes, because
e.g. when I looked at Phoronix benchmarks (PovRay in particular),
GCC on that particular "benchmark" lost to LLVM because the configury
uses -march=k8 -mtune=k8 for x86_64-linux unconditionally, which wasn't
the best tuning for the contemporary Intel CPUs, while LLVM apparently
didn't show much difference between k8 and core2i7 tuning, see
http://phoronix.com/forums/showthread.php?59341-AMD-Llano-Compiler-Performance&p=224367#post224367

	Jakub

Re: Comparison of GCC-4.6.1 and LLVM-2.9 on x86/x86-64 targets

[Lawrence Crowl]

On 9/7/11, Vladimir Makarov <vmakarov@redhat.com> wrote:
> Some people asked me to do comparison of  GCC-4.6 and LLVM-2.9 (both
> released this spring) as I did GCC-LLVM comparison in previous year.
>
> You can find it on http://vmakarov.fedorapeople.org/spec under
> 2011 GCC-LLVM comparison tab entry.

The format of these graphs exaggerates differences.  The reason is
that our hind brains cannot help but compare the heights of bars and
ignore the non-zero bases.  In short, non-zero based graphs are lies.
So, please 0-base all the graphs.  The graphs should show compilation
time from 0 up, execution time from 0 up, SPEC score from 0 up, etc.
A consequence is that you will get rid of the "change" graphs.

In my mind, an interesting graph would be to plot the execution
time of the benchmarks as a function of the compile time of the
benchmarks.  This graph would show you, in particular, what you
buy or lose by changing compilers and/or optimization/debug levels.

-- 
Lawrence Crowl

Re: Comparison of GCC-4.6.1 and LLVM-2.9 on x86/x86-64 targets

[Gabriel Dos Reis]

On Fri, Sep 9, 2011 at 6:30 PM, Lawrence Crowl <crowl@google.com> wrote:

> In my mind, an interesting graph would be to plot the execution
> time of the benchmarks as a function of the compile time of the
> benchmarks.  This graph would show you, in particular, what you
> buy or lose by changing compilers and/or optimization/debug levels.

Amen.

-- Gaby

Re: Comparison of GCC-4.6.1 and LLVM-2.9 on x86/x86-64 targets

[Vladimir Makarov]

On 09/09/2011 07:30 PM, Lawrence Crowl wrote:
> On 9/7/11, Vladimir Makarov<vmakarov@redhat.com>  wrote:
>> Some people asked me to do comparison of  GCC-4.6 and LLVM-2.9 (both
>> released this spring) as I did GCC-LLVM comparison in previous year.
>>
>> You can find it on http://vmakarov.fedorapeople.org/spec under
>> 2011 GCC-LLVM comparison tab entry.
> The format of these graphs exaggerates differences.  The reason is
> that our hind brains cannot help but compare the heights of bars and
> ignore the non-zero bases.  In short, non-zero based graphs are lies.
> So, please 0-base all the graphs.  The graphs should show compilation
> time from 0 up, execution time from 0 up, SPEC score from 0 up, etc.
> A consequence is that you will get rid of the "change" graphs.
>
Thanks for the feedback, Lawrence.

I've changed the graphs which did not started with 0.
> In my mind, an interesting graph would be to plot the execution
> time of the benchmarks as a function of the compile time of the
> benchmarks.  This graph would show you, in particular, what you
> buy or lose by changing compilers and/or optimization/debug levels.
>
I'll think about it.