Re: gcc build / test times on multi-core hosts?

Re: gcc build / test times on multi-core hosts?

[Joern Rennecke]

In http://gcc.gnu.org/ml/gcc/2006-02/msg00357.html, you wrote:

> In fact the "gamer" benchmarks you are dissing are quite well reflecting the
very kind
> of coding excessively found in GCC itself. Some observations suggest
> the you should aim at the CPU with the biggest L2-cache size affordable. In
particular
> comparision on the older Athlon XP line where very conclusive here as there
where similarly rated CPUs available with 256kB vers. 512kB caches. Usually the
512k where 20% faster on GCC build loads then the ones with smaller caches
despite beeing significantly lower clocked.

By that token, the Pentium D 920 should trounce the Athlon X2 3800+ .  But
that's
not what these benchmarks show.

Re: gcc build / test times on multi-core hosts?

[Marcin Dalecki]

On 2006-02-18, at 15:23, Joern Rennecke wrote:

> In http://gcc.gnu.org/ml/gcc/2006-02/msg00357.html, you wrote:
>
>> In fact the "gamer" benchmarks you are dissing are quite well  
>> reflecting the
> very kind
>> of coding excessively found in GCC itself. Some observations suggest
>> the you should aim at the CPU with the biggest L2-cache size  
>> affordable. In
> particular
>> comparision on the older Athlon XP line where very conclusive here  
>> as there
> where similarly rated CPUs available with 256kB vers. 512kB caches.  
> Usually the
> 512k where 20% faster on GCC build loads then the ones with smaller  
> caches
> despite beeing significantly lower clocked.
>
> By that token, the Pentium D 920 should trounce the Athlon X2 3800 
> + .  But
> that's
> not what these benchmarks show.

Well the Pentium D has an severe execution pipeline "complex" in  
combination with recessive memory access latency "syndrome", which it  
"overcompensates" with huge caches and clock "overactivity". I taught  
we where talking only about "mentally sane" architectures here.

More seriously: The Netburst architecture fits well DSP tasks and  
certainly doesn't lend well to decision oriented tasks, which is what  
GCC does most of the time. Pretty pure locality of code in GCC  
strikes bad as well. For compilation tasks an opteron with big caches  
and as many cores as possible of course has right now the best bang  
for the money. The rest basically depends just on budget. 

Re: gcc build / test times on multi-core hosts?

[H. J. Lu]

On Sat, Feb 18, 2006 at 09:23:36AM -0500, Joern Rennecke wrote:
> In http://gcc.gnu.org/ml/gcc/2006-02/msg00357.html, you wrote:
> 
> > In fact the "gamer" benchmarks you are dissing are quite well reflecting the
> very kind
> > of coding excessively found in GCC itself. Some observations suggest
> > the you should aim at the CPU with the biggest L2-cache size affordable. In
> particular
> > comparision on the older Athlon XP line where very conclusive here as there
> where similarly rated CPUs available with 256kB vers. 512kB caches. Usually the
> 512k where 20% faster on GCC build loads then the ones with smaller caches
> despite beeing significantly lower clocked.
> 
> By that token, the Pentium D 920 should trounce the Athlon X2 3800+ .  But
> that's
> not what these benchmarks show.

My 2GHz Yonah with 2MB cache and 2GB RAM holds its own against other
dual core/CPU machines I have when doing gcc build/check, although its
harddrive is slow. With a faster drive, it may be one of the fatest
32bit dual core/CPU machines for gcc build/check. It is also very
quiet.


H.J.

Re: gcc build / test times on multi-core hosts?

[DJ Delorie]

> Some multi-core processors come with less L2 cache than their multi-CPU
> counterparts.

This, and your other arguments, while valid, apply independently of
the CPU.  There are a lot of factors that determine compile speed.

FYI SGIs tend to have crossbars.

> Was that UMA or NUMA, and how far up could you scale N usefully?

UMA I think.  We generally had just 1 and 2 CPU machines.

> Do you know if software RAID (not so much R as A of ID) is effective
> at avoiding I/O bottlenecks, e.g. will two disks for four cores work
> as well as one disk for two?

Disk is so much slower that CPU that I don't think it matters.
Software RAID is already faster than hardware RAID for common
uniprocessor systems, assuming you don't introduce I/O bottlenecks
(like 33MHz PCI bus).  For example, my server here is a dual opteron,
with an 8-sata controller.  It works because the PCI bus is a 133MHz
64 bit bus, which I calculated was sufficient to host all eight
drives.  I was able to benchmark a 7-disk RAID array at 250MB/s
sustained.

Re: gcc build / test times on multi-core hosts?

[H. J. Lu]

On Mon, Feb 20, 2006 at 07:58:35PM +0000, Joern RENNECKE wrote:
> H. J. Lu wrote:
> 
> >On Mon, Feb 20, 2006 at 07:30:41PM +0000, Joern RENNECKE wrote:
> > 
> >
> >>the bottleneck of a shared memory bus, but the operating system must 
> >>allocate
> >>most memory locally to each CPU to avoid a bottleneck in the 
> >>cross-connect
> >>of the processors.
> >>
> >>   
> >>
> >
> >Linux kernel 2.6.16-rc1 and above supports
> >
> >percpu_pagelist_fraction
> > 
> >
> Is this per cpu (which might contain multiple cores) or per core?

I think it is per core. But I don't know if a HT cpu counts here or not.

> 
> >This is the fraction of pages at most (high mark pcp->high) in each zone 
> >that
> >are allocated for each per cpu page list.  The min value for this is 8.  
> >It
> >means that we don't allow more than 1/8th of pages in each zone to be
> >allocated in any single per_cpu_pagelist.
> >
> Do I understand this correctly that in a dual opteron single core system 
> with 2 GB
> memory, only up to 256 MB per processor could be specifically allocated 
> in local
> memory?
> Whereas in an 8-way opteron machine with equal amounts of memory 
> attached to each
> processor, all the local memory could be allocated to its processor?

I don't know the answer.  You may ask it on the Linux kernel mailing list.


H.J.

Re: gcc build / test times on multi-core hosts?

[Joern RENNECKE]

Laurent GUERBY wrote:

>  
>
>On a Pentium III 1GHz, bootstrap is 5h55 and check 5h30
>(on an older version of the tree),
>
Was this before top-level bootstrap?
 

Re: gcc build / test times on multi-core hosts?

[Joern RENNECKE]

H. J. Lu wrote:

>On Mon, Feb 20, 2006 at 07:30:41PM +0000, Joern RENNECKE wrote:
>  
>
>>the bottleneck of a shared memory bus, but the operating system must 
>>allocate
>>most memory locally to each CPU to avoid a bottleneck in the cross-connect
>>of the processors.
>>
>>    
>>
>
>Linux kernel 2.6.16-rc1 and above supports
>
>percpu_pagelist_fraction
>  
>
Is this per cpu (which might contain multiple cores) or per core?

>This is the fraction of pages at most (high mark pcp->high) in each zone that
>are allocated for each per cpu page list.  The min value for this is 8.  It
>means that we don't allow more than 1/8th of pages in each zone to be
>allocated in any single per_cpu_pagelist.
>
Do I understand this correctly that in a dual opteron single core system 
with 2 GB
memory, only up to 256 MB per processor could be specifically allocated 
in local
memory?
Whereas in an 8-way opteron machine with equal amounts of memory 
attached to each
processor, all the local memory could be allocated to its processor?

Re: gcc build / test times on multi-core hosts?

[H. J. Lu]

On Mon, Feb 20, 2006 at 07:30:41PM +0000, Joern RENNECKE wrote:
> the bottleneck of a shared memory bus, but the operating system must 
> allocate
> most memory locally to each CPU to avoid a bottleneck in the cross-connect
> of the processors.
> 

Linux kernel 2.6.16-rc1 and above supports

percpu_pagelist_fraction

This is the fraction of pages at most (high mark pcp->high) in each zone that
are allocated for each per cpu page list.  The min value for this is 8.  It
means that we don't allow more than 1/8th of pages in each zone to be
allocated in any single per_cpu_pagelist.  This entry only changes the value
of hot per cpu pagelists.  User can specify a number like 100 to allocate
1/100th of each zone to each per cpu page list.

The batch value of each per cpu pagelist is also updated as a result.  It is
set to pcp->high/4.  The upper limit of batch is (PAGE_SHIFT * 8)

The initial value is zero.  Kernel does not use this value at boot time to set
the high water marks for each per cpu page list.

The optimal percpu_pagelist_fraction value depends on your configuation.


H.J.

Re: gcc build / test times on multi-core hosts?

[Joern RENNECKE]

DJ Delorie wrote: 

>There's not much difference between multi-core and multi-cpu, and I've
>been building multi-cpu for years.
>  
>
Some multi-core processors come with less L2 cache than their multi-CPU
counterparts.
Also, multi-cpu itself comes in different varieties, with Intels Xeons going
for the classic SMP design with a single shared memory bus providing
uniform memory access (UMA), but also a causing a bootleneck as you add
more processors; whereas the Opterons have non-uniform memory access
(NUMA), with each processor having separate memory busses.  So theat removes
the bottleneck of a shared memory bus, but the operating system must 
allocate
most memory locally to each CPU to avoid a bottleneck in the cross-connect
of the processors.

The Athlon X2 and Opteron dual core processors use internally a cross-bar
switch to connect the two cores to the memory and inter-processor 
interconnect,
so they have slightly higher memory latencies, but better communication
between the two cores inside the processor and the attached memory than 
between
separate Opterons processors with their attached memory.

I wonder: what does the Linux (or insert your favourite BSD here) kernel do
with dual-core Opterons?  Does it keep processor-affinity for physical 
memory
local to the two cores it is attached to?
Dual processor Dual-core Opterons seem like a very cost effective way to 
get a
4-way machine, and two cores each share two memory busses (if the memory is
appropriately installed), so memory bandwidth should also be  good.  But is
there a penalty to pay because this machine is not quite classical SMP 
nor quite
NUMA?
If the kernel pretends it's a fully NUMA machine, that would halve the
local memory available per CPU -  i.e. builds that see largely 
assymetric memory
usage could get slower, since when the kernel runs out of what it thinks 
is the
memory local to one core, it has a good chance (2/3 if you assume random
distribution) of grabbing memory that is indeed not local to that core.
If it pretends the machine is a classic SMP machine with uniform memory 
access,
it's even worse, since then irrespective of the size of the working set, 
it will tend
to grab memory from the wrong place half of the time.

>At a previous job where we were very interested in build times, our
>rule of thumb was N+1 jobs for N cpus with local disk, or 2N+1 jobs
>for N cpus with nfs-mounted disk.  That was for build farms working on
>a 12 hour C++ compile.
>  
>
Was that UMA or NUMA, and how far up could you scale N usefully?
Do you know if software RAID (not so much R as A of ID) is effective at
avoiding I/O bottlenecks, e.g. will two disks for four cores work as 
well as one
disk for two?

Re: gcc build / test times on multi-core hosts?

[Andrew Haley]

Joern RENNECKE writes:
 > Andrew Haley wrote:
 > 
 > > > Is that using the i686 or amd64 instruction set?  If the latter, does it 
 > > > use 32 or 64 bit pointers?
 > >
 > >The latter.  Yes.
 > >  
 > >
 > So which pointer size does the host compiler use, and which pointer size 
 > is used in
 > the gcc that is bootstrapped?

64.  64.

Andrew.

Re: gcc build / test times on multi-core hosts?

[Joern RENNECKE]

Andrew Haley wrote:

> 
> > Is that using the i686 or amd64 instruction set?  If the latter, does it 
> > use 32 or 64 bit pointers?
>
>The latter.  Yes.
>  
>
So which pointer size does the host compiler use, and which pointer size 
is used in
the gcc that is bootstrapped?

> 
>This RAM also has flashing LEDs to make it go faster.  (Something to do
>with photonics, I think.)
>  
>
:-)
 

Re: gcc build / test times on multi-core hosts?

[Andrew Haley]

Joern RENNECKE writes:
 > Andrew Haley wrote:
 > 
 > > 
 > >As a comparison point, I get
 > >
 > >real    73m39.275s
 > >user    113m19.549s
 > >sys     15m26.010s
 > >
 > >for the bootstrap: that's 1h14 elapsed time.  That's on a "AMD
 > >Athlon(tm) 64 X2 Dual Core Processor 4800+" (2.4 GHz) with make -j3.
 > >That's 129min of CPU time in 74min of elapsed time, a pretty good
 > >processor utilization ratio of 1.75 : 2.
 > >  
 > >
 > Is that using the i686 or amd64 instruction set?  If the latter, does it 
 > use 32 or 64 bit pointers?

The latter.  Yes.

 > How much RAM dos this machine have, and how fast is it?

2G.  DDR400  3-3-3-8  (TWINX2048-3200PRO)

This RAM also has flashing LEDs to make it go faster.  (Something to do
with photonics, I think.)

Andrew.

Re: gcc build / test times on multi-core hosts?

[Joern RENNECKE]

Andrew Haley wrote:

> 
>As a comparison point, I get
>
>real    73m39.275s
>user    113m19.549s
>sys     15m26.010s
>
>for the bootstrap: that's 1h14 elapsed time.  That's on a "AMD
>Athlon(tm) 64 X2 Dual Core Processor 4800+" (2.4 GHz) with make -j3.
>That's 129min of CPU time in 74min of elapsed time, a pretty good
>processor utilization ratio of 1.75 : 2.
>  
>
Is that using the i686 or amd64 instruction set?  If the latter, does it 
use 32 or 64 bit pointers?
How much RAM dos this machine have, and how fast is it?

Has anybody any figures what impact the choice of CAS 2 / CAS 2.5 / CAS3 
NON-ECC / ECC
memory has on system performance when doing gcc builds / regression tests?
 

Re: gcc build / test times on multi-core hosts?

[Andrew Haley]

Laurent GUERBY writes:
 > On Fri, 2006-02-17 at 22:43 +0000, Joern RENNECKE wrote:
 > > Has anybody done timings for gcc bootstrap / cross builds and regtests 
 > > with modern multi-core processors?  I wonder what a sensible modern 
 > > configuration would be for gcc development, but the the multimedia and 
 > > games benchmarks I found on the web neither seem particularily relevant, 
 > > nor do they paint a uniform picture.
 > 
 > On a  "AMD Athlon(tm) 64 X2 Dual Core Processor 3800+" (2.0 GHz), 2GB of
 > RAM trunk configured with:
 > 
 > --enable-languages=c,ada,c++,fortran,java,treelang,objc 
 > --enable-__cxa_atexit
 > --disable-nls 
 > --enable-threads=posix 
 > --disable-multilib 
 > --enable-checking
 > 
 > Gives the following timing at -j1:
 > 
 > === GCCBOOT start     === 2006-02-16 22:13:46
 > === GCCBOOT configure === 2006-02-16 22:13:46
 > === GCCBOOT bootstrap === 2006-02-16 22:13:51
 > === GCCBOOT doc       === 2006-02-17 01:48:53
 > === GCCBOOT check     === 2006-02-17 01:48:53
 > === GCCBOOT summary   === 2006-02-17 04:15:25
 > === GCCBOOT install   === 2006-02-17 04:15:37
 > === GCCBOOT done      === 2006-02-17 04:20:03
 > 
 > So that's 6h07 total, bootstrap 3h35 and check 2h27. I assume
 > -j2 could cut bootstrap times by about two, I'm not
 > sure check is parallel (Ada check isn't for sure).
 > 
 > On a Pentium III 1GHz, bootstrap is 5h55 and check 5h30
 > (on an older version of the tree), so p3/amd64
 > 
 > Removing Ada from --enable-languages will also help :).

As a comparison point, I get

real    73m39.275s
user    113m19.549s
sys     15m26.010s

for the bootstrap: that's 1h14 elapsed time.  That's on a "AMD
Athlon(tm) 64 X2 Dual Core Processor 4800+" (2.4 GHz) with make -j3.
That's 129min of CPU time in 74min of elapsed time, a pretty good
processor utilization ratio of 1.75 : 2.

Andrew.

Re: gcc build / test times on multi-core hosts?

[DJ Delorie]

> Has anybody done timings for gcc bootstrap / cross builds and
> regtests with modern multi-core processors?  I wonder what a
> sensible modern configuration would be for gcc development, but the
> the multimedia and games benchmarks I found on the web neither seem
> particularily relevant, nor do they paint a uniform picture.

There's not much difference between multi-core and multi-cpu, and I've
been building multi-cpu for years.

At a previous job where we were very interested in build times, our
rule of thumb was N+1 jobs for N cpus with local disk, or 2N+1 jobs
for N cpus with nfs-mounted disk.  That was for build farms working on
a 12 hour C++ compile.

Re: gcc build / test times on multi-core hosts?

[Mattias Engdegård]

Diego Novillo <dnovillo@redhat.com> writes:

>I/O is probably the biggest bottleneck, so if you have a fairly decent
>I/O system, you may find higher -jN numbers giving better results.

Yes, but there is also an effect in the opposite direction: higher
parallelism may compensate for slow I/O by keeping CPUs busy. My rule
of thumb, no doubt suboptimal, is to build with twice as many jobs as
there are cores.

Re: gcc build / test times on multi-core hosts?

[Laurent GUERBY]

On Fri, 2006-02-17 at 22:43 +0000, Joern RENNECKE wrote:
> Has anybody done timings for gcc bootstrap / cross builds and regtests 
> with modern multi-core processors?  I wonder what a sensible modern 
> configuration would be for gcc development, but the the multimedia and 
> games benchmarks I found on the web neither seem particularily relevant, 
> nor do they paint a uniform picture.

On a  "AMD Athlon(tm) 64 X2 Dual Core Processor 3800+" (2.0 GHz), 2GB of
RAM trunk configured with:

--enable-languages=c,ada,c++,fortran,java,treelang,objc 
--enable-__cxa_atexit
--disable-nls 
--enable-threads=posix 
--disable-multilib 
--enable-checking

Gives the following timing at -j1:

=== GCCBOOT start     === 2006-02-16 22:13:46
=== GCCBOOT configure === 2006-02-16 22:13:46
=== GCCBOOT bootstrap === 2006-02-16 22:13:51
=== GCCBOOT doc       === 2006-02-17 01:48:53
=== GCCBOOT check     === 2006-02-17 01:48:53
=== GCCBOOT summary   === 2006-02-17 04:15:25
=== GCCBOOT install   === 2006-02-17 04:15:37
=== GCCBOOT done      === 2006-02-17 04:20:03

So that's 6h07 total, bootstrap 3h35 and check 2h27. I assume
-j2 could cut bootstrap times by about two, I'm not
sure check is parallel (Ada check isn't for sure).

On a Pentium III 1GHz, bootstrap is 5h55 and check 5h30
(on an older version of the tree), so p3/amd64

Removing Ada from --enable-languages will also help :).

Laurent

Re: gcc build / test times on multi-core hosts?

[Marcin Dalecki]

On 2006-02-17, at 23:43, Joern RENNECKE wrote:

> Has anybody done timings for gcc bootstrap / cross builds and  
> regtests with modern multi-core processors?  I wonder what a  
> sensible modern configuration would be for gcc development, but the  
> the multimedia and games benchmarks I found on the web neither seem  
> particularily relevant, nor do they paint a uniform picture.

In fact the "gamer" benchmarks you are dissing are quite well  
reflecting the very kind
of coding excessively found in GCC itself. Some observations suggest
the you should aim at the CPU with the biggest L2-cache size  
affordable. In particular
comparision on the older Athlon XP line where very conclusive here as  
there where similarly rated CPUs available with 256kB vers. 512kB  
caches. Usually the 512k where 20% faster on GCC build loads then the  
ones with smaller caches despite beeing significantly lower clocked.

Re: gcc build / test times on multi-core hosts?

[Diego Novillo]

Joern RENNECKE wrote:

> Has anybody done timings for gcc bootstrap / cross builds and regtests
> with modern multi-core processors?
>
I've only done very limited testing mostly by eye balling build times.
I don't think build performance scales much beyond -j4/-j6.

I/O is probably the biggest bottleneck, so if you have a fairly decent
I/O system, you may find higher -jN numbers giving better results.

gcc build / test times on multi-core hosts?

[Joern RENNECKE]

Has anybody done timings for gcc bootstrap / cross builds and regtests 
with modern multi-core processors?  I wonder what a sensible modern 
configuration would be for gcc development, but the the multimedia and 
games benchmarks I found on the web neither seem particularily relevant, 
nor do they paint a uniform picture.