[Bug libstdc++/18970] New: 3.4 default memory allocator much slower than 3.3 allocator when large amounts of data are turned over

[Bug libstdc++/18970] New: 3.4 default memory allocator much slower than 3.3 allocator when large amounts of data are turned over

[mj1 at cog dot brown dot edu]

I have a C++ program that seems to be running into severe memory allocation
problems when compiled under g++ 3.4.2 or 3.4.3 , but runs fine when I use the
stock Fedora Core 2 g++ 3.3.3 compiler.  The actual problem is that the program
is 10-20 times slower when compiled using g++ 3.4.2 or 3.4.3 than it is when
compiled using g++ 3.3.3.  I am submitting this bug report to alert you to this
problem.

The program's VSZ is around 550Mb, when compiled under either compiler.  I see
this problem on machines that have either 2Gb or 4Gb RAM, and swapping is not an
issue.

The program processes a stream of data from gigabyte-sized files (it's actually
extracting features from a large number of natural language sentence parses).
The program makes two passes over the data; in the first pass it identifies the
important features in the data, and in the second pass it writes out which
important features occur in each data item.  The second pass is a simple loop
that reads in each datum, writes out the important features in the datum, and
then the temporary data structures involves in the datum are freed; it does not
allocate any additional memory.  And indeed, the memory used by the program (as
measured by ps or top) does not change during the second pass.

When the program is compiled using g++ 3.3.3 both passes take approximately the
same amount of time to run.  However, when the program is compiled using g++
3.4.2 or 3.4.3 the second pass may take 10-20 or more times longer than the
first pass.  Moreover, this seems to depend on the amount of data being
processed.  The problem does not occur when working with smaller data files, for
example.  When the program is compiled with 3.3.3 both passes take approximately
two hours each, but the second pass takes several days to run when compiled with
3.4.3.  (The first pass is actually slightly faster with 3.4.3 than under
3.3.3).    This also makes it difficult to discover what's happening using
gprof, as when compiled the program with profiling turned it is slower still (I
killed a profiled version of my program when it hadn't terminated after a week).

I'm afraid that this isn't very helpful, and I'd like to do what I can do to
help, as I think gcc is a wonderful piece of software and I'd like to contribute
if I can.

My guess is that the default allocator generally works well, but that the
specific pattern of allocation and deallocation that the second pass of my
program is performing is somehow very bad for the default allocator.

One way to test this would be by changing the default allocator, and see if the
behaviour changes.  I tried reading the stdlibc++ library documentation on
allocators, but I couldn't figure out how to globally change the default
allocator.  (That used to be possible with a #DEFINE with the old SGI STL, I
believe).  Is there a way to change the default allocator that my program uses,
or to change its configuration parameters in some way so I could try to
determine if the allocator really is causing the problem?

Anyway, if you have any suggestions for anything else that I might try, I would
be pleased to.

Thanks,

Mark Johnson

-- 
           Summary: 3.4 default memory allocator much slower than 3.3
                    allocator when large amounts of data are turned over
           Product: gcc
           Version: 3.4.3
            Status: UNCONFIRMED
          Severity: minor
          Priority: P2
         Component: libstdc++
        AssignedTo: unassigned at gcc dot gnu dot org
        ReportedBy: mj1 at cog dot brown dot edu
                CC: gcc-bugs at gcc dot gnu dot org,mj1 at cog dot brown dot
                    edu
 GCC build triplet: gcc-3.4.3  --with-arch=pentium4 --enable-
                    languages=c,c++,f77  po
  GCC host triplet: Linux 2.6.9-1.6_FC2smp #1 SMP Dual Pentium 4 Xeon


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=18970

[Bug libstdc++/18970] 3.4 default memory allocator much slower than 3.3 allocator when large amounts of data are turned over

[bkoz at gcc dot gnu dot org]

------- Additional Comments From bkoz at gcc dot gnu dot org  2004-12-13 22:47 -------

the default allocator did change for 3.4.x, from pool_allocator to new_allocator.

see:
http://gcc.gnu.org/onlinedocs/libstdc++/20_util/allocator.html

to change to the 3.3.x allocator, use __pool_alloc. you can configure with

--enable-libstdcxx-allocator=pool

to make this the default.

see:
http://gcc.gnu.org/onlinedocs/libstdc++/configopts.html

anyway.

if you just want to hack in pool for new allocators, locally modify 

/usr/include/c++/3.4.2/i386-redhat-linux/bits/c++allocator 

to be

src/gcc/libstdc++-v3/config/allocator/pool_allocator_base.h

FWIW, it is recommended that you use __mt_alloc for your high-performance
allocator needs.

-benjamin

-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=18970

[Bug libstdc++/18970] 3.4 default memory allocator much slower than 3.3 allocator when large amounts of data are turned over

[pcarlini at suse dot de]

------- Additional Comments From pcarlini at suse dot de  2004-12-31 11:10 -------
I think we can definitely close this one: the various (configure) issues are well
known and no fixes are needed anywhere.

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |RESOLVED
         Resolution|                            |INVALID


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=18970

[Bug libstdc++/18970] 3.4 default memory allocator much slower than 3.3 allocator when large amounts of data are turned over

[mj1 at cog dot brown dot edu]

------- Additional Comments From mj1 at cog dot brown dot edu  2005-01-01 02:53 -------
Subject: Re:  3.4 default memory allocator much slower
 than 3.3 allocator when large amounts of data are turned over

Yes, although you might want to revert to the 3.3 default allocator, as 
the 3.4 default allocator is considerably slower for large programs.

Best,

Mark Johnson

pcarlini at suse dot de wrote:

>------- Additional Comments From pcarlini at suse dot de  2004-12-31 11:10 -------
>I think we can definitely close this one: the various (configure) issues are well
>known and no fixes are needed anywhere.
>
>  
>


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=18970

[Bug libstdc++/18970] 3.4 default memory allocator much slower than 3.3 allocator when large amounts of data are turned over

[pcarlini at suse dot de]

------- Additional Comments From pcarlini at suse dot de  2005-01-01 11:09 -------
> Yes, although you might want to revert to the 3.3 default allocator, as 
> the 3.4 default allocator is considerably slower for large programs.

The default 3.4 ""allocator"", actually just forwards to new/delete. Of course,
there is a reason why we chose it as default: the new allocators we were
working on were not ready in time for 3.4.0; the old one not yet forward
ported to the new framework. The 3.4.x ABI is completely frozen, now, and we 
cannot change the default anymore. For the forthcoming 4.0.0, as indicated by
Benjamin, we have a brand new allocator as default, optimized for MT (+ all the
old possibilities)

-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=18970