Problem with delete[] in class destructor

Problem with delete[] in class destructor

[Nicolas Wack]

[-- Attachment #1: Type: text/plain, Size: 1244 bytes --]

Hi all!

I have a little problem, and I thought maybe someone could help me out 
before I go crazy!!
Here's a simple file that allocates some amount of memory (roughly 
300MB), sleeps for 5 secs, deallocates it and sleeps for 15 seconds.

I know for sure that the program goes through all the destructors, 
however nothing gets freed...
I can also reproduce this behavior (bug?) by replacing the Test class 
with std::vector, for instance.
Also, calling v[i]->destroy() instead of deleting the object doesn't do 
anything neither...

Can somebody tell me what's wrong? Is that a problem with gcc or the 
libc or libstdc++?

Thanks in advance!

Nicolas.

My versions are:

debian unstable
linux 2.6.10
g++ 3.3.5 (debian version)
Configured with: ../src/configure -v 
--enable-languages=c,c++,java,f77,pascal,objc,ada,treelang --prefix=/usr 
--mandir=/usr/share/man --infodir=/usr/share/info 
--with-gxx-include-dir=/usr/include/c++/3.3 --enable-shared 
--enable-__cxa_atexit --with-system-zlib --enable-nls 
--without-included-gettext --enable-clocale=gnu --enable-debug 
--enable-java-gc=boehm --enable-java-awt=xlib --enable-objc-gc i486-linux
Thread model: posix
gcc version 3.3.5 (Debian 1:3.3.5-12)
glibc6 2.3.2.ds1-20
libstdc++ 1:3.3.5-12

[-- Attachment #2: mem.cpp --]
[-- Type: text/x-c++src, Size: 506 bytes --]

#include <iostream>
#include <vector>
using namespace std;

const unsigned int N = 300000;

class Test {
  float* _data;
 public:
  Test() { _data = new float[256]; }
  ~Test() { delete[] _data; }
  void destroy() { delete[] _data; }
};


int main() {

  vector<Test*> v(N);

  for (uint i=0; i<N; i++) {
    v[i] = new Test;
  }

  cout << "1st step" << endl;
  sleep(5);

  for (uint i=0; i<N; i++) {
    delete v[i];
    //v[i]->destroy();
  }

  cout << "2nd step" << endl;
  sleep(15);

  return 0;
}

Re: Problem with delete[] in class destructor

[Nathan Sidwell]

Nicolas Wack wrote:
> Hi all!
> 
> I have a little problem, and I thought maybe someone could help me out 
> before I go crazy!!
> Here's a simple file that allocates some amount of memory (roughly 
> 300MB), sleeps for 5 secs, deallocates it and sleeps for 15 seconds.
> 
> I know for sure that the program goes through all the destructors, 
> however nothing gets freed...

define 'freed'.  Do you mean
1) returned to the program's free memory pool?
2) returned to the OS?

Unless you do something system specific, #1 is what will happen (because
that's what the standard says should happen)

nathan

-- 
Nathan Sidwell    ::   http://www.codesourcery.com   ::     CodeSourcery LLC
nathan@codesourcery.com    ::     http://www.planetfall.pwp.blueyonder.co.uk

Re: Problem with delete[] in class destructor

[Nicolas Wack]

[-- Attachment #1: Type: text/plain, Size: 1306 bytes --]

I'm watching the memory consumption with 'top' (that's the reason for 
the call to sleep)
However if I change the allocation to new float[] instead of new Test 
and then free it, it gets directly freed. (see attached file)

What's weird is that if I launch the original program a second time 
while the first already has supposedly freed its memory, it eats 
everything and bring my box to its knees, killing most of the processes 
that use memory (so I guess the memory is not freed at all).

Also, top tells me that the whole memory stays resident (which means it 
doesn't even return to the program's free memory pool, am I right?)

Nicolas.

Nathan Sidwell wrote:
> Nicolas Wack wrote:
> 
>> Hi all!
>>
>> I have a little problem, and I thought maybe someone could help me out 
>> before I go crazy!!
>> Here's a simple file that allocates some amount of memory (roughly 
>> 300MB), sleeps for 5 secs, deallocates it and sleeps for 15 seconds.
>>
>> I know for sure that the program goes through all the destructors, 
>> however nothing gets freed...
> 
> 
> define 'freed'.  Do you mean
> 1) returned to the program's free memory pool?
> 2) returned to the OS?
> 
> Unless you do something system specific, #1 is what will happen (because
> that's what the standard says should happen)
> 
> nathan
> 

[-- Attachment #2: mem_v2.cpp --]
[-- Type: text/x-c++src, Size: 343 bytes --]

#include <iostream>
#include <vector>
using namespace std;

const unsigned int N = 300000;

int main() {

  vector<float*> v(N);

  for (uint i=0; i<N; i++) {
    v[i] = new float[256];
  }

  cout << "1st step" << endl;
  sleep(5);

  for (uint i=0; i<N; i++) {
    delete v[i];
  }

  cout << "2nd step" << endl;
  sleep(15);

  return 0;
}

Re: Problem with delete[] in class destructor

[Eljay Love-Jensen]

Hi Nicolas,

>I'm watching the memory consumption with 'top' (that's the reason for the call to sleep)

Top does not monitor "free space versus consumed space" in your program's heap.

A different compiler (SAS/C++) gave me an API that I could interact with the heap in my program, asking it such questions as "How many OS extants do you have?", "How big is the extant?", "How much heap free space is in the extant?"  And a few knobs to adjust where small allocations versus large allocations go.

I don't know if the Standard C Library provides an API for interacting with the heap manager -- I presume it does not.

There are alternative heap managers that have different performance characteristics, some of which give the programmer more control over heap behavior -- including when an extant is returned to the OS.  That'd be for OS's such as Mac OS Classic, OS/2, Amiga OS, Win16 applications.  Other OS's, such as Unix-based ones, provide each application a virtual memory space up to their ulimit parameters.  (I haven't looked into the underpinnings of Win32 applications under NT -- I don't know if heap management is done via extants, or done the Unix ulimit way.)

>What's weird is that if I launch the original program a second time while the first already has supposedly freed its memory, it eats everything and bring my box to its knees, killing most of the processes that use memory (so I guess the memory is not freed at all).

I presume you are seeing the behavior of swap space being utilized.

You should use ulimit, to prevent a process from consuming too much space.

>Also, top tells me that the whole memory stays resident (which means it doesn't even return to the program's free memory pool, am I right?)

No.  Top does not monitor the free memory pool.  The resident memory is not a reflection of free or consumed space in the heap.  It's whether the memory is resident (in actual RAM) or swapped (paged out, via the magic of the PMMU).

HTH,
--Eljay

Re: Problem with delete[] in class destructor

[Nicolas Wack]

Ok I get it, thanks!

However I'm still a bit confused about some points:

My first version doesn't shrink in memory (os) when freeing stuff, but 
doesn't grow neither if I reallocate stuff, because of the free space in 
the heap.
That means that the memory allocated in the heap doesn't get returned to 
the system, but is free for my program to reallocate. This far I understand.

2 questions come to my mind:

1) why does the memory actually gets returned to the system and the heap 
shrinks if I use the 2nd program I sent? why does it get returned to the 
system when I use these 2 programs? (files attached)

2) is there a way for a program to yield to the system the free memory 
it has in the heap? (or for the system to reclaim it?)


The actual problem I am trying to solve is that I am writing a program 
that needs to process huge amounts of data, but then I can free nearly 
everything and only keep the results in memory. I don't care if it swaps 
during processing, or uses a huge amount of memory, but I'd like to free 
it afterwards (in the sense: return it to the system) as it is going to 
be ran as a server and stay resident on a machine that maybe could make 
better use of that "wasted" memory in the heap...

Thanks again for your help!

Nicolas.


Eljay Love-Jensen wrote:
> Hi Nicolas,
> 
> 
>>I'm watching the memory consumption with 'top' (that's the reason for the call to sleep)
> 
> 
> Top does not monitor "free space versus consumed space" in your program's heap.
> 
> A different compiler (SAS/C++) gave me an API that I could interact with the heap in my program, asking it such questions as "How many OS extants do you have?", "How big is the extant?", "How much heap free space is in the extant?"  And a few knobs to adjust where small allocations versus large allocations go.
> 
> I don't know if the Standard C Library provides an API for interacting with the heap manager -- I presume it does not.
> 
> There are alternative heap managers that have different performance characteristics, some of which give the programmer more control over heap behavior -- including when an extant is returned to the OS.  That'd be for OS's such as Mac OS Classic, OS/2, Amiga OS, Win16 applications.  Other OS's, such as Unix-based ones, provide each application a virtual memory space up to their ulimit parameters.  (I haven't looked into the underpinnings of Win32 applications under NT -- I don't know if heap management is done via extants, or done the Unix ulimit way.)
> 
> 
>>What's weird is that if I launch the original program a second time while the first already has supposedly freed its memory, it eats everything and bring my box to its knees, killing most of the processes that use memory (so I guess the memory is not freed at all).
> 
> 
> I presume you are seeing the behavior of swap space being utilized.
> 

Yes, that's it, but my thunderbird got killed as well (I guess the 
kernel should only kill the offending process...) However this is 
offtopic here :) ...


> You should use ulimit, to prevent a process from consuming too much space.
> 
> 
>>Also, top tells me that the whole memory stays resident (which means it doesn't even return to the program's free memory pool, am I right?)
> 
> 
> No.  Top does not monitor the free memory pool.  The resident memory is not a reflection of free or consumed space in the heap.  It's whether the memory is resident (in actual RAM) or swapped (paged out, via the magic of the PMMU).
> 
> HTH,
> --Eljay
> 
> 
> 

Re: Problem with delete[] in class destructor (with files attached!)

[Nicolas Wack]

[-- Attachment #1: Type: text/plain, Size: 3494 bytes --]

Ok I get it, thanks!

However I'm still a bit confused about some points:

My first version doesn't shrink in memory (os) when freeing stuff, but
doesn't grow neither if I reallocate stuff, because of the free space in
the heap.
That means that the memory allocated in the heap doesn't get returned to
the system, but is free for my program to reallocate. This far I understand.

2 questions come to my mind:

1) why does the memory actually gets returned to the system and the heap
shrinks if I use the 2nd program I sent? why does it get returned to the
system when I use these 2 programs? (files attached)

2) is there a way for a program to yield to the system the free memory
it has in the heap? (or for the system to reclaim it?)


The actual problem I am trying to solve is that I am writing a program
that needs to process huge amounts of data, but then I can free nearly
everything and only keep the results in memory. I don't care if it swaps
during processing, or uses a huge amount of memory, but I'd like to free
it afterwards (in the sense: return it to the system) as it is going to
be ran as a server and stay resident on a machine that maybe could make
better use of that "wasted" memory in the heap...

Thanks again for your help!

Nicolas.


Eljay Love-Jensen wrote:
> Hi Nicolas,
> 
> 
>>I'm watching the memory consumption with 'top' (that's the reason for the call to sleep)
> 
> 
> Top does not monitor "free space versus consumed space" in your program's heap.
> 
> A different compiler (SAS/C++) gave me an API that I could interact with the heap in my program, asking it such questions as "How many OS extants do you have?", "How big is the extant?", "How much heap free space is in the extant?"  And a few knobs to adjust where small allocations versus large allocations go.
> 
> I don't know if the Standard C Library provides an API for interacting with the heap manager -- I presume it does not.
> 
> There are alternative heap managers that have different performance characteristics, some of which give the programmer more control over heap behavior -- including when an extant is returned to the OS.  That'd be for OS's such as Mac OS Classic, OS/2, Amiga OS, Win16 applications.  Other OS's, such as Unix-based ones, provide each application a virtual memory space up to their ulimit parameters.  (I haven't looked into the underpinnings of Win32 applications under NT -- I don't know if heap management is done via extants, or done the Unix ulimit way.)
> 
> 
>>What's weird is that if I launch the original program a second time while the first already has supposedly freed its memory, it eats everything and bring my box to its knees, killing most of the processes that use memory (so I guess the memory is not freed at all).
> 
> 
> I presume you are seeing the behavior of swap space being utilized.
> 

Yes, that's it, but my thunderbird got killed as well (I guess the
kernel should only kill the offending process...) However this is
offtopic here :) ...


> You should use ulimit, to prevent a process from consuming too much space.
> 
> 
>>Also, top tells me that the whole memory stays resident (which means it doesn't even return to the program's free memory pool, am I right?)
> 
> 
> No.  Top does not monitor the free memory pool.  The resident memory is not a reflection of free or consumed space in the heap.  It's whether the memory is resident (in actual RAM) or swapped (paged out, via the magic of the PMMU).
> 
> HTH,
> --Eljay
> 
> 
> 


[-- Attachment #2: mem_v5.cpp --]
[-- Type: text/x-c++src, Size: 403 bytes --]

#include <iostream>
#include <vector>
using namespace std;

const unsigned int N = 300000;

class Test {
  float* _data;
 public:
  Test() { _data = new float[256]; }
  Test(const Test& t) { _data = new float[256]; }
  ~Test() { delete[] _data; }
};


int main() {

  vector<Test> v(N);

  cout << "1st step" << endl;
  sleep(5);

  v.clear();

  cout << "2nd step" << endl;
  sleep(15);

  return 0;
}

[-- Attachment #3: mem_v6.cpp --]
[-- Type: text/x-c++src, Size: 459 bytes --]

#include <iostream>
#include <vector>
using namespace std;

const unsigned int N = 150000;

class Test {
  float* _data;
 public:
  Test() { _data = new float[256]; }
  Test(const Test& t) { _data = new float[256]; }
  ~Test() { delete[] _data; }
};


int main() {

  vector<Test> v;

  for (uint i=0; i<N; i++) {
    v.push_back(Test());
  }

  cout << "1st step" << endl;
  sleep(5);

  v.clear();

  cout << "2nd step" << endl;
  sleep(15);

  return 0;
}

Re: Problem with delete[] in class destructor

[Nathan Sidwell]

Nicolas Wack wrote:
> I'm watching the memory consumption with 'top' (that's the reason for 
> the call to sleep)
> However if I change the allocation to new float[] instead of new Test 
> and then free it, it gets directly freed. (see attached file)
What do you mean by 'directly freed'?

> 
> What's weird is that if I launch the original program a second time 
> while the first already has supposedly freed its memory, it eats 
> everything and bring my box to its knees, killing most of the processes 
> that use memory (so I guess the memory is not freed at all).
the memory is not returned to the OS until the program terminates.

> Also, top tells me that the whole memory stays resident (which means it 
> doesn't even return to the program's free memory pool, am I right?)
No.

This is not a gcc question, but a system/runtime library question.  You
appear to be confused about heap allocation.  Ask in a more suitable
forum (or search the web for 'malloc heap allocation' and similar terms).

nathan

-- 
Nathan Sidwell    ::   http://www.codesourcery.com   ::     CodeSourcery LLC
nathan@codesourcery.com    ::     http://www.planetfall.pwp.blueyonder.co.uk

Re: Problem with delete[] in class destructor (with files attached!)

[Eljay Love-Jensen]

Hi Nicolas,

>why does the memory actually gets returned to the system and the heap shrinks if I use the 2nd program I sent? why does it get returned to the system when I use these 2 programs? (files attached)

You are at the mercy of the heap manager.

That behavior can vary from platform-to-platform.

>is there a way for a program to yield to the system the free memory it has in the heap? (or for the system to reclaim it?)

Only if the heap manager provides a mechanism for your program to interact act with.  (Any such mechanism would be non-standard.)

The general answers to the questions are:  no, and no.  You are at the mercy of the heap manager.

HTH,
--Eljay

Re: Problem with delete[] in class destructor (with files attached!)

[Nicolas Wack]

Ok, thanks a lot for your time, that answers pretty much all of my doubts.
I guess I'll try to spank my heap manager to see if it's obeying me now 
:) !!

Nicolas.

Eljay Love-Jensen wrote:
> Hi Nicolas,
> 
> 
>>why does the memory actually gets returned to the system and the heap shrinks if I use the 2nd program I sent? why does it get returned to the system when I use these 2 programs? (files attached)
> 
> 
> You are at the mercy of the heap manager.
> 
> That behavior can vary from platform-to-platform.
> 
> 
>>is there a way for a program to yield to the system the free memory it has in the heap? (or for the system to reclaim it?)
> 
> 
> Only if the heap manager provides a mechanism for your program to interact act with.  (Any such mechanism would be non-standard.)
> 
> The general answers to the questions are:  no, and no.  You are at the mercy of the heap manager.
> 
> HTH,
> --Eljay
> 
> 
>