potential problems with malloc and calloc

potential problems with malloc and calloc

[David Joseph Adrian]

[-- Attachment #1: Type: text/plain, Size: 2278 bytes --]

Thank you in advance for your time, patience, and help.

I was writing some test programs to learn more about dynamic memory allocation
and I came upon some interesting behavior.  I think that in the limit of
filling the memory on a system, malloc may be malfunctioning.  (Or it may be my
architecture.)  Could you please help me determine if this is a gcc-thing or a
problem of my architecture?  Has it already been fixed in a gcc update?

Problem definition:
The test program declares a large 2-D array of integers dynamically.  malloc and
calloc are supposed to return a NULL pointer when they run out of space, but I
found that on some architectures, I was able to allocate more space than is
actually on the computer with malloc, and that the computer program dies
ungracefully in the same situation with calloc.

Try compiling and running the malloctest.c and calloctest.c with different sizes
of the 2-D array.

So far, I ran these test problems on 64 and 32 bit architectures.

Test results:
The 64-bit architecture used Red Hat Linux with gcc 3.2.3 with an Intel Xeon
Irwindale processor and 73 GB hard drive.  I found that the malloc calls in
malloctest.c didn't return a NULL pointer until allocating storage for ~500 GiB
of memory.  This should be impossible, right?  Since addresses are stored as 10
hexadecimal digits on this system, there are about 1024 GiB of possible
addresses, leading to more possible address space than actual memory.  Is this
part of the problem?  The calloctest dies ungracefully on the order of the time
the hard drive would be full (actually setting memory to a value outside of
possible space is killing the program).  In this case, it would be great to get
that null pointer instead of program death.  I can also reproduce this result on
an AMD 64-bit architecture.

The 32-bit architecture used Red Hat Linux with gcc 3.2.2 with older Pentium
Xeon Processors (not sure of hard drive size).  In this test, both the malloc
and calloc calls return a null pointer after about 3GiB have been allocated. 
This seems more reasonable.  On this system, the total addressable space is
found from 8 hexadecimal digits, which is about 4GiB.

Thanks again for your time and help.

David Adrian
Ph.D. Candidate
Massachusetts Institute of Technology

[-- Attachment #2: malloctest.c --]
[-- Type: text/x-csrc, Size: 1104 bytes --]

#include <math.h>
#include <stdlib.h>
#include <stdio.h>

int main()
{
  // void* malloc(size_t size);
  //    malloc returns a pointer to a block of memory of size bytes, returns 0d0 if memory unavailable
  //    the pointer must be cast into the appropriate variable type to be used
  // void free(void *s);
  //    given a pointer to memory allocated with malloc, the memory is freed
  //sizeof(datatype) returns the size of a datatype, can be used to make malloc calls easier


  int length;
  int depth;
  int ** list = NULL;
  int i,j;
  int count = 0;
  length = 1000000;
  depth = length;


  printf("The size of an integer in bytes is %d\n",sizeof(int));
  if(list == NULL)
    printf("initial value of pointer properly set to null pointer\n");


  list = (int **) malloc(length * sizeof(int*));
  if(list != NULL)
    printf("There is enough space for %d pointers to integers\n",length);
  
  for(i=0;i<length;i++)
    {
      list[i] = (int *) malloc(depth * sizeof(int));
      if(list[i] == NULL)
	{
	  printf("malloc failed at depth step %d\n",i+1);
	  return 1;
	}

      }

  return 0;
}

[-- Attachment #3: calloctest.c --]
[-- Type: text/x-csrc, Size: 1167 bytes --]

#include <math.h>
#include <stdlib.h>
#include <stdio.h>

int main()
{
  // void* calloc(size_t nelements,size_t size);
  //    calloc returns a pointer to a block of memory of size nelements*size bytes, returns 0d0 if memory unavailable.  Also this memory is set to zero.
  //    the pointer must be cast into the appropriate variable type to be used
  // void free(void *s);
  //    given a pointer to memory allocated with calloc, the memory is freed
  //sizeof(datatype) returns the size of a datatype, can be used to make calloc calls easier


  int length;
  int depth;
  int ** list = NULL;
  int i,j;
  int count = 0;
  length = 1000000;
  depth = length;


  printf("The size of an integer in bytes is %d\n",sizeof(int));
  if(list == NULL)
    printf("initial value of pointer properly set to null pointer\n");


  list = (int **) calloc(length,sizeof(int*));
  if(list != NULL)
    printf("There is enough space for %d pointers to integers\n",length);
  
  for(i=0;i<length;i++)
    {
      list[i] = (int *) calloc(depth,sizeof(int));
      if(list[i] == NULL)
	{
	  printf("calloc failed at depth step %d\n",i+1);
	  return 1;
	}

      }

  return 0;
}

Re: potential problems with malloc and calloc

[Ian Lance Taylor]

David Joseph Adrian <adriand@MIT.EDU> writes:

> I was writing some test programs to learn more about dynamic memory allocation
> and I came upon some interesting behavior.  I think that in the limit of
> filling the memory on a system, malloc may be malfunctioning.  (Or it may be my
> architecture.)  Could you please help me determine if this is a gcc-thing or a
> problem of my architecture?  Has it already been fixed in a gcc update?

This is not a gcc issue.  gcc does not provide the malloc function.
That is part of the C library.  On a GNU/Linux system the C library is
provided by glibc (http://sourceware.org/glibc/).

> Problem definition:
> The test program declares a large 2-D array of integers dynamically.  malloc and
> calloc are supposed to return a NULL pointer when they run out of space, but I
> found that on some architectures, I was able to allocate more space than is
> actually on the computer with malloc, and that the computer program dies
> ungracefully in the same situation with calloc.

And this issue actually doesn't have anything to do with malloc.  The
Linux kernel overcommits memory, and then crashes the program if it
can't come up with any memory when it needs it.

You can control this behaviour by poking at the files
    /proc/sys/vm/overcommit_memory
    /proc/sys/vm/overcommit_ratio
I think that setting overcommit_memory to 2 will tell the kernel to
not overcommit.  (Please take any further discussions of this to a
Linux kernel list, not a gcc list.  Thanks.)

Ian