maximum array size?

maximum array size?

[Anna Sidera]

Hello,

I wrote a program in gcc. I compile it with the option -m64. Is there a limit in how much memory malloc can allocate? I run the program for different values of some input parameters. The program defines some scalar variables and arrays. Then it defines a large array and tries to set a value in an element of the array.
int **bufferc = malloc((inp_par_a*inp_par_b+2)*sizeof(int *));
int *bufferc_aux = malloc((inp_par_a*inp_par_b+2)*(inp_par_c+1)*sizeof(int));
int bfc=1;
bufferc[bfc]=bufferc_aux+bfc*(inp_par_c+1);
bufferc[bfc][0]=0;
At this point sometimes there is no problem but sometimes I get segmentation fault. There is a lot of unused ram in the server when the problem occurs. Do you think the problem is due to a bug that I have before defining the array?

Thanks,
Anna

Re: maximum array size?

[Brian Budge]

Is this running in Linux?  Linux usually behaves optimistically,
actually returning a valid pointer from malloc, even when there is not
enough memory left in the system to satisfy the request.  A good
explanation can be found here:
http://linuxdevcenter.com/pub/a/linux/2006/11/30/linux-out-of-memory.html

On Thu, Apr 15, 2010 at 1:26 AM, Anna Sidera <sidera.anan@ucy.ac.cy> wrote:
> Hello,
>
> I wrote a program in gcc. I compile it with the option -m64. Is there a limit in how much memory malloc can allocate? I run the program for different values of some input parameters. The program defines some scalar variables and arrays. Then it defines a large array and tries to set a value in an element of the array.
> int **bufferc = malloc((inp_par_a*inp_par_b+2)*sizeof(int *));
> int *bufferc_aux = malloc((inp_par_a*inp_par_b+2)*(inp_par_c+1)*sizeof(int));
> int bfc=1;
> bufferc[bfc]=bufferc_aux+bfc*(inp_par_c+1);
> bufferc[bfc][0]=0;
> At this point sometimes there is no problem but sometimes I get segmentation fault. There is a lot of unused ram in the server when the problem occurs. Do you think the problem is due to a bug that I have before defining the array?
>
> Thanks,
> Anna
>

Re: maximum array size?

[Brian Budge]

Oh, you said that there is a lot of *unused* memory... sorry, I
misread.  This is almost certainly a bug in your program.  You should
compile with -g and run the program in gdb to see where the program
crashes.

  Brian

On Thu, Apr 15, 2010 at 1:36 AM, Brian Budge <brian.budge@gmail.com> wrote:
> Is this running in Linux?  Linux usually behaves optimistically,
> actually returning a valid pointer from malloc, even when there is not
> enough memory left in the system to satisfy the request.  A good
> explanation can be found here:
> http://linuxdevcenter.com/pub/a/linux/2006/11/30/linux-out-of-memory.html
>
> On Thu, Apr 15, 2010 at 1:26 AM, Anna Sidera <sidera.anan@ucy.ac.cy> wrote:
>> Hello,
>>
>> I wrote a program in gcc. I compile it with the option -m64. Is there a limit in how much memory malloc can allocate? I run the program for different values of some input parameters. The program defines some scalar variables and arrays. Then it defines a large array and tries to set a value in an element of the array.
>> int **bufferc = malloc((inp_par_a*inp_par_b+2)*sizeof(int *));
>> int *bufferc_aux = malloc((inp_par_a*inp_par_b+2)*(inp_par_c+1)*sizeof(int));
>> int bfc=1;
>> bufferc[bfc]=bufferc_aux+bfc*(inp_par_c+1);
>> bufferc[bfc][0]=0;
>> At this point sometimes there is no problem but sometimes I get segmentation fault. There is a lot of unused ram in the server when the problem occurs. Do you think the problem is due to a bug that I have before defining the array?
>>
>> Thanks,
>> Anna
>>
>

Re: maximum array size?

[John S. Fine]

Anna Sidera wrote:
> int **bufferc = malloc((inp_par_a*inp_par_b+2)*sizeof(int *));
> int *bufferc_aux = malloc((inp_par_a*inp_par_b+2)*(inp_par_c+1)*sizeof(int));
> int bfc=1;
> bufferc[bfc]=bufferc_aux+bfc*(inp_par_c+1);
> bufferc[bfc][0]=0;
>   
How big are you talking about?  What are the values of inp_par_a, 
inp_par_b, and inp_par_c

You ought to be able to allocate a chunk of memory as large as all your 
unused swap space plus a large part of your unused ram.  Even if Linux 
has already "committed" all that unused memory to other tasks that have 
reserved it without yet using it, Linux will commit it again to your task.

So if you aren't talking about a chunk big enough to exceed all unused 
swap space plus a big part of unused ram, then you probably have some 
bug other than trying to commit too much memory.

Re: maximum array size?

[John S. Fine]

Integers in x86_64 are 32 bit.  So in your chain of multiplying you need 
to multiply by something of type std::size_t before the running value 
goes over two billion.

As it is now, you get overflow in the integer multiply so the case that 
seems to work is actually reserving far less memory than you think you 
are requesting.

Once you get rid of the integer overflow, you should see that the amount 
you can reserve is limited by swap size plus large part of ram, so if 
you really want something that big, you'll need a lot more swap space.

Anna Sidera wrote:
> inp_par_a=10000 and inp_par_b=10000. For inp_par_c=60 the program works. For inp_par_c=40 or inp_par_c=80 I get segmentation fault.
>
> I just calculated the memory the array needs and it is more than the available 10GB ram + 3GB swap in the system even for inp_par_c=40.
>
>
>   
>
>   

Re: maximum array size?

[Anna Sidera]

Ok, thanks a lot for your help.

Anna
----- Original Message -----
From: "John S. Fine" <johnsfine@verizon.net>
Date: Thursday, April 15, 2010 6:30 pm
Subject: Re: maximum array size?

> Integers in x86_64 are 32 bit.  So in your chain of multiplying 
> you need 
> to multiply by something of type std::size_t before the running 
> value 
> goes over two billion.
> 
> As it is now, you get overflow in the integer multiply so the case 
> that 
> seems to work is actually reserving far less memory than you think 
> you 
> are requesting.
> 
> Once you get rid of the integer overflow, you should see that the 
> amount 
> you can reserve is limited by swap size plus large part of ram, so 
> if 
> you really want something that big, you'll need a lot more swap space.
> 
> Anna Sidera wrote:
> > inp_par_a=10000 and inp_par_b=10000. For inp_par_c=60 the 
> program works. For inp_par_c=40 or inp_par_c=80 I get segmentation 
> fault.>
> > I just calculated the memory the array needs and it is more than 
> the available 10GB ram + 3GB swap in the system even for inp_par_c=40.
> >
> >
> >   
> >
> >   
> 
> 

Re: maximum array size?

[Anna Sidera]

Ok, thanks a lot for your help.

Anna

----- Original Message -----
From: "John S. Fine" <johnsfine@verizon.net>
Date: Thursday, April 15, 2010 6:30 pm
Subject: Re: maximum array size?

> Integers in x86_64 are 32 bit.  So in your chain of multiplying 
> you need 
> to multiply by something of type std::size_t before the running 
> value 
> goes over two billion.
> 
> As it is now, you get overflow in the integer multiply so the case 
> that 
> seems to work is actually reserving far less memory than you think 
> you 
> are requesting.
> 
> Once you get rid of the integer overflow, you should see that the 
> amount 
> you can reserve is limited by swap size plus large part of ram, so 
> if 
> you really want something that big, you'll need a lot more swap space.
> 
> Anna Sidera wrote:
> > inp_par_a=10000 and inp_par_b=10000. For inp_par_c=60 the 
> program works. For inp_par_c=40 or inp_par_c=80 I get segmentation 
> fault.>
> > I just calculated the memory the array needs and it is more than 
> the available 10GB ram + 3GB swap in the system even for inp_par_c=40.
> >
> >
> >   
> >
> >   
> 
>