maximum number of variables?

maximum number of variables?

[Nicolas Ouellette]

Hello

I am presently running a huge code following the hydrodynamical evolution of 
a system (in 2 dimentions).  I store the quantities evolving (density, 
velocity, pressure) into large 2-D arrays.  Recently, I tried to make my 2-D 
arrays bigger (to study a bigger system) and eventually, I reached a certain 
point where I was getting a "segmentation error".  The only thing I changed 
was the size of my arrays.

I deleted a few arrays that were redundant (making my code less user 
friendly), and I found that I could slightly increase the size of my arrays 
until I reached the "segmentation error".

Here are my questions:  1) Is there a maximum number of variables allowed 
when using gcc? (from my experience I am guessing yes) 2) Is there a way to 
increase the number of variables allowed when using gcc?

Thanking you for your help

Nicolas Ouellette

Re: maximum number of variables?

[Bob Rossi]

On Tue, Aug 30, 2005 at 05:00:57PM -0400, Nicolas Ouellette wrote:
> Hello
> 
> I am presently running a huge code following the hydrodynamical evolution 
> of a system (in 2 dimentions).  I store the quantities evolving (density, 
> velocity, pressure) into large 2-D arrays.  Recently, I tried to make my 
> 2-D arrays bigger (to study a bigger system) and eventually, I reached a 
> certain point where I was getting a "segmentation error".  The only thing I 
> changed was the size of my arrays.
> 
> I deleted a few arrays that were redundant (making my code less user 
> friendly), and I found that I could slightly increase the size of my arrays 
> until I reached the "segmentation error".
> 
> Here are my questions:  1) Is there a maximum number of variables allowed 
> when using gcc? (from my experience I am guessing yes) 2) Is there a way to 
> increase the number of variables allowed when using gcc?

It sounds like you may have overflowed the stack. Could you try
increasing the stack size? I don't know how to do that though with gcc.

Bob Rossi

Re: maximum number of variables?

[Nicolas Ouellette]

>>Hello
>>
>>I am presently running a huge code following the hydrodynamical evolution 
>>of a system (in 2 dimentions).  I store the quantities evolving (density, 
>>velocity, pressure) into large 2-D arrays.  Recently, I tried to make my 
>>2-D arrays bigger (to study a bigger system) and eventually, I reached a 
>>certain point where I was getting a "segmentation error".  The only thing 
>>I changed was the size of my arrays.
>>
>>I deleted a few arrays that were redundant (making my code less user 
>>friendly), and I found that I could slightly increase the size of my 
>>arrays until I reached the "segmentation error".
>>
>>Here are my questions:  1) Is there a maximum number of variables allowed 
>>when using gcc? (from my experience I am guessing yes) 2) Is there a way 
>>to increase the number of variables allowed when using gcc?

>It sounds like you may have overflowed the stack. Could you try
>increasing the stack size? I don't know how to do that though with gcc.

I had to check on the web real to see what you meant.  I am a physicist 
trying to program, and not a programmer trying to do physics :)

According to a web site, the default stack size on OS X is 8 MB.  it is 
possible that I have exceeded that.  I am looking into ways of trying to 
increase this.

Nicolas Ouellette

Re: maximum number of variables?

[Lexington Luthor]

Nicolas Ouellette wrote:
> I had to check on the web real to see what you meant.  I am a physicist 
> trying to program, and not a programmer trying to do physics :)
> 
> According to a web site, the default stack size on OS X is 8 MB.  it is 
> possible that I have exceeded that.  I am looking into ways of trying to 
> increase this.
> 
> Nicolas Ouellette
> 
> 
> 

I think for large arrays, its best to allocate them from the heap. Use 
malloc() or new (if using C++).

LL

Re: maximum number of variables?

[Bob Rossi]

On Tue, Aug 30, 2005 at 10:54:02PM +0100, Lexington Luthor wrote:
> Nicolas Ouellette wrote:
> >I had to check on the web real to see what you meant.  I am a physicist 
> >trying to program, and not a programmer trying to do physics :)
> >
> >According to a web site, the default stack size on OS X is 8 MB.  it is 
> >possible that I have exceeded that.  I am looking into ways of trying to 
> >increase this.
> >
> >Nicolas Ouellette
> >
> >
> >
> 
> I think for large arrays, its best to allocate them from the heap. Use 
> malloc() or new (if using C++).

I've always been told that the heap and stack grow towards each other.
If this is true, why would it be OK to create the item on the heap, vs
on the stack? If it's not true, could someone simply explain how this
works?

Bob Rossi

Re: maximum number of variables?

[Ian Lance Taylor]

Bob Rossi <bob@brasko.net> writes:

> On Tue, Aug 30, 2005 at 10:54:02PM +0100, Lexington Luthor wrote:
> 
> > I think for large arrays, its best to allocate them from the heap. Use 
> > malloc() or new (if using C++).
> 
> I've always been told that the heap and stack grow towards each other.
> If this is true, why would it be OK to create the item on the heap, vs
> on the stack? If it's not true, could someone simply explain how this
> works?

You are correct in theory.  In practice the heap and stack have
different limits, and the limit on the heap is much larger than the
stack (if running bash, compare ulimit -s and ulimit -v).  And if you
worry about portability, on some platforms allocating a large stack
frame will simply fail, and on some other platforms it will require
extra work to emit stack probes to tell the OS that you are
intentionally extending the stack rather than just referencing a
random memory address.

Ian

Re[2]: maximum number of variables?

[Miguel A. Nuñez]

ILT> Bob Rossi <bob@brasko.net> writes:

>> On Tue, Aug 30, 2005 at 10:54:02PM +0100, Lexington Luthor wrote:
>> 
>> > I think for large arrays, its best to allocate them from the heap. Use
>> > malloc() or new (if using C++).
>> 
>> I've always been told that the heap and stack grow towards each other.
>> If this is true, why would it be OK to create the item on the heap, vs
>> on the stack? If it's not true, could someone simply explain how this
>> works?

ILT> You are correct in theory.  In practice the heap and stack have
ILT> different limits, and the limit on the heap is much larger than the
ILT> stack (if running bash, compare ulimit -s and ulimit -v).  And if you
ILT> worry about portability, on some platforms allocating a large stack
ILT> frame will simply fail, and on some other platforms it will require
ILT> extra work to emit stack probes to tell the OS that you are
ILT> intentionally extending the stack rather than just referencing a
ILT> random memory address.

ILT> Ian


Ian,

  I did ulimit -s and ulimit -v in a solaris 9 sun machine. ulimit -s returns 8192, ok, but ulimit -s returns unlimited. Does this make sense?


Miguel Angel

Re: maximum number of variables?

[Bob Rossi]

On Wed, Aug 31, 2005 at 11:13:00AM +0200, Miguel A. Nu?ez wrote:
> 
> 
> ILT> Bob Rossi <bob@brasko.net> writes:
> 
> >> On Tue, Aug 30, 2005 at 10:54:02PM +0100, Lexington Luthor wrote:
> >> 
> >> > I think for large arrays, its best to allocate them from the heap. Use
> >> > malloc() or new (if using C++).
> >> 
> >> I've always been told that the heap and stack grow towards each other.
> >> If this is true, why would it be OK to create the item on the heap, vs
> >> on the stack? If it's not true, could someone simply explain how this
> >> works?
> 
> ILT> You are correct in theory.  In practice the heap and stack have
> ILT> different limits, and the limit on the heap is much larger than the
> ILT> stack (if running bash, compare ulimit -s and ulimit -v).  And if you
> ILT> worry about portability, on some platforms allocating a large stack
> ILT> frame will simply fail, and on some other platforms it will require
> ILT> extra work to emit stack probes to tell the OS that you are
> ILT> intentionally extending the stack rather than just referencing a
> ILT> random memory address.
> 
> ILT> Ian
> 
> 
> Ian,
> 
>   I did ulimit -s and ulimit -v in a solaris 9 sun machine. ulimit -s returns 8192, ok, but ulimit -s returns unlimited. Does this make sense?

Assumming you mean ulimit -s = 8192 and ulimit -v  = unlimited, then
yes, I suppose that makes sense. On my linux machine, ulimit -s and
ulimit -v = unlimited. In this case, I suppose it's possible that if the
array was on the stack and things borked, it would also bork when on the
heap. However, in your case, I know understand why things would work for
you if you moved from the stack to the heap.

Bob Rossi

Re[2]: maximum number of variables?

[Miguel A. Nuñez]

>> 
>> 
>> ILT> Bob Rossi <bob@brasko.net> writes:
>> 
>> >> On Tue, Aug 30, 2005 at 10:54:02PM +0100, Lexington Luthor wrote:
>> >> 
>> >> > I think for large arrays, its best to allocate them from the heap. Use
>> >> > malloc() or new (if using C++).
>> >> 
>> >> I've always been told that the heap and stack grow towards each other.
>> >> If this is true, why would it be OK to create the item on the heap, vs
>> >> on the stack? If it's not true, could someone simply explain how this
>> >> works?
>> 
>> ILT> You are correct in theory.  In practice the heap and stack have
>> ILT> different limits, and the limit on the heap is much larger than the
>> ILT> stack (if running bash, compare ulimit -s and ulimit -v).  And if you
>> ILT> worry about portability, on some platforms allocating a large stack
>> ILT> frame will simply fail, and on some other platforms it will require
>> ILT> extra work to emit stack probes to tell the OS that you are
>> ILT> intentionally extending the stack rather than just referencing a
>> ILT> random memory address.
>> 
>> ILT> Ian
>> 
>> 
>> Ian,
>> 
>>   I did ulimit -s and ulimit -v in a solaris 9 sun machine.
>> ulimit -s returns 8192, ok, but ulimit -s returns unlimited. Does
>> this make sense?

BR> Assumming you mean ulimit -s = 8192 and ulimit -v  = unlimited, then
BR> yes, I suppose that makes sense. On my linux machine, ulimit -s and
BR> ulimit -v = unlimited. In this case, I suppose it's possible that if the
BR> array was on the stack and things borked, it would also bork when on the
BR> heap. However, in your case, I know understand why things would work for
BR> you if you moved from the stack to the heap.

BR> Bob Rossi


Yes, I meant that Bob. Anyway, remember it was Nicolas who originally had this problem. I'm running into a similar problem, so I tried the commands Ian suggested, and the value 'unlimited' was strange to me.

Let's see if anyone can help me with my problem. My application creates a new thread for a receiving infite loop. pthread_create runs the function containing the loop, and this functions defines a local variable that is a huge struct (sizeof returns 984KB). I initialize it using 'mystruct_t mystruct={0}'.

Up to this point everything is ok. When something is received, the loop calls a new function. This function creates a new local variable, the same struct. Here the program crashes.

I have read the default stack size when creating a thread in solaris 32 bits is 1MB, so I thought the crash would be due to allocating more memory than the stack size.

I put the initialization in a separate line, with a for loop giving value 0 byte per byte, from 0 to sizeof(mystruct_t)-1. Strangely (at least to me) the second struct is allocated, and the crash is always produced when writing to the bytes of the struct.

Is it possible that the local variable is allocated partly into the stack and partly beyond the limits of the stack, so that when I write to it I get my segmentation fault?? How could I know where the stack begins and ends?

I will allocate the structs at the heap, instead of the stack, but I would like to understand this situation.

Re: maximum number of variables?

[Bob Rossi]

On Tue, Aug 30, 2005 at 09:19:19PM -0700, Ian Lance Taylor wrote:
> Bob Rossi <bob@brasko.net> writes:
> 
> > On Tue, Aug 30, 2005 at 10:54:02PM +0100, Lexington Luthor wrote:
> > 
> > > I think for large arrays, its best to allocate them from the heap. Use 
> > > malloc() or new (if using C++).
> > 
> > I've always been told that the heap and stack grow towards each other.
> > If this is true, why would it be OK to create the item on the heap, vs
> > on the stack? If it's not true, could someone simply explain how this
> > works?
> 
> You are correct in theory.  In practice the heap and stack have
> different limits, and the limit on the heap is much larger than the
> stack (if running bash, compare ulimit -s and ulimit -v).  And if you
> worry about portability, on some platforms allocating a large stack
> frame will simply fail, and on some other platforms it will require
> extra work to emit stack probes to tell the OS that you are
> intentionally extending the stack rather than just referencing a
> random memory address.

Thanks for the great explanation! Also, please correct me if I'm wrong.
Most likely if ulimit -s == ulimit -v, then it's possible that moving
the array from the stack to the heap would still cause the same problem.
However, if the limit's are different (stack being smaller), then moving
the array could be a successful choice. Does this sound correct to you?

Thanks,
Bob Rossi

Re: maximum number of variables?

[corey taylor]

Nicolas,

  The stack settings are fairly common, but I found a decent FAQ
answer for you pertaining directly to OSX and even an XCode option
which is nice.

http://developer.apple.com/qa/qa2005/qa1419.html

corey

On 8/30/05, Nicolas Ouellette <sideshownic@hotmail.com> wrote:
> Hello
> 
> I am presently running a huge code following the hydrodynamical evolution of
> a system (in 2 dimentions).  I store the quantities evolving (density,
> velocity, pressure) into large 2-D arrays.  Recently, I tried to make my 2-D
> arrays bigger (to study a bigger system) and eventually, I reached a certain
> point where I was getting a "segmentation error".  The only thing I changed
> was the size of my arrays.
> 
> I deleted a few arrays that were redundant (making my code less user
> friendly), and I found that I could slightly increase the size of my arrays
> until I reached the "segmentation error".
> 
> Here are my questions:  1) Is there a maximum number of variables allowed
> when using gcc? (from my experience I am guessing yes) 2) Is there a way to
> increase the number of variables allowed when using gcc?
> 
> Thanking you for your help
> 
> Nicolas Ouellette
> 
> 
>