cannot run many programs simultaneously

cannot run many programs simultaneously

[Anna Sidera]

Hello,

I wrote a gcc program and I am running it on a unix server that has 8 cpus. I have to run the program for different sets of input paramters. I run the program 6 times simultaneously with 6 different sets of input parameters. They run for some time and then they stop. However when I run only 3 of them simultaneously, and then run the other 3, they all finish successfully. Can you please advise me what to do?

I do not know if it is a bug or a problem with the memory the system allows me to use. The output of the ulimit -a command is the following:
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 88063
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 88063
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Thanks,
Anna

Re: cannot run many programs simultaneously

[Ian Lance Taylor]

Anna Sidera <sidera.anan@ucy.ac.cy> writes:

> I wrote a gcc program and I am running it on a unix server that has
> 8 cpus. I have to run the program for different sets of input
> paramters. I run the program 6 times simultaneously with 6 different
> sets of input parameters. They run for some time and then they
> stop. However when I run only 3 of them simultaneously, and then run
> the other 3, they all finish successfully. Can you please advise me
> what to do?

I have no idea.  This is a gcc help mailing list, and you haven't
given us any reason to think that this is a gcc issue.  If you want
help you will not to provide much more information.  But unless you
think this is somehow related to gcc, I recommend asking on a
general programming mailing list.

Ian

Re: cannot run many programs simultaneously

[John S. Fine]

As Ian already explained, you're asking in the wrong place, because 
there is no reason to believe this is a GCC issue.

The programming or linux-general forums at LQ would be a better place if 
this discussion needs a lot of follow up.
http://www.linuxquestions.org/questions/

But for now, I'll give the basic answer:

Anna Sidera wrote:
> They run for some time and then they stop. However when I run only 3 of them simultaneously, and then run the other 3, they all finish successfully.
>   
The obvious explanation for that behavior would be you hit either the 
system over commit limit or actual exhaustion of memory.  Until you rule 
that out, you shouldn't look for any more obscure theory.

Either one of those (over commit limit or exhaustion of memory) could be 
dealt with by increasing the amount of swap space in the system.  If you 
lack enough physical memory to run 6 of those processes at once, 
increasing swap space would stop that from crashing, but might just make 
them crawl instead of crash.  (Depending on what you mean by "stop", 
that might already be what you are experiencing.  You may need to 
explain "stop" better in order to get a better answer).
> I do not know if it is a bug or a problem with the memory the system allows me to use. The output of the ulimit -a command is the following:

That isn't relevant.  No matter what ulimit allows each process to use, 
it can't allow the total of all six processes to use more than is 
available for the whole system.

Re: cannot run many programs simultaneously

[Anna Sidera]

When I said stop, I meant they stop running before finishing.

If increasing the swap space makes them vrey slow this will not help me. The reason I run 6 programms simultaneously is to save time.

Can you explain 'hit either the system over commit limit or actual exhaustion of memory'?

I will ask about this problem at the linux forum also.

Thanks,
Anna

----- Original Message -----
From: "John S. Fine" <johnsfine@verizon.net>
Date: Wednesday, March 3, 2010 7:35 pm
Subject: Re: cannot run many programs simultaneously

> As Ian already explained, you're asking in the wrong place, 
> because 
> there is no reason to believe this is a GCC issue.
> 
> The programming or linux-general forums at LQ would be a better 
> place if 
> this discussion needs a lot of follow up.
> http://www.linuxquestions.org/questions/
> 
> But for now, I'll give the basic answer:
> 
> Anna Sidera wrote:
> > They run for some time and then they stop. However when I run 
> only 3 of them simultaneously, and then run the other 3, they all 
> finish successfully.
> >   
> The obvious explanation for that behavior would be you hit either 
> the 
> system over commit limit or actual exhaustion of memory.  Until 
> you rule 
> that out, you shouldn't look for any more obscure theory.
> 
> Either one of those (over commit limit or exhaustion of memory) 
> could be 
> dealt with by increasing the amount of swap space in the system.  
> If you 
> lack enough physical memory to run 6 of those processes at once, 
> increasing swap space would stop that from crashing, but might 
> just make 
> them crawl instead of crash.  (Depending on what you mean by 
> "stop", 
> that might already be what you are experiencing.  You may need to 
> explain "stop" better in order to get a better answer).
> > I do not know if it is a bug or a problem with the memory the 
> system allows me to use. The output of the ulimit -a command is 
> the following:
> 
> That isn't relevant.  No matter what ulimit allows each process to 
> use, 
> it can't allow the total of all six processes to use more than is 
> available for the whole system.
> 
> 

RE: cannot run many programs simultaneously

[John (Eljay) Love-Jensen]

Hi Anna,

> When I said stop, I meant they stop running before finishing.

I am not sure what you mean by "stop" here.

Stop as in crash?

> If increasing the swap space makes them vrey slow this will not help me. The reason I run 6 programms simultaneously is to save time.

What is the memory footprint of one of the programs running by itself?

For example, let's assume your machine has 64 GB of RAM, and 256 GB of swap space.  If one program running consumes 128 GB of RAM, then when you run the third program, your machine will have exhausted its supply of RAM (both physical and swap).

> Can you explain 'hit either the system over commit limit or actual exhaustion of memory'?

If you have three programs running and each asks for 128 GB of RAM, and you machine physically has 64 GB of RAM and 256 GB of swap, the OS will give each program 128 GB of RAM.  That overcommits the available RAM (physical + swap).

Everything will run okay as long as the programs don't actually use as much RAM as they asked for.  (A surprisingly large number of program exhibit the behavior of asking for a lot more heap memory than they actually use.)

But if the programs do actually need all the RAM they requested, then later on when the program goes to swap in some fresh page of memory the program will fault.  Note that the fault happens long, long after the request for memory was made (and which was made without having an error code).

Sincerely,
--Eljay

Re: RE: cannot run many programs simultaneously

[Anna Sidera]

When I said stop I meant crash.

I am sending you the output of the top command when I run the program only one time.

top - 17:58:23 up 8 days,  2:45,  1 user,  load average: 1.00, 1.00, 1.00
Tasks: 120 total,   2 running, 118 sleeping,   0 stopped,   0 zombie
Cpu(s): 36.6%us,  0.1%sy,  0.0%ni, 63.4%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  10267424k total,  2039184k used,  8228240k free,   153588k buffers
Swap:  2963952k total,    35404k used,  2928548k free,   104768k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
23308 sidera    20   0 8809m 1.5g  720 R  101 15.8 592:06.37 fir2p
    1 root      20   0 10312  200  168 S    0  0.0   0:05.50 init
    2 root      15  -5     0    0    0 S    0  0.0   0:00.00 kthreadd
    3 root      RT  -5     0    0    0 S    0  0.0   0:20.02 migration/0
    4 root      15  -5     0    0    0 S    0  0.0   0:00.20 ksoftirqd/0
    5 root      RT  -5     0    0    0 S    0  0.0   0:08.72 watchdog/0
    6 root      RT  -5     0    0    0 S    0  0.0   0:08.64 migration/1
    7 root      15  -5     0    0    0 S    0  0.0   0:00.00 ksoftirqd/1
    8 root      RT  -5     0    0    0 S    0  0.0   0:00.72 watchdog/1
    9 root      RT  -5     0    0    0 S    0  0.0   0:09.16 migration/2
   10 root      15  -5     0    0    0 S    0  0.0   0:00.00 ksoftirqd/2
   11 root      RT  -5     0    0    0 S    0  0.0   0:00.76 watchdog/2
   12 root      RT  -5     0    0    0 S    0  0.0   0:03.08 migration/3
   13 root      15  -5     0    0    0 S    0  0.0   0:00.08 ksoftirqd/3
   14 root      RT  -5     0    0    0 S    0  0.0   0:00.80 watchdog/3
   15 root      RT  -5     0    0    0 S    0  0.0   0:00.06 migration/4
   16 root      15  -5     0    0    0 S    0  0.0   0:00.00 ksoftirqd/4

I think this means that the system has 10 GB RAM and 2 GB memory that can be used on the disk. In this case, the program asked for 8809 MB of RAM? How much memory is the program currently using? How many of these programs should I be able to run simulatneously?

Thanks,
Anna

----- Original Message -----
From: "John (Eljay) Love-Jensen" <eljay@adobe.com>
Date: Thursday, March 4, 2010 5:33 am
Subject: RE: cannot run many programs simultaneously

> Hi Anna,
> 
> > When I said stop, I meant they stop running before finishing.
> 
> I am not sure what you mean by "stop" here.
> 
> Stop as in crash?
> 
> > If increasing the swap space makes them vrey slow this will not 
> help me. The reason I run 6 programms simultaneously is to save time.
> 
> What is the memory footprint of one of the programs running by itself?
> 
> For example, let's assume your machine has 64 GB of RAM, and 256 
> GB of swap space.  If one program running consumes 128 GB of RAM, 
> then when you run the third program, your machine will have 
> exhausted its supply of RAM (both physical and swap).
> 
> > Can you explain 'hit either the system over commit limit or 
> actual exhaustion of memory'?
> 
> If you have three programs running and each asks for 128 GB of 
> RAM, and you machine physically has 64 GB of RAM and 256 GB of 
> swap, the OS will give each program 128 GB of RAM.  That 
> overcommits the available RAM (physical + swap).
> 
> Everything will run okay as long as the programs don't actually 
> use as much RAM as they asked for.  (A surprisingly large number 
> of program exhibit the behavior of asking for a lot more heap 
> memory than they actually use.)
> 
> But if the programs do actually need all the RAM they requested, 
> then later on when the program goes to swap in some fresh page of 
> memory the program will fault.  Note that the fault happens long, 
> long after the request for memory was made (and which was made 
> without having an error code).
> 
> Sincerely,
> --Eljay
> 

Re: cannot run many programs simultaneously

[John S. Fine]

Anna Sidera wrote:
> If increasing the swap space makes them vrey slow this will not help me. The reason I run 6 programms simultaneously is to save time.
>   
We don't know that increasing swap space would change it from failing to 
crawling.  It might do that. But it might change it from failing to 
working just fine.  That depends on details of the memory access pattern 
that you really can't know until you try.
> Can you explain 'hit either the system over commit limit or actual exhaustion of memory'?
>   
In your later email you showed one process whose total virtual memory 
was 8809MB and resident memory was 1.5GB and the system wide swap use 
was 35404KB.  Not all of the 8809MB is committed memory, but probably 
nearly all of it.  Not all of the 1.5GB of resident memory is used 
committed memory, but probably almost all of it, plus all of the 35404KB 
swap usage is used committed memory by some task, probably some of it is 
for this task.

Should I assume you ran top when that process was at or near its maximum 
memory use?

If so, I can estimate the used committed memory for six of those as 
around 9GB.  That is likely to be OK (not exhaust memory) with 10GB of 
physical ram and nearly 3GB of swap.  But that doesn't leave much 
margin.  So I can't say with confidence that memory isn't exhausted.  I 
think you should significantly increase the 3GB swap space.

We can estimate about 52GB of committed memory for the six processes.  I 
forget the exact parameters and formula that control over commit.  
Roughly, Linux adds most of the memory size to all off the swap size and 
then multiplies that by some over commit factor to get the maximum it 
will commit.  With just 3GB swap and 10GB ram and default settings for 
those parameters, I doubt that it would let you commit 52GB.

Since the ratio of commit to used in this process seems to be higher 
than typical, you might fix the problem by only changing the over commit 
parameters.  You really should be asking in a Linux forum, not a GCC 
mailing list to learn how to do that.

Increasing the swap space, even if that swap space ends up not being 
used, will affect the result of the over commit computation, so you 
might fix the problem just by increasing swap space and not tweaking the 
over commit parameters.  But that might need swap space that is a 
significant fraction of the 52GB you seem to want to commit.  If you 
have new large disks, that much swap space might be trivial and 
allocating it might be the simplest and most robust solution to your 
problem.  If you have older smaller disks, many GB of swap space might 
be a serious cost.
> I will ask about this problem at the linux forum also.
>
>   

You should have.  Maybe post the URL of your question in that forum as a 
message to close the topic from further discussion here.

Re: RE: cannot run many programs simultaneously

[Axel Freyn]

Hi Anna,
On Thu, Mar 04, 2010 at 06:06:09PM +0200, Anna Sidera wrote:
> I am sending you the output of the top command when I run the program only one time.
> 
> top - 17:58:23 up 8 days,  2:45,  1 user,  load average: 1.00, 1.00, 1.00
> Tasks: 120 total,   2 running, 118 sleeping,   0 stopped,   0 zombie
> Cpu(s): 36.6%us,  0.1%sy,  0.0%ni, 63.4%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
> Mem:  10267424k total,  2039184k used,  8228240k free,   153588k buffers
> Swap:  2963952k total,    35404k used,  2928548k free,   104768k cached
This part means, you have a total of 10GB Ram + 2.9GB swap-space on the
disk, so you can use at the same time up to 13GB.
In addition: From the 10.2GB of Ram, 8.2GB are completely free at the
moment. From the 3GB of Swap-space on the disk, only 35MB are used.
> 
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 23308 sidera    20   0 8809m 1.5g  720 R  101 15.8 592:06.37 fir2p
> 
> I think this means that the system has 10 GB RAM and 2 GB memory that
> can be used on the disk. In this case, the program asked for 8809 MB
> of RAM? How much memory is the program currently using? 
Yes, the program allocated (using "alloc" oder "new" or a similar
function) 8.8GB of memory. At the moment, it is using 1.5GB - the
remaining part of the allocated memory (8.8GB-1.5GB=7.3GB) is not used
by the program until now.
> How many of these programs should I be able to run simulatneously?

That is an answer nobody of us can answer;-)

To be on the safe side: IF the program is going to use ALL of these 8.8
GB - you can only run one program at a time (already two simultaneous
programs would need 2*8.8GB = 17.6GB, which is more than you have).

However: Nobody can say for sure whether that will happen or not: if the
program never uses more than 1.5GB, you can run up to 13GB/1.5GB = 8
programs simultaneously (well, maybe 7 as you also have other programs
needing memory).

In addition, there is of course a third possibility: maybe the program
is going to allocate even more memory - and also going to use it. Than
it might happen, that even the single run will crash if it runs to long. 

So, the safest solution would be to read the documentation / sourcecode
of the program in order to determine how much memory it is expected to
use:-)

HTH,

Axel