Problems with the cpp change after 2000-06-18

Problems with the cpp change after 2000-06-18

[H . J . Lu]

Hi, Zack,

I am not sure if I like the cpp change after 2000-06-18 when cpp keeps
so many files open. The problem is on a 4way SMP machie, I can do

# make MAKE="make -j 8"

in the Linux kernel build. My system file descriptor limit is reached
in minutes. Then other processes on my machine start dying. I know you
put in a fix so that cpp won't die. But it doesn't help other processes.
I think you should put a limit on the number of opened files in cpp.

Thanks.


H.J.

Re: Problems with the cpp change after 2000-06-18

[Zack Weinberg]

On Mon, Jul 03, 2000 at 10:09:19AM -0700, H . J . Lu wrote:
> Hi, Zack,
> 
> I am not sure if I like the cpp change after 2000-06-18 when cpp keeps
> so many files open. The problem is on a 4way SMP machie, I can do
> 
> # make MAKE="make -j 8"
> 
> in the Linux kernel build. My system file descriptor limit is reached
> in minutes. Then other processes on my machine start dying. I know you
> put in a fix so that cpp won't die. But it doesn't help other processes.
> I think you should put a limit on the number of opened files in cpp.

Try using ulimit -n.

zw

Re: Problems with the cpp change after 2000-06-18

[Zack Weinberg]

On Mon, Jul 03, 2000 at 10:22:01AM -0700, Zack Weinberg wrote:
> On Mon, Jul 03, 2000 at 10:09:19AM -0700, H . J . Lu wrote:
> > Hi, Zack,
> > 
> > I am not sure if I like the cpp change after 2000-06-18 when cpp keeps
> > so many files open. The problem is on a 4way SMP machie, I can do
> > 
> > # make MAKE="make -j 8"
> > 
> > in the Linux kernel build. My system file descriptor limit is reached
> > in minutes. Then other processes on my machine start dying. I know you
> > put in a fix so that cpp won't die. But it doesn't help other processes.
> > I think you should put a limit on the number of opened files in cpp.
> 
> Try using ulimit -n.

To clarify a bit.  I agree this is a problem.  I want you to try
ulimit -n and see if that is an effective workaround.  That will help
me figure out how to fix the problem in the code.

zw

Re: Problems with the cpp change after 2000-06-18

[H . J . Lu]

On Mon, Jul 03, 2000 at 10:22:01AM -0700, Zack Weinberg wrote:
> On Mon, Jul 03, 2000 at 10:09:19AM -0700, H . J . Lu wrote:
> > Hi, Zack,
> > 
> > I am not sure if I like the cpp change after 2000-06-18 when cpp keeps
> > so many files open. The problem is on a 4way SMP machie, I can do
> > 
> > # make MAKE="make -j 8"
> > 
> > in the Linux kernel build. My system file descriptor limit is reached
> > in minutes. Then other processes on my machine start dying. I know you
> > put in a fix so that cpp won't die. But it doesn't help other processes.
> > I think you should put a limit on the number of opened files in cpp.
> 
> Try using ulimit -n.
> 

I have to build kernel as root. I am not sure "ulimit -n" will help.


H.J.

Re: Problems with the cpp change after 2000-06-18

[Zack Weinberg]

On Mon, Jul 03, 2000 at 10:41:38AM -0700, H . J . Lu wrote:
> On Mon, Jul 03, 2000 at 10:22:01AM -0700, Zack Weinberg wrote:
> > On Mon, Jul 03, 2000 at 10:09:19AM -0700, H . J . Lu wrote:
> > > Hi, Zack,
> > > 
> > > I am not sure if I like the cpp change after 2000-06-18 when cpp keeps
> > > so many files open. The problem is on a 4way SMP machie, I can do
> > > 
> > > # make MAKE="make -j 8"
> > > 
> > > in the Linux kernel build. My system file descriptor limit is reached
> > > in minutes. Then other processes on my machine start dying. I know you
> > > put in a fix so that cpp won't die. But it doesn't help other processes.
> > > I think you should put a limit on the number of opened files in cpp.
> > 
> > Try using ulimit -n.
> > 
> 
> I have to build kernel as root. I am not sure "ulimit -n" will help.

Funny, I've never had any trouble building kernels as an unprivileged
user.

In any case, resource limits apply to uid 0 processes just as they
would any other process; sure, they can call setrlimit() and raise
them, but cpp doesn't do that.

zw

Re: Problems with the cpp change after 2000-06-18

[H . J . Lu]

On Mon, Jul 03, 2000 at 10:54:38AM -0700, Zack Weinberg wrote:
> > > Try using ulimit -n.
> > > 
> > 
> > I have to build kernel as root. I am not sure "ulimit -n" will help.
> 
> Funny, I've never had any trouble building kernels as an unprivileged
> user.
> 
> In any case, resource limits apply to uid 0 processes just as they
> would any other process; sure, they can call setrlimit() and raise
> them, but cpp doesn't do that.

"ulimit -n" seems to work for root. The problem is "ulimit -n" may
not work for me in my shell since other processes have legitimate
reasons to open many files. I tried "ulimit -n 16" which failed. I
am running ""ulimit -n 32" now. It will take at least an hour to know
for sure. Also I don't know how portable setrlimit() will be. Is that
in libiberty? A counter in cpp should work ok.


H.J.

Re: Problems with the cpp change after 2000-06-18

[Zack Weinberg]

On Mon, Jul 03, 2000 at 03:30:13PM -0700, H . J . Lu wrote:
> On Mon, Jul 03, 2000 at 10:54:38AM -0700, Zack Weinberg wrote:
> > > > Try using ulimit -n.
> > > > 
> > > 
> > > I have to build kernel as root. I am not sure "ulimit -n" will help.
> > 
> > Funny, I've never had any trouble building kernels as an unprivileged
> > user.
> > 
> > In any case, resource limits apply to uid 0 processes just as they
> > would any other process; sure, they can call setrlimit() and raise
> > them, but cpp doesn't do that.
> 
> "ulimit -n" seems to work for root. The problem is "ulimit -n" may
> not work for me in my shell since other processes have legitimate
> reasons to open many files.

Which is why this is a test, not the solution - see the other message.

> I tried "ulimit -n 16" which failed. I am running ""ulimit -n 32"
> now. It will take at least an hour to know for sure. Also I don't
> know how portable setrlimit() will be. Is that in libiberty? A
> counter in cpp should work ok.

Believe me, I have no intention of dinking with resource limits inside
cpplib.

We may indeed wind up with a counter.  Another option is to cache the
contents of the file instead of the file descriptor; that would save
even more work, and avoid holding files open.

Question for the audience in general: Are there any supported
platforms where autoconf thinks we can mmap files (HAVE_MMAP_FILE) but
the kernel chokes if you close the file descriptor before you destroy
the mapping?  (I have dim memories of NT acting like this...)  And are
there any supported platforms, etc. with a small limit on the number
of distinct file mappings active at once (entire system or
per-process)?

zw

Re: Problems with the cpp change after 2000-06-18

[H . J . Lu]

On Mon, Jul 03, 2000 at 04:24:29PM -0700, Zack Weinberg wrote:
> 
> > I tried "ulimit -n 16" which failed. I am running ""ulimit -n 32"
> > now. It will take at least an hour to know for sure. Also I don't
> > know how portable setrlimit() will be. Is that in libiberty? A
> > counter in cpp should work ok.
> 
> Believe me, I have no intention of dinking with resource limits inside
> cpplib.
> 

I can report "ulimit -n 32" worked for "make -j 8" on a 4way SMP.


H.J.