Many new test failures (execvp: Too many open files)

Many new test failures (execvp: Too many open files)

[Mark Wielaard]

Hi,

On my Fedora 7 x86 machine (2.6.21-1.3228.fc7 SMP) I am suddenly seeing
lots and lots of new test failures/errors. Lots mention "execvp: Too
many open files", I am not seeing this on my x86_64 Fedora Core 6 setup.
And they only started to massively fail today it seems (although on irc
some people say they saw them already earlier this week). Anyone know
what that is about?

I also saw that at the same time there was a renaming of "broken" to
"unresolved" for the TestCases. Is that a functional change or just a
textual change? Maybe some tests were mislabeled in the process, so they
suddenly started running on my system and before they didn't and so
triggering this "execvp: Too many open files" now?

Cheers,

Mark

Re: Many new test failures (execvp: Too many open files)

[Andrew Cagney]

Mark Wielaard wrote:
> Hi,
>
> On my Fedora 7 x86 machine (2.6.21-1.3228.fc7 SMP) I am suddenly seeing
> lots and lots of new test failures/errors. Lots mention "execvp: Too
> many open files", I am not seeing this on my x86_64 Fedora Core 6 setup.
> And they only started to massively fail today it seems (although on irc
> some people say they saw them already earlier this week). Anyone know
> what that is about?
>
>   
I'm guessing this wasn't a scratch build.  Bug 4742.

> I also saw that at the same time there was a renaming of "broken" to
> "unresolved" for the TestCases. Is that a functional change or just a
> textual change? Maybe some tests were mislabeled in the process, so they
> suddenly started running on my system and before they didn't and so
> triggering this "execvp: Too many open files" now?
>   

Two things to do in this situation are to review the changes you suspect 
could be problematic, and to scratch build before/after trees of the 
specific change showing it caused the breakage.

Andrew


>   

Re: Many new test failures (execvp: Too many open files)

[Mark Wielaard]

On Thu, 2007-07-05 at 10:23 -0400, Andrew Cagney wrote:
> Mark Wielaard wrote:
> > On my Fedora 7 x86 machine (2.6.21-1.3228.fc7 SMP) I am suddenly seeing
> > lots and lots of new test failures/errors. Lots mention "execvp: Too
> > many open files", I am not seeing this on my x86_64 Fedora Core 6 setup.
> > And they only started to massively fail today it seems (although on irc
> > some people say they saw them already earlier this week). Anyone know
> > what that is about?
> >   
> I'm guessing this wasn't a scratch build.

Of course it was a scratch build, otherwise I would not reported it :)
I always do a scratch build when doing any cvs update because our build
system doesn't seem very robust in the face of file deletion, addition
or renaming. I wasn't even aware incremental build should work unless
you only made changes to existing files. Is it supposed to?
I always start my day with a cvs update and then a scratch build and get
myself some breakfast :)

> Bug 4742.

Interesting, make sure you report that one upstream. I don't believe
many, if any, other project rely on the dependency generation feature of
gcj so they probably have not heard about that yet.

> > I also saw that at the same time there was a renaming of "broken" to
> > "unresolved" for the TestCases. Is that a functional change or just a
> > textual change? Maybe some tests were mislabeled in the process, so they
> > suddenly started running on my system and before they didn't and so
> > triggering this "execvp: Too many open files" now?
> >   
> Two things to do in this situation are to review the changes you suspect 
> could be problematic, and to scratch build before/after trees of the 
> specific change showing it caused the breakage.

Could you explain why the changes were made and what they were supposed
to do? That would help in reviewing the issue.

Thanks,

Mark

Re: Many new test failures (execvp: Too many open files)

[Andrew Cagney]

Mark Wielaard wrote:
> On Thu, 2007-07-05 at 10:23 -0400, Andrew Cagney wrote:
>   
>> Mark Wielaard wrote:
>>     
>>> On my Fedora 7 x86 machine (2.6.21-1.3228.fc7 SMP) I am suddenly seeing
>>> lots and lots of new test failures/errors. Lots mention "execvp: Too
>>> many open files", I am not seeing this on my x86_64 Fedora Core 6 setup.
>>> And they only started to massively fail today it seems (although on irc
>>> some people say they saw them already earlier this week). Anyone know
>>> what that is about?
>>>   
>>>       
>> I'm guessing this wasn't a scratch build.
>>     
>
> Of course it was a scratch build, otherwise I would not reported it :)
> I always do a scratch build when doing any cvs update because our build
> system doesn't seem very robust in the face of file deletion, addition
> or renaming. I wasn't even aware incremental build should work unless
> you only made changes to existing files. Is it supposed to?
> I always start my day with a cvs update and then a scratch build and get
> myself some breakfast :)
>
>   
>> Bug 4742.
>>     
>
> Interesting, make sure you report that one upstream. I don't believe
> many, if any, other project rely on the dependency generation feature of
> gcj so they probably have not heard about that yet.
>
>   
>>> I also saw that at the same time there was a renaming of "broken" to
>>> "unresolved" for the TestCases. Is that a functional change or just a
>>> textual change? Maybe some tests were mislabeled in the process, so they
>>> suddenly started running on my system and before they didn't and so
>>> triggering this "execvp: Too many open files" now?
>>>   
>>>       
>> Two things to do in this situation are to review the changes you suspect 
>> could be problematic, and to scratch build before/after trees of the 
>> specific change showing it caused the breakage.
>>     
>
> Could you explain why the changes were made and what they were supposed
> to do? That would help in reviewing the issue.
>   
You described it; s/brokenXXX/unresolvedOnYYY/ ; this finishes the 
introduction of UNRESOLVED I started some time ago.  You'll also see I 
killed some unused functions; and simplified unresolvedOnUtrace (as fc5 
is gone).  Perhaps it is that last change?

Andrew

Re: Many new test failures (execvp: Too many open files)

[Mark Wielaard]

[-- Attachment #1: Type: text/plain, Size: 1124 bytes --]

Hi Andrew,

On Thu, 2007-07-05 at 12:14 -0400, Andrew Cagney wrote:
> > Could you explain why the changes were made and what they were supposed
> > to do? That would help in reviewing the issue.
> >   
> You described it; s/brokenXXX/unresolvedOnYYY/ ; this finishes the 
> introduction of UNRESOLVED I started some time ago.

OK, I must have missed that sorry. So there is not supposed to be any
functional change just a global search/replace on the whole code base
because unresolved is a better term than broken?

>   You'll also see I 
> killed some unused functions; and simplified unresolvedOnUtrace (as fc5 
> is gone).  Perhaps it is that last change?

I'll try that out tomorrow morning. On irc we discussed the problem of
leaking file descriptors in things like Elf objects which currently seem
to rely on garbage collection to close out unused descriptors. So it
might also be some subtle change in memory use that make the garbage
collector trigger earlier or later. In that case it is probably that we
just fail to close()/destroy() the right object at the right time.

Cheers,

Mark

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Many new test failures (execvp: Too many open files)

[Andrew Cagney]

Mark Wielaard wrote:
> I'll try that out tomorrow morning. On irc we discussed the problem of
> leaking file descriptors in things like Elf objects which currently seem
> to rely on garbage collection to close out unused descriptors. So it
> might also be some subtle change in memory use that make the garbage
> collector trigger earlier or later. In that case it is probably that we
> just fail to close()/destroy() the right object at the right time.
>   

Thanks.

Re: Many new test failures (execvp: Too many open files)

[Mark Wielaard]

Hi Andrew,

On Thu, 2007-07-05 at 12:14 -0400, Andrew Cagney wrote:
> Mark Wielaard wrote:
> > Could you explain why the changes were made and what they were supposed
> > to do? That would help in reviewing the issue.
> >   
> You described it; s/brokenXXX/unresolvedOnYYY/ ; this finishes the 
> introduction of UNRESOLVED I started some time ago.  You'll also see I 
> killed some unused functions; and simplified unresolvedOnUtrace (as fc5 
> is gone).  Perhaps it is that last change?

Indeed. The old version returned false on Fedora 7, the new version
returns true. Meaning in particular that several tests in
TestTaskObserver and TestTaskObserver now run while they didn't before
this patch. But this doesn't seem to be the root cause for the execvp
issues.

The thing that worked around it for me for now seems to be to disable
the frysk.rt.TestStepping.testASMSingleStep() test that seems to fail
with ERROR java.lang.ArrayIndexOutOfBoundsException: 0
Apparently this failure causes no more file descriptors to be available.
Reported as http://sourceware.org/bugzilla/show_bug.cgi?id=4751
And adjusted the TestStepping accordingly.

2007-07-09  Mark Wielaard  <mwielaard@redhat.com>

  * TestStepping.java (testASMSingleStep): Add unresolved for bug #4751.

Cheers,

Mark

diff -u -r1.39 TestStepping.java
--- frysk-core/frysk/rt/TestStepping.java       5 Jul 2007 03:01:50 -0000      1.39
+++ frysk-core/frysk/rt/TestStepping.java       9 Jul 2007 09:23:40 -0000
@@ -254,6 +254,9 @@
     if (unresolvedOnPPC(3277))
       return;
     
+    if (unresolved(4751))
+      return;
+
     initial = true;
     this.lineMap = new HashMap();

Re: Many new test failures (execvp: Too many open files)

[Andrew Cagney]

Mike and I tracked down the likely cause - Dwfl's aren't been cleaned up 
aggressively enough (something that is dependent on many things 
including the garbage collector).  We're making two changes:
- adding teardown code to frysk.dwfl that will close all open Dwfls
- enabling code that closes old dwfl's as new ones are created

One thing though:
> Indeed. The old version returned false on Fedora 7, the new version
> returns true. Meaning in particular that several tests in
> TestTaskObserver and TestTaskObserver now run while they didn't before
> this patch. But this doesn't seem to be the root cause for the execvp
> issues.
>   
This is not correct.  unresolvedOnUtrace returning TRUE causes the test 
to be _skipped_; so this causes less tests to be run.  Given that Fedora 
7 has UTRACE, it appears I fixed a latent bug.

Andrew

unresolvedOnUtrace (Was: Many new test failures (execvp: Too many open files))

[Mark Wielaard]

[-- Attachment #1: Type: text/plain, Size: 948 bytes --]

Hi Andrew,

On Mon, 2007-07-09 at 08:50 -0400, Andrew Cagney wrote:
> This is not correct.  unresolvedOnUtrace returning TRUE causes the test 
> to be _skipped_; so this causes less tests to be run.  Given that Fedora 
> 7 has UTRACE, it appears I fixed a latent bug.

Duh, you are right, I had my results crossed. Sorry about that.

But that is interesting because I now really double checked and it looks
like on Fedora 7 (2.6.21-1.3228.fc7 x86 SMP) bugs #3595 and #3737 are
fixed and unresolvedOnUtrace() is setting the following test results to
UNRESOLVED for these 4 tests that actually PASS on my machine:

Running testMultiThreadedStoppedAckDaemon(frysk.proc.TestProcStopped) ...PASS
Running testMultiThreadedStoppedDetached(frysk.proc.TestProcStopped) ...PASS
Running testDetachExitingMainTask(frysk.proc.TestTaskObserver) ...PASS
Running testDetachExitingOtherTask(frysk.proc.TestTaskObserver) ...PASS

Cheers,

Mark

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]