Problems with "make check"

Problems with "make check"

[Richard Kenner]

What is causing this?

                === libstdc++-v3 Summary ===
 
# of expected passes            1255
# of unexpected failures        76
# of expected failures          2
make[4]: *** [check-DEJAGNU] Error 1
make[4]: Leaving directory `/han.a/kenner/bld/i686-pc-linux-gnu/libstdc++-v3/tes
tsuite'
make[3]: *** [check-am] Error 2
make[3]: Leaving directory `/han.a/kenner/bld/i686-pc-linux-gnu/libstdc++-v3/tes
tsuite'
make[2]: *** [check-recursive] Error 1
make[2]: Leaving directory `/han.a/kenner/bld/i686-pc-linux-gnu/libstdc++-v3'
make[1]: *** [check-target-libstdc++-v3] Error 2
make[1]: Leaving directory `/han.a/kenner/bld'
make: *** [check] Error 2

Re: Problems with "make check"

[Zack Weinberg]

kenner@vlsi1.ultra.nyu.edu (Richard Kenner) writes:

> What is causing this?
>
>                 === libstdc++-v3 Summary ===
>  
> # of expected passes            1255
> # of unexpected failures        76
> # of expected failures          2
> make[4]: *** [check-DEJAGNU] Error 1

dejagnu exits unsuccessfully if there are unexpected failures.  Use
make -k check.

76 looks suspiciously like the number of failures that happen if you
don't have enough locales installed, see the list I posted last month.

zw

Re: Problems with "make check"

[Andreas Jaeger]

Zack Weinberg <zack@codesourcery.com> writes:

> kenner@vlsi1.ultra.nyu.edu (Richard Kenner) writes:
>
>> What is causing this?
>>
>>                 === libstdc++-v3 Summary ===
>>  
>> # of expected passes            1255
>> # of unexpected failures        76
>> # of expected failures          2
>> make[4]: *** [check-DEJAGNU] Error 1
>
> dejagnu exits unsuccessfully if there are unexpected failures.  Use
> make -k check.

But libstdc++-v3 is the only directory where then make fails.  If you
run the other testsuites, the error is always ignored and libstdc++-v3
forces you to use make -k check.

Andreas
-- 
 Andreas Jaeger
  SuSE Labs aj@suse.de
   private aj@arthur.inka.de
    http://www.suse.de/~aj

Re: Problems with "make check"

[Zack Weinberg]

Andreas Jaeger <aj@suse.de> writes:

> Zack Weinberg <zack@codesourcery.com> writes:
>
>> kenner@vlsi1.ultra.nyu.edu (Richard Kenner) writes:
>>
>>> What is causing this?
>>>
>>>                 === libstdc++-v3 Summary ===
>>>  
>>> # of expected passes            1255
>>> # of unexpected failures        76
>>> # of expected failures          2
>>> make[4]: *** [check-DEJAGNU] Error 1
>>
>> dejagnu exits unsuccessfully if there are unexpected failures.  Use
>> make -k check.
>
> But libstdc++-v3 is the only directory where then make fails.  If you
> run the other testsuites, the error is always ignored and libstdc++-v3
> forces you to use make -k check.

Ah, yes, I see that gcc/Makefile.in for instance uses -$(RUNTEST) so
errors will be ignored.  It would probably be good to make this
consistent.

zw

Re: Problems with "make check"

[Andreas Jaeger]

On Wed, Apr 02, 2003 at 10:04:55PM -0800, Zack Weinberg wrote:
> Andreas Jaeger <aj@suse.de> writes:
> 
> > Zack Weinberg <zack@codesourcery.com> writes:
> >
> >> kenner@vlsi1.ultra.nyu.edu (Richard Kenner) writes:
> >>
> >>> What is causing this?
> >>>
> >>>                 === libstdc++-v3 Summary ===
> >>>  
> >>> # of expected passes            1255
> >>> # of unexpected failures        76
> >>> # of expected failures          2
> >>> make[4]: *** [check-DEJAGNU] Error 1
> >>
> >> dejagnu exits unsuccessfully if there are unexpected failures.  Use
> >> make -k check.
> >
> > But libstdc++-v3 is the only directory where then make fails.  If you
> > run the other testsuites, the error is always ignored and libstdc++-v3
> > forces you to use make -k check.
> 
> Ah, yes, I see that gcc/Makefile.in for instance uses -$(RUNTEST) so
> errors will be ignored.  It would probably be good to make this
> consistent.

Yes, it would be.  The problem that I see is that libstdc++-v3 uses
automake to generate the calls for runtest.  I do not know how to
teach automake to ignore the error (otherwise I would have send a patch
already ;-).

Andreas
-- 
 Andreas Jaeger
  SuSE Labs aj@suse.de
   private aj@arthur.inka.de
    http://www.suse.de/~aj

Re: Problems with "make check"

[Phil Edwards]

On Thu, Apr 03, 2003 at 08:10:59AM +0200, Andreas Jaeger wrote:
> On Wed, Apr 02, 2003 at 10:04:55PM -0800, Zack Weinberg wrote:
> > Andreas Jaeger <aj@suse.de> writes:
> > > But libstdc++-v3 is the only directory where then make fails.  If you
> > > run the other testsuites, the error is always ignored and libstdc++-v3
> > > forces you to use make -k check.
> > 
> > Ah, yes, I see that gcc/Makefile.in for instance uses -$(RUNTEST) so
> > errors will be ignored.  It would probably be good to make this
> > consistent.
> 
> Yes, it would be.  The problem that I see is that libstdc++-v3 uses
> automake to generate the calls for runtest.  I do not know how to
> teach automake to ignore the error (otherwise I would have send a patch
> already ;-).

You're not the first.  :-)  It would require a one-line change to automake's
$share/am/dejagnu.am, and I'm not sure what else.

-- 
If ye love wealth greater than liberty, the tranquility of servitude greater
than the animating contest for freedom, go home and leave us in peace.  We seek
not your counsel, nor your arms.  Crouch down and lick the hand that feeds you;
and may posterity forget that ye were our countrymen.            - Samuel Adams

Re: Problems with "make check"

[Andreas Schwab]

Phil Edwards <phil@jaj.com> writes:

|> On Thu, Apr 03, 2003 at 08:10:59AM +0200, Andreas Jaeger wrote:
|> > On Wed, Apr 02, 2003 at 10:04:55PM -0800, Zack Weinberg wrote:
|> > > Andreas Jaeger <aj@suse.de> writes:
|> > > > But libstdc++-v3 is the only directory where then make fails.  If you
|> > > > run the other testsuites, the error is always ignored and libstdc++-v3
|> > > > forces you to use make -k check.
|> > > 
|> > > Ah, yes, I see that gcc/Makefile.in for instance uses -$(RUNTEST) so
|> > > errors will be ignored.  It would probably be good to make this
|> > > consistent.
|> > 
|> > Yes, it would be.  The problem that I see is that libstdc++-v3 uses
|> > automake to generate the calls for runtest.  I do not know how to
|> > teach automake to ignore the error (otherwise I would have send a patch
|> > already ;-).
|> 
|> You're not the first.  :-)  It would require a one-line change to automake's
|> $share/am/dejagnu.am, and I'm not sure what else.

But then, what's so bad about make -k check?

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux AG, Deutschherrnstr. 15-19, D-90429 Nürnberg
Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Problems with "make check"

[Gerald Pfeifer]

On Thu, 3 Apr 2003, Andreas Schwab wrote:
> But then, what's so bad about make -k check?

One can forget the -k? ;-)

Seriously, I believe I recall several reports/discussions on that here,
and while it is definitely a user error, if we can somehow avoid it, why
not do that?

Gerald
-- 
Gerald "Jerry"   pfeifer@dbai.tuwien.ac.at   http://www.pfeifer.com/gerald/

Re: Problems with "make check"

[Tom Tromey]

>>>>> "Phil" == Phil Edwards <phil@jaj.com> writes:

Phil> You're not the first.  :-) It would require a one-line change to
Phil> automake's $share/am/dejagnu.am, and I'm not sure what else.

That should be it.  However, that's automake 1.4.  I doubt the current
maintainer would want to do another 1.4.x release.  libgcj already has
a mini-fork though; you could put it there.

The code in there is a little weird in another way:

	runtest=$(RUNTEST); \
## If runtest can't be found, print a warning but don't die.  It is
## pointless to cause a failure if the tests cannot be run at all.
	if $(SHELL) -c "$$runtest --version" > /dev/null 2>&1; then \
	  l='$(DEJATOOL)'; for tool in $$l; do \
	    $$runtest $(AM_RUNTESTFLAGS) $(RUNTESTDEFAULTFLAGS) $(RUNTESTFLAGS); \
	  done; \
	else echo "WARNING: could not find \`runtest'" 1>&2; :;\

This seems backwards now.  I mean, wouldn't you want a failure if you
can't run the tests?  So that you know they didn't get run?

Some of this older code just copied existing practice, though,
independent of whether said practice made sense.

Tom

Re: Problems with "make check"

[Mike Stump]

On Thursday, April 3, 2003, at 12:38 AM, Gerald Pfeifer wrote:
> On Thu, 3 Apr 2003, Andreas Schwab wrote:
>> But then, what's so bad about make -k check?
>
> One can forget the -k? ;-)
>
> Seriously, I believe I recall several reports/discussions on that here,
> and while it is definitely a user error, if we can somehow avoid it, 
> why
> not do that?

Let me state the obvious.  If the testsuite doesn't have any unexpected 
failures, it will return success.  libstdc++ used to not fail.  I think 
we should move to the point where unexpected errors on primary 
platforms won't be allowed to exist (48 rule imposed by a dispationate 
cronjob).

For example, uninstalled locales need to be detected, and tests that 
fail because of them, should be fixed to ignore that situation, or 
report untested.

Re: Problems with "make check"

[Phil Edwards]

On Thu, Apr 03, 2003 at 09:25:54AM -0700, Tom Tromey wrote:
> >>>>> "Phil" == Phil Edwards <phil@jaj.com> writes:
> 
> Phil> You're not the first.  :-) It would require a one-line change to
> Phil> automake's $share/am/dejagnu.am, and I'm not sure what else.
> 
> That should be it.  However, that's automake 1.4.  I doubt the current
> maintainer would want to do another 1.4.x release.  libgcj already has
> a mini-fork though; you could put it there.

I was looking at /usr/share/automake-1.7/am/dejagnu.am.

> This seems backwards now.  I mean, wouldn't you want a failure if you
> can't run the tests?  So that you know they didn't get run?

*shrug*  Some of the dejangu philosophy escapes me.  I don't know how much
of it is imposed by the POSIX test-framework specification, and how much
is just dejagnu.


Phil

-- 
If ye love wealth greater than liberty, the tranquility of servitude greater
than the animating contest for freedom, go home and leave us in peace.  We seek
not your counsel, nor your arms.  Crouch down and lick the hand that feeds you;
and may posterity forget that ye were our countrymen.            - Samuel Adams

Re: Problems with "make check"

[Andreas Jaeger]

kenner@vlsi1.ultra.nyu.edu (Richard Kenner) writes:

>     But libstdc++-v3 is the only directory where then make fails.  
>
> Can you explain the reason for this incompatibility?

Usage of automake,

Andreas
-- 
 Andreas Jaeger
  SuSE Labs aj@suse.de
   private aj@arthur.inka.de
    http://www.suse.de/~aj

Re: Problems with "make check"

[Richard Kenner]

    But libstdc++-v3 is the only directory where then make fails.  

Can you explain the reason for this incompatibility?

Re: Problems with "make check"

[Richard Kenner]

    dejagnu exits unsuccessfully if there are unexpected failures.  

That's what I thought, except that there were also unexpected failures
in previous languages and it didn't exit unsuccessfully.

    76 looks suspiciously like the number of failures that happen if you
    don't have enough locales installed, see the list I posted last month.

Could easily be.  I'm viewing the failures I get on an unpatched tree
as a baseline anyway unless they look very excessive, and this doesn't.