[Bug build/30157] parallel "make check" randomly skips tests

["carlos at redhat dot com" <sourceware-bugzilla@sourceware.org>]

https://sourceware.org/bugzilla/show_bug.cgi?id=30157

--- Comment #11 from Carlos O'Donell <carlos at redhat dot com> ---
(In reply to Vincent Lefèvre from comment #9)
> (In reply to Carlos O'Donell from comment #8)
> > I agree it doesn't answer your question, but it does set the expectation
> > that anything you find that is not driven by dependencies is a mistake
> > (barring some complexity around tests-container).
> 
> Tests are there to find bugs, and bugs may depend on the run-time
> environment in a broad sense (this may be environment variables, but also
> the current time, the scheduling, in case of race conditions). So it does
> not make much sense to say that if a test gave some result in some run, it
> will always give the same result just because the source files have not
> changed.

Your are correct that the test results may vary depending on the system
configuration, but what you're asking for is outside the current assumptions of
the implementation. Describing the expectations highlights that the system is
behaving *as expected*, but what you are asking for is an enhancement to the
current build and test system.

The build and test system *expects* a correctly configured developer system and
optimizes for that to deduce developer iteration time to look at regression
test results.

If your system is incorrectly configured you may need to clean the whole build
and start again or `find . -name *.out | xargs rm` or something similar.

Could we do better? I expect we could.

> > Have you seen tests that were incorrectly executed based on their
> > dependencies?
> 
> Yes, for instance, with 3 "make check" without a change of the source files
> (thus needing a build of everything only after the first one[*]), but
> LD_PRELOAD was unset after the first one, elf/tst-audit23
> * was executed after the first one (obviously) and failed due to the
> LD_PRELOAD;
> * was not executed after the second one;
> * was executed after the third one (and no longer failed as LD_PRELOAD was
> no longer set).
> 
> [*] unless there is a bug in the dependencies for the build.

(In reply to Vincent Lefèvre from comment #10)
> I've done another "make check". So now there are tests whose last execution
> is
> * 2023-02-24 (today)
> * 2023-02-23
> * 2023-02-22 around 17:43
> * 2023-02-22 around 17:12 (corresponding to the first "make check")
> 
> (No source files have been changed after the initial checkout.)

Just to be clear, while you say "No source files have been changed", what is
actually important is *dependencies*.

In the case of tst-audit23 we have the following:

2283 $(objpfx)tst-audit23.out: $(objpfx)tst-auditmod23.so \
2284                           $(objpfx)tst-audit23mod.so

The tests need their output file (the *.out file) and this says that output
needs the two audit modules compiled, so they get built before the test is
built and run.

I would be curious to know if the *.out file was changed, or the *.so files
were changed, since they are are of the set of dependencies for the test.

-- 
You are receiving this mail because:
You are on the CC list for the bug.