patch: use "make -k check" broadly

patch: use "make -k check" broadly

[Frank Ch. Eigler]

Hi -

The following diff gets the annobin and a few other package
make_check* steps to use "make -k check" rather than "make check", so
that early partial failures do not doom the entire run.  Many other
package builders already use the equivalent.

diff --git a/builder/master.cfg b/builder/master.cfg
index e4cd0802a051..d0b046f7af19 100644
--- a/builder/master.cfg
+++ b/builder/master.cfg
@@ -1194,26 +1194,26 @@ make_pdf_step = steps.Compile(
         command=["make", "pdf"],
         name="make pdf")
 make_check_step = steps.Test(
-        command=addOutputSync.withArgs(["make", "check",
+        command=addOutputSync.withArgs(["make", "-k", "check",
             util.Interpolate('-j%(prop:ncpus)s')]),
         name="make check", haltOnFailure=False, flunkOnFailure=True)
 # make check explicitly not using -j
 make_single_check_step = steps.Test(
-        command=["make", "check"],
+        command=["make", "-k", "check"],
         name="make check")
 make_single_test_suite_step = steps.Test(
-        command=["make", "check"],
+        command=["make", "-k" "check"],
         name="make check",
         logfiles={"test-suite.log": "tests/test-suite.log"})
 # Same as make_check_step but with tests/test-suite.log recorded
 make_check_test_suite_step = steps.Test(
-        command=addOutputSync.withArgs(["make", "check",
+        command=addOutputSync.withArgs(["make", "-k", "check",
             util.Interpolate('-j%(prop:ncpus)s')]),
         name="make check", haltOnFailure=False, flunkOnFailure=True,
         logfiles={"test-suite.log": "tests/test-suite.log"})
 # Same but with tests/testsuite.log (note, no dash)
 make_check_testsuite_step = steps.Test(
-        command=addOutputSync.withArgs(["make", "check",
+        command=addOutputSync.withArgs(["make", "-k", "check",
             util.Interpolate('-j%(prop:ncpus)s')]),
         name="make check", haltOnFailure=False, flunkOnFailure=True,
         logfiles={"testsuite.log": "tests/testsuite.log"})


- FChE

Re: patch: use "make -k check" broadly

[Mark Wielaard]

On Wed, Mar 06, 2024 at 12:14:31PM -0500, Frank Ch. Eigler wrote:
> The following diff gets the annobin and a few other package
> make_check* steps to use "make -k check" rather than "make check", so
> that early partial failures do not doom the entire run.  Many other
> package builders already use the equivalent.

Yes that makes sense. I do hope most testsuites are normally zero
fail, so any failure would indicate an issue. But we would like to get
as much test results as possible in the CI builders. And make -k check
still makes the command fail if any make target fails. It just doesn't
bail out early.

Thanks,

Mark

> diff --git a/builder/master.cfg b/builder/master.cfg
> index e4cd0802a051..d0b046f7af19 100644
> --- a/builder/master.cfg
> +++ b/builder/master.cfg
> @@ -1194,26 +1194,26 @@ make_pdf_step = steps.Compile(
>          command=["make", "pdf"],
>          name="make pdf")
>  make_check_step = steps.Test(
> -        command=addOutputSync.withArgs(["make", "check",
> +        command=addOutputSync.withArgs(["make", "-k", "check",
>              util.Interpolate('-j%(prop:ncpus)s')]),
>          name="make check", haltOnFailure=False, flunkOnFailure=True)
>  # make check explicitly not using -j
>  make_single_check_step = steps.Test(
> -        command=["make", "check"],
> +        command=["make", "-k", "check"],
>          name="make check")
>  make_single_test_suite_step = steps.Test(
> -        command=["make", "check"],
> +        command=["make", "-k" "check"],
>          name="make check",
>          logfiles={"test-suite.log": "tests/test-suite.log"})
>  # Same as make_check_step but with tests/test-suite.log recorded
>  make_check_test_suite_step = steps.Test(
> -        command=addOutputSync.withArgs(["make", "check",
> +        command=addOutputSync.withArgs(["make", "-k", "check",
>              util.Interpolate('-j%(prop:ncpus)s')]),
>          name="make check", haltOnFailure=False, flunkOnFailure=True,
>          logfiles={"test-suite.log": "tests/test-suite.log"})
>  # Same but with tests/testsuite.log (note, no dash)
>  make_check_testsuite_step = steps.Test(
> -        command=addOutputSync.withArgs(["make", "check",
> +        command=addOutputSync.withArgs(["make", "-k", "check",
>              util.Interpolate('-j%(prop:ncpus)s')]),
>          name="make check", haltOnFailure=False, flunkOnFailure=True,
>          logfiles={"testsuite.log": "tests/testsuite.log"})
> 
> 
> - FChE