[PATCH 0/4] Testsuite update

[PATCH 0/4] Testsuite update

[Jon Turney]

This gets us down to no permanent failures in the testsuite in CI.

When run locally, msgtest, semtest and shmtest fail because they need a running cygserver using the test DLL,
and I haven't got a good idea about how to automate that.  devdsp fails due to a hang in child while exiting.

Jon Turney (4):
  Cygwin: testsuite: Add '-notimeout' option to cygrun
  Cygwin: testsuite: Update README
  Cygwin: testsuite: Fix cygload test
  Cygwin: CI: XFAIL umask03

 winsup/testsuite/Makefile.am |  6 ++-
 winsup/testsuite/README      | 82 +++++++++++++++++++++++++-----------
 winsup/testsuite/cygrun.c    | 26 ++++++++++--
 winsup/testsuite/cygrun.sh   |  2 +-
 4 files changed, 86 insertions(+), 30 deletions(-)

-- 
2.39.0

[PATCH 1/4] Cygwin: testsuite: Add '-notimeout' option to cygrun

[Jon Turney]

Add '-notimeout' option for cygrun.  This is very useful when using it
to run a test standalone and under a debugger.

Also: warn about excess arguments
---
 winsup/testsuite/cygrun.c | 26 ++++++++++++++++++++++----
 1 file changed, 22 insertions(+), 4 deletions(-)

diff --git a/winsup/testsuite/cygrun.c b/winsup/testsuite/cygrun.c
index d8de7d158..450201342 100644
--- a/winsup/testsuite/cygrun.c
+++ b/winsup/testsuite/cygrun.c
@@ -23,24 +23,42 @@ main (int argc, char **argv)
   DWORD res;
   DWORD ec = 1;
   char *p;
+  DWORD timeout = 60 * 1000;
 
   if (argc < 2)
     {
       fprintf (stderr, "Usage: cygrun [program]\n");
-      exit (0);
+      exit (1);
+    }
+
+  int i;
+  for (i = 1; i < argc; ++i)
+    {
+      if (strcmp (argv[i], "-notimeout") == 0)
+	timeout = INFINITE;
+      else
+	break;
+    }
+
+  char *command = argv[i];
+
+  if (i < (argc-1))
+    {
+      fprintf (stderr, "cygrun: excess arguments\n");
+      exit (1);
     }
 
   SetEnvironmentVariable ("CYGWIN_TESTING", "1");
 
   memset (&sa, 0, sizeof (sa));
   memset (&pi, 0, sizeof (pi));
-  if (!CreateProcess (0, argv[1], 0, 0, 1, 0, 0, 0, &sa, &pi))
+  if (!CreateProcess (0, command, 0, 0, 1, 0, 0, 0, &sa, &pi))
     {
-      fprintf (stderr, "CreateProcess %s failed\n", argv[1]);
+      fprintf (stderr, "CreateProcess %s failed\n", command);
       exit (1);
     }
 
-  res = WaitForSingleObject (pi.hProcess, 60 * 1000);
+  res = WaitForSingleObject (pi.hProcess, timeout);
 
   if (res == WAIT_TIMEOUT)
     {
-- 
2.39.0

[PATCH 2/4] Cygwin: testsuite: Update README

[Jon Turney]

v2:
Polish instructions on adding a test

Signed-off-by: Jon Turney <jon.turney@dronecode.org.uk>
---
 winsup/testsuite/README | 82 +++++++++++++++++++++++++++++------------
 1 file changed, 58 insertions(+), 24 deletions(-)

diff --git a/winsup/testsuite/README b/winsup/testsuite/README
index ff2df4119..511133e4d 100644
--- a/winsup/testsuite/README
+++ b/winsup/testsuite/README
@@ -1,35 +1,69 @@
-Here are some notes about adding and using this testsuite.
+Here are some notes about adding to and using this testsuite.
 
-The testsuite adds a directory containing the just built cygwin1.dll to the PATH
-(during the run step) so that it can be found by the Windows loader during
-testing.
+The testsuite adds a directory containing the just-built cygwin1.dll to the PATH
+so that it can be found by the Windows loader during testing.
 
-Because we'll probably run into complaints about using two DLLs, we
-run cygrun.exe for each test.  All this does is run the test with
-CreateProcess() so that we don't attempt to do the special code for
-when a cygwin program calls another cygwin program, as this might be a
-"multiple cygwins" problem.
+Because we'll probably run into complaints about using two DLLs, we use
+cygrun.exe to run each test.  All this does is run the test with CreateProcess()
+so that we don't attempt to do the special code for when a cygwin program calls
+another cygwin program, as this might be a "multiple cygwins" problem.
 
-Any test that needs to test command line args or redirection needs to
-run such a child program itself, as the testsuite will not do any
-arguments or redirection for it.  Same for fork, signals, etc.
+The testsuite/winsup.api subdirectory is for testing the API to cygwin1.dll
+ONLY.  Create other subdirs under testsuite/ for other classes of testing.
 
-The testsuite/winsup.api subdirectory is for testing the API to
-cygwin1.dll ONLY.  Create other subdirs under testsuite/ for other
-classes of testing.
+Tests in testsuite/winsup.api/pthread/ are derived from the pthread-win32
+testsuite.
 
-Tests under winsup.api/ either run successfully and exit(0), exit(77) to
-indicate a skipped test, or any other exit status to indicate a failure.
+Tests in testsuite/winsup.api/ltp/ are derived from (a very old version of) the
+ltp testsuite.
 
 Don't print anything to the screen if you can avoid it (except for failure
-reasons, of course).  One .c file per test, no compile options are allowed
-(we're testing the api, not the compiler).
+reasons, of course).
 
-Tests whose filename is mentioned in XFAIL_TESTS are expected to fail,
-effectively reversing the result of those.
+"make check" will only work if you run it *on* an NT machine.  Cross-checking is
+not supported.
 
-"make check" will only work if you run it *on* an NT machine.
-Cross-checking is not supported.
+Tests whose name is mentioned in XFAIL_TESTS are expected to fail, effectively
+reversing the result of those.
+
+Adding a test
+=============
+
+Add the source for the test under testsuite/winsup.api/.
+
+Add the additional tests program to check_PROGRAMS in testsuite/Makefile.am.
+
+(Note that if the test 'foo' has a single source file, foo.c, there's no need to
+write a foo_SOURCES as that's the default Automake assumes.)
+
+Tests can use libltp, but's that not required.
+
+The Cygwin 'installation' that the tests are run in is minimal, so don't assume
+anything is present.
+
+Any test that needs to test command line args or redirection needs to run such a
+child program itself, as the testsuite will not do any arguments or redirection
+for it.  Same for fork, signals, etc.
+
+Tests should either run successfully and exit(0), exit(77) to indicate a skipped
+test, or any other exit status to indicate a failure.
+
+Tips
+====
+
+* To run selected tests, use e.g:
 
-To run selected tests, use e.g:
 $ make check TESTS="winsup.api/ltp/umask03 winsup.api/ltp/stat06"
+
+* To build the tests without running them, use the check_programs target, e.g:
+
+$ make -C winsup/testsuite/ check_programs
+
+* To run an individual test program directly (against the installed, rather than
+  just built DLL), e.g.:
+
+$ winsup/testsuite/winsup.api/pthread/cancel1
+
+* To run an individual test program against the test DLL under gdb, e.g.:
+
+$ PATH="<build_tooldir>/winsup/testsuite/testinst/bin/:$PATH" cygrun -notimeout "gdb winsup.api/systemcall"
-- 
2.39.0

[PATCH 3/4] Cygwin: testsuite: Fix cygload test

[Jon Turney]

Use cygrun to isolate the cygload test from the current DLL, which
allows it to pass.
---
 winsup/testsuite/cygrun.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/winsup/testsuite/cygrun.sh b/winsup/testsuite/cygrun.sh
index c82c1872b..bf1d5cc6b 100755
--- a/winsup/testsuite/cygrun.sh
+++ b/winsup/testsuite/cygrun.sh
@@ -11,7 +11,7 @@ export PATH="$runtime_root:${PATH}"
 if [ "$1" = "./mingw/cygload" ]
 then
     windows_runtime_root=$(cygpath -m $runtime_root)
-    $exe -v -cygwin $windows_runtime_root/cygwin1.dll
+    $cygrun "$exe -v -cygwin $windows_runtime_root/cygwin1.dll"
 else
     cygdrop $cygrun $exe
 fi
-- 
2.39.0

[PATCH 4/4] Cygwin: CI: XFAIL umask03

[Jon Turney]

umask03 fails in CI due to some not understood weirdness.
---
 winsup/testsuite/Makefile.am | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/winsup/testsuite/Makefile.am b/winsup/testsuite/Makefile.am
index 8f2967a6d..228955668 100644
--- a/winsup/testsuite/Makefile.am
+++ b/winsup/testsuite/Makefile.am
@@ -333,12 +333,16 @@ TESTS = $(check_PROGRAMS) \
 	mingw/cygload
 
 # expected fail tests
+XFAIL_TESTS_CI_true = \
+	winsup.api/ltp/umask03$(EXEEXT)
+
 XFAIL_TESTS = \
 	winsup.api/ltp/setgroups01 \
 	winsup.api/ltp/setuid02 \
 	winsup.api/ltp/ulimit01 \
 	winsup.api/ltp/unlink08 \
-	winsup.api/samples/sample-fail
+	winsup.api/samples/sample-fail \
+	$(XFAIL_TESTS_CI_$(GITHUB_ACTIONS))
 
 # cygrun.sh test-runner script, and variables used by it:
 LOG_COMPILER = $(srcdir)/cygrun.sh
-- 
2.39.0

Re: [PATCH 0/4] Testsuite update

[Corinna Vinschen]

On Aug  4 13:47, Jon Turney wrote:
> This gets us down to no permanent failures in the testsuite in CI.
> 
> When run locally, msgtest, semtest and shmtest fail because they need a running cygserver using the test DLL,
> and I haven't got a good idea about how to automate that.  devdsp fails due to a hang in child while exiting.
> 
> Jon Turney (4):
>   Cygwin: testsuite: Add '-notimeout' option to cygrun
>   Cygwin: testsuite: Update README
>   Cygwin: testsuite: Fix cygload test
>   Cygwin: CI: XFAIL umask03
> 
>  winsup/testsuite/Makefile.am |  6 ++-
>  winsup/testsuite/README      | 82 +++++++++++++++++++++++++-----------
>  winsup/testsuite/cygrun.c    | 26 ++++++++++--
>  winsup/testsuite/cygrun.sh   |  2 +-
>  4 files changed, 86 insertions(+), 30 deletions(-)
> 
> -- 
> 2.39.0

:+1:


Corinna