Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Takashi Yano <takashi.yano@nifty.ne.jp>]

[-- Attachment #1: Type: text/plain, Size: 4991 bytes --]

On Sun, 5 Sep 2021 08:15:23 +0900
Takashi Yano wrote:
> Hi Ken,
> 
> On Sat, 4 Sep 2021 10:04:12 -0400
> Ken Brown wrote:
> > On 9/4/2021 8:37 AM, Takashi Yano wrote:
> > > On Sat, 4 Sep 2021 21:02:58 +0900
> > > Takashi Yano wrote:
> > >> Hi Corinna, Ken,
> > >>
> > >> On Fri, 3 Sep 2021 09:27:37 -0400
> > >> Ken Brown wrote:
> > >>> On 9/3/2021 8:22 AM, Takashi Yano wrote:
> > >>>> POSIX says:
> > >>>>       The value returned may be less than nbyte if the number of bytes left
> > >>>>       in the file is less than nbyte, if the read() request was interrupted
> > >>>>       by a signal, or if the file is a pipe or FIFO or special file and has
> > >>>>                                                                         ~~~
> > >>>>       fewer than nbyte bytes immediately available for reading.
> > >>>>       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > >>>>
> > >>>> https://pubs.opengroup.org/onlinepubs/009604599/functions/read.html
> > >>>>
> > >>>> If it is turned over, read() should read all data immediately available,
> > >>>> I think.
> > >>>
> > >>> I understand the reasoning now, but I think your patch isn't quite right.  As it
> > >>> stands, if the call to NtQueryInformationFile fails but total_length != 0,
> > >>> you're trying to read again without knowing that there's data in the pipe.
> > >>>
> > >>> Also, I think you need the following:
> > >>>
> > >>> diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
> > >>> index ef7823ae5..46bb96961 100644
> > >>> --- a/winsup/cygwin/fhandler_pipe.cc
> > >>> +++ b/winsup/cygwin/fhandler_pipe.cc
> > >>> @@ -348,8 +348,13 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
> > >>>        CloseHandle (evt);
> > >>>      if (status == STATUS_THREAD_SIGNALED)
> > >>>        {
> > >>> -      set_errno (EINTR);
> > >>> -      len = (size_t) -1;
> > >>> +      if (total_len == 0)
> > >>> +       {
> > >>> +         set_errno (EINTR);
> > >>> +         len = (size_t) -1;
> > >>> +       }
> > >>> +      else
> > >>> +       len = total_len;
> > >>>        }
> > >>>      else if (status == STATUS_THREAD_CANCELED)
> > >>>        pthread::static_cancel_self ();
> > >>
> > >> Thanks for your advice. I fixed the issue and attached new patch.
> > >>
> > >> On Fri, 3 Sep 2021 17:37:13 +0200
> > >> Corinna Vinschen wrote:
> > >>> Hmm, I see the point, but we might have another problem with that.
> > >>>
> > >>> We can't keep the mutex while waiting on the pending read, and there
> > >>> could be more than one pending read running at the time.  if so,
> > >>> chances are extremly high, that the data written to the buffer gets
> > >>> split like this:
> > >>>
> > >>>     reader 1		               reader 2
> > >>>
> > >>>     calls read(65536)                   calls read(65536)
> > >>>
> > >>>     calls NtReadFile(16384 bytes)
> > >>>                                         calls NtReadFile(16384 bytes)
> > >>>
> > >>> writer writes 65536 bytes
> > >>>
> > >>>     wakes up and gets 16384 bytes
> > >>>                                         wakes up and gets 16384 bytes
> > >>>     gets the mutex, calls
> > >>>     NtReadFile(32768) which
> > >>>     returns immediately with
> > >>>     32768 bytes added to the
> > >>>     caller's buffer.
> > >>>
> > >>> so the buffer returned to reader 1 is 49152 bytes, with 16384 bytes
> > >>> missing in the middle of it, *without* the reader knowing about that
> > >>> fact.  If reader 1 gets the first 16384 bytes, the 16384 bytes have
> > >>> been read in a single call, at least, so the byte order is not
> > >>> unknowingly broken on the application level.
> > >>>
> > >>> Does that make sense?
> > >>
> > >> Why can't we keep the mutex while waiting on the pending read?
> > >> If we can keep the mutex, the issue above mentioned does not
> > >> happen, right?
> > >>
> > >> What about the patch attached? This keeps the mutex while read()
> > >> but I do not see any defects so far.
> > 
> > LGTM.
> > 
> > If Corinna agrees, I have a couple of suggestions.
> > 
> > 1. With this patch, we can no longer have more than one pending ReadFile.  So 
> > there's no longer a need to count read handles, and the problem with select is 
> > completely fixed as long as the number of bytes requested is less than the pipe 
> > buffer size.
> > 
> > 2. raw_read is now reading in chunks, like raw_write.  For readability of the 
> > code, I think it would be better to make the two functions as similar as 
> > possible.  For example, you could replace the do/while loop by a 
> > while(total_len<orig_len) loop.  And you could even use similar names for the 
> > variables, e.g., nbytes instead of total_len, or vice versa.
> 
> Thanks for the suggestion. I have rebuilt the patch.
> Please see the patch attached.

This patch seems to fail to adopt to current git head of topic/pipe
branch. I rebuilt the patch to fit current top/pipe.

Please see the patch attached.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

[-- Attachment #2: 0001-Cygwin-pipe-Stop-counting-reader-and-read-all-availa.patch --]
[-- Type: application/octet-stream, Size: 5024 bytes --]

From 8bd38b3748adb1aa0037fb0683d20d680ce553c4 Mon Sep 17 00:00:00 2001
From: Takashi Yano <takashi.yano@nifty.ne.jp>
Date: Sun, 5 Sep 2021 22:35:28 +0900
Subject: [PATCH] Cygwin: pipe: Stop counting reader and read all available
 data.

- By guarding read with read_mtx, no more than one ReadFile can
  be called simultaneously. So couting read handles is no longer
  necessary.
- Make raw_read code as similar as possible to raw_write code.
---
 winsup/cygwin/fhandler_pipe.cc | 82 +++++++++++++++++++---------------
 1 file changed, 45 insertions(+), 37 deletions(-)

diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index 68974eb80..c094515b3 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -221,12 +221,10 @@ fhandler_pipe::get_proc_fd_name (char *buf)
 void __reg3
 fhandler_pipe::raw_read (void *ptr, size_t& len)
 {
-  NTSTATUS status;
+  size_t nbytes = 0;
+  NTSTATUS status = STATUS_SUCCESS;
   IO_STATUS_BLOCK io;
   HANDLE evt = NULL;
-  DWORD waitret = WAIT_OBJECT_0;
-  bool keep_looping = false;
-  size_t orig_len = len;
 
   if (!len)
     return;
@@ -239,43 +237,47 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
       return;
     }
 
-  do
+  DWORD timeout = is_nonblocking () ? 0 : INFINITE;
+  if (WAIT_TIMEOUT == WaitForSingleObject (read_mtx, timeout))
     {
-      len = orig_len;
-      keep_looping = false;
+      set_errno (EAGAIN);
+      len = (size_t) -1;
+      return;
+    }
+  while (nbytes < len)
+    {
+      ULONG_PTR nbytes_now = 0;
+      size_t left = len - nbytes;
+      ULONG len1 = (ULONG) left;
+      DWORD waitret = WAIT_OBJECT_0;
+
       if (evt)
 	ResetEvent (evt);
       if (!is_nonblocking ())
 	{
 	  FILE_PIPE_LOCAL_INFORMATION fpli;
-	  ULONG reader_count;
-	  ULONG max_len = 64;
-
-	  WaitForSingleObject (read_mtx, INFINITE);
 
 	  /* If the pipe is empty, don't request more bytes than half the
-	     pipe buffer size.  Every pending read lowers WriteQuotaAvailable
+	     pipe buffer size. pending read lowers WriteQuotaAvailable
 	     on the write side and thus affects select's ability to return
-	     more or less reliable info whether a write succeeds or not.
-
-	     Let the size of the request depend on the number of readers
-	     at the time. */
+	     more or less reliable info whether a write succeeds or not. */
+	  ULONG chunk = max_atomic_write / 2;
 	  status = NtQueryInformationFile (get_handle (), &io,
 					   &fpli, sizeof (fpli),
 					   FilePipeLocalInformation);
 	  if (NT_SUCCESS (status) && fpli.ReadDataAvailable == 0)
 	    {
-	      reader_count = get_obj_handle_count (get_handle ());
-	      if (reader_count < 10)
-		max_len = fpli.InboundQuota / (2 * reader_count);
-	      if (len > max_len)
-		len = max_len;
+	      if (nbytes != 0)
+		break;
+	      chunk = fpli.InboundQuota / 2;
 	    }
+	  if (!NT_SUCCESS (status) && nbytes != 0)
+	    break;
+
+	  len1 = (left < chunk) ? (ULONG) left : chunk;
 	}
       status = NtReadFile (get_handle (), evt, NULL, NULL, &io, ptr,
-			   len, NULL, NULL);
-      if (!is_nonblocking ())
-	ReleaseMutex (read_mtx);
+			   len1, NULL, NULL);
       if (evt && status == STATUS_PENDING)
 	{
 	  waitret = cygwait (evt);
@@ -292,13 +294,13 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
 	    set_errno (EBADF);
 	  else
 	    __seterrno ();
-	  len = (size_t) -1;
+	  nbytes = (size_t) -1;
 	}
       else if (NT_SUCCESS (status))
 	{
-	  len = io.Information;
-	  if (len == 0)
-	    keep_looping = true;
+	  nbytes_now = io.Information;
+	  ptr = ((char *) ptr) + nbytes_now;
+	  nbytes += nbytes_now;
 	}
       else
 	{
@@ -308,40 +310,46 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
 	    case STATUS_END_OF_FILE:
 	    case STATUS_PIPE_BROKEN:
 	      /* This is really EOF.  */
-	      len = 0;
 	      break;
 	    case STATUS_MORE_ENTRIES:
 	    case STATUS_BUFFER_OVERFLOW:
 	      /* `io.Information' is supposedly valid.  */
-	      len = io.Information;
-	      if (len == 0)
-		keep_looping = true;
+	      nbytes_now = io.Information;
+	      ptr = ((char *) ptr) + nbytes_now;
+	      nbytes += nbytes_now;
 	      break;
 	    case STATUS_PIPE_LISTENING:
 	    case STATUS_PIPE_EMPTY:
+	      if (nbytes != 0)
+		break;
 	      if (is_nonblocking ())
 		{
 		  set_errno (EAGAIN);
-		  len = (size_t) -1;
+		  nbytes = (size_t) -1;
 		  break;
 		}
 	      fallthrough;
 	    default:
 	      __seterrno_from_nt_status (status);
-	      len = (size_t) -1;
+	      nbytes = (size_t) -1;
 	      break;
 	    }
 	}
-    } while (keep_looping);
+
+      if (nbytes_now == 0)
+	break;
+    }
+  ReleaseMutex (read_mtx);
   if (evt)
     CloseHandle (evt);
-  if (status == STATUS_THREAD_SIGNALED)
+  if (status == STATUS_THREAD_SIGNALED && nbytes == 0)
     {
       set_errno (EINTR);
-      len = (size_t) -1;
+      nbytes = (size_t) -1;
     }
   else if (status == STATUS_THREAD_CANCELED)
     pthread::static_cancel_self ();
+  len = nbytes;
 }
 
 ssize_t __reg3
-- 
2.33.0