public inbox for cygwin-developers@cygwin.com
 help / color / mirror / Atom feed
* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
       [not found]                             ` <6e9bb35e-6f4f-cf78-e515-549da487b5ef@cornell.edu>
@ 2021-08-30  7:57                               ` Corinna Vinschen
  0 siblings, 0 replies; 250+ messages in thread
From: Corinna Vinschen @ 2021-08-30  7:57 UTC (permalink / raw)
  To: cygwin-developers

[Moving discussion to cygwin-developers where it belongs]

On Aug 29 18:42, Ken Brown via Cygwin wrote:
> On 8/29/2021 4:41 AM, Takashi Yano wrote:
> > Hi Ken,
> > 
> > On Sat, 28 Aug 2021 16:55:52 -0400
> > Ken Brown wrote:
> > > On 8/28/2021 11:43 AM, Takashi Yano via Cygwin wrote:
> > > > On Sat, 28 Aug 2021 13:58:08 +0200
> > > > Corinna Vinschen wrote:
> > > > > On Aug 28 18:41, Takashi Yano via Cygwin wrote:
> > > > > > On Sat, 28 Aug 2021 10:43:27 +0200
> > > > > > Corinna Vinschen wrote:
> > > > > > > [...]
> > > > > > If 'non-blocking' means overlapped I/O, only the problem will be:
> > > > > > https://cygwin.com/pipermail/cygwin/2021-March/247987.html
> > > > > 
> > > > > Sorry if that wasn't clear, but I was not talking about overlapped I/O,
> > > > > which we should get rid off, but of real non-blocking mode, which
> > > > > Windows pipes are fortunately capable of.
> > > > 
> > > > Do you mean, PIPE_NOWAIT flag? If this flags is specified in
> > > > the read pipe, non-cygwin apps cannot read the pipe correctly.
> > > 
> > > While waiting for Corinna's response to this, I have one more question.  Do you
> > > understand why nt_create() failed and you had to revert to create()?  Was it an
> > > access problem because nt_create requested FILE_WRITE_ATTRIBUTES?  Or did I make
> > > some careless mistake in writing nt_create?
> > 
> > I am sorry but no. I don't understand why piping C# program via
> > the pipe created by nt_create() has the issue. I tried to change
> > setup parameters in nt_create(), however, I did not succeed it to
> > work. I also couldn't find any mistake in nt_create() so far.
> > 
> > Win32 programs which use ReadFile() and WriteFile() work even
> > with the pipe created by nt_create() as well as overlapped I/O.
> > 
> > What does C# program differ from legacy win32 program at all?
> 
> I don't know.
> 
> By the way, when I introduced nt_create(), my preference would have been to
> simply change create() to use the NT API, but I was afraid to do that
> because I didn't want to take a chance on breaking something.  That's still
> my preference, if we can find a way to work around this problem with C#
> programs.

Maybe Procmon from sysinternals helps to find the difference in the
working vs. the non-working calls...?


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
       [not found]                       ` <789f056a-f164-d71d-1dc9-230f5a41846d@cornell.edu>
@ 2021-08-30  8:27                         ` Corinna Vinschen
  2021-08-30 13:00                           ` Corinna Vinschen
       [not found]                         ` <20210830043756.8aa0ada77db0bfbbe3889f62@nifty.ne.jp>
  1 sibling, 1 reply; 250+ messages in thread
From: Corinna Vinschen @ 2021-08-30  8:27 UTC (permalink / raw)
  To: cygwin-developers

[Moving discussion to cygwin-developers]

On Aug 29 11:57, Ken Brown via Cygwin wrote:
> On 8/29/2021 5:07 AM, Takashi Yano via Cygwin wrote:
> > On Sat, 28 Aug 2021 18:41:02 +0900
> > Takashi Yano wrote:
> > > On Sat, 28 Aug 2021 10:43:27 +0200
> > > Corinna Vinschen wrote:
> > > > On Aug 28 02:21, Takashi Yano via Cygwin wrote:
> > > > > On Fri, 27 Aug 2021 12:00:50 -0400
> > > > > Ken Brown wrote:
> > > > > > Two years ago I thought I needed nt_create to avoid problems when calling
> > > > > > set_pipe_non_blocking.  Are you saying that's not an issue?  Is
> > > > > > set_pipe_non_blocking unnecessary?  Is that the point of your modification to
> > > > > > raw_read?
> > > > > 
> > > > > Yes. Instead of making windows read function itself non-blocking,
> > > > > it is possible to check if the pipe can be read before read using
> > > > > PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is
> > > > > returned.
> > > > 
> > > > The problem is this:
> > > > 
> > > >    if (PeekNamedPipe())
> > > >      ReadFile(blocking);
> > > > 
> > > > is not atomic.  I. e., if PeekNamedPipe succeeds, nothing keeps another
> > > > thread from draining the pipe between the PeekNamedPipe and the ReadFile
> > > > call.  And as soon as ReadFile runs, it hangs indefinitely and we can't
> > > > stop it via a signal.
> > > 
> > > Hmm, you are right. Mutex guard seems to be necessary like pty code
> > > if we go this way.
> > 
> > I have found that set_pipe_non_blocking() succeeds for both read and
> > write pipes if the write pipe is created by CreateNamedPipe() and the
> > read pipe is created by CreateFile() contrary to the current create()
> > code. Therefore, not only nt_create() but also PeekNamedPipe() become
> > unnecessary.
> > 
> > Please see the revised patch attached.
> 
> That's a great idea.
> 
> I've applied your two patches to the topic/pipe branch.  I also rebased it
> and did a forced push in order to bring in Corinna's loader script fix.  So
> you'll have to do 'git fetch' and 'git rebase --hard origin/topic/pipe'.
> 
> Does this now fix all known problems with pipes?
> 
> Corinna, do you still see any benefit to switching to PIPE_NOWAIT?  AFAICT,
> it wouldn't decrease the code size at this point, so the only question is
> whether it might improve performance.

Pipes are already using PIPE_NOWAIT aka FILE_PIPE_COMPLETE_OPERATION
mode, see set_pipe_non_blocking.  The problem is that it's not used for
blocking pipes.  Rather, blocking pipes use overlapped IO.  Overlapped
IO is conceptually upside-down from the POSIX concept of non-blocking.
Also, the information returned in FilePipeLocalInformation is historically
borderline.  For kicks, see
https://cygwin.com/pipermail/cygwin-patches/2004q4/005002.html

So my suggestion is to try switching to non-blocking Windows pipes
entirely, even for blocking pipes on the user level.  It works nicely
for sockets.

> If you think it's worth trying, I'd be glad to code it up on a new branch,
> and we could compare the two.

We can do this in two version steps.  There's no pressure.

> Aside from that, I'm wondering how and when to merge the new pipe
> implementation to master.  It obviously needs much more widespread testing
> than it's gotten so far.  I'm a little nervous about it because I haven't
> thought about the details for two years, and no one other than me has tested
> it until a few days ago.

I could push out version 3.3.0, and afterwards you can just merge into
master.


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
       [not found]                           ` <47e5dd74-b940-f305-fd5a-c6c9d8f41305@cornell.edu>
@ 2021-08-30  8:48                             ` Corinna Vinschen
  0 siblings, 0 replies; 250+ messages in thread
From: Corinna Vinschen @ 2021-08-30  8:48 UTC (permalink / raw)
  To: cygwin-developers

[Moved to cygwin-developers]

On Aug 29 17:09, Ken Brown via Cygwin wrote:
> On 8/29/2021 3:37 PM, Takashi Yano wrote:
> > Only the small thing remaining is pipe mode. If the pipe mode
> > is changed to byte mode, the following issue will be solved.
> > https://cygwin.com/pipermail/cygwin/2021-March/247987.html
> > 
> > How about the simple patch attached?
> > 
> > The comment in pipe code says:
> >       Note that the write side of the pipe is opened as PIPE_TYPE_MESSAGE.
> >       This *seems* to more closely mimic Linux pipe behavior and is
> >       definitely required for pty handling since fhandler_pty_master
> >       writes to the pipe in chunks, terminated by newline when CANON mode
> >       is specified.
> > 
> > This mentions about pty behaiviour in canonical mode, however, the
> > pty pipe is created as message mode even with this patch. Are there
> > any other reasons that message mode is preferred for pipe?
> 
> No idea.  All I remember is that there was a lot of discussion around the
> time that it was decided to use PIPE_TYPE_MESSAGE by default.  Corinna
> probably remembers the reasons.

No, sorry, I don't remember the exact discussion.  But it seemed to fix
quite a few issues at the time.


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
       [not found]                               ` <20210830170204.fa91eaf110f310f13b67abc3@nifty.ne.jp>
@ 2021-08-30 10:20                                 ` Corinna Vinschen
  2021-08-30 10:38                                   ` Corinna Vinschen
  2021-08-30 12:04                                   ` Takashi Yano
  0 siblings, 2 replies; 250+ messages in thread
From: Corinna Vinschen @ 2021-08-30 10:20 UTC (permalink / raw)
  To: cygwin-developers

[Move discussion to cygwin-developers]

On Aug 30 17:02, Takashi Yano via Cygwin wrote:
> On Sun, 29 Aug 2021 22:15:29 -0400
> Ken Brown wrote:
> > On 8/29/2021 8:22 PM, Takashi Yano via Cygwin wrote:
> > > We have two easy options:
> > > 1) Configure the pipe with PIPE_ACCESS_DUPLEX.
> > > 2) Use nt_create() again and forget C# program issue.
> > 
> > I vote for 2), but let's see what Corinna thinks.
> 
> BTW. what's wrong if just:
> 
> static int
> nt_create (LPSECURITY_ATTRIBUTES sa_ptr, PHANDLE r, PHANDLE w,
>                 DWORD psize, int64_t *unique_id)
> {
>   if (r && w)
>     {
>       static volatile ULONG pipe_unique_id;
>       LONG id = InterlockedIncrement ((LONG *) &pipe_unique_id);
>       if (unique_id)
>         *unique_id = ((int64_t) id << 32 | GetCurrentProcessId ());
>       if (!CreatePipe (r, w, sa_ptr, psize))
>         {
>           *r = *w = NULL;
>           return GetLastError ();
>         }
>     }
>   return 0;
> }
> 
> ?
> 
> In my environment, I cannot find any defects.
> - No performance degradation.
> - set_pipe_non_blocking() works for both read and write pipes.
> - NtQueryInformationFile() in select() works for both r/w pipes.
> - Piping C# program works.
> 
> Is naming the pipe really necessary?

It's not, but CreatePipe is doing this anyway.

"Anonymous pipes are implemented using a named pipe with a unique name."
https://docs.microsoft.com/en-us/windows/win32/api/namedpipeapi/nf-namedpipeapi-createpipe

The reason CreateNamedPipe was used in the first place was that
FILE_READ_ATTRIBUTES isn't set by CreatePipe for the write side
of the pipe, however, it creates full duplex pipe:

https://cygwin.com/pipermail/cygwin-patches/2004q3/004912.html

Given the fact that CreatePipe is implemented in terms of
NtCreateNamedPipeFile anyway, why should the pipe created with
NtCreateNamedPipeFile fail where the pipe created with CreatePipe works?

The only reason can be some missing flag, I think.  Checking
fhandler_pipe.cc::nt_create and comparing that with the default flags
for files and other devices, it occurs to me that the SYNCHRONIZE stuff
is missing.  So, Takashi, what if you call NtCreateNamedPipeFile like
this in nt_create:

  status = NtCreateNamedPipeFile (r, access | SYNCHRONIZE, &attr, &io,
				  FILE_SHARE_READ | FILE_SHARE_WRITE,
				  FILE_CREATE, FILE_SYNCHRONOUS_IO_NONALERT,
				  pipe_type, FILE_PIPE_BYTE_STREAM_MODE,
				  0, 1, psize, psize, &timeout);

Does that fix the above problems, too?


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-08-30 10:20                                 ` Corinna Vinschen
@ 2021-08-30 10:38                                   ` Corinna Vinschen
  2021-08-30 12:04                                   ` Takashi Yano
  1 sibling, 0 replies; 250+ messages in thread
From: Corinna Vinschen @ 2021-08-30 10:38 UTC (permalink / raw)
  To: cygwin-developers

On Aug 30 12:20, Corinna Vinschen wrote:
> [Move discussion to cygwin-developers]
> 
> On Aug 30 17:02, Takashi Yano via Cygwin wrote:
> > On Sun, 29 Aug 2021 22:15:29 -0400
> > Ken Brown wrote:
> > > On 8/29/2021 8:22 PM, Takashi Yano via Cygwin wrote:
> > > > We have two easy options:
> > > > 1) Configure the pipe with PIPE_ACCESS_DUPLEX.
> > > > 2) Use nt_create() again and forget C# program issue.
> > > 
> > > I vote for 2), but let's see what Corinna thinks.
> > 
> > BTW. what's wrong if just:
> > [...]
> >       if (!CreatePipe (r, w, sa_ptr, psize))
> > [...]
> > In my environment, I cannot find any defects.
> > - No performance degradation.
> > - set_pipe_non_blocking() works for both read and write pipes.
> > - NtQueryInformationFile() in select() works for both r/w pipes.
> > - Piping C# program works.
> > 
> > Is naming the pipe really necessary?
> 
> It's not, but CreatePipe is doing this anyway.
> [...]
> Given the fact that CreatePipe is implemented in terms of
> NtCreateNamedPipeFile anyway, why should the pipe created with
> NtCreateNamedPipeFile fail where the pipe created with CreatePipe works?
> 
> The only reason can be some missing flag, I think.  Checking
> fhandler_pipe.cc::nt_create and comparing that with the default flags
> for files and other devices, it occurs to me that the SYNCHRONIZE stuff
> is missing.  So, Takashi, what if you call NtCreateNamedPipeFile like
> this in nt_create:
> 
>   status = NtCreateNamedPipeFile (r, access | SYNCHRONIZE, &attr, &io,
> 				  FILE_SHARE_READ | FILE_SHARE_WRITE,
> 				  FILE_CREATE, FILE_SYNCHRONOUS_IO_NONALERT,
> 				  pipe_type, FILE_PIPE_BYTE_STREAM_MODE,
> 				  0, 1, psize, psize, &timeout);
> 
> Does that fix the above problems, too?

Btw., checking the calls with Procmon from sysinternals may give a clue,
too.  It was pretty helpful in the past.

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-08-30 10:20                                 ` Corinna Vinschen
  2021-08-30 10:38                                   ` Corinna Vinschen
@ 2021-08-30 12:04                                   ` Takashi Yano
  2021-08-30 12:55                                     ` Corinna Vinschen
  1 sibling, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-08-30 12:04 UTC (permalink / raw)
  To: cygwin-developers

On Mon, 30 Aug 2021 12:20:30 +0200
Corinna Vinschen wrote:
> [Move discussion to cygwin-developers]
> 
> On Aug 30 17:02, Takashi Yano via Cygwin wrote:
> > On Sun, 29 Aug 2021 22:15:29 -0400
> > Ken Brown wrote:
> > > On 8/29/2021 8:22 PM, Takashi Yano via Cygwin wrote:
> > > > We have two easy options:
> > > > 1) Configure the pipe with PIPE_ACCESS_DUPLEX.
> > > > 2) Use nt_create() again and forget C# program issue.
> > > 
> > > I vote for 2), but let's see what Corinna thinks.
> > 
> > BTW. what's wrong if just:
> > 
> > static int
> > nt_create (LPSECURITY_ATTRIBUTES sa_ptr, PHANDLE r, PHANDLE w,
> >                 DWORD psize, int64_t *unique_id)
> > {
> >   if (r && w)
> >     {
> >       static volatile ULONG pipe_unique_id;
> >       LONG id = InterlockedIncrement ((LONG *) &pipe_unique_id);
> >       if (unique_id)
> >         *unique_id = ((int64_t) id << 32 | GetCurrentProcessId ());
> >       if (!CreatePipe (r, w, sa_ptr, psize))
> >         {
> >           *r = *w = NULL;
> >           return GetLastError ();
> >         }
> >     }
> >   return 0;
> > }
> > 
> > ?
> > 
> > In my environment, I cannot find any defects.
> > - No performance degradation.
> > - set_pipe_non_blocking() works for both read and write pipes.
> > - NtQueryInformationFile() in select() works for both r/w pipes.
> > - Piping C# program works.
> > 
> > Is naming the pipe really necessary?
> 
> It's not, but CreatePipe is doing this anyway.
> 
> "Anonymous pipes are implemented using a named pipe with a unique name."
> https://docs.microsoft.com/en-us/windows/win32/api/namedpipeapi/nf-namedpipeapi-createpipe
> 
> The reason CreateNamedPipe was used in the first place was that
> FILE_READ_ATTRIBUTES isn't set by CreatePipe for the write side
> of the pipe, however, it creates full duplex pipe:
> 
> https://cygwin.com/pipermail/cygwin-patches/2004q3/004912.html
> 
> Given the fact that CreatePipe is implemented in terms of
> NtCreateNamedPipeFile anyway, why should the pipe created with
> NtCreateNamedPipeFile fail where the pipe created with CreatePipe works?
> 
> The only reason can be some missing flag, I think.  Checking
> fhandler_pipe.cc::nt_create and comparing that with the default flags
> for files and other devices, it occurs to me that the SYNCHRONIZE stuff
> is missing.  So, Takashi, what if you call NtCreateNamedPipeFile like
> this in nt_create:
> 
>   status = NtCreateNamedPipeFile (r, access | SYNCHRONIZE, &attr, &io,
> 				  FILE_SHARE_READ | FILE_SHARE_WRITE,
> 				  FILE_CREATE, FILE_SYNCHRONOUS_IO_NONALERT,
> 				  pipe_type, FILE_PIPE_BYTE_STREAM_MODE,
> 				  0, 1, psize, psize, &timeout);
> 
> Does that fix the above problems, too?

Yes it does! Now, if CYGWIN=pipe_byte is also set, the piping issue
of C# program is gone!

In fact, I've already tested adding the SYNCHRONIZE access flag,
but it didn't solve the problem. It seems that the cause was
that FILE_SYNCHRONOUS_IO_NONALERT was missing.

Thank you for figuring out the solution!

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-08-30 12:04                                   ` Takashi Yano
@ 2021-08-30 12:55                                     ` Corinna Vinschen
  2021-08-30 13:31                                       ` Corinna Vinschen
  2021-08-30 13:51                                       ` Ken Brown
  0 siblings, 2 replies; 250+ messages in thread
From: Corinna Vinschen @ 2021-08-30 12:55 UTC (permalink / raw)
  To: cygwin-developers

On Aug 30 21:04, Takashi Yano wrote:
> On Mon, 30 Aug 2021 12:20:30 +0200
> Corinna Vinschen wrote:
> > [Move discussion to cygwin-developers]
> > 
> > On Aug 30 17:02, Takashi Yano via Cygwin wrote:
> > > [...]
> > > Is naming the pipe really necessary?
> > 
> > It's not, but CreatePipe is doing this anyway.
> > 
> > "Anonymous pipes are implemented using a named pipe with a unique name."
> > https://docs.microsoft.com/en-us/windows/win32/api/namedpipeapi/nf-namedpipeapi-createpipe
> > 
> > The reason CreateNamedPipe was used in the first place was that
> > FILE_READ_ATTRIBUTES isn't set by CreatePipe for the write side
> > of the pipe, however, it creates full duplex pipe:
> > 
> > https://cygwin.com/pipermail/cygwin-patches/2004q3/004912.html
> > 
> > Given the fact that CreatePipe is implemented in terms of
> > NtCreateNamedPipeFile anyway, why should the pipe created with
> > NtCreateNamedPipeFile fail where the pipe created with CreatePipe works?
> > 
> > The only reason can be some missing flag, I think.  Checking
> > fhandler_pipe.cc::nt_create and comparing that with the default flags
> > for files and other devices, it occurs to me that the SYNCHRONIZE stuff
> > is missing.  So, Takashi, what if you call NtCreateNamedPipeFile like
> > this in nt_create:
> > 
> >   status = NtCreateNamedPipeFile (r, access | SYNCHRONIZE, &attr, &io,
> > 				  FILE_SHARE_READ | FILE_SHARE_WRITE,
> > 				  FILE_CREATE, FILE_SYNCHRONOUS_IO_NONALERT,
> > 				  pipe_type, FILE_PIPE_BYTE_STREAM_MODE,
> > 				  0, 1, psize, psize, &timeout);
> > 
> > Does that fix the above problems, too?
> 
> Yes it does! Now, if CYGWIN=pipe_byte is also set, the piping issue
> of C# program is gone!
> 
> In fact, I've already tested adding the SYNCHRONIZE access flag,
> but it didn't solve the problem. It seems that the cause was
> that FILE_SYNCHRONOUS_IO_NONALERT was missing.
> 
> Thank you for figuring out the solution!

No worries.  The same should apply to the NtCreateFile side of the
pipe, btw.


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-08-30  8:27                         ` Corinna Vinschen
@ 2021-08-30 13:00                           ` Corinna Vinschen
  2021-08-30 13:20                             ` Corinna Vinschen
  0 siblings, 1 reply; 250+ messages in thread
From: Corinna Vinschen @ 2021-08-30 13:00 UTC (permalink / raw)
  To: cygwin-developers

On Aug 30 10:27, Corinna Vinschen wrote:
> [Moving discussion to cygwin-developers]
> 
> On Aug 29 11:57, Ken Brown via Cygwin wrote:
> > Corinna, do you still see any benefit to switching to PIPE_NOWAIT?  AFAICT,
> > it wouldn't decrease the code size at this point, so the only question is
> > whether it might improve performance.
> 
> Pipes are already using PIPE_NOWAIT aka FILE_PIPE_COMPLETE_OPERATION
> mode, see set_pipe_non_blocking.  The problem is that it's not used for
> blocking pipes.  Rather, blocking pipes use overlapped IO.  Overlapped
> IO is conceptually upside-down from the POSIX concept of non-blocking.
> Also, the information returned in FilePipeLocalInformation is historically
> borderline.  For kicks, see
> https://cygwin.com/pipermail/cygwin-patches/2004q4/005002.html
> 
> So my suggestion is to try switching to non-blocking Windows pipes
> entirely, even for blocking pipes on the user level.  It works nicely
> for sockets.

On second thought, I'm not so sure how to block on non-blocking pipes
on writing.  Assuming a write fails because the buffer is full, we
don't have a waitable object to wait on.  Unless the pipe handle is
signalled if writing is allowed, but that would be a first in Windows.
So in theory this would still require overlapped IO.  Does that still
work as desired if the pipe mode is non-blocking?  I don't think I ever
tried that...


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-08-30 13:00                           ` Corinna Vinschen
@ 2021-08-30 13:20                             ` Corinna Vinschen
  2021-08-30 13:41                               ` Ken Brown
  0 siblings, 1 reply; 250+ messages in thread
From: Corinna Vinschen @ 2021-08-30 13:20 UTC (permalink / raw)
  To: cygwin-developers

On Aug 30 15:00, Corinna Vinschen wrote:
> On Aug 30 10:27, Corinna Vinschen wrote:
> > [Moving discussion to cygwin-developers]
> > 
> > On Aug 29 11:57, Ken Brown via Cygwin wrote:
> > > Corinna, do you still see any benefit to switching to PIPE_NOWAIT?  AFAICT,
> > > it wouldn't decrease the code size at this point, so the only question is
> > > whether it might improve performance.
> > 
> > Pipes are already using PIPE_NOWAIT aka FILE_PIPE_COMPLETE_OPERATION
> > mode, see set_pipe_non_blocking.  The problem is that it's not used for
> > blocking pipes.  Rather, blocking pipes use overlapped IO.  Overlapped
> > IO is conceptually upside-down from the POSIX concept of non-blocking.
> > Also, the information returned in FilePipeLocalInformation is historically
> > borderline.  For kicks, see
> > https://cygwin.com/pipermail/cygwin-patches/2004q4/005002.html
> > 
> > So my suggestion is to try switching to non-blocking Windows pipes
> > entirely, even for blocking pipes on the user level.  It works nicely
> > for sockets.
> 
> On second thought, I'm not so sure how to block on non-blocking pipes
> on writing.  Assuming a write fails because the buffer is full, we
> don't have a waitable object to wait on.  Unless the pipe handle is
> signalled if writing is allowed, but that would be a first in Windows.
> So in theory this would still require overlapped IO.  Does that still
> work as desired if the pipe mode is non-blocking?  I don't think I ever
> tried that...

That probably doesn't make sense.  If WriteFile returns without writing
something, what should overlapped io be waiting on?


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-08-30 12:55                                     ` Corinna Vinschen
@ 2021-08-30 13:31                                       ` Corinna Vinschen
  2021-08-31  8:50                                         ` Takashi Yano
  2021-08-30 13:51                                       ` Ken Brown
  1 sibling, 1 reply; 250+ messages in thread
From: Corinna Vinschen @ 2021-08-30 13:31 UTC (permalink / raw)
  To: cygwin-developers

Hi Takashi,

On Aug 30 14:55, Corinna Vinschen wrote:
> On Aug 30 21:04, Takashi Yano wrote:
> > On Mon, 30 Aug 2021 12:20:30 +0200
> > Corinna Vinschen wrote:
> > > [Move discussion to cygwin-developers]
> > > 
> > > On Aug 30 17:02, Takashi Yano via Cygwin wrote:
> > > > [...]
> > > > Is naming the pipe really necessary?
> > > 
> > > It's not, but CreatePipe is doing this anyway.
> > > 
> > > "Anonymous pipes are implemented using a named pipe with a unique name."
> > > https://docs.microsoft.com/en-us/windows/win32/api/namedpipeapi/nf-namedpipeapi-createpipe
> > > 
> > > The reason CreateNamedPipe was used in the first place was that
> > > FILE_READ_ATTRIBUTES isn't set by CreatePipe for the write side
> > > of the pipe, however, it creates full duplex pipe:
> > > 
> > > https://cygwin.com/pipermail/cygwin-patches/2004q3/004912.html
> > > 
> > > Given the fact that CreatePipe is implemented in terms of
> > > NtCreateNamedPipeFile anyway, why should the pipe created with
> > > NtCreateNamedPipeFile fail where the pipe created with CreatePipe works?
> > > 
> > > The only reason can be some missing flag, I think.  Checking
> > > fhandler_pipe.cc::nt_create and comparing that with the default flags
> > > for files and other devices, it occurs to me that the SYNCHRONIZE stuff
> > > is missing.  So, Takashi, what if you call NtCreateNamedPipeFile like
> > > this in nt_create:
> > > 
> > >   status = NtCreateNamedPipeFile (r, access | SYNCHRONIZE, &attr, &io,
> > > 				  FILE_SHARE_READ | FILE_SHARE_WRITE,
> > > 				  FILE_CREATE, FILE_SYNCHRONOUS_IO_NONALERT,
> > > 				  pipe_type, FILE_PIPE_BYTE_STREAM_MODE,
> > > 				  0, 1, psize, psize, &timeout);
> > > 
> > > Does that fix the above problems, too?
> > 
> > Yes it does! Now, if CYGWIN=pipe_byte is also set, the piping issue
> > of C# program is gone!

I don't quite understand this one.  Is that C# example using the write
side of the pipe?  If it reads from the pipe, this behaiour would be
pretty puzzeling, given the read mode is always BYTE.

Either way, assuming we switch the write side to BYTE mode only, is
the pty code robust enough to work with that?  The comment

  Note that the write side of the pipe is opened as PIPE_TYPE_MESSAGE.
  This *seems* to more closely mimic Linux pipe behavior and is
  definitely required for pty handling since fhandler_pty_master
  writes to the pipe in chunks, terminated by newline when CANON mode
  is specified. 

is old, so the problems the message mode was trying to solve for
CANON mode may not apply anymore...


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
       [not found]                             ` <d217ef03-7858-5e22-0aa6-f0507eedd9da@cornell.edu>
       [not found]                               ` <20210830170204.fa91eaf110f310f13b67abc3@nifty.ne.jp>
@ 2021-08-30 13:36                               ` Ken Brown
  2021-08-30 14:05                                 ` Corinna Vinschen
  1 sibling, 1 reply; 250+ messages in thread
From: Ken Brown @ 2021-08-30 13:36 UTC (permalink / raw)
  To: cygwin-devel

On 8/29/2021 10:15 PM, Ken Brown via Cygwin wrote:
> On 8/29/2021 8:22 PM, Takashi Yano via Cygwin wrote:
>> On Mon, 30 Aug 2021 09:13:14 +0900
>> Takashi Yano wrote:
>>> On Sun, 29 Aug 2021 17:04:56 -0400
>>> Ken Brown wrote:
>>>> On 8/29/2021 5:07 AM, Takashi Yano via Cygwin wrote:
>>>>> On Sat, 28 Aug 2021 18:41:02 +0900
>>>>> Takashi Yano wrote:
>>>>>> On Sat, 28 Aug 2021 10:43:27 +0200
>>>>>> Corinna Vinschen wrote:
>>>>>>> On Aug 28 02:21, Takashi Yano via Cygwin wrote:
>>>>>>>> On Fri, 27 Aug 2021 12:00:50 -0400
>>>>>>>> Ken Brown wrote:
>>>>>>>>> Two years ago I thought I needed nt_create to avoid problems when calling
>>>>>>>>> set_pipe_non_blocking.  Are you saying that's not an issue?  Is
>>>>>>>>> set_pipe_non_blocking unnecessary?  Is that the point of your 
>>>>>>>>> modification to
>>>>>>>>> raw_read?
>>>>>>>>
>>>>>>>> Yes. Instead of making windows read function itself non-blocking,
>>>>>>>> it is possible to check if the pipe can be read before read using
>>>>>>>> PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is
>>>>>>>> returned.
>>>>>>>
>>>>>>> The problem is this:
>>>>>>>
>>>>>>>     if (PeekNamedPipe())
>>>>>>>       ReadFile(blocking);
>>>>>>>
>>>>>>> is not atomic.  I. e., if PeekNamedPipe succeeds, nothing keeps another
>>>>>>> thread from draining the pipe between the PeekNamedPipe and the ReadFile
>>>>>>> call.  And as soon as ReadFile runs, it hangs indefinitely and we can't
>>>>>>> stop it via a signal.
>>>>>>
>>>>>> Hmm, you are right. Mutex guard seems to be necessary like pty code
>>>>>> if we go this way.
>>>>>
>>>>> I have found that set_pipe_non_blocking() succeeds for both read and
>>>>> write pipes if the write pipe is created by CreateNamedPipe() and the
>>>>> read pipe is created by CreateFile() contrary to the current create()
>>>>> code. Therefore, not only nt_create() but also PeekNamedPipe() become
>>>>> unnecessary.
>>>>>
>>>>> Please see the revised patch attached.
>>>>
>>>> I haven't had a chance to test this myself yet, but occurs to me that we might
>>>> have a different problem after this patch: Does the write handle that we get
>>>> from CreateNamedPipe() have FILE_READ_ATTRIBUTES access?
>>>
>>> I have just checked this, and the answer is "No". Due to this problem,
>>> NtQueryInformationFile() call in select() fails on the write pipe.
>>>
>>> It seems that we need more consideration...
>>
>> We have two easy options:
>> 1) Configure the pipe with PIPE_ACCESS_DUPLEX.
>> 2) Use nt_create() again and forget C# program issue.
> 
> I vote for 2), but let's see what Corinna thinks.
> 
>> Even without this problem, select() for writing pipe has a bug
>> and does not wrok as expected. The following patch seems to be
>> needed.
>>
>> diff --git a/winsup/cygwin/select.cc b/winsup/cygwin/select.cc
>> index 83e1c00e0..ac2fd227e 100644
>> --- a/winsup/cygwin/select.cc
>> +++ b/winsup/cygwin/select.cc
>> @@ -612,7 +612,6 @@ pipe_data_available (int fd, fhandler_base *fh, HANDLE h, 
>> bool writing)
>>             that.  This means that a pipe could still block since you could
>>             be trying to write more to the pipe than is available in the
>>             buffer but that is the hazard of select().  */
>> -      fpli.WriteQuotaAvailable = fpli.OutboundQuota - fpli.ReadDataAvailable;
>>         if (fpli.WriteQuotaAvailable > 0)
>>          {
>>            paranoid_printf ("fd %d, %s, write: size %u, avail %u", fd,
>>
> 
> I agree.

Now I'm starting to wonder.  The use of fpli.OutboundQuota - 
fpli.ReadDataAvailable instead of fpli.WriteQuotaAvailable was introduced in 
commit a010e6abe, with no comment in the commit message or the code to explain 
why.  Corinna, does this make sense to you?  Is it related to the issues raised 
in the message

  https://cygwin.com/pipermail/cygwin-patches/2004q4/005002.html

that you cited elsewhere in this thread?

BTW, when I was working on the pipe approach to AF_UNIX sockets (topic/af_unix 
branch), I had occasion to step through select.cc:pipe_data_available in gdb, 
and the use of fpli.OutboundQuota - fpli.ReadDataAvailable definitely seemed 
wrong to me.  So when I wrote peek_socket_unix on that branch, I used 
fpli.WriteQuotaAvailable, as Takashi is suggesting now.

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-08-30 13:20                             ` Corinna Vinschen
@ 2021-08-30 13:41                               ` Ken Brown
  2021-08-30 14:12                                 ` Corinna Vinschen
  0 siblings, 1 reply; 250+ messages in thread
From: Ken Brown @ 2021-08-30 13:41 UTC (permalink / raw)
  To: cygwin-developers

On 8/30/2021 9:20 AM, Corinna Vinschen wrote:
> On Aug 30 15:00, Corinna Vinschen wrote:
>> On Aug 30 10:27, Corinna Vinschen wrote:
>>> [Moving discussion to cygwin-developers]
>>>
>>> On Aug 29 11:57, Ken Brown via Cygwin wrote:
>>>> Corinna, do you still see any benefit to switching to PIPE_NOWAIT?  AFAICT,
>>>> it wouldn't decrease the code size at this point, so the only question is
>>>> whether it might improve performance.
>>>
>>> Pipes are already using PIPE_NOWAIT aka FILE_PIPE_COMPLETE_OPERATION
>>> mode, see set_pipe_non_blocking.  The problem is that it's not used for
>>> blocking pipes.  Rather, blocking pipes use overlapped IO.  Overlapped
>>> IO is conceptually upside-down from the POSIX concept of non-blocking.
>>> Also, the information returned in FilePipeLocalInformation is historically
>>> borderline.  For kicks, see
>>> https://cygwin.com/pipermail/cygwin-patches/2004q4/005002.html
>>>
>>> So my suggestion is to try switching to non-blocking Windows pipes
>>> entirely, even for blocking pipes on the user level.  It works nicely
>>> for sockets.
>>
>> On second thought, I'm not so sure how to block on non-blocking pipes
>> on writing.  Assuming a write fails because the buffer is full, we
>> don't have a waitable object to wait on.  Unless the pipe handle is
>> signalled if writing is allowed, but that would be a first in Windows.
>> So in theory this would still require overlapped IO.  Does that still
>> work as desired if the pipe mode is non-blocking?  I don't think I ever
>> tried that...
> 
> That probably doesn't make sense.  If WriteFile returns without writing
> something, what should overlapped io be waiting on?

The approach I've taken on the topic/pipe branch is to stop using overlapped I/O 
and to always keep the blocking mode of the Windows pipe in sync with the 
blocking mode of the fhandler.  This seems to work pretty well so far, although 
problems could certainly show up after further testing.

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-08-30 12:55                                     ` Corinna Vinschen
  2021-08-30 13:31                                       ` Corinna Vinschen
@ 2021-08-30 13:51                                       ` Ken Brown
  2021-08-30 15:00                                         ` Ken Brown
  1 sibling, 1 reply; 250+ messages in thread
From: Ken Brown @ 2021-08-30 13:51 UTC (permalink / raw)
  To: cygwin-developers

On 8/30/2021 8:55 AM, Corinna Vinschen wrote:
> On Aug 30 21:04, Takashi Yano wrote:
>> On Mon, 30 Aug 2021 12:20:30 +0200
>> Corinna Vinschen wrote:
>>> [Move discussion to cygwin-developers]
>>>
>>> On Aug 30 17:02, Takashi Yano via Cygwin wrote:
>>>> [...]
>>>> Is naming the pipe really necessary?
>>>
>>> It's not, but CreatePipe is doing this anyway.
>>>
>>> "Anonymous pipes are implemented using a named pipe with a unique name."
>>> https://docs.microsoft.com/en-us/windows/win32/api/namedpipeapi/nf-namedpipeapi-createpipe
>>>
>>> The reason CreateNamedPipe was used in the first place was that
>>> FILE_READ_ATTRIBUTES isn't set by CreatePipe for the write side
>>> of the pipe, however, it creates full duplex pipe:
>>>
>>> https://cygwin.com/pipermail/cygwin-patches/2004q3/004912.html
>>>
>>> Given the fact that CreatePipe is implemented in terms of
>>> NtCreateNamedPipeFile anyway, why should the pipe created with
>>> NtCreateNamedPipeFile fail where the pipe created with CreatePipe works?
>>>
>>> The only reason can be some missing flag, I think.  Checking
>>> fhandler_pipe.cc::nt_create and comparing that with the default flags
>>> for files and other devices, it occurs to me that the SYNCHRONIZE stuff
>>> is missing.  So, Takashi, what if you call NtCreateNamedPipeFile like
>>> this in nt_create:
>>>
>>>    status = NtCreateNamedPipeFile (r, access | SYNCHRONIZE, &attr, &io,
>>> 				  FILE_SHARE_READ | FILE_SHARE_WRITE,
>>> 				  FILE_CREATE, FILE_SYNCHRONOUS_IO_NONALERT,
>>> 				  pipe_type, FILE_PIPE_BYTE_STREAM_MODE,
>>> 				  0, 1, psize, psize, &timeout);
>>>
>>> Does that fix the above problems, too?
>>
>> Yes it does! Now, if CYGWIN=pipe_byte is also set, the piping issue
>> of C# program is gone!
>>
>> In fact, I've already tested adding the SYNCHRONIZE access flag,
>> but it didn't solve the problem. It seems that the cause was
>> that FILE_SYNCHRONOUS_IO_NONALERT was missing.
>>
>> Thank you for figuring out the solution!
> 
> No worries.  The same should apply to the NtCreateFile side of the
> pipe, btw.

I'll add my thanks.  I should have checked the default flags that are typically 
used for other devices when I wrote nt_create.  I'm glad you caught this.

So I'll reinstate the use of nt_create and then let Takashi recheck everything.

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-08-30 13:36                               ` Ken Brown
@ 2021-08-30 14:05                                 ` Corinna Vinschen
  2021-08-30 15:53                                   ` Corinna Vinschen
  0 siblings, 1 reply; 250+ messages in thread
From: Corinna Vinschen @ 2021-08-30 14:05 UTC (permalink / raw)
  To: cygwin-developers

On Aug 30 09:36, Ken Brown wrote:
> On 8/29/2021 10:15 PM, Ken Brown via Cygwin wrote:
> > On 8/29/2021 8:22 PM, Takashi Yano via Cygwin wrote:
> > > [...]
> > > Even without this problem, select() for writing pipe has a bug
> > > and does not wrok as expected. The following patch seems to be
> > > needed.
> > > 
> > > diff --git a/winsup/cygwin/select.cc b/winsup/cygwin/select.cc
> > > index 83e1c00e0..ac2fd227e 100644
> > > --- a/winsup/cygwin/select.cc
> > > +++ b/winsup/cygwin/select.cc
> > > @@ -612,7 +612,6 @@ pipe_data_available (int fd, fhandler_base *fh,
> > > HANDLE h, bool writing)
> > >             that.  This means that a pipe could still block since you could
> > >             be trying to write more to the pipe than is available in the
> > >             buffer but that is the hazard of select().  */
> > > -      fpli.WriteQuotaAvailable = fpli.OutboundQuota - fpli.ReadDataAvailable;
> > >         if (fpli.WriteQuotaAvailable > 0)
> > >          {
> > >            paranoid_printf ("fd %d, %s, write: size %u, avail %u", fd,
> > > 
> > 
> > I agree.
> 
> Now I'm starting to wonder.  The use of fpli.OutboundQuota -
> fpli.ReadDataAvailable instead of fpli.WriteQuotaAvailable was introduced in
> commit a010e6abe, with no comment in the commit message or the code to
> explain why.  Corinna, does this make sense to you?  Is it related to the
> issues raised in the message
> 
>  https://cygwin.com/pipermail/cygwin-patches/2004q4/005002.html
> 
> that you cited elsewhere in this thread?

I thought so, but no.  This patch was introduced by cgf in 2008,
and I don't see any hint as to the why in any of my mail archives,
not even in private email.  The only vague hint is the release message
later on:

  - Reworked pipe handling for better speed and better support for signal
    processing.

I wonder if that was a typo, considering the observation from the above
mail:

  "But there is a strange twist:  When a read is pending on an empty
   pipe, then WriteQuotaAvailable is also decremented!"

> BTW, when I was working on the pipe approach to AF_UNIX sockets
> (topic/af_unix branch), I had occasion to step through
> select.cc:pipe_data_available in gdb, and the use of fpli.OutboundQuota -
> fpli.ReadDataAvailable definitely seemed wrong to me.  So when I wrote
> peek_socket_unix on that branch, I used fpli.WriteQuotaAvailable, as Takashi
> is suggesting now.

If that's working reliable these days (keeping fingers crossed for W7),
it's ok if we use that.  We may want to check if the above observation
in terms on WriteQuotaAvailable on a pipe with pending read is still an
issue.


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-08-30 13:41                               ` Ken Brown
@ 2021-08-30 14:12                                 ` Corinna Vinschen
  2021-08-30 14:52                                   ` Ken Brown
  0 siblings, 1 reply; 250+ messages in thread
From: Corinna Vinschen @ 2021-08-30 14:12 UTC (permalink / raw)
  To: cygwin-developers

On Aug 30 09:41, Ken Brown wrote:
> On 8/30/2021 9:20 AM, Corinna Vinschen wrote:
> > On Aug 30 15:00, Corinna Vinschen wrote:
> > > On Aug 30 10:27, Corinna Vinschen wrote:
> > > > [Moving discussion to cygwin-developers]
> > > > 
> > > > On Aug 29 11:57, Ken Brown via Cygwin wrote:
> > > > > Corinna, do you still see any benefit to switching to PIPE_NOWAIT?  AFAICT,
> > > > > it wouldn't decrease the code size at this point, so the only question is
> > > > > whether it might improve performance.
> > > > 
> > > > Pipes are already using PIPE_NOWAIT aka FILE_PIPE_COMPLETE_OPERATION
> > > > mode, see set_pipe_non_blocking.  The problem is that it's not used for
> > > > blocking pipes.  Rather, blocking pipes use overlapped IO.  Overlapped
> > > > IO is conceptually upside-down from the POSIX concept of non-blocking.
> > > > Also, the information returned in FilePipeLocalInformation is historically
> > > > borderline.  For kicks, see
> > > > https://cygwin.com/pipermail/cygwin-patches/2004q4/005002.html
> > > > 
> > > > So my suggestion is to try switching to non-blocking Windows pipes
> > > > entirely, even for blocking pipes on the user level.  It works nicely
> > > > for sockets.
> > > 
> > > On second thought, I'm not so sure how to block on non-blocking pipes
> > > on writing.  Assuming a write fails because the buffer is full, we
> > > don't have a waitable object to wait on.  Unless the pipe handle is
> > > signalled if writing is allowed, but that would be a first in Windows.
> > > So in theory this would still require overlapped IO.  Does that still
> > > work as desired if the pipe mode is non-blocking?  I don't think I ever
> > > tried that...
> > 
> > That probably doesn't make sense.  If WriteFile returns without writing
> > something, what should overlapped io be waiting on?
> 
> The approach I've taken on the topic/pipe branch is to stop using overlapped
> I/O and to always keep the blocking mode of the Windows pipe in sync with
> the blocking mode of the fhandler.  This seems to work pretty well so far,
> although problems could certainly show up after further testing.

Erm... afaics, fhandler_pipe::raw_read and fhandler_pipe::raw_write are
still using overlapped IO on blocking sockets.  Otherwise, how'd you
handle signals?

Just to be clear, Windows pipes can be read/write in three modes:

- non-blocking	 (FILE_PIPE_COMPLETE_OPERATION)
- synchronous	 (FILE_PIPE_QUEUE_OPERATION, non-overlapped)
- asynchronous   (FILE_PIPE_QUEUE_OPERATION, overlapped)

Right now the pipe code uses non-blocking mode for non-blocking sockets
and asynchronous mode for blocking sockets.  This looks like the most
promising approach, afaics.


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-08-30 14:12                                 ` Corinna Vinschen
@ 2021-08-30 14:52                                   ` Ken Brown
  2021-08-30 15:15                                     ` Corinna Vinschen
  0 siblings, 1 reply; 250+ messages in thread
From: Ken Brown @ 2021-08-30 14:52 UTC (permalink / raw)
  To: cygwin-developers

On 8/30/2021 10:12 AM, Corinna Vinschen wrote:
> On Aug 30 09:41, Ken Brown wrote:
>> On 8/30/2021 9:20 AM, Corinna Vinschen wrote:
>>> On Aug 30 15:00, Corinna Vinschen wrote:
>>>> On Aug 30 10:27, Corinna Vinschen wrote:
>>>>> [Moving discussion to cygwin-developers]
>>>>>
>>>>> On Aug 29 11:57, Ken Brown via Cygwin wrote:
>>>>>> Corinna, do you still see any benefit to switching to PIPE_NOWAIT?  AFAICT,
>>>>>> it wouldn't decrease the code size at this point, so the only question is
>>>>>> whether it might improve performance.
>>>>>
>>>>> Pipes are already using PIPE_NOWAIT aka FILE_PIPE_COMPLETE_OPERATION
>>>>> mode, see set_pipe_non_blocking.  The problem is that it's not used for
>>>>> blocking pipes.  Rather, blocking pipes use overlapped IO.  Overlapped
>>>>> IO is conceptually upside-down from the POSIX concept of non-blocking.
>>>>> Also, the information returned in FilePipeLocalInformation is historically
>>>>> borderline.  For kicks, see
>>>>> https://cygwin.com/pipermail/cygwin-patches/2004q4/005002.html
>>>>>
>>>>> So my suggestion is to try switching to non-blocking Windows pipes
>>>>> entirely, even for blocking pipes on the user level.  It works nicely
>>>>> for sockets.
>>>>
>>>> On second thought, I'm not so sure how to block on non-blocking pipes
>>>> on writing.  Assuming a write fails because the buffer is full, we
>>>> don't have a waitable object to wait on.  Unless the pipe handle is
>>>> signalled if writing is allowed, but that would be a first in Windows.
>>>> So in theory this would still require overlapped IO.  Does that still
>>>> work as desired if the pipe mode is non-blocking?  I don't think I ever
>>>> tried that...
>>>
>>> That probably doesn't make sense.  If WriteFile returns without writing
>>> something, what should overlapped io be waiting on?
>>
>> The approach I've taken on the topic/pipe branch is to stop using overlapped
>> I/O and to always keep the blocking mode of the Windows pipe in sync with
>> the blocking mode of the fhandler.  This seems to work pretty well so far,
>> although problems could certainly show up after further testing.
> 
> Erm... afaics, fhandler_pipe::raw_read and fhandler_pipe::raw_write are
> still using overlapped IO on blocking sockets.  Otherwise, how'd you
> handle signals?
> 
> Just to be clear, Windows pipes can be read/write in three modes:
> 
> - non-blocking	 (FILE_PIPE_COMPLETE_OPERATION)
> - synchronous	 (FILE_PIPE_QUEUE_OPERATION, non-overlapped)
> - asynchronous   (FILE_PIPE_QUEUE_OPERATION, overlapped)

OK, I've been thoroughly confused about what "overlapped" means.  I thought it 
meant specifying FILE_FLAG_OVERLAPPED and a pointer to an OVERLAPPED structure, 
both of which (I thought) only made sense when using the Win32 API rather than 
the NT API.

I *think* I understand what you mean now.  By using an event in the calls to 
NtReadFile and NtWriteFile in the blocking case, I'm selecting asynchronous 
mode, right?

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-08-30 13:51                                       ` Ken Brown
@ 2021-08-30 15:00                                         ` Ken Brown
  2021-08-30 15:19                                           ` Corinna Vinschen
  0 siblings, 1 reply; 250+ messages in thread
From: Ken Brown @ 2021-08-30 15:00 UTC (permalink / raw)
  To: cygwin-developers

On 8/30/2021 9:51 AM, Ken Brown wrote:
> On 8/30/2021 8:55 AM, Corinna Vinschen wrote:
>> On Aug 30 21:04, Takashi Yano wrote:
>>> On Mon, 30 Aug 2021 12:20:30 +0200
>>> Corinna Vinschen wrote:
>>>> [Move discussion to cygwin-developers]
>>>>
>>>> On Aug 30 17:02, Takashi Yano via Cygwin wrote:
>>>>> [...]
>>>>> Is naming the pipe really necessary?
>>>>
>>>> It's not, but CreatePipe is doing this anyway.
>>>>
>>>> "Anonymous pipes are implemented using a named pipe with a unique name."
>>>> https://docs.microsoft.com/en-us/windows/win32/api/namedpipeapi/nf-namedpipeapi-createpipe 
>>>>
>>>>
>>>> The reason CreateNamedPipe was used in the first place was that
>>>> FILE_READ_ATTRIBUTES isn't set by CreatePipe for the write side
>>>> of the pipe, however, it creates full duplex pipe:
>>>>
>>>> https://cygwin.com/pipermail/cygwin-patches/2004q3/004912.html
>>>>
>>>> Given the fact that CreatePipe is implemented in terms of
>>>> NtCreateNamedPipeFile anyway, why should the pipe created with
>>>> NtCreateNamedPipeFile fail where the pipe created with CreatePipe works?
>>>>
>>>> The only reason can be some missing flag, I think.  Checking
>>>> fhandler_pipe.cc::nt_create and comparing that with the default flags
>>>> for files and other devices, it occurs to me that the SYNCHRONIZE stuff
>>>> is missing.  So, Takashi, what if you call NtCreateNamedPipeFile like
>>>> this in nt_create:
>>>>
>>>>    status = NtCreateNamedPipeFile (r, access | SYNCHRONIZE, &attr, &io,
>>>>                   FILE_SHARE_READ | FILE_SHARE_WRITE,
>>>>                   FILE_CREATE, FILE_SYNCHRONOUS_IO_NONALERT,
>>>>                   pipe_type, FILE_PIPE_BYTE_STREAM_MODE,
>>>>                   0, 1, psize, psize, &timeout);
>>>>
>>>> Does that fix the above problems, too?
>>>
>>> Yes it does! Now, if CYGWIN=pipe_byte is also set, the piping issue
>>> of C# program is gone!
>>>
>>> In fact, I've already tested adding the SYNCHRONIZE access flag,
>>> but it didn't solve the problem. It seems that the cause was
>>> that FILE_SYNCHRONOUS_IO_NONALERT was missing.
>>>
>>> Thank you for figuring out the solution!
>>
>> No worries.  The same should apply to the NtCreateFile side of the
>> pipe, btw.
> 
> I'll add my thanks.  I should have checked the default flags that are typically 
> used for other devices when I wrote nt_create.  I'm glad you caught this.
> 
> So I'll reinstate the use of nt_create and then let Takashi recheck everything.

I've done this now.  I'm still not sure I've got all the flags right.  For 
unknown reasons, I've used FILE_SHARE_READ | FILE_SHARE_WRITE in the call to 
NtCreateNamedPipeFile, and no sharing in the call to NtOpenFile.  Should I also 
use FILE_SHARE_READ | FILE_SHARE in NtOpenFile?  Is sharing even relevant in 
this context?

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-08-30 14:52                                   ` Ken Brown
@ 2021-08-30 15:15                                     ` Corinna Vinschen
  0 siblings, 0 replies; 250+ messages in thread
From: Corinna Vinschen @ 2021-08-30 15:15 UTC (permalink / raw)
  To: cygwin-developers

On Aug 30 10:52, Ken Brown wrote:
> On 8/30/2021 10:12 AM, Corinna Vinschen wrote:
> > On Aug 30 09:41, Ken Brown wrote:
> > > The approach I've taken on the topic/pipe branch is to stop using overlapped
> > > I/O and to always keep the blocking mode of the Windows pipe in sync with
> > > the blocking mode of the fhandler.  This seems to work pretty well so far,
> > > although problems could certainly show up after further testing.
> > 
> > Erm... afaics, fhandler_pipe::raw_read and fhandler_pipe::raw_write are
> > still using overlapped IO on blocking sockets.  Otherwise, how'd you
> > handle signals?
> > 
> > Just to be clear, Windows pipes can be read/write in three modes:
> > 
> > - non-blocking	 (FILE_PIPE_COMPLETE_OPERATION)
> > - synchronous	 (FILE_PIPE_QUEUE_OPERATION, non-overlapped)
> > - asynchronous   (FILE_PIPE_QUEUE_OPERATION, overlapped)
> 
> OK, I've been thoroughly confused about what "overlapped" means.  I thought
> it meant specifying FILE_FLAG_OVERLAPPED and a pointer to an OVERLAPPED
> structure, both of which (I thought) only made sense when using the Win32
> API rather than the NT API.
> 
> I *think* I understand what you mean now.  By using an event in the calls to
> NtReadFile and NtWriteFile in the blocking case, I'm selecting asynchronous
> mode, right?

Sorry, yes, OVERLAPPED is a Win32 expression only.  The NT calls only
differ between synchronous and asynchronous calls.  For asynchronous
calls you can either call a wait function on the file handle or you can
add an event object or a completion routine.

But that makes me wonder...  Looks like my idea to add the
FILE_SYNCHRONOUS_IO_NONALERT flag was a red herring.  This enforces
synchronous operation, which is not what we want.  Bummer.

However, if C# can't work with asynchronous handles, I wonder how to
fix this issue at all.


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-08-30 15:00                                         ` Ken Brown
@ 2021-08-30 15:19                                           ` Corinna Vinschen
  2021-08-30 15:43                                             ` Ken Brown
  2021-08-31  8:52                                             ` Takashi Yano
  0 siblings, 2 replies; 250+ messages in thread
From: Corinna Vinschen @ 2021-08-30 15:19 UTC (permalink / raw)
  To: cygwin-developers

On Aug 30 11:00, Ken Brown wrote:
> On 8/30/2021 9:51 AM, Ken Brown wrote:
> > On 8/30/2021 8:55 AM, Corinna Vinschen wrote:
> > > On Aug 30 21:04, Takashi Yano wrote:
> > > No worries.  The same should apply to the NtCreateFile side of the
> > > pipe, btw.
> > 
> > I'll add my thanks.  I should have checked the default flags that are
> > typically used for other devices when I wrote nt_create.  I'm glad you
> > caught this.
> > 
> > So I'll reinstate the use of nt_create and then let Takashi recheck everything.
> 
> I've done this now.  I'm still not sure I've got all the flags right.  For
> unknown reasons, I've used FILE_SHARE_READ | FILE_SHARE_WRITE in the call to
> NtCreateNamedPipeFile, and no sharing in the call to NtOpenFile.  Should I
> also use FILE_SHARE_READ | FILE_SHARE in NtOpenFile?  Is sharing even
> relevant in this context?

This is only relevant if you want to open the pipe from another context,
calling CreateNamedPipe/CreateFile.  As long as the pipe is only
duplicated, it shouldn't matter at all.

But, as I just wrote in my previous mail, the FILE_SYNCHRONOUS_IO_NONALERT
flag is probably a good thing for C# apps, but not for Cygwin, because it
enforces synchronous operation.  Sorry about that...


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-08-30 15:19                                           ` Corinna Vinschen
@ 2021-08-30 15:43                                             ` Ken Brown
  2021-08-31  9:43                                               ` Corinna Vinschen
  2021-08-31  8:52                                             ` Takashi Yano
  1 sibling, 1 reply; 250+ messages in thread
From: Ken Brown @ 2021-08-30 15:43 UTC (permalink / raw)
  To: cygwin-developers

On 8/30/2021 11:19 AM, Corinna Vinschen wrote:
> On Aug 30 11:00, Ken Brown wrote:
>> On 8/30/2021 9:51 AM, Ken Brown wrote:
>>> On 8/30/2021 8:55 AM, Corinna Vinschen wrote:
>>>> On Aug 30 21:04, Takashi Yano wrote:
>>>> No worries.  The same should apply to the NtCreateFile side of the
>>>> pipe, btw.
>>>
>>> I'll add my thanks.  I should have checked the default flags that are
>>> typically used for other devices when I wrote nt_create.  I'm glad you
>>> caught this.
>>>
>>> So I'll reinstate the use of nt_create and then let Takashi recheck everything.
>>
>> I've done this now.  I'm still not sure I've got all the flags right.  For
>> unknown reasons, I've used FILE_SHARE_READ | FILE_SHARE_WRITE in the call to
>> NtCreateNamedPipeFile, and no sharing in the call to NtOpenFile.  Should I
>> also use FILE_SHARE_READ | FILE_SHARE in NtOpenFile?  Is sharing even
>> relevant in this context?
> 
> This is only relevant if you want to open the pipe from another context,
> calling CreateNamedPipe/CreateFile.  As long as the pipe is only
> duplicated, it shouldn't matter at all.

OK, then I think I should remove the sharing from NtCreateNamedPipeFile, since 
it could confuse someone reading the code.

> But, as I just wrote in my previous mail, the FILE_SYNCHRONOUS_IO_NONALERT
> flag is probably a good thing for C# apps, but not for Cygwin, because it
> enforces synchronous operation.  Sorry about that...

No problem.  I'll remove that flag for now, and we may have to live with the C# 
problem unless someone can find a different fix for it.

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-08-30 14:05                                 ` Corinna Vinschen
@ 2021-08-30 15:53                                   ` Corinna Vinschen
  2021-08-30 17:00                                     ` Corinna Vinschen
  0 siblings, 1 reply; 250+ messages in thread
From: Corinna Vinschen @ 2021-08-30 15:53 UTC (permalink / raw)
  To: cygwin-developers

On Aug 30 16:05, Corinna Vinschen wrote:
> On Aug 30 09:36, Ken Brown wrote:
> > BTW, when I was working on the pipe approach to AF_UNIX sockets
> > (topic/af_unix branch), I had occasion to step through
> > select.cc:pipe_data_available in gdb, and the use of fpli.OutboundQuota -
> > fpli.ReadDataAvailable definitely seemed wrong to me.  So when I wrote
> > peek_socket_unix on that branch, I used fpli.WriteQuotaAvailable, as Takashi
> > is suggesting now.
> 
> If that's working reliable these days (keeping fingers crossed for W7),
> it's ok if we use that.  We may want to check if the above observation
> in terms on WriteQuotaAvailable on a pipe with pending read is still an
> issue.

Ok, I wrote a small testcase.  It creates a named pipe, reads from the
pipe, then, later, writes to the pipe.  Interlaced with these calls, it
calls NtQueryInformationFile(FilePipeLocalInformation) on the write side
of the pipe.  Kind of like this:

  CreatePipe
  NtQueryInformationFile
  ReadFile
  NtQueryInformationFile
  WriteFile
  NtQueryInformationFile

Here's the result:

Before ReadFile:

  InboundQuota: 65536
  ReadDataAvailable: 0
  OutboundQuota: 65536
  WriteQuotaAvailable: 65536

While ReadFile is running:

  InboundQuota: 65536
  ReadDataAvailable: 0
  OutboundQuota: 65536
  WriteQuotaAvailable: 65494	!!!

After WriteFile and ReadFile succeeded:

  InboundQuota: 65536
  ReadDataAvailable: 0
  OutboundQuota: 65536
  WriteQuotaAvailable: 65536

That means, while a reader on the reader side is waiting for data, the
WriteQuotaAvailable on the write side is decremented by the amount of
data requested by the reader (42 bytes in my case), just as outlined in that
mail from 2004.  And this is on W10 now.

What to do with this information?  TBD.

Side note:  My testcase is starting a second thread to call ReadFile.
For that reason I was using synchronous IO on the pipe since, well,
never mind if that thread is blocked in ReadFile, right?  Nothing keeps
us from calling NtQueryInformationFile on the write side of the pipe,
right?

Wrong.  While the second thread was blocked in ReadFile, the call to
NtQueryInformationFile was blocking, too :-P

I had to convert the read side of the pipe to asynchronous mode to be
able to call NtQueryInformationFile(FilePipeLocalInformation) on the
write side of the pipe, while the read side is performing a ReadFile
operation.


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-08-30 15:53                                   ` Corinna Vinschen
@ 2021-08-30 17:00                                     ` Corinna Vinschen
  2021-08-30 17:11                                       ` Corinna Vinschen
                                                         ` (2 more replies)
  0 siblings, 3 replies; 250+ messages in thread
From: Corinna Vinschen @ 2021-08-30 17:00 UTC (permalink / raw)
  To: cygwin-developers

On Aug 30 17:53, Corinna Vinschen wrote:
> On Aug 30 16:05, Corinna Vinschen wrote:
> > On Aug 30 09:36, Ken Brown wrote:
> > > BTW, when I was working on the pipe approach to AF_UNIX sockets
> > > (topic/af_unix branch), I had occasion to step through
> > > select.cc:pipe_data_available in gdb, and the use of fpli.OutboundQuota -
> > > fpli.ReadDataAvailable definitely seemed wrong to me.  So when I wrote
> > > peek_socket_unix on that branch, I used fpli.WriteQuotaAvailable, as Takashi
> > > is suggesting now.
> > 
> > If that's working reliable these days (keeping fingers crossed for W7),
> > it's ok if we use that.  We may want to check if the above observation
> > in terms on WriteQuotaAvailable on a pipe with pending read is still an
> > issue.
> 
> Ok, I wrote a small testcase.  It creates a named pipe, reads from the
> pipe, then, later, writes to the pipe.  Interlaced with these calls, it
> calls NtQueryInformationFile(FilePipeLocalInformation) on the write side
> of the pipe.  Kind of like this:
> 
>   CreatePipe
>   NtQueryInformationFile
>   ReadFile
>   NtQueryInformationFile
>   WriteFile
>   NtQueryInformationFile
> 
> Here's the result:
> 
> Before ReadFile:
> 
>   InboundQuota: 65536
>   ReadDataAvailable: 0
>   OutboundQuota: 65536
>   WriteQuotaAvailable: 65536
> 
> While ReadFile is running:
> 
>   InboundQuota: 65536
>   ReadDataAvailable: 0
>   OutboundQuota: 65536
>   WriteQuotaAvailable: 65494	!!!
> 
> After WriteFile and ReadFile succeeded:
> 
>   InboundQuota: 65536
>   ReadDataAvailable: 0
>   OutboundQuota: 65536
>   WriteQuotaAvailable: 65536
> 
> That means, while a reader on the reader side is waiting for data, the
> WriteQuotaAvailable on the write side is decremented by the amount of
> data requested by the reader (42 bytes in my case), just as outlined in that
> mail from 2004.  And this is on W10 now.
> 
> What to do with this information?  TBD.

Ok, let's discuss this.  I added more code to my testcase and here's
what I see.  I dropped all data from the output which doesn't change.

What I'm trying to get a grip on are the dependencies here.

After creating the pipe:

  read side: ReadDataAvailable: 0
  write side: WriteQuotaAvailable: 65536

After writing 20 bytes...

  read side: ReadDataAvailable: 20
  write side: WriteQuotaAvailable: 65516

After writing 40 more bytes...

  read side: ReadDataAvailable: 60
  write side: WriteQuotaAvailable: 65476

After reading 42 bytes...

  read side: ReadDataAvailable: 18
  write side: WriteQuotaAvailable: 65518

After writing 20 bytes...

  read side: ReadDataAvailable: 38
  write side: WriteQuotaAvailable: 65498

*While* reading 42 bytes with an empty buffer...

  read side: ReadDataAvailable: 0
  write side: WriteQuotaAvailable: 65494

Another important fun fact:  Assuming the read and write buffer sizes
are differently specified.  I called CreateNamedPipe with an outbuffer
size of 32K and an inbuffer size of 64K:

After creating the pipe:

  read side:
    InboundQuota: 65536
    ReadDataAvailable: 0
    OutboundQuota: 32768
    WriteQuotaAvailable: 32768
  write side:
    InboundQuota: 65536
    ReadDataAvailable: 0
    OutboundQuota: 32768
    WriteQuotaAvailable: 65536	!!!

This last data point shows that:

- InboundQuota and OutboundQuota are always constant values and
  do not depend on the side the information has been queried on.
  That certainly makes sense.

- WriteQuotaAvailable does not depend on the OutboundQuota, but on
  the InboundQuota, and very likely on the InboundQuota of the read
  side.  The OutboundQuota *probably* only makes sense when using
  named pipes with remote clients, which we never do anyway.

The preceeding output shows that ReadDataAvailable on the read side and
WriteQuotaAvailable on the write side are connected.  If we write 20
bytes, ReadDataAvailable is incremented by 20 and WriteQuotaAvailable is
decremented by 20.

So: write.WriteQuotaAvailable == InboundQuota - read.ReadDataAvailable.

Except when a ReadFile is pending on the read side.  It's as if the
running ReadFile already reserved write quota.  So the write side
WriteQuotaAvailable is the number of bytes we can write without blocking,
after all pending ReadFiles have been satisfied.

Unfortunately that doesn't really make sense when looked at it from the
user space.

What that means in the first place is that WriteQuotaAvailable on the
write side is unreliable.  What we really need is InboundQuota -
read.ReadDataAvailable.  The problem with that is that the write side
usually has no access to the read side of the pipe.

Long story short, I have no idea how to fix that ATM.


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-08-30 17:00                                     ` Corinna Vinschen
@ 2021-08-30 17:11                                       ` Corinna Vinschen
  2021-08-30 18:59                                       ` Ken Brown
  2021-08-30 20:14                                       ` Corinna Vinschen
  2 siblings, 0 replies; 250+ messages in thread
From: Corinna Vinschen @ 2021-08-30 17:11 UTC (permalink / raw)
  To: cygwin-developers

On Aug 30 19:00, Corinna Vinschen wrote:
> - WriteQuotaAvailable does not depend on the OutboundQuota, but on
>   the InboundQuota, and very likely on the InboundQuota of the read
>   side.  The OutboundQuota *probably* only makes sense when using
>   named pipes with remote clients, which we never do anyway.

D'oh.  It makes sense for duplex pipes, of course.  I always forget them.


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-08-30 17:00                                     ` Corinna Vinschen
  2021-08-30 17:11                                       ` Corinna Vinschen
@ 2021-08-30 18:59                                       ` Ken Brown
  2021-08-30 19:12                                         ` Ken Brown
  2021-08-30 20:21                                         ` Corinna Vinschen
  2021-08-30 20:14                                       ` Corinna Vinschen
  2 siblings, 2 replies; 250+ messages in thread
From: Ken Brown @ 2021-08-30 18:59 UTC (permalink / raw)
  To: cygwin-developers

On 8/30/2021 1:00 PM, Corinna Vinschen wrote:
> On Aug 30 17:53, Corinna Vinschen wrote:
>> On Aug 30 16:05, Corinna Vinschen wrote:
>>> On Aug 30 09:36, Ken Brown wrote:
>>>> BTW, when I was working on the pipe approach to AF_UNIX sockets
>>>> (topic/af_unix branch), I had occasion to step through
>>>> select.cc:pipe_data_available in gdb, and the use of fpli.OutboundQuota -
>>>> fpli.ReadDataAvailable definitely seemed wrong to me.  So when I wrote
>>>> peek_socket_unix on that branch, I used fpli.WriteQuotaAvailable, as Takashi
>>>> is suggesting now.
>>>
>>> If that's working reliable these days (keeping fingers crossed for W7),
>>> it's ok if we use that.  We may want to check if the above observation
>>> in terms on WriteQuotaAvailable on a pipe with pending read is still an
>>> issue.
>>
>> Ok, I wrote a small testcase.  It creates a named pipe, reads from the
>> pipe, then, later, writes to the pipe.  Interlaced with these calls, it
>> calls NtQueryInformationFile(FilePipeLocalInformation) on the write side
>> of the pipe.  Kind of like this:
>>
>>    CreatePipe
>>    NtQueryInformationFile
>>    ReadFile
>>    NtQueryInformationFile
>>    WriteFile
>>    NtQueryInformationFile
>>
>> Here's the result:
>>
>> Before ReadFile:
>>
>>    InboundQuota: 65536
>>    ReadDataAvailable: 0
>>    OutboundQuota: 65536
>>    WriteQuotaAvailable: 65536
>>
>> While ReadFile is running:
>>
>>    InboundQuota: 65536
>>    ReadDataAvailable: 0
>>    OutboundQuota: 65536
>>    WriteQuotaAvailable: 65494	!!!
>>
>> After WriteFile and ReadFile succeeded:
>>
>>    InboundQuota: 65536
>>    ReadDataAvailable: 0
>>    OutboundQuota: 65536
>>    WriteQuotaAvailable: 65536
>>
>> That means, while a reader on the reader side is waiting for data, the
>> WriteQuotaAvailable on the write side is decremented by the amount of
>> data requested by the reader (42 bytes in my case), just as outlined in that
>> mail from 2004.  And this is on W10 now.
>>
>> What to do with this information?  TBD.
> 
> Ok, let's discuss this.  I added more code to my testcase and here's
> what I see.  I dropped all data from the output which doesn't change.
> 
> What I'm trying to get a grip on are the dependencies here.
> 
> After creating the pipe:
> 
>    read side: ReadDataAvailable: 0
>    write side: WriteQuotaAvailable: 65536
> 
> After writing 20 bytes...
> 
>    read side: ReadDataAvailable: 20
>    write side: WriteQuotaAvailable: 65516
> 
> After writing 40 more bytes...
> 
>    read side: ReadDataAvailable: 60
>    write side: WriteQuotaAvailable: 65476
> 
> After reading 42 bytes...
> 
>    read side: ReadDataAvailable: 18
>    write side: WriteQuotaAvailable: 65518
> 
> After writing 20 bytes...
> 
>    read side: ReadDataAvailable: 38
>    write side: WriteQuotaAvailable: 65498
> 
> *While* reading 42 bytes with an empty buffer...
> 
>    read side: ReadDataAvailable: 0
>    write side: WriteQuotaAvailable: 65494
> 
> Another important fun fact:  Assuming the read and write buffer sizes
> are differently specified.  I called CreateNamedPipe with an outbuffer
> size of 32K and an inbuffer size of 64K:
> 
> After creating the pipe:
> 
>    read side:
>      InboundQuota: 65536
>      ReadDataAvailable: 0
>      OutboundQuota: 32768
>      WriteQuotaAvailable: 32768
>    write side:
>      InboundQuota: 65536
>      ReadDataAvailable: 0
>      OutboundQuota: 32768
>      WriteQuotaAvailable: 65536	!!!
> 
> This last data point shows that:
> 
> - InboundQuota and OutboundQuota are always constant values and
>    do not depend on the side the information has been queried on.
>    That certainly makes sense.
> 
> - WriteQuotaAvailable does not depend on the OutboundQuota, but on
>    the InboundQuota, and very likely on the InboundQuota of the read
>    side.  The OutboundQuota *probably* only makes sense when using
>    named pipes with remote clients, which we never do anyway.
> 
> The preceeding output shows that ReadDataAvailable on the read side and
> WriteQuotaAvailable on the write side are connected.  If we write 20
> bytes, ReadDataAvailable is incremented by 20 and WriteQuotaAvailable is
> decremented by 20.
> 
> So: write.WriteQuotaAvailable == InboundQuota - read.ReadDataAvailable.
> 
> Except when a ReadFile is pending on the read side.  It's as if the
> running ReadFile already reserved write quota.  So the write side
> WriteQuotaAvailable is the number of bytes we can write without blocking,
> after all pending ReadFiles have been satisfied.
> 
> Unfortunately that doesn't really make sense when looked at it from the
> user space.
> 
> What that means in the first place is that WriteQuotaAvailable on the
> write side is unreliable.  What we really need is InboundQuota -
> read.ReadDataAvailable.  The problem with that is that the write side
> usually has no access to the read side of the pipe.

For the purposes of select.cc:pipe_data_available, we only need to know whether 
InboundQuota - read.ReadDataAvailable is positive.  If WriteQuotaAvailable is 
positive, then we're OK, even though its precise value might be too small.  But 
if WriteQuotaAvailable == 0, we don't know whether the buffer is actually full 
or there's a pending ReadFile.  It's only in this case that we need access to 
the read side.

What if we reverse the roles of the read and write sides of the pipe, so that 
the write side is the server and the read side is the client.  We can then try 
to use ImpersonateNamedPipeClient to get information about the read side when 
WriteQuotaAvailable == 0.  If we succeed, then we can determine whether or not 
there's space in the buffer.  If we fail, we simply report, possibly 
incorrectly, that there is space.  This is no worse than the current situation 
(on the master branch), in which we use OutboundQuota - ReadDataAvailable, which 
is always positive.

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-08-30 18:59                                       ` Ken Brown
@ 2021-08-30 19:12                                         ` Ken Brown
  2021-08-30 20:21                                         ` Corinna Vinschen
  1 sibling, 0 replies; 250+ messages in thread
From: Ken Brown @ 2021-08-30 19:12 UTC (permalink / raw)
  To: cygwin-developers

On 8/30/2021 2:59 PM, Ken Brown wrote:
> On 8/30/2021 1:00 PM, Corinna Vinschen wrote:
>> On Aug 30 17:53, Corinna Vinschen wrote:
>>> On Aug 30 16:05, Corinna Vinschen wrote:
>>>> On Aug 30 09:36, Ken Brown wrote:
>>>>> BTW, when I was working on the pipe approach to AF_UNIX sockets
>>>>> (topic/af_unix branch), I had occasion to step through
>>>>> select.cc:pipe_data_available in gdb, and the use of fpli.OutboundQuota -
>>>>> fpli.ReadDataAvailable definitely seemed wrong to me.  So when I wrote
>>>>> peek_socket_unix on that branch, I used fpli.WriteQuotaAvailable, as Takashi
>>>>> is suggesting now.
>>>>
>>>> If that's working reliable these days (keeping fingers crossed for W7),
>>>> it's ok if we use that.  We may want to check if the above observation
>>>> in terms on WriteQuotaAvailable on a pipe with pending read is still an
>>>> issue.
>>>
>>> Ok, I wrote a small testcase.  It creates a named pipe, reads from the
>>> pipe, then, later, writes to the pipe.  Interlaced with these calls, it
>>> calls NtQueryInformationFile(FilePipeLocalInformation) on the write side
>>> of the pipe.  Kind of like this:
>>>
>>>    CreatePipe
>>>    NtQueryInformationFile
>>>    ReadFile
>>>    NtQueryInformationFile
>>>    WriteFile
>>>    NtQueryInformationFile
>>>
>>> Here's the result:
>>>
>>> Before ReadFile:
>>>
>>>    InboundQuota: 65536
>>>    ReadDataAvailable: 0
>>>    OutboundQuota: 65536
>>>    WriteQuotaAvailable: 65536
>>>
>>> While ReadFile is running:
>>>
>>>    InboundQuota: 65536
>>>    ReadDataAvailable: 0
>>>    OutboundQuota: 65536
>>>    WriteQuotaAvailable: 65494    !!!
>>>
>>> After WriteFile and ReadFile succeeded:
>>>
>>>    InboundQuota: 65536
>>>    ReadDataAvailable: 0
>>>    OutboundQuota: 65536
>>>    WriteQuotaAvailable: 65536
>>>
>>> That means, while a reader on the reader side is waiting for data, the
>>> WriteQuotaAvailable on the write side is decremented by the amount of
>>> data requested by the reader (42 bytes in my case), just as outlined in that
>>> mail from 2004.  And this is on W10 now.
>>>
>>> What to do with this information?  TBD.
>>
>> Ok, let's discuss this.  I added more code to my testcase and here's
>> what I see.  I dropped all data from the output which doesn't change.
>>
>> What I'm trying to get a grip on are the dependencies here.
>>
>> After creating the pipe:
>>
>>    read side: ReadDataAvailable: 0
>>    write side: WriteQuotaAvailable: 65536
>>
>> After writing 20 bytes...
>>
>>    read side: ReadDataAvailable: 20
>>    write side: WriteQuotaAvailable: 65516
>>
>> After writing 40 more bytes...
>>
>>    read side: ReadDataAvailable: 60
>>    write side: WriteQuotaAvailable: 65476
>>
>> After reading 42 bytes...
>>
>>    read side: ReadDataAvailable: 18
>>    write side: WriteQuotaAvailable: 65518
>>
>> After writing 20 bytes...
>>
>>    read side: ReadDataAvailable: 38
>>    write side: WriteQuotaAvailable: 65498
>>
>> *While* reading 42 bytes with an empty buffer...
>>
>>    read side: ReadDataAvailable: 0
>>    write side: WriteQuotaAvailable: 65494
>>
>> Another important fun fact:  Assuming the read and write buffer sizes
>> are differently specified.  I called CreateNamedPipe with an outbuffer
>> size of 32K and an inbuffer size of 64K:
>>
>> After creating the pipe:
>>
>>    read side:
>>      InboundQuota: 65536
>>      ReadDataAvailable: 0
>>      OutboundQuota: 32768
>>      WriteQuotaAvailable: 32768
>>    write side:
>>      InboundQuota: 65536
>>      ReadDataAvailable: 0
>>      OutboundQuota: 32768
>>      WriteQuotaAvailable: 65536    !!!
>>
>> This last data point shows that:
>>
>> - InboundQuota and OutboundQuota are always constant values and
>>    do not depend on the side the information has been queried on.
>>    That certainly makes sense.
>>
>> - WriteQuotaAvailable does not depend on the OutboundQuota, but on
>>    the InboundQuota, and very likely on the InboundQuota of the read
>>    side.  The OutboundQuota *probably* only makes sense when using
>>    named pipes with remote clients, which we never do anyway.
>>
>> The preceeding output shows that ReadDataAvailable on the read side and
>> WriteQuotaAvailable on the write side are connected.  If we write 20
>> bytes, ReadDataAvailable is incremented by 20 and WriteQuotaAvailable is
>> decremented by 20.
>>
>> So: write.WriteQuotaAvailable == InboundQuota - read.ReadDataAvailable.
>>
>> Except when a ReadFile is pending on the read side.  It's as if the
>> running ReadFile already reserved write quota.  So the write side
>> WriteQuotaAvailable is the number of bytes we can write without blocking,
>> after all pending ReadFiles have been satisfied.
>>
>> Unfortunately that doesn't really make sense when looked at it from the
>> user space.
>>
>> What that means in the first place is that WriteQuotaAvailable on the
>> write side is unreliable.  What we really need is InboundQuota -
>> read.ReadDataAvailable.  The problem with that is that the write side
>> usually has no access to the read side of the pipe.
> 
> For the purposes of select.cc:pipe_data_available, we only need to know whether 
> InboundQuota - read.ReadDataAvailable is positive.  If WriteQuotaAvailable is 
> positive, then we're OK, even though its precise value might be too small.  But 
> if WriteQuotaAvailable == 0, we don't know whether the buffer is actually full 
> or there's a pending ReadFile.  It's only in this case that we need access to 
> the read side.
> 
> What if we reverse the roles of the read and write sides of the pipe, so that 
> the write side is the server and the read side is the client.  We can then try 
> to use ImpersonateNamedPipeClient to get information about the read side when 
> WriteQuotaAvailable == 0.  If we succeed, then we can determine whether or not 
> there's space in the buffer.  If we fail, we simply report, possibly 
> incorrectly, that there is space.  This is no worse than the current situation 
> (on the master branch), in which we use OutboundQuota - ReadDataAvailable, which 
> is always positive.

I should add that I know absolutely nothing about ImpersonateNamedPipeClient, so 
this might be complete nonsense.

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-08-30 17:00                                     ` Corinna Vinschen
  2021-08-30 17:11                                       ` Corinna Vinschen
  2021-08-30 18:59                                       ` Ken Brown
@ 2021-08-30 20:14                                       ` Corinna Vinschen
  2021-08-30 20:47                                         ` Ken Brown
  2021-08-31  8:55                                         ` Takashi Yano
  2 siblings, 2 replies; 250+ messages in thread
From: Corinna Vinschen @ 2021-08-30 20:14 UTC (permalink / raw)
  To: cygwin-developers

[-- Attachment #1: Type: text/plain, Size: 2269 bytes --]

Hi Ken, Hi Takashi,

On Aug 30 19:00, Corinna Vinschen wrote:
> Ok, let's discuss this.  I added more code to my testcase and here's
> what I see.  I dropped all data from the output which doesn't change.
> [...]
> - InboundQuota and OutboundQuota are always constant values and
>   do not depend on the side the information has been queried on.
>   That certainly makes sense.
> 
> - WriteQuotaAvailable does not depend on the OutboundQuota, but on
>   the InboundQuota, and very likely on the InboundQuota of the read
>   side.  The OutboundQuota *probably* only makes sense when using
>   named pipes with remote clients, which we never do anyway.
> 
> The preceeding output shows that ReadDataAvailable on the read side and
> WriteQuotaAvailable on the write side are connected.  If we write 20
> bytes, ReadDataAvailable is incremented by 20 and WriteQuotaAvailable is
> decremented by 20.
> 
> So: write.WriteQuotaAvailable == InboundQuota - read.ReadDataAvailable.
> 
> Except when a ReadFile is pending on the read side.  It's as if the
> running ReadFile already reserved write quota.  So the write side
> WriteQuotaAvailable is the number of bytes we can write without blocking,
> after all pending ReadFiles have been satisfied.
> 
> Unfortunately that doesn't really make sense when looked at it from the
> user space.
> 
> What that means in the first place is that WriteQuotaAvailable on the
> write side is unreliable.  What we really need is InboundQuota -
> read.ReadDataAvailable.  The problem with that is that the write side
> usually has no access to the read side of the pipe.
> 
> Long story short, I have no idea how to fix that ATM.

Well, what about keeping a duplicate of the read side handle on the 
write side just for calling NtQueryInformationFile?

Attached is an untested patch, can you have a look if that makes sense?

Btw., I think I found a bug in the new fhandler_pipe::create.  If the
function fails to create the write side fhandler, it deletes the read
side fhandler, but neglects to close the read handle.  My patch fixes
that.

While looking into this I found a problem in fhandler_disk_file in
terms of handle inheritance of the special handle for pread/pwrite.
I already force pushed this onto topic/pipe.


Thanks,
Corinna

[-- Attachment #2: pipe.diff --]
[-- Type: text/plain, Size: 5407 bytes --]

diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
index 132e6002133b..a2de4301521b 100644
--- a/winsup/cygwin/fhandler.h
+++ b/winsup/cygwin/fhandler.h
@@ -462,6 +462,7 @@ public:
   virtual HANDLE& get_output_handle_nat () { return io_handle; }
   virtual HANDLE get_stat_handle () { return pc.handle () ?: io_handle; }
   virtual HANDLE get_echo_handle () const { return NULL; }
+  virtual HANDLE get_query_handle () const { return NULL; }
   virtual bool hit_eof () {return false;}
   virtual select_record *select_read (select_stuff *);
   virtual select_record *select_write (select_stuff *);
@@ -1171,6 +1172,7 @@ class fhandler_socket_unix : public fhandler_socket
 class fhandler_pipe: public fhandler_base
 {
 private:
+  HANDLE query_hdl;
   pid_t popen_pid;
   size_t max_atomic_write;
   void set_pipe_non_blocking (bool nonblocking);
@@ -1179,6 +1181,12 @@ public:
 
   bool ispipe() const { return true; }
 
+  HANDLE get_query_handle ()
+  {
+    return (get_device () == FH_PIPEW) ? query_hdl : get_handle ();
+  }
+  void set_query_handle (HANDLE qh) { query_hdl = qh; }
+
   void set_popen_pid (pid_t pid) {popen_pid = pid;}
   pid_t get_popen_pid () const {return popen_pid;}
   off_t lseek (off_t offset, int whence);
@@ -1187,7 +1195,9 @@ public:
   select_record *select_except (select_stuff *);
   char *get_proc_fd_name (char *buf);
   int open (int flags, mode_t mode = 0);
+  void fixup_after_fork (HANDLE);
   int dup (fhandler_base *child, int);
+  int close ();
   void __reg3 raw_read (void *ptr, size_t& len);
   ssize_t __reg3 raw_write (const void *ptr, size_t len);
   int ioctl (unsigned int cmd, void *);
diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index b99f00c099f8..f698f9063207 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -405,22 +405,45 @@ fhandler_pipe::raw_write (const void *ptr, size_t len)
   return ret;
 }
 
+void
+fhandler_pipe::fixup_after_fork (HANDLE parent)
+{
+  if (query_hdl)
+    fork_fixup (parent, query_hdl, "query_hdl");
+  fhandler_base::fixup_after_fork (parent);
+}
+
 int
 fhandler_pipe::dup (fhandler_base *child, int flags)
 {
   fhandler_pipe *ftp = (fhandler_pipe *) child;
   ftp->set_popen_pid (0);
 
-  int res;
-  if (get_handle () && fhandler_base::dup (child, flags))
+  int res = 0;
+  if (fhandler_base::dup (child, flags))
     res = -1;
-  else
-    res = 0;
+  else if (query_hdl &&
+	   !DuplicateHandle (GetCurrentProcess (), query_hdl,
+			     GetCurrentProcess (), &ftp->query_hdl,
+			     0, !(flags & O_CLOEXEC), DUPLICATE_SAME_ACCESS))
+    {
+      __seterrno ();
+      ftp->close ();
+      res = -1;
+    }
 
   debug_printf ("res %d", res);
   return res;
 }
 
+int
+fhandler_pipe::close ()
+{
+  if (query_hdl)
+    NtClose (query_hdl);
+  return fhandler_base::close ();
+}
+
 #define PIPE_INTRO "\\\\.\\pipe\\cygwin-"
 
 /* Create a pipe, and return handles to the read and write ends,
@@ -608,6 +631,7 @@ fhandler_pipe::create (fhandler_pipe *fhs[2], unsigned psize, int mode)
   else if ((fhs[1] = (fhandler_pipe *) build_fh_dev (*pipew_dev)) == NULL)
     {
       delete fhs[0];
+      CloseHandle (r);
       CloseHandle (w);
     }
   else
@@ -617,7 +641,17 @@ fhandler_pipe::create (fhandler_pipe *fhs[2], unsigned psize, int mode)
 		    unique_id);
       fhs[1]->init (w, FILE_CREATE_PIPE_INSTANCE | GENERIC_WRITE, mode,
 		    unique_id);
-      res = 0;
+      if (!DuplicateHandle (GetCurrentProcess (), r, GetCurrentProcess (),
+			    &fhs[1]->query_hdl, FILE_READ_ATTRIBUTES,
+			    !(mode & O_CLOEXEC), 0))
+	{
+	  delete fhs[0];
+	  CloseHandle (r);
+	  delete fhs[1];
+	  CloseHandle (w);
+	}
+      else
+	res = 0;
     }
 
   debug_printf ("%R = pipe([%p, %p], %d, %y)", res, fhs[0], fhs[1], psize, mode);
diff --git a/winsup/cygwin/select.cc b/winsup/cygwin/select.cc
index 83e1c00e0ac7..dc0563a45729 100644
--- a/winsup/cygwin/select.cc
+++ b/winsup/cygwin/select.cc
@@ -608,11 +608,14 @@ pipe_data_available (int fd, fhandler_base *fh, HANDLE h, bool writing)
     }
   if (writing)
     {
-	/* If there is anything available in the pipe buffer then signal
-	   that.  This means that a pipe could still block since you could
-	   be trying to write more to the pipe than is available in the
-	   buffer but that is the hazard of select().  */
-      fpli.WriteQuotaAvailable = fpli.OutboundQuota - fpli.ReadDataAvailable;
+      /* If there is anything available in the pipe buffer then signal
+	 that.  This means that a pipe could still block since you could
+	 be trying to write more to the pipe than is available in the
+	 buffer but that is the hazard of select().
+	 Note that WriteQuotaAvailable is unreliable.  The only reliable
+	 information is available on the read side, which is why we fetch
+	 the info from the read side via the pipe-specific query handle. */
+      fpli.WriteQuotaAvailable = fpli.InboundQuota - fpli.ReadDataAvailable;
       if (fpli.WriteQuotaAvailable > 0)
 	{
 	  paranoid_printf ("fd %d, %s, write: size %u, avail %u", fd,
@@ -718,7 +721,7 @@ out:
       fhandler_pty_master *fhm = (fhandler_pty_master *) fh;
       fhm->set_mask_flusho (s->read_ready);
     }
-  h = fh->get_output_handle ();
+  h = fh->get_query_handle ();
   if (s->write_selected && dev != FH_PIPER)
     {
       gotone += s->write_ready =  pipe_data_available (s->fd, fh, h, true);

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-08-30 18:59                                       ` Ken Brown
  2021-08-30 19:12                                         ` Ken Brown
@ 2021-08-30 20:21                                         ` Corinna Vinschen
  1 sibling, 0 replies; 250+ messages in thread
From: Corinna Vinschen @ 2021-08-30 20:21 UTC (permalink / raw)
  To: cygwin-developers

On Aug 30 14:59, Ken Brown wrote:
> On 8/30/2021 1:00 PM, Corinna Vinschen wrote:
> > [...]
> > So: write.WriteQuotaAvailable == InboundQuota - read.ReadDataAvailable.
> > 
> > Except when a ReadFile is pending on the read side.  It's as if the
> > running ReadFile already reserved write quota.  So the write side
> > WriteQuotaAvailable is the number of bytes we can write without blocking,
> > after all pending ReadFiles have been satisfied.
> > 
> > Unfortunately that doesn't really make sense when looked at it from the
> > user space.
> > 
> > What that means in the first place is that WriteQuotaAvailable on the
> > write side is unreliable.  What we really need is InboundQuota -
> > read.ReadDataAvailable.  The problem with that is that the write side
> > usually has no access to the read side of the pipe.
> 
> For the purposes of select.cc:pipe_data_available, we only need to know
> whether InboundQuota - read.ReadDataAvailable is positive.  If
> WriteQuotaAvailable is positive, then we're OK, even though its precise
> value might be too small.  But if WriteQuotaAvailable == 0, we don't know
> whether the buffer is actually full or there's a pending ReadFile.  It's
> only in this case that we need access to the read side.

Any ReadFile with buffer size >= pipe buffer size will trigger that
immediately.

> What if we reverse the roles of the read and write sides of the pipe, so
> that the write side is the server and the read side is the client.  We can
> then try to use ImpersonateNamedPipeClient to get information about the read
> side when WriteQuotaAvailable == 0.

It's not a problem of impersonation, it's the problem of not having
access to the read side, because the write side just doesn't know
what handle in which process constitutes the read side of the pipe.
For all the write side knows, it could be some other fhandler in the
same process, or some far away HANDLE in some far away process in
the same session, sometimes not even a Cygwin process.


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-08-30 20:14                                       ` Corinna Vinschen
@ 2021-08-30 20:47                                         ` Ken Brown
  2021-08-31  8:55                                         ` Takashi Yano
  1 sibling, 0 replies; 250+ messages in thread
From: Ken Brown @ 2021-08-30 20:47 UTC (permalink / raw)
  To: cygwin-developers

On 8/30/2021 4:14 PM, Corinna Vinschen wrote:
> Hi Ken, Hi Takashi,
> 
> On Aug 30 19:00, Corinna Vinschen wrote:
>> Ok, let's discuss this.  I added more code to my testcase and here's
>> what I see.  I dropped all data from the output which doesn't change.
>> [...]
>> - InboundQuota and OutboundQuota are always constant values and
>>    do not depend on the side the information has been queried on.
>>    That certainly makes sense.
>>
>> - WriteQuotaAvailable does not depend on the OutboundQuota, but on
>>    the InboundQuota, and very likely on the InboundQuota of the read
>>    side.  The OutboundQuota *probably* only makes sense when using
>>    named pipes with remote clients, which we never do anyway.
>>
>> The preceeding output shows that ReadDataAvailable on the read side and
>> WriteQuotaAvailable on the write side are connected.  If we write 20
>> bytes, ReadDataAvailable is incremented by 20 and WriteQuotaAvailable is
>> decremented by 20.
>>
>> So: write.WriteQuotaAvailable == InboundQuota - read.ReadDataAvailable.
>>
>> Except when a ReadFile is pending on the read side.  It's as if the
>> running ReadFile already reserved write quota.  So the write side
>> WriteQuotaAvailable is the number of bytes we can write without blocking,
>> after all pending ReadFiles have been satisfied.
>>
>> Unfortunately that doesn't really make sense when looked at it from the
>> user space.
>>
>> What that means in the first place is that WriteQuotaAvailable on the
>> write side is unreliable.  What we really need is InboundQuota -
>> read.ReadDataAvailable.  The problem with that is that the write side
>> usually has no access to the read side of the pipe.
>>
>> Long story short, I have no idea how to fix that ATM.
> 
> Well, what about keeping a duplicate of the read side handle on the
> write side just for calling NtQueryInformationFile?
> 
> Attached is an untested patch, can you have a look if that makes sense?

I probably won't get a chance to test this until tomorrow, but at a glance it 
looks great!!

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-08-30 13:31                                       ` Corinna Vinschen
@ 2021-08-31  8:50                                         ` Takashi Yano
  0 siblings, 0 replies; 250+ messages in thread
From: Takashi Yano @ 2021-08-31  8:50 UTC (permalink / raw)
  To: cygwin-developers

On Mon, 30 Aug 2021 15:31:23 +0200
Corinna Vinschen wrote:
> Hi Takashi,
> 
> On Aug 30 14:55, Corinna Vinschen wrote:
> > On Aug 30 21:04, Takashi Yano wrote:
> > > On Mon, 30 Aug 2021 12:20:30 +0200
> > > Corinna Vinschen wrote:
> > > > [Move discussion to cygwin-developers]
> > > > 
> > > > On Aug 30 17:02, Takashi Yano via Cygwin wrote:
> > > > > [...]
> > > > > Is naming the pipe really necessary?
> > > > 
> > > > It's not, but CreatePipe is doing this anyway.
> > > > 
> > > > "Anonymous pipes are implemented using a named pipe with a unique name."
> > > > https://docs.microsoft.com/en-us/windows/win32/api/namedpipeapi/nf-namedpipeapi-createpipe
> > > > 
> > > > The reason CreateNamedPipe was used in the first place was that
> > > > FILE_READ_ATTRIBUTES isn't set by CreatePipe for the write side
> > > > of the pipe, however, it creates full duplex pipe:
> > > > 
> > > > https://cygwin.com/pipermail/cygwin-patches/2004q3/004912.html
> > > > 
> > > > Given the fact that CreatePipe is implemented in terms of
> > > > NtCreateNamedPipeFile anyway, why should the pipe created with
> > > > NtCreateNamedPipeFile fail where the pipe created with CreatePipe works?
> > > > 
> > > > The only reason can be some missing flag, I think.  Checking
> > > > fhandler_pipe.cc::nt_create and comparing that with the default flags
> > > > for files and other devices, it occurs to me that the SYNCHRONIZE stuff
> > > > is missing.  So, Takashi, what if you call NtCreateNamedPipeFile like
> > > > this in nt_create:
> > > > 
> > > >   status = NtCreateNamedPipeFile (r, access | SYNCHRONIZE, &attr, &io,
> > > > 				  FILE_SHARE_READ | FILE_SHARE_WRITE,
> > > > 				  FILE_CREATE, FILE_SYNCHRONOUS_IO_NONALERT,
> > > > 				  pipe_type, FILE_PIPE_BYTE_STREAM_MODE,
> > > > 				  0, 1, psize, psize, &timeout);
> > > > 
> > > > Does that fix the above problems, too?
> > > 
> > > Yes it does! Now, if CYGWIN=pipe_byte is also set, the piping issue
> > > of C# program is gone!
> 
> I don't quite understand this one.  Is that C# example using the write
> side of the pipe?  If it reads from the pipe, this behaiour would be
> pretty puzzeling, given the read mode is always BYTE.

Both side. Writer and reader are C# program. However, only even if only
reader side is C# program, the problem occurs when the reader starts
to read pipe before the writer writes something to the pipe; i.e.
something like:

(sleep 0.1; echo AAAAAAAA) | ./reader

> Either way, assuming we switch the write side to BYTE mode only, is
> the pty code robust enough to work with that?  The comment
> 
>   Note that the write side of the pipe is opened as PIPE_TYPE_MESSAGE.
>   This *seems* to more closely mimic Linux pipe behavior and is
>   definitely required for pty handling since fhandler_pty_master
>   writes to the pipe in chunks, terminated by newline when CANON mode
>   is specified. 
> 
> is old, so the problems the message mode was trying to solve for
> CANON mode may not apply anymore...

In the current topic/pipe head, named pipe for pty is configured
as PIPE_TYPE_MESSAGE even if CYGWIN=pipe_byte is set. Pty needs
message mode for canonical read.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-08-30 15:19                                           ` Corinna Vinschen
  2021-08-30 15:43                                             ` Ken Brown
@ 2021-08-31  8:52                                             ` Takashi Yano
  2021-08-31  9:04                                               ` Corinna Vinschen
  1 sibling, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-08-31  8:52 UTC (permalink / raw)
  To: cygwin-developers

On Mon, 30 Aug 2021 17:19:44 +0200
Corinna Vinschen wrote:
> On Aug 30 11:00, Ken Brown wrote:
> > On 8/30/2021 9:51 AM, Ken Brown wrote:
> > > On 8/30/2021 8:55 AM, Corinna Vinschen wrote:
> > > > On Aug 30 21:04, Takashi Yano wrote:
> > > > No worries.  The same should apply to the NtCreateFile side of the
> > > > pipe, btw.
> > > 
> > > I'll add my thanks.  I should have checked the default flags that are
> > > typically used for other devices when I wrote nt_create.  I'm glad you
> > > caught this.
> > > 
> > > So I'll reinstate the use of nt_create and then let Takashi recheck everything.
> > 
> > I've done this now.  I'm still not sure I've got all the flags right.  For
> > unknown reasons, I've used FILE_SHARE_READ | FILE_SHARE_WRITE in the call to
> > NtCreateNamedPipeFile, and no sharing in the call to NtOpenFile.  Should I
> > also use FILE_SHARE_READ | FILE_SHARE in NtOpenFile?  Is sharing even
> > relevant in this context?
> 
> This is only relevant if you want to open the pipe from another context,
> calling CreateNamedPipe/CreateFile.  As long as the pipe is only
> duplicated, it shouldn't matter at all.
> 
> But, as I just wrote in my previous mail, the FILE_SYNCHRONOUS_IO_NONALERT
> flag is probably a good thing for C# apps, but not for Cygwin, because it
> enforces synchronous operation.  Sorry about that...

With FILE_SYNCHRONOUS_IO_NONALERT, what kind of problems are you
specifically concerned about cygwin pipe? 

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-08-30 20:14                                       ` Corinna Vinschen
  2021-08-30 20:47                                         ` Ken Brown
@ 2021-08-31  8:55                                         ` Takashi Yano
  2021-08-31  9:08                                           ` Corinna Vinschen
  1 sibling, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-08-31  8:55 UTC (permalink / raw)
  To: cygwin-developers

On Mon, 30 Aug 2021 22:14:15 +0200
Corinna Vinschen wrote:
> Hi Ken, Hi Takashi,
> 
> On Aug 30 19:00, Corinna Vinschen wrote:
> > Ok, let's discuss this.  I added more code to my testcase and here's
> > what I see.  I dropped all data from the output which doesn't change.
> > [...]
> > - InboundQuota and OutboundQuota are always constant values and
> >   do not depend on the side the information has been queried on.
> >   That certainly makes sense.
> > 
> > - WriteQuotaAvailable does not depend on the OutboundQuota, but on
> >   the InboundQuota, and very likely on the InboundQuota of the read
> >   side.  The OutboundQuota *probably* only makes sense when using
> >   named pipes with remote clients, which we never do anyway.
> > 
> > The preceeding output shows that ReadDataAvailable on the read side and
> > WriteQuotaAvailable on the write side are connected.  If we write 20
> > bytes, ReadDataAvailable is incremented by 20 and WriteQuotaAvailable is
> > decremented by 20.
> > 
> > So: write.WriteQuotaAvailable == InboundQuota - read.ReadDataAvailable.
> > 
> > Except when a ReadFile is pending on the read side.  It's as if the
> > running ReadFile already reserved write quota.  So the write side
> > WriteQuotaAvailable is the number of bytes we can write without blocking,
> > after all pending ReadFiles have been satisfied.
> > 
> > Unfortunately that doesn't really make sense when looked at it from the
> > user space.
> > 
> > What that means in the first place is that WriteQuotaAvailable on the
> > write side is unreliable.  What we really need is InboundQuota -
> > read.ReadDataAvailable.  The problem with that is that the write side
> > usually has no access to the read side of the pipe.
> > 
> > Long story short, I have no idea how to fix that ATM.
> 
> Well, what about keeping a duplicate of the read side handle on the 
> write side just for calling NtQueryInformationFile?
> 
> Attached is an untested patch, can you have a look if that makes sense?
> 
> Btw., I think I found a bug in the new fhandler_pipe::create.  If the
> function fails to create the write side fhandler, it deletes the read
> side fhandler, but neglects to close the read handle.  My patch fixes
> that.
> 
> While looking into this I found a problem in fhandler_disk_file in
> terms of handle inheritance of the special handle for pread/pwrite.
> I already force pushed this onto topic/pipe.

I tested your patch attached. Unfortunately, select() does not work
as expected for write pipe. Even if the select reports write pipe
is available, writing to pipe fails. It seems that your patch fails
to detect pipe full.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-08-31  8:52                                             ` Takashi Yano
@ 2021-08-31  9:04                                               ` Corinna Vinschen
  2021-08-31 11:05                                                 ` Takashi Yano
  0 siblings, 1 reply; 250+ messages in thread
From: Corinna Vinschen @ 2021-08-31  9:04 UTC (permalink / raw)
  To: cygwin-developers

On Aug 31 17:52, Takashi Yano wrote:
> On Mon, 30 Aug 2021 17:19:44 +0200
> Corinna Vinschen wrote:
> > On Aug 30 11:00, Ken Brown wrote:
> > > On 8/30/2021 9:51 AM, Ken Brown wrote:
> > > > On 8/30/2021 8:55 AM, Corinna Vinschen wrote:
> > > > > On Aug 30 21:04, Takashi Yano wrote:
> > > > > No worries.  The same should apply to the NtCreateFile side of the
> > > > > pipe, btw.
> > > > 
> > > > I'll add my thanks.  I should have checked the default flags that are
> > > > typically used for other devices when I wrote nt_create.  I'm glad you
> > > > caught this.
> > > > 
> > > > So I'll reinstate the use of nt_create and then let Takashi recheck everything.
> > > 
> > > I've done this now.  I'm still not sure I've got all the flags right.  For
> > > unknown reasons, I've used FILE_SHARE_READ | FILE_SHARE_WRITE in the call to
> > > NtCreateNamedPipeFile, and no sharing in the call to NtOpenFile.  Should I
> > > also use FILE_SHARE_READ | FILE_SHARE in NtOpenFile?  Is sharing even
> > > relevant in this context?
> > 
> > This is only relevant if you want to open the pipe from another context,
> > calling CreateNamedPipe/CreateFile.  As long as the pipe is only
> > duplicated, it shouldn't matter at all.
> > 
> > But, as I just wrote in my previous mail, the FILE_SYNCHRONOUS_IO_NONALERT
> > flag is probably a good thing for C# apps, but not for Cygwin, because it
> > enforces synchronous operation.  Sorry about that...
> 
> With FILE_SYNCHRONOUS_IO_NONALERT, what kind of problems are you
> specifically concerned about cygwin pipe? 

We're using asynchronous IO to be able to call WFMO and thus to be able
to handle signals and thread cancellation events.  Wit hsynchronous IO
this is not possible.


Corinna


^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-08-31  8:55                                         ` Takashi Yano
@ 2021-08-31  9:08                                           ` Corinna Vinschen
  2021-08-31  9:25                                             ` Takashi Yano
  0 siblings, 1 reply; 250+ messages in thread
From: Corinna Vinschen @ 2021-08-31  9:08 UTC (permalink / raw)
  To: cygwin-developers

On Aug 31 17:55, Takashi Yano wrote:
> On Mon, 30 Aug 2021 22:14:15 +0200
> Corinna Vinschen wrote:
> > Hi Ken, Hi Takashi,
> > 
> > On Aug 30 19:00, Corinna Vinschen wrote:
> > Well, what about keeping a duplicate of the read side handle on the 
> > write side just for calling NtQueryInformationFile?
> > 
> > Attached is an untested patch, can you have a look if that makes sense?
> > 
> > Btw., I think I found a bug in the new fhandler_pipe::create.  If the
> > function fails to create the write side fhandler, it deletes the read
> > side fhandler, but neglects to close the read handle.  My patch fixes
> > that.
> > 
> > While looking into this I found a problem in fhandler_disk_file in
> > terms of handle inheritance of the special handle for pread/pwrite.
> > I already force pushed this onto topic/pipe.
> 
> I tested your patch attached. Unfortunately, select() does not work
> as expected for write pipe. Even if the select reports write pipe
> is available, writing to pipe fails. It seems that your patch fails
> to detect pipe full.

Bummer.  Is that with byte mode pipes or with message mode pipes?  If
the latter, if you try to write more data than available in the buffer,
it's bound to fail.

Did you add debug output to pipe_data_available to see how the
information looks like?  Or do you have a simple, self-contained
testcase in plain C?


Thanks,
Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-08-31  9:08                                           ` Corinna Vinschen
@ 2021-08-31  9:25                                             ` Takashi Yano
  2021-08-31 10:05                                               ` Corinna Vinschen
  0 siblings, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-08-31  9:25 UTC (permalink / raw)
  To: cygwin-developers

[-- Attachment #1: Type: text/plain, Size: 1750 bytes --]

On Tue, 31 Aug 2021 11:08:42 +0200
Corinna Vinschenwrote:
> On Aug 31 17:55, Takashi Yano wrote:
> > On Mon, 30 Aug 2021 22:14:15 +0200
> > Corinna Vinschen wrote:
> > > Hi Ken, Hi Takashi,
> > > 
> > > On Aug 30 19:00, Corinna Vinschen wrote:
> > > Well, what about keeping a duplicate of the read side handle on the 
> > > write side just for calling NtQueryInformationFile?
> > > 
> > > Attached is an untested patch, can you have a look if that makes sense?
> > > 
> > > Btw., I think I found a bug in the new fhandler_pipe::create.  If the
> > > function fails to create the write side fhandler, it deletes the read
> > > side fhandler, but neglects to close the read handle.  My patch fixes
> > > that.
> > > 
> > > While looking into this I found a problem in fhandler_disk_file in
> > > terms of handle inheritance of the special handle for pread/pwrite.
> > > I already force pushed this onto topic/pipe.
> > 
> > I tested your patch attached. Unfortunately, select() does not work
> > as expected for write pipe. Even if the select reports write pipe
> > is available, writing to pipe fails. It seems that your patch fails
> > to detect pipe full.
> 
> Bummer.  Is that with byte mode pipes or with message mode pipes?  If
> the latter, if you try to write more data than available in the buffer,
> it's bound to fail.

Both message pipe and byte pipe.

> Did you add debug output to pipe_data_available to see how the
> information looks like?  Or do you have a simple, self-contained
> testcase in plain C?

The test case is attached. If select() works as expected, the program
does not show "r" or "w". However, with your patch, the program prints
many "w" (means write() fails with EAGAIN).

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

[-- Attachment #2: stc.c --]
[-- Type: text/x-csrc, Size: 1797 bytes --]

#include <unistd.h>
#include <sys/wait.h>
#include <stdio.h>
#include <time.h>
#include <sys/select.h>
#include <fcntl.h>
#include <errno.h>

#define BLKSIZ 16384
#define NBLK (100*(1<<20)/BLKSIZ)

int main()
{
	int fd[2];
	pid_t pid;
	int nonblock = 1;

	pipe(fd);

	if (!(pid = fork ())) {
		int i;
		char buf[BLKSIZ] = {0,};
		close(fd[0]);
		if (nonblock) {
			int flags;
			flags = fcntl(fd[1], F_GETFL);
			flags |= O_NONBLOCK;
			fcntl(fd[1], F_SETFL, flags);
		}
		fd_set wfds;
		for (i=0; i<NBLK; i++) {
			FD_ZERO(&wfds);
			FD_SET(fd[1], &wfds);
			if (select(fd[1]+1, NULL, &wfds, NULL, NULL) > 0
					&& FD_ISSET(fd[1], &wfds)) {
				ssize_t len = write(fd[1], buf, sizeof(buf));
				if (len <= 0 && errno == EAGAIN) printf("w", i);
				if (len <= 0 && errno == EAGAIN) i --;
			}
		}
		close(fd[1]);
	} else {
		int i;
		char buf[BLKSIZ] = {0,};
		int total = 0;
		fd_set rfds;
		struct timespec tv0, tv1;
		double elasped;
		close(fd[1]);
		if (nonblock) {
			int flags;
			flags = fcntl(fd[0], F_GETFL);
			flags |= O_NONBLOCK;
			fcntl(fd[0], F_SETFL, flags);
		}
		usleep(100000); /* Delay to start reader */
		clock_gettime(CLOCK_MONOTONIC, &tv0);
		for (i=0; i<NBLK; i++) {
			FD_ZERO(&rfds);
			FD_SET(fd[0], &rfds);
			if (select(fd[0]+1, &rfds, NULL, NULL, NULL) > 0
					&& FD_ISSET(fd[0], &rfds)) {
				ssize_t len = read(fd[0], buf, sizeof(buf));
				if (len <= 0 && errno == EAGAIN) printf("r");
				if (len <= 0 && errno == EAGAIN) i --;
				else if(len < 0) break;
				else total += len;
			}
		}
		clock_gettime(CLOCK_MONOTONIC, &tv1);
		close(fd[0]);
		elasped = (tv1.tv_sec - tv0.tv_sec) + (tv1.tv_nsec - tv0.tv_nsec)*1e-9;
		printf("Total: %dMB in %f second, %fMB/s\n",
			total/(1<<20), elasped, total/(1<<20)/elasped);
		waitpid(pid, NULL, 0);
	}

	return 0;
}


^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-08-30 15:43                                             ` Ken Brown
@ 2021-08-31  9:43                                               ` Corinna Vinschen
  0 siblings, 0 replies; 250+ messages in thread
From: Corinna Vinschen @ 2021-08-31  9:43 UTC (permalink / raw)
  To: cygwin-developers

On Aug 30 11:43, Ken Brown wrote:
> On 8/30/2021 11:19 AM, Corinna Vinschen wrote:
> > On Aug 30 11:00, Ken Brown wrote:
> > > On 8/30/2021 9:51 AM, Ken Brown wrote:
> > > > On 8/30/2021 8:55 AM, Corinna Vinschen wrote:
> > > > > On Aug 30 21:04, Takashi Yano wrote:
> > > > > No worries.  The same should apply to the NtCreateFile side of the
> > > > > pipe, btw.
> > > > 
> > > > I'll add my thanks.  I should have checked the default flags that are
> > > > typically used for other devices when I wrote nt_create.  I'm glad you
> > > > caught this.
> > > > 
> > > > So I'll reinstate the use of nt_create and then let Takashi recheck everything.
> > > 
> > > I've done this now.  I'm still not sure I've got all the flags right.  For
> > > unknown reasons, I've used FILE_SHARE_READ | FILE_SHARE_WRITE in the call to
> > > NtCreateNamedPipeFile, and no sharing in the call to NtOpenFile.  Should I
> > > also use FILE_SHARE_READ | FILE_SHARE in NtOpenFile?  Is sharing even
> > > relevant in this context?
> > 
> > This is only relevant if you want to open the pipe from another context,
> > calling CreateNamedPipe/CreateFile.  As long as the pipe is only
> > duplicated, it shouldn't matter at all.
> 
> OK, then I think I should remove the sharing from NtCreateNamedPipeFile,
> since it could confuse someone reading the code.

I reverted both "fix flags" patches for the time being.  Removing the
sharing flags results in NtOpenFile for the write side to fail.


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-08-31  9:25                                             ` Takashi Yano
@ 2021-08-31 10:05                                               ` Corinna Vinschen
  2021-08-31 10:18                                                 ` Corinna Vinschen
  0 siblings, 1 reply; 250+ messages in thread
From: Corinna Vinschen @ 2021-08-31 10:05 UTC (permalink / raw)
  To: cygwin-developers

On Aug 31 18:25, Takashi Yano wrote:
> On Tue, 31 Aug 2021 11:08:42 +0200
> Corinna Vinschenwrote:
> > On Aug 31 17:55, Takashi Yano wrote:
> > > On Mon, 30 Aug 2021 22:14:15 +0200
> > > Corinna Vinschen wrote:
> > > > Hi Ken, Hi Takashi,
> > > > 
> > > > On Aug 30 19:00, Corinna Vinschen wrote:
> > > > Well, what about keeping a duplicate of the read side handle on the 
> > > > write side just for calling NtQueryInformationFile?
> > > > 
> > > > Attached is an untested patch, can you have a look if that makes sense?
> > > > 
> > > > Btw., I think I found a bug in the new fhandler_pipe::create.  If the
> > > > function fails to create the write side fhandler, it deletes the read
> > > > side fhandler, but neglects to close the read handle.  My patch fixes
> > > > that.
> > > > 
> > > > While looking into this I found a problem in fhandler_disk_file in
> > > > terms of handle inheritance of the special handle for pread/pwrite.
> > > > I already force pushed this onto topic/pipe.
> > > 
> > > I tested your patch attached. Unfortunately, select() does not work
> > > as expected for write pipe. Even if the select reports write pipe
> > > is available, writing to pipe fails. It seems that your patch fails
> > > to detect pipe full.
> > 
> > Bummer.  Is that with byte mode pipes or with message mode pipes?  If
> > the latter, if you try to write more data than available in the buffer,
> > it's bound to fail.
> 
> Both message pipe and byte pipe.
> 
> > Did you add debug output to pipe_data_available to see how the
> > information looks like?  Or do you have a simple, self-contained
> > testcase in plain C?
> 
> The test case is attached. If select() works as expected, the program
> does not show "r" or "w". However, with your patch, the program prints
> many "w" (means write() fails with EAGAIN).

Thanks!  I found th culprit, but we have another problem.  Even if
select returns correct info,  A write, trying to write more bytes
than are available in the buffer, hangs.  This shouldn't happen.
Still digging...


Corinna


^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-08-31 10:05                                               ` Corinna Vinschen
@ 2021-08-31 10:18                                                 ` Corinna Vinschen
  2021-08-31 11:45                                                   ` Takashi Yano
  0 siblings, 1 reply; 250+ messages in thread
From: Corinna Vinschen @ 2021-08-31 10:18 UTC (permalink / raw)
  To: cygwin-developers

[-- Attachment #1: Type: text/plain, Size: 2316 bytes --]

On Aug 31 12:05, Corinna Vinschen wrote:
> On Aug 31 18:25, Takashi Yano wrote:
> > On Tue, 31 Aug 2021 11:08:42 +0200
> > Corinna Vinschenwrote:
> > > On Aug 31 17:55, Takashi Yano wrote:
> > > > On Mon, 30 Aug 2021 22:14:15 +0200
> > > > Corinna Vinschen wrote:
> > > > > Hi Ken, Hi Takashi,
> > > > > 
> > > > > On Aug 30 19:00, Corinna Vinschen wrote:
> > > > > Well, what about keeping a duplicate of the read side handle on the 
> > > > > write side just for calling NtQueryInformationFile?
> > > > > 
> > > > > Attached is an untested patch, can you have a look if that makes sense?
> > > > > 
> > > > > Btw., I think I found a bug in the new fhandler_pipe::create.  If the
> > > > > function fails to create the write side fhandler, it deletes the read
> > > > > side fhandler, but neglects to close the read handle.  My patch fixes
> > > > > that.
> > > > > 
> > > > > While looking into this I found a problem in fhandler_disk_file in
> > > > > terms of handle inheritance of the special handle for pread/pwrite.
> > > > > I already force pushed this onto topic/pipe.
> > > > 
> > > > I tested your patch attached. Unfortunately, select() does not work
> > > > as expected for write pipe. Even if the select reports write pipe
> > > > is available, writing to pipe fails. It seems that your patch fails
> > > > to detect pipe full.
> > > 
> > > Bummer.  Is that with byte mode pipes or with message mode pipes?  If
> > > the latter, if you try to write more data than available in the buffer,
> > > it's bound to fail.
> > 
> > Both message pipe and byte pipe.
> > 
> > > Did you add debug output to pipe_data_available to see how the
> > > information looks like?  Or do you have a simple, self-contained
> > > testcase in plain C?
> > 
> > The test case is attached. If select() works as expected, the program
> > does not show "r" or "w". However, with your patch, the program prints
> > many "w" (means write() fails with EAGAIN).
> 
> Thanks!  I found th culprit, but we have another problem.  Even if
> select returns correct info,  A write, trying to write more bytes
> than are available in the buffer, hangs.  This shouldn't happen.
> Still digging...

That's, of course, correct behaviour for pipes in blocking mode.  D'oh! 

Please try the attached patch on top of topic/pipe.


Thanks,
Corinna

[-- Attachment #2: pipe.diff --]
[-- Type: text/plain, Size: 6770 bytes --]

diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
index 132e6002133b..1f0f28077a7c 100644
--- a/winsup/cygwin/fhandler.h
+++ b/winsup/cygwin/fhandler.h
@@ -1171,6 +1171,7 @@ class fhandler_socket_unix : public fhandler_socket
 class fhandler_pipe: public fhandler_base
 {
 private:
+  HANDLE query_hdl;
   pid_t popen_pid;
   size_t max_atomic_write;
   void set_pipe_non_blocking (bool nonblocking);
@@ -1179,6 +1180,8 @@ public:
 
   bool ispipe() const { return true; }
 
+  HANDLE get_query_handle () const { return query_hdl; }
+
   void set_popen_pid (pid_t pid) {popen_pid = pid;}
   pid_t get_popen_pid () const {return popen_pid;}
   off_t lseek (off_t offset, int whence);
@@ -1187,7 +1190,9 @@ public:
   select_record *select_except (select_stuff *);
   char *get_proc_fd_name (char *buf);
   int open (int flags, mode_t mode = 0);
+  void fixup_after_fork (HANDLE);
   int dup (fhandler_base *child, int);
+  int close ();
   void __reg3 raw_read (void *ptr, size_t& len);
   ssize_t __reg3 raw_write (const void *ptr, size_t len);
   int ioctl (unsigned int cmd, void *);
diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index 2dec0a84817c..479b62bbd4aa 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -405,22 +405,45 @@ fhandler_pipe::raw_write (const void *ptr, size_t len)
   return ret;
 }
 
+void
+fhandler_pipe::fixup_after_fork (HANDLE parent)
+{
+  if (query_hdl)
+    fork_fixup (parent, query_hdl, "query_hdl");
+  fhandler_base::fixup_after_fork (parent);
+}
+
 int
 fhandler_pipe::dup (fhandler_base *child, int flags)
 {
   fhandler_pipe *ftp = (fhandler_pipe *) child;
   ftp->set_popen_pid (0);
 
-  int res;
-  if (get_handle () && fhandler_base::dup (child, flags))
+  int res = 0;
+  if (fhandler_base::dup (child, flags))
     res = -1;
-  else
-    res = 0;
+  else if (query_hdl &&
+	   !DuplicateHandle (GetCurrentProcess (), query_hdl,
+			     GetCurrentProcess (), &ftp->query_hdl,
+			     0, !(flags & O_CLOEXEC), DUPLICATE_SAME_ACCESS))
+    {
+      __seterrno ();
+      ftp->close ();
+      res = -1;
+    }
 
   debug_printf ("res %d", res);
   return res;
 }
 
+int
+fhandler_pipe::close ()
+{
+  if (query_hdl)
+    NtClose (query_hdl);
+  return fhandler_base::close ();
+}
+
 #define PIPE_INTRO "\\\\.\\pipe\\cygwin-"
 
 /* Create a pipe, and return handles to the read and write ends,
@@ -608,6 +631,7 @@ fhandler_pipe::create (fhandler_pipe *fhs[2], unsigned psize, int mode)
   else if ((fhs[1] = (fhandler_pipe *) build_fh_dev (*pipew_dev)) == NULL)
     {
       delete fhs[0];
+      CloseHandle (r);
       CloseHandle (w);
     }
   else
@@ -617,10 +641,23 @@ fhandler_pipe::create (fhandler_pipe *fhs[2], unsigned psize, int mode)
 		    unique_id);
       fhs[1]->init (w, FILE_CREATE_PIPE_INSTANCE | GENERIC_WRITE, mode,
 		    unique_id);
-      res = 0;
+      /* For the write side of the pipe, duplicate the handle to the read side
+	 into query_hdl just for calling NtQueryInformationFile.  See longish
+	 comment in select.cc, pipe_data_available() for the reasoning. */
+      if (!DuplicateHandle (GetCurrentProcess (), r, GetCurrentProcess (),
+			    &fhs[1]->query_hdl, GENERIC_READ,
+			    !(mode & O_CLOEXEC), 0))
+	{
+	  delete fhs[0];
+	  CloseHandle (r);
+	  delete fhs[1];
+	  CloseHandle (w);
+	}
+      else
+	res = 0;
     }
 
-  debug_printf ("%R = pipe([%p, %p], %d, %y)", res, fhs[0], fhs[1], psize, mode);
+  debug_printf ("%R = pipe(%d, %y)", res, psize, mode);
   return res;
 }
 
@@ -661,7 +698,7 @@ nt_create (LPSECURITY_ATTRIBUTES sa_ptr, PHANDLE r, PHANDLE w,
   access = GENERIC_READ | FILE_WRITE_ATTRIBUTES;
 
   ULONG pipe_type = pipe_byte ? FILE_PIPE_BYTE_STREAM_TYPE
-    : FILE_PIPE_MESSAGE_TYPE;
+			      : FILE_PIPE_MESSAGE_TYPE;
 
   /* Retry NtCreateNamedPipeFile as long as the pipe name is in use.
      Retrying will probably never be necessary, but we want
diff --git a/winsup/cygwin/select.cc b/winsup/cygwin/select.cc
index 83e1c00e0ac7..7e69ad834d9a 100644
--- a/winsup/cygwin/select.cc
+++ b/winsup/cygwin/select.cc
@@ -608,15 +608,33 @@ pipe_data_available (int fd, fhandler_base *fh, HANDLE h, bool writing)
     }
   if (writing)
     {
-	/* If there is anything available in the pipe buffer then signal
-	   that.  This means that a pipe could still block since you could
-	   be trying to write more to the pipe than is available in the
-	   buffer but that is the hazard of select().  */
-      fpli.WriteQuotaAvailable = fpli.OutboundQuota - fpli.ReadDataAvailable;
+      /* If there is anything available in the pipe buffer then signal
+	 that.  This means that a pipe could still block since you could
+	 be trying to write more to the pipe than is available in the
+	 buffer but that is the hazard of select().
+
+	 Note that WriteQuotaAvailable is unreliable.
+
+	 Usually WriteQuotaAvailable on the write side reflects the space
+	 available in the inbound buffer on the read side.  However, if a
+	 pipe read is currently pending, WriteQuotaAvailable on the write side
+	 is decremented by the number of bytes the read side is requesting.
+	 So it's possible (even likely) that WriteQuotaAvailable is 0, even
+	 if the inbound buffer on the read side is not full.  This can lead to
+	 a deadlock situation: The reader is waiting for data, but select
+	 on the writer side assumes that no space is available in the read
+	 side inbound buffer.
+
+	 Consequentially, the only reliable information is available on the
+	 read side, so fetch info from the read side via the pipe-specific
+	 query handle.  Use fpli.WriteQuotaAvailable as storage for the actual
+	 interesting value, which is the InboundQuote on the read side,
+	 decremented by the number of bytes of data in that buffer. */
+      fpli.WriteQuotaAvailable = fpli.InboundQuota - fpli.ReadDataAvailable;
       if (fpli.WriteQuotaAvailable > 0)
 	{
 	  paranoid_printf ("fd %d, %s, write: size %u, avail %u", fd,
-			   fh->get_name (), fpli.OutboundQuota,
+			   fh->get_name (), fpli.InboundQuota,
 			   fpli.WriteQuotaAvailable);
 	  return 1;
 	}
@@ -718,9 +736,14 @@ out:
       fhandler_pty_master *fhm = (fhandler_pty_master *) fh;
       fhm->set_mask_flusho (s->read_ready);
     }
-  h = fh->get_output_handle ();
   if (s->write_selected && dev != FH_PIPER)
     {
+      /* For the write side of a pipe, fetch the handle to the read side.
+	 See the longish comment in pipe_data_available for the reasoning. */
+      if (dev == FH_PIPEW)
+	h = ((fhandler_pipe *) fh)->get_query_handle ();
+      else
+	h = fh->get_output_handle ();
       gotone += s->write_ready =  pipe_data_available (s->fd, fh, h, true);
       select_printf ("write: %s, gotone %d", fh->get_name (), gotone);
     }

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-08-31  9:04                                               ` Corinna Vinschen
@ 2021-08-31 11:05                                                 ` Takashi Yano
  2021-08-31 15:20                                                   ` Corinna Vinschen
  0 siblings, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-08-31 11:05 UTC (permalink / raw)
  To: cygwin-developers

On Tue, 31 Aug 2021 11:04:05 +0200
Corinna Vinschen wrote:
> On Aug 31 17:52, Takashi Yano wrote:
> > On Mon, 30 Aug 2021 17:19:44 +0200
> > Corinna Vinschen wrote:
> > > On Aug 30 11:00, Ken Brown wrote:
> > > > On 8/30/2021 9:51 AM, Ken Brown wrote:
> > > > > On 8/30/2021 8:55 AM, Corinna Vinschen wrote:
> > > > > > On Aug 30 21:04, Takashi Yano wrote:
> > > > > > No worries.  The same should apply to the NtCreateFile side of the
> > > > > > pipe, btw.
> > > > > 
> > > > > I'll add my thanks.  I should have checked the default flags that are
> > > > > typically used for other devices when I wrote nt_create.  I'm glad you
> > > > > caught this.
> > > > > 
> > > > > So I'll reinstate the use of nt_create and then let Takashi recheck everything.
> > > > 
> > > > I've done this now.  I'm still not sure I've got all the flags right.  For
> > > > unknown reasons, I've used FILE_SHARE_READ | FILE_SHARE_WRITE in the call to
> > > > NtCreateNamedPipeFile, and no sharing in the call to NtOpenFile.  Should I
> > > > also use FILE_SHARE_READ | FILE_SHARE in NtOpenFile?  Is sharing even
> > > > relevant in this context?
> > > 
> > > This is only relevant if you want to open the pipe from another context,
> > > calling CreateNamedPipe/CreateFile.  As long as the pipe is only
> > > duplicated, it shouldn't matter at all.
> > > 
> > > But, as I just wrote in my previous mail, the FILE_SYNCHRONOUS_IO_NONALERT
> > > flag is probably a good thing for C# apps, but not for Cygwin, because it
> > > enforces synchronous operation.  Sorry about that...
> > 
> > With FILE_SYNCHRONOUS_IO_NONALERT, what kind of problems are you
> > specifically concerned about cygwin pipe? 
> 
> We're using asynchronous IO to be able to call WFMO and thus to be able
> to handle signals and thread cancellation events.  Wit hsynchronous IO
> this is not possible.

Thanks. How can I regenerate above issue? Stopping by Ctrl-C or killing
the process by kill seems to work even with FILE_SYNCHRONOUS_IO_NONALERT.
Where is the WFMO called for pipe handle?

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-08-31 10:18                                                 ` Corinna Vinschen
@ 2021-08-31 11:45                                                   ` Takashi Yano
  2021-08-31 12:31                                                     ` Takashi Yano
  2021-08-31 12:33                                                     ` Ken Brown
  0 siblings, 2 replies; 250+ messages in thread
From: Takashi Yano @ 2021-08-31 11:45 UTC (permalink / raw)
  To: cygwin-developers

[-- Attachment #1: Type: text/plain, Size: 3634 bytes --]

On Tue, 31 Aug 2021 12:18:57 +0200
Corinna Vinschen wrote:
> On Aug 31 12:05, Corinna Vinschen wrote:
> > On Aug 31 18:25, Takashi Yano wrote:
> > > On Tue, 31 Aug 2021 11:08:42 +0200
> > > Corinna Vinschenwrote:
> > > > On Aug 31 17:55, Takashi Yano wrote:
> > > > > On Mon, 30 Aug 2021 22:14:15 +0200
> > > > > Corinna Vinschen wrote:
> > > > > > Hi Ken, Hi Takashi,
> > > > > > 
> > > > > > On Aug 30 19:00, Corinna Vinschen wrote:
> > > > > > Well, what about keeping a duplicate of the read side handle on the 
> > > > > > write side just for calling NtQueryInformationFile?
> > > > > > 
> > > > > > Attached is an untested patch, can you have a look if that makes sense?
> > > > > > 
> > > > > > Btw., I think I found a bug in the new fhandler_pipe::create.  If the
> > > > > > function fails to create the write side fhandler, it deletes the read
> > > > > > side fhandler, but neglects to close the read handle.  My patch fixes
> > > > > > that.
> > > > > > 
> > > > > > While looking into this I found a problem in fhandler_disk_file in
> > > > > > terms of handle inheritance of the special handle for pread/pwrite.
> > > > > > I already force pushed this onto topic/pipe.
> > > > > 
> > > > > I tested your patch attached. Unfortunately, select() does not work
> > > > > as expected for write pipe. Even if the select reports write pipe
> > > > > is available, writing to pipe fails. It seems that your patch fails
> > > > > to detect pipe full.
> > > > 
> > > > Bummer.  Is that with byte mode pipes or with message mode pipes?  If
> > > > the latter, if you try to write more data than available in the buffer,
> > > > it's bound to fail.
> > > 
> > > Both message pipe and byte pipe.
> > > 
> > > > Did you add debug output to pipe_data_available to see how the
> > > > information looks like?  Or do you have a simple, self-contained
> > > > testcase in plain C?
> > > 
> > > The test case is attached. If select() works as expected, the program
> > > does not show "r" or "w". However, with your patch, the program prints
> > > many "w" (means write() fails with EAGAIN).
> > 
> > Thanks!  I found th culprit, but we have another problem.  Even if
> > select returns correct info,  A write, trying to write more bytes
> > than are available in the buffer, hangs.  This shouldn't happen.
> > Still digging...
> 
> That's, of course, correct behaviour for pipes in blocking mode.  D'oh! 
> 
> Please try the attached patch on top of topic/pipe.

Thanks for the new patch. I have confirmed that above issue
is fixed and select() for write pipe seems to work as expected.


BTW, I found one minor difference between Linux and this pipe
implementation.

The test case is attached. The test case uses non-bloking I/O.
If this STC runs on Linux, the result is:

1024/1024
1740/1740
2958/2958
5028/5028
8547/8547
14529/14529
24699/24699
41988/41988
22227/71379
65536/121344
65536/206284
Total: 247KB in 0.000612 second, 403517.628166KB/s

On cygwin 3.2.0, the result is similar to Linux.

1024/1024
1740/1740
2957/2957
5026/5026
8544/8544
14524/14524
24690/24690
41972/41972
65536/71352
65536/121298
65536/206206
Total: 290KB in 0.062653 second, 4628.669018KB/s


However, on topic/pipe implementation, the result is

1024/1024
1740/1740
2957/2957
5026/5026
8544/8544
14524/14524
24690/24690
-1/41972
w-1/71352
w-1/121298
w-1/206206
wTotal: 57KB in 0.000330 second, 172989.377845KB/s

In non-blocking mode, writing more than pipe space will fail with
EAGAIN in this implementation.

In Linux and cygwin 3.2.0, it seems to write as much as writable.

Is this difficult to be fixed?

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

[-- Attachment #2: stc3.c --]
[-- Type: text/x-csrc, Size: 1786 bytes --]

#include <unistd.h>
#include <sys/wait.h>
#include <stdio.h>
#include <time.h>
#include <sys/select.h>
#include <fcntl.h>
#include <errno.h>

#define BLKSIZ (65536*4)
#define NBLK (100*(1<<20)/BLKSIZ)

int main()
{
	int fd[2];
	pid_t pid;
	int nonblock = 1;

	pipe(fd);

	if (!(pid = fork ())) {
		int i;
		char buf[BLKSIZ] = {0,};
		close(fd[0]);
		if (nonblock) {
			int flags;
			flags = fcntl(fd[1], F_GETFL);
			flags |= O_NONBLOCK;
			fcntl(fd[1], F_SETFL, flags);
		}
		fd_set wfds;
		for (int wlen=1024; wlen<=BLKSIZ; wlen *= 1.7) {
			FD_ZERO(&wfds);
			FD_SET(fd[1], &wfds);
			if (select(fd[1]+1, NULL, &wfds, NULL, NULL) > 0
					&& FD_ISSET(fd[1], &wfds)) {
				ssize_t len = write(fd[1], buf, wlen);
				printf("%d/%d\n", len, wlen);
				if (len < 0 && errno == EAGAIN) printf("w", i);
				if (len < 0 && errno == EAGAIN) i --;
			}
		}
		close(fd[1]);
	} else {
		char buf[BLKSIZ] = {0,};
		int total = 0;
		fd_set rfds;
		struct timespec tv0, tv1;
		double elasped;
		close(fd[1]);
		if (nonblock) {
			int flags;
			flags = fcntl(fd[0], F_GETFL);
			flags |= O_NONBLOCK;
			fcntl(fd[0], F_SETFL, flags);
		}
		usleep(1000000); /* Delay to start reader */
		clock_gettime(CLOCK_MONOTONIC, &tv0);
		for (;;) {
			FD_ZERO(&rfds);
			FD_SET(fd[0], &rfds);
			if (select(fd[0]+1, &rfds, NULL, NULL, NULL) > 0
					&& FD_ISSET(fd[0], &rfds)) {
				ssize_t len = read(fd[0], buf, sizeof(buf));
				if (len < 0 && errno == EAGAIN) printf("r");
				else if (len <= 0) break;
				else total += len;
			}
		}
		clock_gettime(CLOCK_MONOTONIC, &tv1);
		close(fd[0]);
		elasped = (tv1.tv_sec - tv0.tv_sec) + (tv1.tv_nsec - tv0.tv_nsec)*1e-9;
		printf("Total: %dKB in %f second, %fKB/s\n",
			total/(1<<10), elasped, total/(1<<10)/elasped);
		waitpid(pid, NULL, 0);
	}

	return 0;
}


^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-08-31 11:45                                                   ` Takashi Yano
@ 2021-08-31 12:31                                                     ` Takashi Yano
  2021-08-31 15:08                                                       ` Corinna Vinschen
  2021-08-31 12:33                                                     ` Ken Brown
  1 sibling, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-08-31 12:31 UTC (permalink / raw)
  To: cygwin-developers

On Tue, 31 Aug 2021 20:45:41 +0900
Takashi Yano wrote:
> On Tue, 31 Aug 2021 12:18:57 +0200
> Corinna Vinschen wrote:
> > On Aug 31 12:05, Corinna Vinschen wrote:
> > > On Aug 31 18:25, Takashi Yano wrote:
> > > > On Tue, 31 Aug 2021 11:08:42 +0200
> > > > Corinna Vinschenwrote:
> > > > > On Aug 31 17:55, Takashi Yano wrote:
> > > > > > On Mon, 30 Aug 2021 22:14:15 +0200
> > > > > > Corinna Vinschen wrote:
> > > > > > > Hi Ken, Hi Takashi,
> > > > > > > 
> > > > > > > On Aug 30 19:00, Corinna Vinschen wrote:
> > > > > > > Well, what about keeping a duplicate of the read side handle on the 
> > > > > > > write side just for calling NtQueryInformationFile?
> > > > > > > 
> > > > > > > Attached is an untested patch, can you have a look if that makes sense?
> > > > > > > 
> > > > > > > Btw., I think I found a bug in the new fhandler_pipe::create.  If the
> > > > > > > function fails to create the write side fhandler, it deletes the read
> > > > > > > side fhandler, but neglects to close the read handle.  My patch fixes
> > > > > > > that.
> > > > > > > 
> > > > > > > While looking into this I found a problem in fhandler_disk_file in
> > > > > > > terms of handle inheritance of the special handle for pread/pwrite.
> > > > > > > I already force pushed this onto topic/pipe.
> > > > > > 
> > > > > > I tested your patch attached. Unfortunately, select() does not work
> > > > > > as expected for write pipe. Even if the select reports write pipe
> > > > > > is available, writing to pipe fails. It seems that your patch fails
> > > > > > to detect pipe full.
> > > > > 
> > > > > Bummer.  Is that with byte mode pipes or with message mode pipes?  If
> > > > > the latter, if you try to write more data than available in the buffer,
> > > > > it's bound to fail.
> > > > 
> > > > Both message pipe and byte pipe.
> > > > 
> > > > > Did you add debug output to pipe_data_available to see how the
> > > > > information looks like?  Or do you have a simple, self-contained
> > > > > testcase in plain C?
> > > > 
> > > > The test case is attached. If select() works as expected, the program
> > > > does not show "r" or "w". However, with your patch, the program prints
> > > > many "w" (means write() fails with EAGAIN).
> > > 
> > > Thanks!  I found th culprit, but we have another problem.  Even if
> > > select returns correct info,  A write, trying to write more bytes
> > > than are available in the buffer, hangs.  This shouldn't happen.
> > > Still digging...
> > 
> > That's, of course, correct behaviour for pipes in blocking mode.  D'oh! 
> > 
> > Please try the attached patch on top of topic/pipe.
> 
> Thanks for the new patch. I have confirmed that above issue
> is fixed and select() for write pipe seems to work as expected.
> 
> 
> BTW, I found one minor difference between Linux and this pipe
> implementation.
> 
> The test case is attached. The test case uses non-bloking I/O.
> If this STC runs on Linux, the result is:
> 
> 1024/1024
> 1740/1740
> 2958/2958
> 5028/5028
> 8547/8547
> 14529/14529
> 24699/24699
> 41988/41988
> 22227/71379
> 65536/121344
> 65536/206284
> Total: 247KB in 0.000612 second, 403517.628166KB/s
> 
> On cygwin 3.2.0, the result is similar to Linux.
> 
> 1024/1024
> 1740/1740
> 2957/2957
> 5026/5026
> 8544/8544
> 14524/14524
> 24690/24690
> 41972/41972
> 65536/71352
> 65536/121298
> 65536/206206
> Total: 290KB in 0.062653 second, 4628.669018KB/s
> 
> 
> However, on topic/pipe implementation, the result is
> 
> 1024/1024
> 1740/1740
> 2957/2957
> 5026/5026
> 8544/8544
> 14524/14524
> 24690/24690
> -1/41972
> w-1/71352
> w-1/121298
> w-1/206206
> wTotal: 57KB in 0.000330 second, 172989.377845KB/s
> 
> In non-blocking mode, writing more than pipe space will fail with
> EAGAIN in this implementation.
> 
> In Linux and cygwin 3.2.0, it seems to write as much as writable.
> 
> Is this difficult to be fixed?

The following patch almost fixes the issue, but atomicity is the problem.

diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index 2dec0a848..0a74a654d 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -323,10 +323,15 @@ fhandler_pipe::raw_write (const void *ptr, size_t len)
   if (!len)
     return 0;

-  if (len <= max_atomic_write)
+  FILE_PIPE_LOCAL_INFORMATION fpli;
+  NtQueryInformationFile (query_hdl, &io, &fpli, sizeof (fpli),
+                         FilePipeLocalInformation);
+  ULONG room = fpli.InboundQuota - fpli.ReadDataAvailable;
+
+  if (len <= room)
     chunk = len;
   else if (is_nonblocking ())
-    chunk = len = max_atomic_write;
+    chunk = len = room;
   else
     chunk = max_atomic_write;


-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-08-31 11:45                                                   ` Takashi Yano
  2021-08-31 12:31                                                     ` Takashi Yano
@ 2021-08-31 12:33                                                     ` Ken Brown
  2021-08-31 15:18                                                       ` Corinna Vinschen
  1 sibling, 1 reply; 250+ messages in thread
From: Ken Brown @ 2021-08-31 12:33 UTC (permalink / raw)
  To: cygwin-developers

On 8/31/2021 7:45 AM, Takashi Yano wrote:
> On Tue, 31 Aug 2021 12:18:57 +0200
> Corinna Vinschen wrote:
>> On Aug 31 12:05, Corinna Vinschen wrote:
>>> On Aug 31 18:25, Takashi Yano wrote:
>>>> On Tue, 31 Aug 2021 11:08:42 +0200
>>>> Corinna Vinschenwrote:
>>>>> On Aug 31 17:55, Takashi Yano wrote:
>>>>>> On Mon, 30 Aug 2021 22:14:15 +0200
>>>>>> Corinna Vinschen wrote:
>>>>>>> Hi Ken, Hi Takashi,
>>>>>>>
>>>>>>> On Aug 30 19:00, Corinna Vinschen wrote:
>>>>>>> Well, what about keeping a duplicate of the read side handle on the
>>>>>>> write side just for calling NtQueryInformationFile?
>>>>>>>
>>>>>>> Attached is an untested patch, can you have a look if that makes sense?
>>>>>>>
>>>>>>> Btw., I think I found a bug in the new fhandler_pipe::create.  If the
>>>>>>> function fails to create the write side fhandler, it deletes the read
>>>>>>> side fhandler, but neglects to close the read handle.  My patch fixes
>>>>>>> that.
>>>>>>>
>>>>>>> While looking into this I found a problem in fhandler_disk_file in
>>>>>>> terms of handle inheritance of the special handle for pread/pwrite.
>>>>>>> I already force pushed this onto topic/pipe.
>>>>>>
>>>>>> I tested your patch attached. Unfortunately, select() does not work
>>>>>> as expected for write pipe. Even if the select reports write pipe
>>>>>> is available, writing to pipe fails. It seems that your patch fails
>>>>>> to detect pipe full.
>>>>>
>>>>> Bummer.  Is that with byte mode pipes or with message mode pipes?  If
>>>>> the latter, if you try to write more data than available in the buffer,
>>>>> it's bound to fail.
>>>>
>>>> Both message pipe and byte pipe.
>>>>
>>>>> Did you add debug output to pipe_data_available to see how the
>>>>> information looks like?  Or do you have a simple, self-contained
>>>>> testcase in plain C?
>>>>
>>>> The test case is attached. If select() works as expected, the program
>>>> does not show "r" or "w". However, with your patch, the program prints
>>>> many "w" (means write() fails with EAGAIN).
>>>
>>> Thanks!  I found th culprit, but we have another problem.  Even if
>>> select returns correct info,  A write, trying to write more bytes
>>> than are available in the buffer, hangs.  This shouldn't happen.
>>> Still digging...
>>
>> That's, of course, correct behaviour for pipes in blocking mode.  D'oh!
>>
>> Please try the attached patch on top of topic/pipe.
> 
> Thanks for the new patch. I have confirmed that above issue
> is fixed and select() for write pipe seems to work as expected.
> 
> 
> BTW, I found one minor difference between Linux and this pipe
> implementation.
> 
> The test case is attached. The test case uses non-bloking I/O.
> If this STC runs on Linux, the result is:
> 
> 1024/1024
> 1740/1740
> 2958/2958
> 5028/5028
> 8547/8547
> 14529/14529
> 24699/24699
> 41988/41988
> 22227/71379
> 65536/121344
> 65536/206284
> Total: 247KB in 0.000612 second, 403517.628166KB/s
> 
> On cygwin 3.2.0, the result is similar to Linux.
> 
> 1024/1024
> 1740/1740
> 2957/2957
> 5026/5026
> 8544/8544
> 14524/14524
> 24690/24690
> 41972/41972
> 65536/71352
> 65536/121298
> 65536/206206
> Total: 290KB in 0.062653 second, 4628.669018KB/s
> 
> 
> However, on topic/pipe implementation, the result is
> 
> 1024/1024
> 1740/1740
> 2957/2957
> 5026/5026
> 8544/8544
> 14524/14524
> 24690/24690
> -1/41972
> w-1/71352
> w-1/121298
> w-1/206206
> wTotal: 57KB in 0.000330 second, 172989.377845KB/s
> 
> In non-blocking mode, writing more than pipe space will fail with
> EAGAIN in this implementation.
> 
> In Linux and cygwin 3.2.0, it seems to write as much as writable.
> 
> Is this difficult to be fixed?
Two other remarks:

1. I think query_hdl needs to be initialized in the fhandler_pipe constructor.

2. When the read side of the pipe is non-blocking, there can be no pending 
reads, so shouldn't we be able to use WriteQuotaAvailable reliably on the write 
side?  (I can't test this at the moment.)  This applies in particular to the 
call to pipe_data_available at the end of peek_fifo, since all fifo readers use 
non-blocking pipes.  Maybe pipe_data-available needs an extra parameter so the 
caller can specify that WriteQuotaAvailable should be used.

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-08-31 12:31                                                     ` Takashi Yano
@ 2021-08-31 15:08                                                       ` Corinna Vinschen
  0 siblings, 0 replies; 250+ messages in thread
From: Corinna Vinschen @ 2021-08-31 15:08 UTC (permalink / raw)
  To: cygwin-developers

On Aug 31 21:31, Takashi Yano wrote:
> On Tue, 31 Aug 2021 20:45:41 +0900
> Takashi Yano wrote:
> > On Tue, 31 Aug 2021 12:18:57 +0200
> > Corinna Vinschen wrote:
> > > Please try the attached patch on top of topic/pipe.
> > 
> > Thanks for the new patch. I have confirmed that above issue
> > is fixed and select() for write pipe seems to work as expected.
> > 
> > 
> > BTW, I found one minor difference between Linux and this pipe
> > implementation.
> > 
> > The test case is attached. The test case uses non-bloking I/O.
> > If this STC runs on Linux, the result is:
> > 
> > 1024/1024
> > 1740/1740
> > 2958/2958
> > 5028/5028
> > 8547/8547
> > 14529/14529
> > 24699/24699
> > 41988/41988
> > 22227/71379
> > 65536/121344
> > 65536/206284
> > Total: 247KB in 0.000612 second, 403517.628166KB/s
> > 
> > On cygwin 3.2.0, the result is similar to Linux.
> > 
> > 1024/1024
> > 1740/1740
> > 2957/2957
> > 5026/5026
> > 8544/8544
> > 14524/14524
> > 24690/24690
> > 41972/41972
> > 65536/71352
> > 65536/121298
> > 65536/206206
> > Total: 290KB in 0.062653 second, 4628.669018KB/s
> > 
> > 
> > However, on topic/pipe implementation, the result is
> > 
> > 1024/1024
> > 1740/1740
> > 2957/2957
> > 5026/5026
> > 8544/8544
> > 14524/14524
> > 24690/24690
> > -1/41972
> > w-1/71352
> > w-1/121298
> > w-1/206206
> > wTotal: 57KB in 0.000330 second, 172989.377845KB/s
> > 
> > In non-blocking mode, writing more than pipe space will fail with
> > EAGAIN in this implementation.
> > 
> > In Linux and cygwin 3.2.0, it seems to write as much as writable.
> > 
> > Is this difficult to be fixed?
> 
> The following patch almost fixes the issue, but atomicity is the problem.

Thanks, I took the liberty to use your idea to implement a loop trying
to write again.  For me the output is now

  1024/1024
  1740/1740
  2958/2958
  5028/5028
  8547/8547
  14529/14529
  24699/24699
  7011/41988
  65536/71379
  65536/121344
  65536/206284
  Total: 256KB in 0.017771 second, 14405.248913KB/s

Could you try again with this patch?  I'm glad if we can straighten
out the bugs :)

diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index 2dec0a84817c..0aed8456bb0b 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -352,8 +352,30 @@ fhandler_pipe::raw_write (const void *ptr, size_t len)
       else
 	len1 = (ULONG) left;
       nbytes_now = 0;
-      status = NtWriteFile (get_handle (), evt, NULL, NULL, &io,
-			    (PVOID) ptr, len1, NULL, NULL);
+      while (true)
+	{
+	  status = NtWriteFile (get_handle (), evt, NULL, NULL, &io,
+				(PVOID) ptr, len1, NULL, NULL);
+	  if (evt || !NT_SUCCESS (status) || io.Information > 0)
+	    break;
+
+	  FILE_PIPE_LOCAL_INFORMATION fpli;
+	  IO_STATUS_BLOCK qio;
+
+	  if (!NT_SUCCESS (NtQueryInformationFile (query_hdl, &qio, &fpli,
+			   sizeof (fpli), FilePipeLocalInformation)))
+	    len1 >>= 1;
+	  else
+	    {
+	      fpli.WriteQuotaAvailable = fpli.InboundQuota
+					 - fpli.ReadDataAvailable;
+	      if (len1 > fpli.WriteQuotaAvailable
+		  && fpli.WriteQuotaAvailable > 0)
+		len1 = fpli.InboundQuota - fpli.ReadDataAvailable;
+	      else
+		break;
+	    }
+	}
       if (evt && status == STATUS_PENDING)
 	{
 	  waitret = cygwait (evt);
@@ -378,7 +400,7 @@ fhandler_pipe::raw_write (const void *ptr, size_t len)
 	  /* NtWriteFile returns success with # of bytes written == 0
 	     if writing on a non-blocking pipe fails because the pipe
 	     buffer doesn't have sufficient space. */
-	  if (nbytes_now == 0)
+	  if (nbytes_now == 0 && nbytes == 0)
 	    set_errno (EAGAIN);
 	  ptr = ((char *) ptr) + chunk;
 	  nbytes += nbytes_now;

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-08-31 12:33                                                     ` Ken Brown
@ 2021-08-31 15:18                                                       ` Corinna Vinschen
  2021-08-31 15:27                                                         ` Corinna Vinschen
  0 siblings, 1 reply; 250+ messages in thread
From: Corinna Vinschen @ 2021-08-31 15:18 UTC (permalink / raw)
  To: cygwin-developers

On Aug 31 08:33, Ken Brown wrote:
> On 8/31/2021 7:45 AM, Takashi Yano wrote:
> > On Tue, 31 Aug 2021 12:18:57 +0200
> > Corinna Vinschen wrote:
> > > Please try the attached patch on top of topic/pipe.
> > 
> > Thanks for the new patch. I have confirmed that above issue
> > is fixed and select() for write pipe seems to work as expected.
> > 
> > 
> > BTW, I found one minor difference between Linux and this pipe
> > implementation.
> > [...]
> > Is this difficult to be fixed?
> Two other remarks:
> 
> 1. I think query_hdl needs to be initialized in the fhandler_pipe constructor.

No, that's not necessary.  The fhandlers are always ccalloc'ed so they
are all 0 anyway.

> 2. When the read side of the pipe is non-blocking, there can be no pending
> reads, so shouldn't we be able to use WriteQuotaAvailable reliably on the
> write side?  (I can't test this at the moment.)

In theory, yes, but is it a safe bet that non-blocking reads won't change
WriteQuotaAvailable on the write side, at least for a very short time?
The question is, of course, if that really makes much of a difference.

> This applies in particular
> to the call to pipe_data_available at the end of peek_fifo, since all fifo
> readers use non-blocking pipes.  Maybe pipe_data-available needs an extra
> parameter so the caller can specify that WriteQuotaAvailable should be used.

I can fix up my patch to accommodate that.


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-08-31 11:05                                                 ` Takashi Yano
@ 2021-08-31 15:20                                                   ` Corinna Vinschen
  2021-09-01  2:39                                                     ` Takashi Yano
  0 siblings, 1 reply; 250+ messages in thread
From: Corinna Vinschen @ 2021-08-31 15:20 UTC (permalink / raw)
  To: cygwin-developers

On Aug 31 20:05, Takashi Yano wrote:
> On Tue, 31 Aug 2021 11:04:05 +0200
> Corinna Vinschen wrote:
> > On Aug 31 17:52, Takashi Yano wrote:
> > > On Mon, 30 Aug 2021 17:19:44 +0200
> > > Corinna Vinschen wrote:
> > > > On Aug 30 11:00, Ken Brown wrote:
> > > > > On 8/30/2021 9:51 AM, Ken Brown wrote:
> > > > > > On 8/30/2021 8:55 AM, Corinna Vinschen wrote:
> > > > > > > On Aug 30 21:04, Takashi Yano wrote:
> > > > > > > No worries.  The same should apply to the NtCreateFile side of the
> > > > > > > pipe, btw.
> > > > > > 
> > > > > > I'll add my thanks.  I should have checked the default flags that are
> > > > > > typically used for other devices when I wrote nt_create.  I'm glad you
> > > > > > caught this.
> > > > > > 
> > > > > > So I'll reinstate the use of nt_create and then let Takashi recheck everything.
> > > > > 
> > > > > I've done this now.  I'm still not sure I've got all the flags right.  For
> > > > > unknown reasons, I've used FILE_SHARE_READ | FILE_SHARE_WRITE in the call to
> > > > > NtCreateNamedPipeFile, and no sharing in the call to NtOpenFile.  Should I
> > > > > also use FILE_SHARE_READ | FILE_SHARE in NtOpenFile?  Is sharing even
> > > > > relevant in this context?
> > > > 
> > > > This is only relevant if you want to open the pipe from another context,
> > > > calling CreateNamedPipe/CreateFile.  As long as the pipe is only
> > > > duplicated, it shouldn't matter at all.
> > > > 
> > > > But, as I just wrote in my previous mail, the FILE_SYNCHRONOUS_IO_NONALERT
> > > > flag is probably a good thing for C# apps, but not for Cygwin, because it
> > > > enforces synchronous operation.  Sorry about that...
> > > 
> > > With FILE_SYNCHRONOUS_IO_NONALERT, what kind of problems are you
> > > specifically concerned about cygwin pipe? 
> > 
> > We're using asynchronous IO to be able to call WFMO and thus to be able
> > to handle signals and thread cancellation events.  Wit hsynchronous IO
> > this is not possible.
> 
> Thanks. How can I regenerate above issue? Stopping by Ctrl-C or killing
> the process by kill seems to work even with FILE_SYNCHRONOUS_IO_NONALERT.

It may depend on the thread you're running this in.  But really, just
call a blocking (SYNCHRONIZE + FILE_SYNCHRONOUS_IO_NONALERT) ReadFile
in the main thread of a Cygwin app, and you'll see that neither Ctrl-C
nor kill signalling will get through.

> Where is the WFMO called for pipe handle?

The cygwait function.


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-08-31 15:18                                                       ` Corinna Vinschen
@ 2021-08-31 15:27                                                         ` Corinna Vinschen
  2021-08-31 15:50                                                           ` Corinna Vinschen
  0 siblings, 1 reply; 250+ messages in thread
From: Corinna Vinschen @ 2021-08-31 15:27 UTC (permalink / raw)
  To: cygwin-developers

On Aug 31 17:18, Corinna Vinschen wrote:
> On Aug 31 08:33, Ken Brown wrote:
> > On 8/31/2021 7:45 AM, Takashi Yano wrote:
> > > On Tue, 31 Aug 2021 12:18:57 +0200
> > > Corinna Vinschen wrote:
> > > > Please try the attached patch on top of topic/pipe.
> > > 
> > > Thanks for the new patch. I have confirmed that above issue
> > > is fixed and select() for write pipe seems to work as expected.
> > > 
> > > 
> > > BTW, I found one minor difference between Linux and this pipe
> > > implementation.
> > > [...]
> > > Is this difficult to be fixed?
> > Two other remarks:
> > 
> > 1. I think query_hdl needs to be initialized in the fhandler_pipe constructor.
> 
> No, that's not necessary.  The fhandlers are always ccalloc'ed so they
> are all 0 anyway.
> 
> > 2. When the read side of the pipe is non-blocking, there can be no pending
> > reads, so shouldn't we be able to use WriteQuotaAvailable reliably on the
> > write side?  (I can't test this at the moment.)
> 
> In theory, yes, but is it a safe bet that non-blocking reads won't change
> WriteQuotaAvailable on the write side, at least for a very short time?
> The question is, of course, if that really makes much of a difference.

Oh, btw... why do you want to use WriteQuotaAvailable for normal
pipes, even though the read side information is available anyway?

We can do that for fifos, no problem, but it doesn't make much sense
to differ between blocking and non-blocking pipes, the code flow is the
same.


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-08-31 15:27                                                         ` Corinna Vinschen
@ 2021-08-31 15:50                                                           ` Corinna Vinschen
  2021-08-31 16:19                                                             ` Ken Brown
  2021-08-31 23:02                                                             ` Takashi Yano
  0 siblings, 2 replies; 250+ messages in thread
From: Corinna Vinschen @ 2021-08-31 15:50 UTC (permalink / raw)
  To: cygwin-developers

[-- Attachment #1: Type: text/plain, Size: 1933 bytes --]

On Aug 31 17:27, Corinna Vinschen wrote:
> On Aug 31 17:18, Corinna Vinschen wrote:
> > On Aug 31 08:33, Ken Brown wrote:
> > > On 8/31/2021 7:45 AM, Takashi Yano wrote:
> > > > On Tue, 31 Aug 2021 12:18:57 +0200
> > > > Corinna Vinschen wrote:
> > > > > Please try the attached patch on top of topic/pipe.
> > > > 
> > > > Thanks for the new patch. I have confirmed that above issue
> > > > is fixed and select() for write pipe seems to work as expected.
> > > > 
> > > > 
> > > > BTW, I found one minor difference between Linux and this pipe
> > > > implementation.
> > > > [...]
> > > > Is this difficult to be fixed?
> > > Two other remarks:
> > > 
> > > 1. I think query_hdl needs to be initialized in the fhandler_pipe constructor.
> > 
> > No, that's not necessary.  The fhandlers are always ccalloc'ed so they
> > are all 0 anyway.
> > 
> > > 2. When the read side of the pipe is non-blocking, there can be no pending
> > > reads, so shouldn't we be able to use WriteQuotaAvailable reliably on the
> > > write side?  (I can't test this at the moment.)
> > 
> > In theory, yes, but is it a safe bet that non-blocking reads won't change
> > WriteQuotaAvailable on the write side, at least for a very short time?
> > The question is, of course, if that really makes much of a difference.
> 
> Oh, btw... why do you want to use WriteQuotaAvailable for normal
> pipes, even though the read side information is available anyway?
> 
> We can do that for fifos, no problem, but it doesn't make much sense
> to differ between blocking and non-blocking pipes, the code flow is the
> same.

So for the time being I suggest the below patch on top of topic/pipe.
It contains everything we discussed so far.

One question left is, do we want to switch to FILE_PIPE_BYTE_STREAM_TYPE
entirely for pipes?  I don't see that it's still necessary to use
FILE_PIPE_MESSAGE_TYPE for pipes.  Everything seems to work normally
with byte-type pipes.

[-- Attachment #2: pipe.diff --]
[-- Type: text/plain, Size: 10392 bytes --]

diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
index 132e6002133b..1f0f28077a7c 100644
--- a/winsup/cygwin/fhandler.h
+++ b/winsup/cygwin/fhandler.h
@@ -1171,6 +1171,7 @@ class fhandler_socket_unix : public fhandler_socket
 class fhandler_pipe: public fhandler_base
 {
 private:
+  HANDLE query_hdl;
   pid_t popen_pid;
   size_t max_atomic_write;
   void set_pipe_non_blocking (bool nonblocking);
@@ -1179,6 +1180,8 @@ public:
 
   bool ispipe() const { return true; }
 
+  HANDLE get_query_handle () const { return query_hdl; }
+
   void set_popen_pid (pid_t pid) {popen_pid = pid;}
   pid_t get_popen_pid () const {return popen_pid;}
   off_t lseek (off_t offset, int whence);
@@ -1187,7 +1190,9 @@ public:
   select_record *select_except (select_stuff *);
   char *get_proc_fd_name (char *buf);
   int open (int flags, mode_t mode = 0);
+  void fixup_after_fork (HANDLE);
   int dup (fhandler_base *child, int);
+  int close ();
   void __reg3 raw_read (void *ptr, size_t& len);
   ssize_t __reg3 raw_write (const void *ptr, size_t len);
   int ioctl (unsigned int cmd, void *);
diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index 2dec0a84817c..2d9e87bb3450 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -352,8 +352,30 @@ fhandler_pipe::raw_write (const void *ptr, size_t len)
       else
 	len1 = (ULONG) left;
       nbytes_now = 0;
-      status = NtWriteFile (get_handle (), evt, NULL, NULL, &io,
-			    (PVOID) ptr, len1, NULL, NULL);
+      while (true)
+	{
+	  status = NtWriteFile (get_handle (), evt, NULL, NULL, &io,
+				(PVOID) ptr, len1, NULL, NULL);
+	  if (evt || !NT_SUCCESS (status) || io.Information > 0)
+	    break;
+
+	  FILE_PIPE_LOCAL_INFORMATION fpli;
+	  IO_STATUS_BLOCK qio;
+
+	  if (!NT_SUCCESS (NtQueryInformationFile (query_hdl, &qio, &fpli,
+			   sizeof (fpli), FilePipeLocalInformation)))
+	    len1 >>= 1;
+	  else
+	    {
+	      fpli.WriteQuotaAvailable = fpli.InboundQuota
+					 - fpli.ReadDataAvailable;
+	      if (len1 > fpli.WriteQuotaAvailable
+		  && fpli.WriteQuotaAvailable > 0)
+		len1 = fpli.InboundQuota - fpli.ReadDataAvailable;
+	      else
+		break;
+	    }
+	}
       if (evt && status == STATUS_PENDING)
 	{
 	  waitret = cygwait (evt);
@@ -378,7 +400,7 @@ fhandler_pipe::raw_write (const void *ptr, size_t len)
 	  /* NtWriteFile returns success with # of bytes written == 0
 	     if writing on a non-blocking pipe fails because the pipe
 	     buffer doesn't have sufficient space. */
-	  if (nbytes_now == 0)
+	  if (nbytes_now == 0 && nbytes == 0)
 	    set_errno (EAGAIN);
 	  ptr = ((char *) ptr) + chunk;
 	  nbytes += nbytes_now;
@@ -405,22 +427,45 @@ fhandler_pipe::raw_write (const void *ptr, size_t len)
   return ret;
 }
 
+void
+fhandler_pipe::fixup_after_fork (HANDLE parent)
+{
+  if (query_hdl)
+    fork_fixup (parent, query_hdl, "query_hdl");
+  fhandler_base::fixup_after_fork (parent);
+}
+
 int
 fhandler_pipe::dup (fhandler_base *child, int flags)
 {
   fhandler_pipe *ftp = (fhandler_pipe *) child;
   ftp->set_popen_pid (0);
 
-  int res;
-  if (get_handle () && fhandler_base::dup (child, flags))
+  int res = 0;
+  if (fhandler_base::dup (child, flags))
     res = -1;
-  else
-    res = 0;
+  else if (query_hdl &&
+	   !DuplicateHandle (GetCurrentProcess (), query_hdl,
+			     GetCurrentProcess (), &ftp->query_hdl,
+			     0, !(flags & O_CLOEXEC), DUPLICATE_SAME_ACCESS))
+    {
+      __seterrno ();
+      ftp->close ();
+      res = -1;
+    }
 
   debug_printf ("res %d", res);
   return res;
 }
 
+int
+fhandler_pipe::close ()
+{
+  if (query_hdl)
+    NtClose (query_hdl);
+  return fhandler_base::close ();
+}
+
 #define PIPE_INTRO "\\\\.\\pipe\\cygwin-"
 
 /* Create a pipe, and return handles to the read and write ends,
@@ -608,6 +653,7 @@ fhandler_pipe::create (fhandler_pipe *fhs[2], unsigned psize, int mode)
   else if ((fhs[1] = (fhandler_pipe *) build_fh_dev (*pipew_dev)) == NULL)
     {
       delete fhs[0];
+      CloseHandle (r);
       CloseHandle (w);
     }
   else
@@ -617,10 +663,23 @@ fhandler_pipe::create (fhandler_pipe *fhs[2], unsigned psize, int mode)
 		    unique_id);
       fhs[1]->init (w, FILE_CREATE_PIPE_INSTANCE | GENERIC_WRITE, mode,
 		    unique_id);
-      res = 0;
+      /* For the write side of the pipe, duplicate the handle to the read side
+	 into query_hdl just for calling NtQueryInformationFile.  See longish
+	 comment in select.cc, pipe_data_available() for the reasoning. */
+      if (!DuplicateHandle (GetCurrentProcess (), r, GetCurrentProcess (),
+			    &fhs[1]->query_hdl, GENERIC_READ,
+			    !(mode & O_CLOEXEC), 0))
+	{
+	  delete fhs[0];
+	  CloseHandle (r);
+	  delete fhs[1];
+	  CloseHandle (w);
+	}
+      else
+	res = 0;
     }
 
-  debug_printf ("%R = pipe([%p, %p], %d, %y)", res, fhs[0], fhs[1], psize, mode);
+  debug_printf ("%R = pipe(%d, %y)", res, psize, mode);
   return res;
 }
 
@@ -658,10 +717,10 @@ nt_create (LPSECURITY_ATTRIBUTES sa_ptr, PHANDLE r, PHANDLE w,
 				 &cygheap->installation_key,
 				 GetCurrentProcessId ());
 
-  access = GENERIC_READ | FILE_WRITE_ATTRIBUTES;
+  access = GENERIC_READ | FILE_WRITE_ATTRIBUTES | SYNCHRONIZE;
 
   ULONG pipe_type = pipe_byte ? FILE_PIPE_BYTE_STREAM_TYPE
-    : FILE_PIPE_MESSAGE_TYPE;
+			      : FILE_PIPE_MESSAGE_TYPE;
 
   /* Retry NtCreateNamedPipeFile as long as the pipe name is in use.
      Retrying will probably never be necessary, but we want
@@ -737,7 +796,7 @@ nt_create (LPSECURITY_ATTRIBUTES sa_ptr, PHANDLE r, PHANDLE w,
     {
       debug_printf ("NtOpenFile: name %S", &pipename);
 
-      access = GENERIC_WRITE | FILE_READ_ATTRIBUTES;
+      access = GENERIC_WRITE | FILE_READ_ATTRIBUTES | SYNCHRONIZE;
       status = NtOpenFile (w, access, &attr, &io, 0, 0);
       if (!NT_SUCCESS (status))
 	{
diff --git a/winsup/cygwin/select.cc b/winsup/cygwin/select.cc
index 83e1c00e0ac7..dc1f7961351b 100644
--- a/winsup/cygwin/select.cc
+++ b/winsup/cygwin/select.cc
@@ -585,7 +585,8 @@ no_verify (select_record *, fd_set *, fd_set *, fd_set *)
 }
 
 static int
-pipe_data_available (int fd, fhandler_base *fh, HANDLE h, bool writing)
+pipe_data_available (int fd, fhandler_base *fh, HANDLE h, bool writing,
+		     bool use_readside)
 {
   IO_STATUS_BLOCK iosb = {{0}, 0};
   FILE_PIPE_LOCAL_INFORMATION fpli = {0};
@@ -608,15 +609,34 @@ pipe_data_available (int fd, fhandler_base *fh, HANDLE h, bool writing)
     }
   if (writing)
     {
-	/* If there is anything available in the pipe buffer then signal
-	   that.  This means that a pipe could still block since you could
-	   be trying to write more to the pipe than is available in the
-	   buffer but that is the hazard of select().  */
-      fpli.WriteQuotaAvailable = fpli.OutboundQuota - fpli.ReadDataAvailable;
+      /* If there is anything available in the pipe buffer then signal
+	 that.  This means that a pipe could still block since you could
+	 be trying to write more to the pipe than is available in the
+	 buffer but that is the hazard of select().
+
+	 Note that WriteQuotaAvailable is unreliable.
+
+	 Usually WriteQuotaAvailable on the write side reflects the space
+	 available in the inbound buffer on the read side.  However, if a
+	 pipe read is currently pending, WriteQuotaAvailable on the write side
+	 is decremented by the number of bytes the read side is requesting.
+	 So it's possible (even likely) that WriteQuotaAvailable is 0, even
+	 if the inbound buffer on the read side is not full.  This can lead to
+	 a deadlock situation: The reader is waiting for data, but select
+	 on the writer side assumes that no space is available in the read
+	 side inbound buffer.
+
+	 Consequentially, the only reliable information is available on the
+	 read side, so fetch info from the read side via the pipe-specific
+	 query handle.  Use fpli.WriteQuotaAvailable as storage for the actual
+	 interesting value, which is the InboundQuote on the read side,
+	 decremented by the number of bytes of data in that buffer. */
+      if (use_readside)
+	fpli.WriteQuotaAvailable = fpli.InboundQuota - fpli.ReadDataAvailable;
       if (fpli.WriteQuotaAvailable > 0)
 	{
 	  paranoid_printf ("fd %d, %s, write: size %u, avail %u", fd,
-			   fh->get_name (), fpli.OutboundQuota,
+			   fh->get_name (), fpli.InboundQuota,
 			   fpli.WriteQuotaAvailable);
 	  return 1;
 	}
@@ -684,10 +704,11 @@ peek_pipe (select_record *s, bool from_select)
 	  gotone = s->read_ready = true;
 	  goto out;
 	}
-      int n = pipe_data_available (s->fd, fh, h, false);
+      int n = pipe_data_available (s->fd, fh, h, false, false);
       /* On PTY masters, check if input from the echo pipe is available. */
       if (n == 0 && fh->get_echo_handle ())
-	n = pipe_data_available (s->fd, fh, fh->get_echo_handle (), false);
+	n = pipe_data_available (s->fd, fh, fh->get_echo_handle (), false,
+				 false);
 
       if (n < 0)
 	{
@@ -718,10 +739,16 @@ out:
       fhandler_pty_master *fhm = (fhandler_pty_master *) fh;
       fhm->set_mask_flusho (s->read_ready);
     }
-  h = fh->get_output_handle ();
   if (s->write_selected && dev != FH_PIPER)
     {
-      gotone += s->write_ready =  pipe_data_available (s->fd, fh, h, true);
+      /* For the write side of a pipe, fetch the handle to the read side.
+	 See the longish comment in pipe_data_available for the reasoning. */
+      if (dev == FH_PIPEW)
+	h = ((fhandler_pipe *) fh)->get_query_handle ();
+      else
+	h = fh->get_output_handle ();
+      gotone += s->write_ready = pipe_data_available (s->fd, fh, h, true,
+						      dev == FH_PIPEW);
       select_printf ("write: %s, gotone %d", fh->get_name (), gotone);
     }
   return gotone;
@@ -922,7 +949,7 @@ out:
   if (s->write_selected)
     {
       gotone += s->write_ready
-	= pipe_data_available (s->fd, fh, fh->get_handle (), true);
+	= pipe_data_available (s->fd, fh, fh->get_handle (), true, false);
       select_printf ("write: %s, gotone %d", fh->get_name (), gotone);
     }
   return gotone;
@@ -1368,7 +1395,8 @@ out:
   HANDLE h = ptys->get_output_handle ();
   if (s->write_selected)
     {
-      gotone += s->write_ready =  pipe_data_available (s->fd, fh, h, true);
+      gotone += s->write_ready =  pipe_data_available (s->fd, fh, h, true,
+						       false);
       select_printf ("write: %s, gotone %d", fh->get_name (), gotone);
     }
   return gotone;

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-08-31 15:50                                                           ` Corinna Vinschen
@ 2021-08-31 16:19                                                             ` Ken Brown
  2021-08-31 16:38                                                               ` Ken Brown
  2021-08-31 23:02                                                             ` Takashi Yano
  1 sibling, 1 reply; 250+ messages in thread
From: Ken Brown @ 2021-08-31 16:19 UTC (permalink / raw)
  To: cygwin-developers

On 8/31/2021 11:50 AM, Corinna Vinschen wrote:
> On Aug 31 17:27, Corinna Vinschen wrote:
>> On Aug 31 17:18, Corinna Vinschen wrote:
>>> On Aug 31 08:33, Ken Brown wrote:
>>>> On 8/31/2021 7:45 AM, Takashi Yano wrote:
>>>>> On Tue, 31 Aug 2021 12:18:57 +0200
>>>>> Corinna Vinschen wrote:
>>>>>> Please try the attached patch on top of topic/pipe.
>>>>>
>>>>> Thanks for the new patch. I have confirmed that above issue
>>>>> is fixed and select() for write pipe seems to work as expected.
>>>>>
>>>>>
>>>>> BTW, I found one minor difference between Linux and this pipe
>>>>> implementation.
>>>>> [...]
>>>>> Is this difficult to be fixed?
>>>> Two other remarks:
>>>>
>>>> 1. I think query_hdl needs to be initialized in the fhandler_pipe constructor.
>>>
>>> No, that's not necessary.  The fhandlers are always ccalloc'ed so they
>>> are all 0 anyway.
>>>
>>>> 2. When the read side of the pipe is non-blocking, there can be no pending
>>>> reads, so shouldn't we be able to use WriteQuotaAvailable reliably on the
>>>> write side?  (I can't test this at the moment.)
>>>
>>> In theory, yes, but is it a safe bet that non-blocking reads won't change
>>> WriteQuotaAvailable on the write side, at least for a very short time?
>>> The question is, of course, if that really makes much of a difference.
>>
>> Oh, btw... why do you want to use WriteQuotaAvailable for normal
>> pipes, even though the read side information is available anyway?
>>
>> We can do that for fifos, no problem, but it doesn't make much sense
>> to differ between blocking and non-blocking pipes, the code flow is the
>> same.

Agreed.  It was mainly the fifo case that I was concerned about, and your way of 
handling that is much better than what I suggested.  [I also wondered about the 
pty case, but I see you've dealt with that in your latest patch.]

> So for the time being I suggest the below patch on top of topic/pipe.
> It contains everything we discussed so far.
> 
> One question left is, do we want to switch to FILE_PIPE_BYTE_STREAM_TYPE
> entirely for pipes?  I don't see that it's still necessary to use
> FILE_PIPE_MESSAGE_TYPE for pipes.  Everything seems to work normally
> with byte-type pipes.

Since no one remembers why we're defaulting to FILE_PIPE_MESSAGE_TYPE, I agree. 
  If a problem shows up, we can always rethink it.  I suggest that we still 
retain the CYGWIN option, at least for a while, in case someone encounters a 
problem and wants to switch back to message type for testing.

I'm afraid I still haven't had a chance to do any testing of your patch, but I 
expect to be able to do that later today.

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-08-31 16:19                                                             ` Ken Brown
@ 2021-08-31 16:38                                                               ` Ken Brown
  2021-08-31 17:30                                                                 ` Corinna Vinschen
  0 siblings, 1 reply; 250+ messages in thread
From: Ken Brown @ 2021-08-31 16:38 UTC (permalink / raw)
  To: cygwin-developers

On 8/31/2021 12:19 PM, Ken Brown wrote:
> On 8/31/2021 11:50 AM, Corinna Vinschen wrote:
>> On Aug 31 17:27, Corinna Vinschen wrote:
>>> On Aug 31 17:18, Corinna Vinschen wrote:
>>>> On Aug 31 08:33, Ken Brown wrote:
>>>>> On 8/31/2021 7:45 AM, Takashi Yano wrote:
>>>>>> On Tue, 31 Aug 2021 12:18:57 +0200
>>>>>> Corinna Vinschen wrote:
>>>>>>> Please try the attached patch on top of topic/pipe.
>>>>>>
>>>>>> Thanks for the new patch. I have confirmed that above issue
>>>>>> is fixed and select() for write pipe seems to work as expected.
>>>>>>
>>>>>>
>>>>>> BTW, I found one minor difference between Linux and this pipe
>>>>>> implementation.
>>>>>> [...]
>>>>>> Is this difficult to be fixed?
>>>>> Two other remarks:
>>>>>
>>>>> 1. I think query_hdl needs to be initialized in the fhandler_pipe constructor.
>>>>
>>>> No, that's not necessary.  The fhandlers are always ccalloc'ed so they
>>>> are all 0 anyway.
>>>>
>>>>> 2. When the read side of the pipe is non-blocking, there can be no pending
>>>>> reads, so shouldn't we be able to use WriteQuotaAvailable reliably on the
>>>>> write side?  (I can't test this at the moment.)
>>>>
>>>> In theory, yes, but is it a safe bet that non-blocking reads won't change
>>>> WriteQuotaAvailable on the write side, at least for a very short time?
>>>> The question is, of course, if that really makes much of a difference.
>>>
>>> Oh, btw... why do you want to use WriteQuotaAvailable for normal
>>> pipes, even though the read side information is available anyway?
>>>
>>> We can do that for fifos, no problem, but it doesn't make much sense
>>> to differ between blocking and non-blocking pipes, the code flow is the
>>> same.
> 
> Agreed.  It was mainly the fifo case that I was concerned about, and your way of 
> handling that is much better than what I suggested.  [I also wondered about the 
> pty case, but I see you've dealt with that in your latest patch.]
> 
>> So for the time being I suggest the below patch on top of topic/pipe.
>> It contains everything we discussed so far.
>>
>> One question left is, do we want to switch to FILE_PIPE_BYTE_STREAM_TYPE
>> entirely for pipes?  I don't see that it's still necessary to use
>> FILE_PIPE_MESSAGE_TYPE for pipes.  Everything seems to work normally
>> with byte-type pipes.
> 
> Since no one remembers why we're defaulting to FILE_PIPE_MESSAGE_TYPE, I agree. 
>   If a problem shows up, we can always rethink it.  I suggest that we still 
> retain the CYGWIN option, at least for a while, in case someone encounters a 
> problem and wants to switch back to message type for testing.
> 
> I'm afraid I still haven't had a chance to do any testing of your patch, but I 
> expect to be able to do that later today.

And here's a really trivial comment about your patch to raw_write: Where you have

   len1 = fpli.InboundQuota - fpli.ReadDataAvailable;

I think the code would be slightly clearer if you wrote

   len1 = fpli.WriteQuotaAvailable;

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-08-31 16:38                                                               ` Ken Brown
@ 2021-08-31 17:30                                                                 ` Corinna Vinschen
  2021-08-31 18:54                                                                   ` Ken Brown
  0 siblings, 1 reply; 250+ messages in thread
From: Corinna Vinschen @ 2021-08-31 17:30 UTC (permalink / raw)
  To: cygwin-developers

On Aug 31 12:38, Ken Brown wrote:
> And here's a really trivial comment about your patch to raw_write: Where you have
> 
>   len1 = fpli.InboundQuota - fpli.ReadDataAvailable;
> 
> I think the code would be slightly clearer if you wrote
> 
>   len1 = fpli.WriteQuotaAvailable;

D'oh!  That was the idea.  Aparently I forgot it in mid-air...


Thanks,
Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-08-31 17:30                                                                 ` Corinna Vinschen
@ 2021-08-31 18:54                                                                   ` Ken Brown
  2021-08-31 19:51                                                                     ` Corinna Vinschen
  0 siblings, 1 reply; 250+ messages in thread
From: Ken Brown @ 2021-08-31 18:54 UTC (permalink / raw)
  To: cygwin-developers

On 8/31/2021 1:30 PM, Corinna Vinschen wrote:
> On Aug 31 12:38, Ken Brown wrote:
>> And here's a really trivial comment about your patch to raw_write: Where you have
>>
>>    len1 = fpli.InboundQuota - fpli.ReadDataAvailable;
>>
>> I think the code would be slightly clearer if you wrote
>>
>>    len1 = fpli.WriteQuotaAvailable;
> 
> D'oh!  That was the idea.  Aparently I forgot it in mid-air...

One more thing.  For a non-blocking write, according to POSIX, "A write request 
for {PIPE_BUF} or fewer bytes shall have the following effect: if there is 
sufficient space available in the pipe, write() shall transfer all the data and 
return the number of bytes requested. Otherwise, write() shall transfer no data 
and return -1 with errno set to [EAGAIN]."

So I think the condition for breaking from the retry loop has to be changed from

   evt || !NT_SUCCESS (status) || io.Information > 0

to

   evt || !NT_SUCCESS (status) || io.Information > 0 || len <= PIPE_BUF

And I wonder if we've now uncovered a reason for using message mode: If the pipe 
was created in byte mode, might we get a partial write when len <= PIPE_BUF?  I 
see the following under "Pipes" at

   https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-writefile

"When writing to a non-blocking, byte-mode pipe handle with insufficient buffer 
space, WriteFile returns TRUE with *lpNumberOfBytesWritten < nNumberOfBytesToWrite."

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-08-31 18:54                                                                   ` Ken Brown
@ 2021-08-31 19:51                                                                     ` Corinna Vinschen
  0 siblings, 0 replies; 250+ messages in thread
From: Corinna Vinschen @ 2021-08-31 19:51 UTC (permalink / raw)
  To: cygwin-developers

On Aug 31 14:54, Ken Brown wrote:
> On 8/31/2021 1:30 PM, Corinna Vinschen wrote:
> > On Aug 31 12:38, Ken Brown wrote:
> > > And here's a really trivial comment about your patch to raw_write: Where you have
> > > 
> > >    len1 = fpli.InboundQuota - fpli.ReadDataAvailable;
> > > 
> > > I think the code would be slightly clearer if you wrote
> > > 
> > >    len1 = fpli.WriteQuotaAvailable;
> > 
> > D'oh!  That was the idea.  Aparently I forgot it in mid-air...
> 
> One more thing.  For a non-blocking write, according to POSIX, "A write
> request for {PIPE_BUF} or fewer bytes shall have the following effect: if
> there is sufficient space available in the pipe, write() shall transfer all
> the data and return the number of bytes requested. Otherwise, write() shall
> transfer no data and return -1 with errno set to [EAGAIN]."
> 
> So I think the condition for breaking from the retry loop has to be changed from
> 
>   evt || !NT_SUCCESS (status) || io.Information > 0
> 
> to
> 
>   evt || !NT_SUCCESS (status) || io.Information > 0 || len <= PIPE_BUF

Hmm.  I wonder if we shouldn't untangle the raw_write code and handle
blocking and non-blocking writes in two different branches of an if.
That should make things much clearer, shouldn't it?

> And I wonder if we've now uncovered a reason for using message mode: If the
> pipe was created in byte mode, might we get a partial write when len <=
> PIPE_BUF?  I see the following under "Pipes" at
> 
>   https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-writefile
> 
> "When writing to a non-blocking, byte-mode pipe handle with insufficient
> buffer space, WriteFile returns TRUE with *lpNumberOfBytesWritten <
> nNumberOfBytesToWrite."

Good point.


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-08-31 15:50                                                           ` Corinna Vinschen
  2021-08-31 16:19                                                             ` Ken Brown
@ 2021-08-31 23:02                                                             ` Takashi Yano
  2021-09-01  0:16                                                               ` Takashi Yano
  1 sibling, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-08-31 23:02 UTC (permalink / raw)
  To: cygwin-developers

On Tue, 31 Aug 2021 17:50:14 +0200
Corinna Vinschen wrote:
> On Aug 31 17:27, Corinna Vinschen wrote:
> > On Aug 31 17:18, Corinna Vinschen wrote:
> > > On Aug 31 08:33, Ken Brown wrote:
> > > > On 8/31/2021 7:45 AM, Takashi Yano wrote:
> > > > > On Tue, 31 Aug 2021 12:18:57 +0200
> > > > > Corinna Vinschen wrote:
> > > > > > Please try the attached patch on top of topic/pipe.
> > > > > 
> > > > > Thanks for the new patch. I have confirmed that above issue
> > > > > is fixed and select() for write pipe seems to work as expected.
> > > > > 
> > > > > 
> > > > > BTW, I found one minor difference between Linux and this pipe
> > > > > implementation.
> > > > > [...]
> > > > > Is this difficult to be fixed?
> > > > Two other remarks:
> > > > 
> > > > 1. I think query_hdl needs to be initialized in the fhandler_pipe constructor.
> > > 
> > > No, that's not necessary.  The fhandlers are always ccalloc'ed so they
> > > are all 0 anyway.
> > > 
> > > > 2. When the read side of the pipe is non-blocking, there can be no pending
> > > > reads, so shouldn't we be able to use WriteQuotaAvailable reliably on the
> > > > write side?  (I can't test this at the moment.)
> > > 
> > > In theory, yes, but is it a safe bet that non-blocking reads won't change
> > > WriteQuotaAvailable on the write side, at least for a very short time?
> > > The question is, of course, if that really makes much of a difference.
> > 
> > Oh, btw... why do you want to use WriteQuotaAvailable for normal
> > pipes, even though the read side information is available anyway?
> > 
> > We can do that for fifos, no problem, but it doesn't make much sense
> > to differ between blocking and non-blocking pipes, the code flow is the
> > same.
> 
> So for the time being I suggest the below patch on top of topic/pipe.
> It contains everything we discussed so far.

One more thing. 'git log' cannot stop normally with 'q' with your patch.

> One question left is, do we want to switch to FILE_PIPE_BYTE_STREAM_TYPE
> entirely for pipes?  I don't see that it's still necessary to use
> FILE_PIPE_MESSAGE_TYPE for pipes.  Everything seems to work normally
> with byte-type pipes.

Byte pipe seems to work for me too.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-08-31 23:02                                                             ` Takashi Yano
@ 2021-09-01  0:16                                                               ` Takashi Yano
  2021-09-01  8:07                                                                 ` Corinna Vinschen
  0 siblings, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-01  0:16 UTC (permalink / raw)
  To: cygwin-developers

On Wed, 1 Sep 2021 08:02:20 +0900
Takashi Yano wrote:
> On Tue, 31 Aug 2021 17:50:14 +0200
> Corinna Vinschen wrote:
> > On Aug 31 17:27, Corinna Vinschen wrote:
> > > On Aug 31 17:18, Corinna Vinschen wrote:
> > > > On Aug 31 08:33, Ken Brown wrote:
> > > > > On 8/31/2021 7:45 AM, Takashi Yano wrote:
> > > > > > On Tue, 31 Aug 2021 12:18:57 +0200
> > > > > > Corinna Vinschen wrote:
> > > > > > > Please try the attached patch on top of topic/pipe.
> > > > > > 
> > > > > > Thanks for the new patch. I have confirmed that above issue
> > > > > > is fixed and select() for write pipe seems to work as expected.
> > > > > > 
> > > > > > 
> > > > > > BTW, I found one minor difference between Linux and this pipe
> > > > > > implementation.
> > > > > > [...]
> > > > > > Is this difficult to be fixed?
> > > > > Two other remarks:
> > > > > 
> > > > > 1. I think query_hdl needs to be initialized in the fhandler_pipe constructor.
> > > > 
> > > > No, that's not necessary.  The fhandlers are always ccalloc'ed so they
> > > > are all 0 anyway.
> > > > 
> > > > > 2. When the read side of the pipe is non-blocking, there can be no pending
> > > > > reads, so shouldn't we be able to use WriteQuotaAvailable reliably on the
> > > > > write side?  (I can't test this at the moment.)
> > > > 
> > > > In theory, yes, but is it a safe bet that non-blocking reads won't change
> > > > WriteQuotaAvailable on the write side, at least for a very short time?
> > > > The question is, of course, if that really makes much of a difference.
> > > 
> > > Oh, btw... why do you want to use WriteQuotaAvailable for normal
> > > pipes, even though the read side information is available anyway?
> > > 
> > > We can do that for fifos, no problem, but it doesn't make much sense
> > > to differ between blocking and non-blocking pipes, the code flow is the
> > > same.
> > 
> > So for the time being I suggest the below patch on top of topic/pipe.
> > It contains everything we discussed so far.
> 
> One more thing. 'git log' cannot stop normally with 'q' with your patch.

The same happes with 'yes |less'.

The cause is that write side cannot detect closing read side because
query_hdl (read handle) is still opened.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-08-31 15:20                                                   ` Corinna Vinschen
@ 2021-09-01  2:39                                                     ` Takashi Yano
  2021-09-01  8:03                                                       ` Corinna Vinschen
  0 siblings, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-01  2:39 UTC (permalink / raw)
  To: cygwin-developers

On Tue, 31 Aug 2021 17:20:25 +0200
Corinna Vinschen wrote:
> On Aug 31 20:05, Takashi Yano wrote:
> > On Tue, 31 Aug 2021 11:04:05 +0200
> > Corinna Vinschen wrote:
> > > On Aug 31 17:52, Takashi Yano wrote:
> > > > On Mon, 30 Aug 2021 17:19:44 +0200
> > > > Corinna Vinschen wrote:
> > > > > On Aug 30 11:00, Ken Brown wrote:
> > > > > > On 8/30/2021 9:51 AM, Ken Brown wrote:
> > > > > > > On 8/30/2021 8:55 AM, Corinna Vinschen wrote:
> > > > > > > > On Aug 30 21:04, Takashi Yano wrote:
> > > > > > > > No worries.  The same should apply to the NtCreateFile side of the
> > > > > > > > pipe, btw.
> > > > > > > 
> > > > > > > I'll add my thanks.  I should have checked the default flags that are
> > > > > > > typically used for other devices when I wrote nt_create.  I'm glad you
> > > > > > > caught this.
> > > > > > > 
> > > > > > > So I'll reinstate the use of nt_create and then let Takashi recheck everything.
> > > > > > 
> > > > > > I've done this now.  I'm still not sure I've got all the flags right.  For
> > > > > > unknown reasons, I've used FILE_SHARE_READ | FILE_SHARE_WRITE in the call to
> > > > > > NtCreateNamedPipeFile, and no sharing in the call to NtOpenFile.  Should I
> > > > > > also use FILE_SHARE_READ | FILE_SHARE in NtOpenFile?  Is sharing even
> > > > > > relevant in this context?
> > > > > 
> > > > > This is only relevant if you want to open the pipe from another context,
> > > > > calling CreateNamedPipe/CreateFile.  As long as the pipe is only
> > > > > duplicated, it shouldn't matter at all.
> > > > > 
> > > > > But, as I just wrote in my previous mail, the FILE_SYNCHRONOUS_IO_NONALERT
> > > > > flag is probably a good thing for C# apps, but not for Cygwin, because it
> > > > > enforces synchronous operation.  Sorry about that...
> > > > 
> > > > With FILE_SYNCHRONOUS_IO_NONALERT, what kind of problems are you
> > > > specifically concerned about cygwin pipe? 
> > > 
> > > We're using asynchronous IO to be able to call WFMO and thus to be able
> > > to handle signals and thread cancellation events.  Wit hsynchronous IO
> > > this is not possible.
> > 
> > Thanks. How can I regenerate above issue? Stopping by Ctrl-C or killing
> > the process by kill seems to work even with FILE_SYNCHRONOUS_IO_NONALERT.
> 
> It may depend on the thread you're running this in.  But really, just
> call a blocking (SYNCHRONIZE + FILE_SYNCHRONOUS_IO_NONALERT) ReadFile
> in the main thread of a Cygwin app, and you'll see that neither Ctrl-C
> nor kill signalling will get through.

I confirmed the issue with FILE_SYNCHRONOUS_IO_NONALERT.

Thanks.


-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-01  2:39                                                     ` Takashi Yano
@ 2021-09-01  8:03                                                       ` Corinna Vinschen
  2021-09-01  8:13                                                         ` Corinna Vinschen
  0 siblings, 1 reply; 250+ messages in thread
From: Corinna Vinschen @ 2021-09-01  8:03 UTC (permalink / raw)
  To: cygwin-developers

On Sep  1 11:39, Takashi Yano wrote:
> On Tue, 31 Aug 2021 17:20:25 +0200
> Corinna Vinschen wrote:
> > On Aug 31 20:05, Takashi Yano wrote:
> > > On Tue, 31 Aug 2021 11:04:05 +0200
> > > Corinna Vinschen wrote:
> > > > On Aug 31 17:52, Takashi Yano wrote:
> > > > > On Mon, 30 Aug 2021 17:19:44 +0200
> > > > > Corinna Vinschen wrote:
> > > > > > But, as I just wrote in my previous mail, the FILE_SYNCHRONOUS_IO_NONALERT
> > > > > > flag is probably a good thing for C# apps, but not for Cygwin, because it
> > > > > > enforces synchronous operation.  Sorry about that...
> > > > > 
> > > > > With FILE_SYNCHRONOUS_IO_NONALERT, what kind of problems are you
> > > > > specifically concerned about cygwin pipe? 
> > > > 
> > > > We're using asynchronous IO to be able to call WFMO and thus to be able
> > > > to handle signals and thread cancellation events.  Wit hsynchronous IO
> > > > this is not possible.
> > > 
> > > Thanks. How can I regenerate above issue? Stopping by Ctrl-C or killing
> > > the process by kill seems to work even with FILE_SYNCHRONOUS_IO_NONALERT.
> > 
> > It may depend on the thread you're running this in.  But really, just
> > call a blocking (SYNCHRONIZE + FILE_SYNCHRONOUS_IO_NONALERT) ReadFile
> > in the main thread of a Cygwin app, and you'll see that neither Ctrl-C
> > nor kill signalling will get through.
> 
> I confirmed the issue with FILE_SYNCHRONOUS_IO_NONALERT.

There's, of course, a workaround.  If you start a thread for each
synchronous ReadFile/WriteFile, you can cygwait in the caller for the
thread, rather than for the event object of the async IO.  If a signal
or a thread cancellation request arrives, you can then call
CancelSynchronousIo on the ReadFile/WriteFile thread and wait for thread
termination.  I implemented something along these lines in
fhandler_disk_file::mand_lock.

Maybe something to consider?  It would certainly fix the C# issue,
but I'm reluctant to do the thread juggle for each read/write.


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-01  0:16                                                               ` Takashi Yano
@ 2021-09-01  8:07                                                                 ` Corinna Vinschen
  2021-09-01  8:23                                                                   ` Takashi Yano
  0 siblings, 1 reply; 250+ messages in thread
From: Corinna Vinschen @ 2021-09-01  8:07 UTC (permalink / raw)
  To: cygwin-developers

On Sep  1 09:16, Takashi Yano wrote:
> On Wed, 1 Sep 2021 08:02:20 +0900
> Takashi Yano wrote:
> > On Tue, 31 Aug 2021 17:50:14 +0200
> > Corinna Vinschen wrote:
> > > So for the time being I suggest the below patch on top of topic/pipe.
> > > It contains everything we discussed so far.
> > 
> > One more thing. 'git log' cannot stop normally with 'q' with your patch.
> 
> The same happes with 'yes |less'.
> 
> The cause is that write side cannot detect closing read side because
> query_hdl (read handle) is still opened.

Oh

my

god.


That kills the entire idea of keeping the read handle :(


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-01  8:03                                                       ` Corinna Vinschen
@ 2021-09-01  8:13                                                         ` Corinna Vinschen
  0 siblings, 0 replies; 250+ messages in thread
From: Corinna Vinschen @ 2021-09-01  8:13 UTC (permalink / raw)
  To: cygwin-developers

On Sep  1 10:03, Corinna Vinschen wrote:
> On Sep  1 11:39, Takashi Yano wrote:
> > On Tue, 31 Aug 2021 17:20:25 +0200
> > Corinna Vinschen wrote:
> > > On Aug 31 20:05, Takashi Yano wrote:
> > > > On Tue, 31 Aug 2021 11:04:05 +0200
> > > > Corinna Vinschen wrote:
> > > > > On Aug 31 17:52, Takashi Yano wrote:
> > > > > > With FILE_SYNCHRONOUS_IO_NONALERT, what kind of problems are you
> > > > > > specifically concerned about cygwin pipe? 
> > > > > 
> > > > > We're using asynchronous IO to be able to call WFMO and thus to be able
> > > > > to handle signals and thread cancellation events.  Wit hsynchronous IO
> > > > > this is not possible.
> > > > 
> > > > Thanks. How can I regenerate above issue? Stopping by Ctrl-C or killing
> > > > the process by kill seems to work even with FILE_SYNCHRONOUS_IO_NONALERT.
> > > 
> > > It may depend on the thread you're running this in.  But really, just
> > > call a blocking (SYNCHRONIZE + FILE_SYNCHRONOUS_IO_NONALERT) ReadFile
> > > in the main thread of a Cygwin app, and you'll see that neither Ctrl-C
> > > nor kill signalling will get through.
> > 
> > I confirmed the issue with FILE_SYNCHRONOUS_IO_NONALERT.
> 
> There's, of course, a workaround.  If you start a thread for each
> synchronous ReadFile/WriteFile, you can cygwait in the caller for the
> thread, rather than for the event object of the async IO.  If a signal
> or a thread cancellation request arrives, you can then call
> CancelSynchronousIo on the ReadFile/WriteFile thread and wait for thread
> termination.  I implemented something along these lines in
> fhandler_disk_file::mand_lock.
> 
> Maybe something to consider?  It would certainly fix the C# issue,
> but I'm reluctant to do the thread juggle for each read/write.

Oh, yeah, there's still the problem that NtQueryInformationFile on the
read side hangs when called during a synchronous ReadFile, as I learned
two days ago...


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-01  8:07                                                                 ` Corinna Vinschen
@ 2021-09-01  8:23                                                                   ` Takashi Yano
  2021-09-01  8:46                                                                     ` Corinna Vinschen
  0 siblings, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-01  8:23 UTC (permalink / raw)
  To: cygwin-developers

On Wed, 1 Sep 2021 10:07:48 +0200
Corinna Vinschen wrote:
> On Sep  1 09:16, Takashi Yano wrote:
> > On Wed, 1 Sep 2021 08:02:20 +0900
> > Takashi Yano wrote:
> > > On Tue, 31 Aug 2021 17:50:14 +0200
> > > Corinna Vinschen wrote:
> > > > So for the time being I suggest the below patch on top of topic/pipe.
> > > > It contains everything we discussed so far.
> > > 
> > > One more thing. 'git log' cannot stop normally with 'q' with your patch.
> > 
> > The same happes with 'yes |less'.
> > 
> > The cause is that write side cannot detect closing read side because
> > query_hdl (read handle) is still opened.
> 
> Oh
> 
> my
> 
> god.
> 
> 
> That kills the entire idea of keeping the read handle :(

One idea is:

Count read handle and write handle opned using NtQueryObject().
If the numbers of opened handle are equal each other, only
the write side (pair of write handle and query_hdl) is alive.
In this case, write() returns error.
If read side is alive, number of read handles is greater than
number of write handles. 

But atomicity should be considered.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-01  8:23                                                                   ` Takashi Yano
@ 2021-09-01  8:46                                                                     ` Corinna Vinschen
  2021-09-01 12:56                                                                       ` Ken Brown
  2021-09-02  8:15                                                                       ` Takashi Yano
  0 siblings, 2 replies; 250+ messages in thread
From: Corinna Vinschen @ 2021-09-01  8:46 UTC (permalink / raw)
  To: cygwin-developers

On Sep  1 17:23, Takashi Yano wrote:
> On Wed, 1 Sep 2021 10:07:48 +0200
> Corinna Vinschen wrote:
> > On Sep  1 09:16, Takashi Yano wrote:
> > > On Wed, 1 Sep 2021 08:02:20 +0900
> > > Takashi Yano wrote:
> > > > On Tue, 31 Aug 2021 17:50:14 +0200
> > > > Corinna Vinschen wrote:
> > > > > So for the time being I suggest the below patch on top of topic/pipe.
> > > > > It contains everything we discussed so far.
> > > > 
> > > > One more thing. 'git log' cannot stop normally with 'q' with your patch.
> > > 
> > > The same happes with 'yes |less'.
> > > 
> > > The cause is that write side cannot detect closing read side because
> > > query_hdl (read handle) is still opened.
> > 
> > Oh
> > 
> > my
> > 
> > god.
> > 
> > 
> > That kills the entire idea of keeping the read handle :(
> 
> One idea is:
> 
> Count read handle and write handle opned using NtQueryObject().
> If the numbers of opened handle are equal each other, only
> the write side (pair of write handle and query_hdl) is alive.
> In this case, write() returns error.
> If read side is alive, number of read handles is greater than
> number of write handles. 

Interesting idea.  But where do you do the count?  The event object
will not get signalled, so WFMO will not return when performing a
blocking write.


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-01  8:46                                                                     ` Corinna Vinschen
@ 2021-09-01 12:56                                                                       ` Ken Brown
  2021-09-01 13:52                                                                         ` Corinna Vinschen
  2021-09-02  8:15                                                                       ` Takashi Yano
  1 sibling, 1 reply; 250+ messages in thread
From: Ken Brown @ 2021-09-01 12:56 UTC (permalink / raw)
  To: cygwin-developers

On 9/1/2021 4:46 AM, Corinna Vinschen wrote:
> On Sep  1 17:23, Takashi Yano wrote:
>> On Wed, 1 Sep 2021 10:07:48 +0200
>> Corinna Vinschen wrote:
>>> On Sep  1 09:16, Takashi Yano wrote:
>>>> On Wed, 1 Sep 2021 08:02:20 +0900
>>>> Takashi Yano wrote:
>>>>> On Tue, 31 Aug 2021 17:50:14 +0200
>>>>> Corinna Vinschen wrote:
>>>>>> So for the time being I suggest the below patch on top of topic/pipe.
>>>>>> It contains everything we discussed so far.
>>>>>
>>>>> One more thing. 'git log' cannot stop normally with 'q' with your patch.
>>>>
>>>> The same happes with 'yes |less'.
>>>>
>>>> The cause is that write side cannot detect closing read side because
>>>> query_hdl (read handle) is still opened.
>>>
>>> Oh
>>>
>>> my
>>>
>>> god.
>>>
>>>
>>> That kills the entire idea of keeping the read handle :(
>>
>> One idea is:
>>
>> Count read handle and write handle opned using NtQueryObject().
>> If the numbers of opened handle are equal each other, only
>> the write side (pair of write handle and query_hdl) is alive.
>> In this case, write() returns error.
>> If read side is alive, number of read handles is greater than
>> number of write handles.
> 
> Interesting idea.  But where do you do the count?  The event object
> will not get signalled, so WFMO will not return when performing a
> blocking write.

What if we create an event that we signal every time a reader closes, and we add 
that to the events that WFMO is waiting for?

If this doesn't work for some reason, a different (but more complicated) idea is 
to keep a count of the number of open readers in shared memory.  When this is 0, 
write returns an error.  I'm thinking of shared memory as in topic/af_unix 
(which I copied in the fifo implementation), but maybe something simpler would 
work since we only have a single variable to keep track of.

Ken


^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-01 12:56                                                                       ` Ken Brown
@ 2021-09-01 13:52                                                                         ` Corinna Vinschen
  2021-09-01 23:02                                                                           ` Ken Brown
  0 siblings, 1 reply; 250+ messages in thread
From: Corinna Vinschen @ 2021-09-01 13:52 UTC (permalink / raw)
  To: cygwin-developers

On Sep  1 08:56, Ken Brown wrote:
> On 9/1/2021 4:46 AM, Corinna Vinschen wrote:
> > On Sep  1 17:23, Takashi Yano wrote:
> > > On Wed, 1 Sep 2021 10:07:48 +0200
> > > Corinna Vinschen wrote:
> > > > On Sep  1 09:16, Takashi Yano wrote:
> > > > > On Wed, 1 Sep 2021 08:02:20 +0900
> > > > > Takashi Yano wrote:
> > > > > > On Tue, 31 Aug 2021 17:50:14 +0200
> > > > > > Corinna Vinschen wrote:
> > > > > > > So for the time being I suggest the below patch on top of topic/pipe.
> > > > > > > It contains everything we discussed so far.
> > > > > > 
> > > > > > One more thing. 'git log' cannot stop normally with 'q' with your patch.
> > > > > 
> > > > > The same happes with 'yes |less'.
> > > > > 
> > > > > The cause is that write side cannot detect closing read side because
> > > > > query_hdl (read handle) is still opened.
> > > > 
> > > > Oh
> > > > 
> > > > my
> > > > 
> > > > god.
> > > > 
> > > > 
> > > > That kills the entire idea of keeping the read handle :(
> > > 
> > > One idea is:
> > > 
> > > Count read handle and write handle opned using NtQueryObject().
> > > If the numbers of opened handle are equal each other, only
> > > the write side (pair of write handle and query_hdl) is alive.
> > > In this case, write() returns error.
> > > If read side is alive, number of read handles is greater than
> > > number of write handles.
> > 
> > Interesting idea.  But where do you do the count?  The event object
> > will not get signalled, so WFMO will not return when performing a
> > blocking write.
> 
> What if we create an event that we signal every time a reader closes, and we
> add that to the events that WFMO is waiting for?
> 
> If this doesn't work for some reason, a different (but more complicated)
> idea is to keep a count of the number of open readers in shared memory.
> When this is 0, write returns an error.  I'm thinking of shared memory as in
> topic/af_unix (which I copied in the fifo implementation), but maybe
> something simpler would work since we only have a single variable to keep
> track of.

Great idea that.  What we need would be some semaphore upside down.
One that can be used to count items and which is signalled if it's
down to zero.

Hmm, something to think about...


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-01 13:52                                                                         ` Corinna Vinschen
@ 2021-09-01 23:02                                                                           ` Ken Brown
  2021-09-02  8:17                                                                             ` Corinna Vinschen
  0 siblings, 1 reply; 250+ messages in thread
From: Ken Brown @ 2021-09-01 23:02 UTC (permalink / raw)
  To: cygwin-developers

On 9/1/2021 9:52 AM, Corinna Vinschen wrote:
> On Sep  1 08:56, Ken Brown wrote:
>> On 9/1/2021 4:46 AM, Corinna Vinschen wrote:
>>> On Sep  1 17:23, Takashi Yano wrote:
>>>> On Wed, 1 Sep 2021 10:07:48 +0200
>>>> Corinna Vinschen wrote:
>>>>> On Sep  1 09:16, Takashi Yano wrote:
>>>>>> On Wed, 1 Sep 2021 08:02:20 +0900
>>>>>> Takashi Yano wrote:
>>>>>>> On Tue, 31 Aug 2021 17:50:14 +0200
>>>>>>> Corinna Vinschen wrote:
>>>>>>>> So for the time being I suggest the below patch on top of topic/pipe.
>>>>>>>> It contains everything we discussed so far.
>>>>>>>
>>>>>>> One more thing. 'git log' cannot stop normally with 'q' with your patch.
>>>>>>
>>>>>> The same happes with 'yes |less'.
>>>>>>
>>>>>> The cause is that write side cannot detect closing read side because
>>>>>> query_hdl (read handle) is still opened.
>>>>>
>>>>> Oh
>>>>>
>>>>> my
>>>>>
>>>>> god.
>>>>>
>>>>>
>>>>> That kills the entire idea of keeping the read handle :(
>>>>
>>>> One idea is:
>>>>
>>>> Count read handle and write handle opned using NtQueryObject().
>>>> If the numbers of opened handle are equal each other, only
>>>> the write side (pair of write handle and query_hdl) is alive.
>>>> In this case, write() returns error.
>>>> If read side is alive, number of read handles is greater than
>>>> number of write handles.
>>>
>>> Interesting idea.  But where do you do the count?  The event object
>>> will not get signalled, so WFMO will not return when performing a
>>> blocking write.
>>
>> What if we create an event that we signal every time a reader closes, and we
>> add that to the events that WFMO is waiting for?
>>
>> If this doesn't work for some reason, a different (but more complicated)
>> idea is to keep a count of the number of open readers in shared memory.
>> When this is 0, write returns an error.  I'm thinking of shared memory as in
>> topic/af_unix (which I copied in the fifo implementation), but maybe
>> something simpler would work since we only have a single variable to keep
>> track of.
> 
> Great idea that.  What we need would be some semaphore upside down.
> One that can be used to count items and which is signalled if it's
> down to zero.

Here's an idea (untested), based on 
https://stackoverflow.com/questions/6559854/is-there-something-opposite-to-semaphore:

We create an ordinary Windows semaphore and use it to count the readers: It 
starts at 0, we increment it by calling ReleaseSemaphore when a reader is opened 
(by fhandler_pipe::create, fork/exec, dup), and we decrement it by calling WFSO 
when a reader closes.  When we decrement it, we test whether it's been reduced 
to 0.  We do this by calling ReleaseSemaphore and using its lpPreviousCount 
argument.

We also create an event that we can use to make WFMO return during a blocking 
write.  We signal this event if a reader closes and we've discovered that there 
are no more readers.  In this case we cancel the pending write [*] and return an 
error.

I'm sure I've overlooked something, but does this seem feasible?

Ken

[*] I don't know offhand if Windows provides a way to cancel a pending write. 
If not, we could use query_hdl to drain the pipe.

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-01  8:46                                                                     ` Corinna Vinschen
  2021-09-01 12:56                                                                       ` Ken Brown
@ 2021-09-02  8:15                                                                       ` Takashi Yano
  2021-09-02 18:54                                                                         ` Corinna Vinschen
  1 sibling, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-02  8:15 UTC (permalink / raw)
  To: cygwin-developers

[-- Attachment #1: Type: text/plain, Size: 1862 bytes --]

On Wed, 1 Sep 2021 10:46:24 +0200
Corinna Vinschen wrote:
> On Sep  1 17:23, Takashi Yano wrote:
> > On Wed, 1 Sep 2021 10:07:48 +0200
> > Corinna Vinschen wrote:
> > > On Sep  1 09:16, Takashi Yano wrote:
> > > > On Wed, 1 Sep 2021 08:02:20 +0900
> > > > Takashi Yano wrote:
> > > > > On Tue, 31 Aug 2021 17:50:14 +0200
> > > > > Corinna Vinschen wrote:
> > > > > > So for the time being I suggest the below patch on top of topic/pipe.
> > > > > > It contains everything we discussed so far.
> > > > > 
> > > > > One more thing. 'git log' cannot stop normally with 'q' with your patch.
> > > > 
> > > > The same happes with 'yes |less'.
> > > > 
> > > > The cause is that write side cannot detect closing read side because
> > > > query_hdl (read handle) is still opened.
> > > 
> > > Oh
> > > 
> > > my
> > > 
> > > god.
> > > 
> > > 
> > > That kills the entire idea of keeping the read handle :(
> > 
> > One idea is:
> > 
> > Count read handle and write handle opned using NtQueryObject().
> > If the numbers of opened handle are equal each other, only
> > the write side (pair of write handle and query_hdl) is alive.
> > In this case, write() returns error.
> > If read side is alive, number of read handles is greater than
> > number of write handles. 
> 
> Interesting idea.  But where do you do the count?  The event object
> will not get signalled, so WFMO will not return when performing a
> blocking write.

I imagined something like attached patch.

Unfortunately, the attached patch seems to have bug that
occasionally causes the following error while building
cygwin1.dll.

<command-line>: error: "GCC_VERSION" redefined [-Werror]
<command-line>: note: this is the location of the previous definition
cc1plus: all warnings being treated as errors
make[1]: *** [Makefile:1942: fhandler_proc.o] Error 1

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

[-- Attachment #2: count-rw-handle.patch --]
[-- Type: application/octet-stream, Size: 5129 bytes --]

diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
index 1f0f28077..38390848f 100644
--- a/winsup/cygwin/fhandler.h
+++ b/winsup/cygwin/fhandler.h
@@ -1172,6 +1172,7 @@ class fhandler_pipe: public fhandler_base
 {
 private:
   HANDLE query_hdl;
+  HANDLE reader_evt;
   pid_t popen_pid;
   size_t max_atomic_write;
   void set_pipe_non_blocking (bool nonblocking);
@@ -1181,6 +1182,7 @@ public:
   bool ispipe() const { return true; }
 
   HANDLE get_query_handle () const { return query_hdl; }
+  bool reader_closed ();
 
   void set_popen_pid (pid_t pid) {popen_pid = pid;}
   pid_t get_popen_pid () const {return popen_pid;}
diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index 2d9e87bb3..4773d04da 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -51,6 +51,8 @@ fhandler_pipe::set_pipe_non_blocking (bool nonblocking)
   fpi.ReadMode = FILE_PIPE_BYTE_STREAM_MODE;
   fpi.CompletionMode = nonblocking ? FILE_PIPE_COMPLETE_OPERATION
     : FILE_PIPE_QUEUE_OPERATION;
+  if (get_device () == FH_PIPEW)
+    fpi.CompletionMode = FILE_PIPE_COMPLETE_OPERATION;
   status = NtSetInformationFile (get_handle (), &io, &fpi, sizeof fpi,
 				 FilePipeInformation);
   if (!NT_SUCCESS (status))
@@ -308,6 +310,22 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
     }
   else if (status == STATUS_THREAD_CANCELED)
     pthread::static_cancel_self ();
+
+  if ((ssize_t)len > 0)
+    SetEvent (reader_evt);
+}
+
+bool
+fhandler_pipe::reader_closed (void)
+{
+  if (get_device () == FH_PIPER)
+    return false;
+  OBJECT_BASIC_INFORMATION obi;
+  NtQueryObject (get_handle (), ObjectBasicInformation, &obi, sizeof obi, NULL);
+  int n_writer = obi.HandleCount;
+  NtQueryObject (query_hdl, ObjectBasicInformation, &obi, sizeof obi, NULL);
+  int n_reader = obi.HandleCount;
+  return n_writer == n_reader;
 }
 
 ssize_t __reg3
@@ -323,6 +341,13 @@ fhandler_pipe::raw_write (const void *ptr, size_t len)
   if (!len)
     return 0;
 
+  if (reader_closed ())
+    {
+      set_errno(EPIPE);
+      raise(SIGPIPE);
+      return -1;
+    }
+
   if (len <= max_atomic_write)
     chunk = len;
   else if (is_nonblocking ())
@@ -400,10 +425,24 @@ fhandler_pipe::raw_write (const void *ptr, size_t len)
 	  /* NtWriteFile returns success with # of bytes written == 0
 	     if writing on a non-blocking pipe fails because the pipe
 	     buffer doesn't have sufficient space. */
-	  if (nbytes_now == 0 && nbytes == 0)
+	  if (nbytes_now == 0 && nbytes == 0 && is_nonblocking ())
 	    set_errno (EAGAIN);
-	  ptr = ((char *) ptr) + chunk;
+	  ptr = ((char *) ptr) + nbytes_now;
 	  nbytes += nbytes_now;
+	  if (reader_closed () && nbytes == 0)
+	    {
+	      set_errno(EPIPE);
+	      raise(SIGPIPE);
+	    }
+	  if (!is_nonblocking () && nbytes < len)
+	    {
+	      if (nbytes_now == 0)
+		{
+		  cygwait (reader_evt);
+		  ResetEvent (reader_evt);
+		}
+	      continue;
+	    }
 	}
       else if (STATUS_PIPE_IS_CLOSED (status))
 	{
@@ -430,9 +469,14 @@ fhandler_pipe::raw_write (const void *ptr, size_t len)
 void
 fhandler_pipe::fixup_after_fork (HANDLE parent)
 {
+  if (close_on_exec() && query_hdl)
+    CloseHandle (query_hdl);
   if (query_hdl)
     fork_fixup (parent, query_hdl, "query_hdl");
   fhandler_base::fixup_after_fork (parent);
+  if (!close_on_exec ())
+    DuplicateHandle (parent, reader_evt, GetCurrentProcess (), &reader_evt,
+		     0, 1, DUPLICATE_SAME_ACCESS);
 }
 
 int
@@ -453,6 +497,9 @@ fhandler_pipe::dup (fhandler_base *child, int flags)
       ftp->close ();
       res = -1;
     }
+  DuplicateHandle (GetCurrentProcess (), reader_evt,
+		   GetCurrentProcess (), &ftp->reader_evt,
+		   0, 1, DUPLICATE_SAME_ACCESS);
 
   debug_printf ("res %d", res);
   return res;
@@ -463,7 +510,11 @@ fhandler_pipe::close ()
 {
   if (query_hdl)
     NtClose (query_hdl);
-  return fhandler_base::close ();
+  int ret = fhandler_base::close ();
+  if (get_device () == FH_PIPER)
+    SetEvent (reader_evt);
+  CloseHandle (reader_evt);
+  return ret;
 }
 
 #define PIPE_INTRO "\\\\.\\pipe\\cygwin-"
@@ -678,6 +729,10 @@ fhandler_pipe::create (fhandler_pipe *fhs[2], unsigned psize, int mode)
       else
 	res = 0;
     }
+  fhs[0]->reader_evt = CreateEvent (NULL, true, false, NULL);
+  DuplicateHandle (GetCurrentProcess (), fhs[0]->reader_evt,
+		   GetCurrentProcess (), &fhs[1]->reader_evt,
+		   0, 1, DUPLICATE_SAME_ACCESS);
 
   debug_printf ("%R = pipe(%d, %y)", res, psize, mode);
   return res;
diff --git a/winsup/cygwin/select.cc b/winsup/cygwin/select.cc
index dc1f79613..0905b3d38 100644
--- a/winsup/cygwin/select.cc
+++ b/winsup/cygwin/select.cc
@@ -741,6 +741,13 @@ out:
     }
   if (s->write_selected && dev != FH_PIPER)
     {
+      if (dev == FH_PIPEW && ((fhandler_pipe *) fh)->reader_closed ())
+	{
+	  gotone += s->write_ready = true;
+	  if (s->except_selected)
+	    gotone += s->except_ready = true;
+	  return gotone;
+	}
       /* For the write side of a pipe, fetch the handle to the read side.
 	 See the longish comment in pipe_data_available for the reasoning. */
       if (dev == FH_PIPEW)

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-01 23:02                                                                           ` Ken Brown
@ 2021-09-02  8:17                                                                             ` Corinna Vinschen
  2021-09-02 13:01                                                                               ` Ken Brown
  0 siblings, 1 reply; 250+ messages in thread
From: Corinna Vinschen @ 2021-09-02  8:17 UTC (permalink / raw)
  To: cygwin-developers

On Sep  1 19:02, Ken Brown wrote:
> On 9/1/2021 9:52 AM, Corinna Vinschen wrote:
> > Great idea that.  What we need would be some semaphore upside down.
> > One that can be used to count items and which is signalled if it's
> > down to zero.
> 
> Here's an idea (untested), based on https://stackoverflow.com/questions/6559854/is-there-something-opposite-to-semaphore:
> 
> We create an ordinary Windows semaphore and use it to count the readers: It
> starts at 0, we increment it by calling ReleaseSemaphore when a reader is
> opened (by fhandler_pipe::create, fork/exec, dup), and we decrement it by
> calling WFSO when a reader closes.  When we decrement it, we test whether
> it's been reduced to 0.  We do this by calling ReleaseSemaphore and using
> its lpPreviousCount argument.
> 
> We also create an event that we can use to make WFMO return during a
> blocking write.  We signal this event if a reader closes and we've
> discovered that there are no more readers.  In this case we cancel the
> pending write [*] and return an error.
> 
> I'm sure I've overlooked something, but does this seem feasible?

It could work, but the problem with all these approaches is that they
are tricky and bound to fail as soon as a process is killed or crashes.

> [*] I don't know offhand if Windows provides a way to cancel a pending
> write. If not, we could use query_hdl to drain the pipe.

There's a CancelIoEx function to cancel all async IO on a handle.

In a lucid moment tonight, I had another idea.

First of all, scratch my patch.  Also, revert select to check only
for WriteQuotaAvailable.

Next, for sanity, let's assume that non-blocking reads don't change
WriteQuotaAvailable.  So the only important case is the blocking read,
which reduces WriteQuotaAvailable by the number of requested bytes.

Next, fact is, we're only interested in WriteQuotaAvailable > 0.
And we have a buffersize of 64K.

We can also safely assume that we only have a very small number of
readers, typically only one.

So here's the crazily simple idea:

What if the readers never request more than, say, 50 or even 25% of the
available buffer space?  Our buffer is 64K and there's no guarantee that
any read > PIPE_BUF (== 4K) is atomic anyway.  This can work without
having to check the other side of the pipe.  Something like this,
ignoring border cases:

pipe::create()
{
   [...]
   mutex = CreateMutex();
}

pipe::raw_read(char *buf, size_t num_requested)
{
  if (blocking)
    {
      WFSO(mutex);
      NtQueryInformationFile(FilePipeLocalInformation);
      if (!fpli.ReadDataAvailable
	  && num_requested > fpli.InboundQuota / 4)
	num_requested = fpli.InboundQuota / 4;
      NtReadFile(pipe, buf, num_requested);
      ReleaseMutex(mutex);
    }
}

It's not entirely foolproof, but it should fix 99% of the cases.


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-02  8:17                                                                             ` Corinna Vinschen
@ 2021-09-02 13:01                                                                               ` Ken Brown
  2021-09-02 19:00                                                                                 ` Corinna Vinschen
  0 siblings, 1 reply; 250+ messages in thread
From: Ken Brown @ 2021-09-02 13:01 UTC (permalink / raw)
  To: cygwin-developers

On 9/2/2021 4:17 AM, Corinna Vinschen wrote:
> On Sep  1 19:02, Ken Brown wrote:
>> On 9/1/2021 9:52 AM, Corinna Vinschen wrote:
>>> Great idea that.  What we need would be some semaphore upside down.
>>> One that can be used to count items and which is signalled if it's
>>> down to zero.
>>
>> Here's an idea (untested), based on https://stackoverflow.com/questions/6559854/is-there-something-opposite-to-semaphore:
>>
>> We create an ordinary Windows semaphore and use it to count the readers: It
>> starts at 0, we increment it by calling ReleaseSemaphore when a reader is
>> opened (by fhandler_pipe::create, fork/exec, dup), and we decrement it by
>> calling WFSO when a reader closes.  When we decrement it, we test whether
>> it's been reduced to 0.  We do this by calling ReleaseSemaphore and using
>> its lpPreviousCount argument.
>>
>> We also create an event that we can use to make WFMO return during a
>> blocking write.  We signal this event if a reader closes and we've
>> discovered that there are no more readers.  In this case we cancel the
>> pending write [*] and return an error.
>>
>> I'm sure I've overlooked something, but does this seem feasible?
> 
> It could work, but the problem with all these approaches is that they
> are tricky and bound to fail as soon as a process is killed or crashes.
> 
>> [*] I don't know offhand if Windows provides a way to cancel a pending
>> write. If not, we could use query_hdl to drain the pipe.
> 
> There's a CancelIoEx function to cancel all async IO on a handle.
> 
> In a lucid moment tonight, I had another idea.
> 
> First of all, scratch my patch.  Also, revert select to check only
> for WriteQuotaAvailable.
> 
> Next, for sanity, let's assume that non-blocking reads don't change
> WriteQuotaAvailable.  So the only important case is the blocking read,
> which reduces WriteQuotaAvailable by the number of requested bytes.
> 
> Next, fact is, we're only interested in WriteQuotaAvailable > 0.
> And we have a buffersize of 64K.
> 
> We can also safely assume that we only have a very small number of
> readers, typically only one.
> 
> So here's the crazily simple idea:
> 
> What if the readers never request more than, say, 50 or even 25% of the
> available buffer space?  Our buffer is 64K and there's no guarantee that
> any read > PIPE_BUF (== 4K) is atomic anyway.  This can work without
> having to check the other side of the pipe.  Something like this,
> ignoring border cases:
> 
> pipe::create()
> {
>     [...]
>     mutex = CreateMutex();
> }
> 
> pipe::raw_read(char *buf, size_t num_requested)
> {
>    if (blocking)
>      {
>        WFSO(mutex);
>        NtQueryInformationFile(FilePipeLocalInformation);
>        if (!fpli.ReadDataAvailable
> 	  && num_requested > fpli.InboundQuota / 4)
> 	num_requested = fpli.InboundQuota / 4;
>        NtReadFile(pipe, buf, num_requested);
>        ReleaseMutex(mutex);
>      }
> }
> 
> It's not entirely foolproof, but it should fix 99% of the cases.

I like it!

Do you think there's anything we can or should do to avoid a deadlock in the 
rare cases where this fails?  The only thing I can think of immediately is to 
always impose a timeout if select is called with infinite timeout on the write 
side of a pipe, after which we report that the pipe is write ready.  After all, 
we've lived since 2008 with a bug that caused select to *always* report write ready.

Alternatively, we could just wait and see if there's an actual use case in which 
someone encounters a deadlock.

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-02  8:15                                                                       ` Takashi Yano
@ 2021-09-02 18:54                                                                         ` Corinna Vinschen
  0 siblings, 0 replies; 250+ messages in thread
From: Corinna Vinschen @ 2021-09-02 18:54 UTC (permalink / raw)
  To: cygwin-developers

Hi Takashi,

On Sep  2 17:15, Takashi Yano wrote:
> On Wed, 1 Sep 2021 10:46:24 +0200
> Corinna Vinschen wrote:
> > On Sep  1 17:23, Takashi Yano wrote:
> > > One idea is:
> > > 
> > > Count read handle and write handle opned using NtQueryObject().
> > > If the numbers of opened handle are equal each other, only
> > > the write side (pair of write handle and query_hdl) is alive.
> > > In this case, write() returns error.
> > > If read side is alive, number of read handles is greater than
> > > number of write handles. 
> > 
> > Interesting idea.  But where do you do the count?  The event object
> > will not get signalled, so WFMO will not return when performing a
> > blocking write.
> 
> I imagined something like attached patch.
> 
> Unfortunately, the attached patch seems to have bug that
> occasionally causes the following error while building
> cygwin1.dll.
> 
> <command-line>: error: "GCC_VERSION" redefined [-Werror]
> <command-line>: note: this is the location of the previous definition
> cc1plus: all warnings being treated as errors
> make[1]: *** [Makefile:1942: fhandler_proc.o] Error 1

> diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
> index 1f0f28077..38390848f 100644
> --- a/winsup/cygwin/fhandler.h
> +++ b/winsup/cygwin/fhandler.h
> @@ -1172,6 +1172,7 @@ class fhandler_pipe: public fhandler_base
>  {
>  private:
>    HANDLE query_hdl;
> +  HANDLE reader_evt;
>    pid_t popen_pid;
>    size_t max_atomic_write;
>    void set_pipe_non_blocking (bool nonblocking);
> @@ -1181,6 +1182,7 @@ public:
>    bool ispipe() const { return true; }
>  
>    HANDLE get_query_handle () const { return query_hdl; }
> +  bool reader_closed ();
>  
>    void set_popen_pid (pid_t pid) {popen_pid = pid;}
>    pid_t get_popen_pid () const {return popen_pid;}
> diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
> index 2d9e87bb3..4773d04da 100644
> --- a/winsup/cygwin/fhandler_pipe.cc
> +++ b/winsup/cygwin/fhandler_pipe.cc
> @@ -51,6 +51,8 @@ fhandler_pipe::set_pipe_non_blocking (bool nonblocking)
>    fpi.ReadMode = FILE_PIPE_BYTE_STREAM_MODE;
>    fpi.CompletionMode = nonblocking ? FILE_PIPE_COMPLETE_OPERATION
>      : FILE_PIPE_QUEUE_OPERATION;
> +  if (get_device () == FH_PIPEW)
> +    fpi.CompletionMode = FILE_PIPE_COMPLETE_OPERATION;

Ok, so  the write side is always non-blocking...


>    status = NtSetInformationFile (get_handle (), &io, &fpi, sizeof fpi,
>  				 FilePipeInformation);
>    if (!NT_SUCCESS (status))
> @@ -308,6 +310,22 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
>      }
>    else if (status == STATUS_THREAD_CANCELED)
>      pthread::static_cancel_self ();
> +
> +  if ((ssize_t)len > 0)
> +    SetEvent (reader_evt);
> +}

...and every successful read sets the event object to signalled.

Sounds good.

> +    }
> +
>    if (len <= max_atomic_write)
>      chunk = len;
>    else if (is_nonblocking ())
> @@ -400,10 +425,24 @@ fhandler_pipe::raw_write (const void *ptr, size_t len)
>  	  /* NtWriteFile returns success with # of bytes written == 0
>  	     if writing on a non-blocking pipe fails because the pipe
>  	     buffer doesn't have sufficient space. */
> -	  if (nbytes_now == 0 && nbytes == 0)
> +	  if (nbytes_now == 0 && nbytes == 0 && is_nonblocking ())
>  	    set_errno (EAGAIN);
> -	  ptr = ((char *) ptr) + chunk;
> +	  ptr = ((char *) ptr) + nbytes_now;
>  	  nbytes += nbytes_now;
> +	  if (reader_closed () && nbytes == 0)
> +	    {
> +	      set_errno(EPIPE);
> +	      raise(SIGPIPE);
> +	    }
> +	  if (!is_nonblocking () && nbytes < len)
> +	    {
> +	      if (nbytes_now == 0)
> +		{
> +		  cygwait (reader_evt);
> +		  ResetEvent (reader_evt);

But what if a read calls SetEvent between cygwait and ResetEvent?
This looks like a potential deadlock issue, no?

> +	    }
>  	}
>        else if (STATUS_PIPE_IS_CLOSED (status))
>  	{
> @@ -430,9 +469,14 @@ fhandler_pipe::raw_write (const void *ptr, size_t len)
>  void
>  fhandler_pipe::fixup_after_fork (HANDLE parent)
>  {
> +  if (close_on_exec() && query_hdl)
> +    CloseHandle (query_hdl);

Why do you close the handle here?  It gets already created with
inheritence settings according to the O_CLOEXEC flag.

>    if (query_hdl)

This is broken.  If you close query_hdl above, it's still != NULL
and fork_fixup will be called.

>      fork_fixup (parent, query_hdl, "query_hdl");
>    fhandler_base::fixup_after_fork (parent);
> +  if (!close_on_exec ())
> +    DuplicateHandle (parent, reader_evt, GetCurrentProcess (), &reader_evt,
> +		     0, 1, DUPLICATE_SAME_ACCESS);

Uhm... this is fixup_after_fork.  I'm a bit puzzled.  You create the
event object with inheritence set to TRUE unconditionally.  So the
forked process will have this handle anyway.  If you duplicate the
handle here, you'll have a handle leak.  What about creating and
duplicating with inheritance == !(flags & O_CLOEXEC) and just call
fork_fixup?

> @@ -463,7 +510,11 @@ fhandler_pipe::close ()
>  {
>    if (query_hdl)
>      NtClose (query_hdl);
> -  return fhandler_base::close ();
> +  int ret = fhandler_base::close ();
> +  if (get_device () == FH_PIPER)
> +    SetEvent (reader_evt);
> +  CloseHandle (reader_evt);
> +  return ret;
>  }

What do you do if the reader process gets killed or crashes?  I fear
this solution has the same problem as a solution using a self-implemented
counter.


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-02 13:01                                                                               ` Ken Brown
@ 2021-09-02 19:00                                                                                 ` Corinna Vinschen
  2021-09-02 19:34                                                                                   ` Ken Brown
  2021-09-02 19:35                                                                                   ` Corinna Vinschen
  0 siblings, 2 replies; 250+ messages in thread
From: Corinna Vinschen @ 2021-09-02 19:00 UTC (permalink / raw)
  To: cygwin-developers

On Sep  2 09:01, Ken Brown wrote:
> On 9/2/2021 4:17 AM, Corinna Vinschen wrote:
> > What if the readers never request more than, say, 50 or even 25% of the
> > available buffer space?  Our buffer is 64K and there's no guarantee that
> > any read > PIPE_BUF (== 4K) is atomic anyway.  This can work without
> > having to check the other side of the pipe.  Something like this,
> > ignoring border cases:
> > 
> > pipe::create()
> > {
> >     [...]
> >     mutex = CreateMutex();
> > }
> > 
> > pipe::raw_read(char *buf, size_t num_requested)
> > {
> >    if (blocking)
> >      {
> >        WFSO(mutex);
> >        NtQueryInformationFile(FilePipeLocalInformation);
> >        if (!fpli.ReadDataAvailable
> > 	  && num_requested > fpli.InboundQuota / 4)
> > 	num_requested = fpli.InboundQuota / 4;
> >        NtReadFile(pipe, buf, num_requested);
> >        ReleaseMutex(mutex);
> >      }
> > }
> > 
> > It's not entirely foolproof, but it should fix 99% of the cases.
> 
> I like it!
> 
> Do you think there's anything we can or should do to avoid a deadlock in the
> rare cases where this fails?  The only thing I can think of immediately is
> to always impose a timeout if select is called with infinite timeout on the
> write side of a pipe, after which we report that the pipe is write ready.
> After all, we've lived since 2008 with a bug that caused select to *always*
> report write ready.

Indeed.  Hmm.  What timeout are you thinking of?  Seconds?  Minutes?

> Alternatively, we could just wait and see if there's an actual use case in
> which someone encounters a deadlock.

Or that.  Fixing up select isn't too hard in that case, I guess.


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-02 19:00                                                                                 ` Corinna Vinschen
@ 2021-09-02 19:34                                                                                   ` Ken Brown
  2021-09-02 19:35                                                                                   ` Corinna Vinschen
  1 sibling, 0 replies; 250+ messages in thread
From: Ken Brown @ 2021-09-02 19:34 UTC (permalink / raw)
  To: cygwin-developers

On 9/2/2021 3:00 PM, Corinna Vinschen wrote:
> On Sep  2 09:01, Ken Brown wrote:
>> On 9/2/2021 4:17 AM, Corinna Vinschen wrote:
>>> What if the readers never request more than, say, 50 or even 25% of the
>>> available buffer space?  Our buffer is 64K and there's no guarantee that
>>> any read > PIPE_BUF (== 4K) is atomic anyway.  This can work without
>>> having to check the other side of the pipe.  Something like this,
>>> ignoring border cases:
>>>
>>> pipe::create()
>>> {
>>>      [...]
>>>      mutex = CreateMutex();
>>> }
>>>
>>> pipe::raw_read(char *buf, size_t num_requested)
>>> {
>>>     if (blocking)
>>>       {
>>>         WFSO(mutex);
>>>         NtQueryInformationFile(FilePipeLocalInformation);
>>>         if (!fpli.ReadDataAvailable
>>> 	  && num_requested > fpli.InboundQuota / 4)
>>> 	num_requested = fpli.InboundQuota / 4;
>>>         NtReadFile(pipe, buf, num_requested);
>>>         ReleaseMutex(mutex);
>>>       }
>>> }
>>>
>>> It's not entirely foolproof, but it should fix 99% of the cases.
>>
>> I like it!
>>
>> Do you think there's anything we can or should do to avoid a deadlock in the
>> rare cases where this fails?  The only thing I can think of immediately is
>> to always impose a timeout if select is called with infinite timeout on the
>> write side of a pipe, after which we report that the pipe is write ready.
>> After all, we've lived since 2008 with a bug that caused select to *always*
>> report write ready.
> 
> Indeed.  Hmm.  What timeout are you thinking of?  Seconds?  Minutes?

Probably seconds (maybe 20, like the connect timeout for sockets?), but it's 
hard to know without seeing naturally occurring cases where this happens.

>> Alternatively, we could just wait and see if there's an actual use case in
>> which someone encounters a deadlock.
> 
> Or that.  Fixing up select isn't too hard in that case, I guess.

I think this would be my preference, at least during testing (if we can persuade 
people to test it).

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-02 19:00                                                                                 ` Corinna Vinschen
  2021-09-02 19:34                                                                                   ` Ken Brown
@ 2021-09-02 19:35                                                                                   ` Corinna Vinschen
  2021-09-02 20:19                                                                                     ` Ken Brown
                                                                                                       ` (2 more replies)
  1 sibling, 3 replies; 250+ messages in thread
From: Corinna Vinschen @ 2021-09-02 19:35 UTC (permalink / raw)
  To: cygwin-developers

[-- Attachment #1: Type: text/plain, Size: 2566 bytes --]

On Sep  2 21:00, Corinna Vinschen wrote:
> On Sep  2 09:01, Ken Brown wrote:
> > On 9/2/2021 4:17 AM, Corinna Vinschen wrote:
> > > What if the readers never request more than, say, 50 or even 25% of the
> > > available buffer space?  Our buffer is 64K and there's no guarantee that
> > > any read > PIPE_BUF (== 4K) is atomic anyway.  This can work without
> > > having to check the other side of the pipe.  Something like this,
> > > ignoring border cases:
> > > 
> > > pipe::create()
> > > {
> > >     [...]
> > >     mutex = CreateMutex();
> > > }
> > > 
> > > pipe::raw_read(char *buf, size_t num_requested)
> > > {
> > >    if (blocking)
> > >      {
> > >        WFSO(mutex);
> > >        NtQueryInformationFile(FilePipeLocalInformation);
> > >        if (!fpli.ReadDataAvailable
> > > 	  && num_requested > fpli.InboundQuota / 4)
> > > 	num_requested = fpli.InboundQuota / 4;
> > >        NtReadFile(pipe, buf, num_requested);
> > >        ReleaseMutex(mutex);
> > >      }
> > > }
> > > 
> > > It's not entirely foolproof, but it should fix 99% of the cases.
> > 
> > I like it!
> > 
> > Do you think there's anything we can or should do to avoid a deadlock in the
> > rare cases where this fails?  The only thing I can think of immediately is
> > to always impose a timeout if select is called with infinite timeout on the
> > write side of a pipe, after which we report that the pipe is write ready.
> > After all, we've lived since 2008 with a bug that caused select to *always*
> > report write ready.
> 
> Indeed.  Hmm.  What timeout are you thinking of?  Seconds?  Minutes?
> 
> > Alternatively, we could just wait and see if there's an actual use case in
> > which someone encounters a deadlock.
> 
> Or that.  Fixing up select isn't too hard in that case, I guess.

It's getting too late again.  I drop off for tonight, but I attached
my POC code I have so far.  It also adds the snippets from my previous
patch which fixes stuff Takashi found during testing.  It also fixes
something which looks like a bug in raw_write:

-	  ptr = ((char *) ptr) + chunk;
+	  ptr = ((char *) ptr) + nbytes_now;

Incrementing ptr by chunk bytes while only nbytes_now have been written
looks incorrect.

As for the reader, it makes the # of bytes to read dependent on the
number of reader handles.  I don't know if that's such a bright idea,
but this can be changed easily.

Anyway, this runs all my testcases successfully but they are anything
but thorough.

Patch relativ to topic/pipe attached.  Would you both mind to take a
scrutinizing look?


Thanks,
Corinna

[-- Attachment #2: pipe.diff --]
[-- Type: text/plain, Size: 10810 bytes --]

diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
index 132e6002133b..032ab5fb07ae 100644
--- a/winsup/cygwin/fhandler.h
+++ b/winsup/cygwin/fhandler.h
@@ -1171,6 +1171,7 @@ class fhandler_socket_unix : public fhandler_socket
 class fhandler_pipe: public fhandler_base
 {
 private:
+  HANDLE read_mtx;
   pid_t popen_pid;
   size_t max_atomic_write;
   void set_pipe_non_blocking (bool nonblocking);
@@ -1178,6 +1179,7 @@ public:
   fhandler_pipe ();
 
   bool ispipe() const { return true; }
+  void set_read_mutex (HANDLE mtx) { read_mtx = mtx; }
 
   void set_popen_pid (pid_t pid) {popen_pid = pid;}
   pid_t get_popen_pid () const {return popen_pid;}
@@ -1187,7 +1189,9 @@ public:
   select_record *select_except (select_stuff *);
   char *get_proc_fd_name (char *buf);
   int open (int flags, mode_t mode = 0);
+  void fixup_after_fork (HANDLE);
   int dup (fhandler_base *child, int);
+  int close ();
   void __reg3 raw_read (void *ptr, size_t& len);
   ssize_t __reg3 raw_write (const void *ptr, size_t len);
   int ioctl (unsigned int cmd, void *);
diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index 2dec0a84817c..7a5cefb3d07c 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -240,8 +240,37 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
       keep_looping = false;
       if (evt)
 	ResetEvent (evt);
+      if (!is_nonblocking ())
+	{
+	  FILE_PIPE_LOCAL_INFORMATION fpli;
+	  ULONG reader_count;
+	  ULONG max_len = 64;
+
+	  WaitForSingleObject (read_mtx, INFINITE);
+
+	  /* Make sure never to request more bytes than half the pipe
+	     buffer size.  Every pending read lowers WriteQuotaAvailable
+	     on the write side and thus affects select's ability to return
+	     more or less reliable info whether a write succeeds or not.
+
+	     Let the size of the request depend on the number of readers
+	     at the time. */
+	  status = NtQueryInformationFile (get_handle (), &io,
+					   &fpli, sizeof (fpli),
+					   FilePipeLocalInformation);
+	  if (NT_SUCCESS (status) && fpli.ReadDataAvailable == 0)
+	    {
+	      reader_count = get_obj_handle_count (get_handle ());
+	      if (reader_count < 10)
+		max_len = fpli.InboundQuota / (2 * reader_count);
+	      if (len > max_len)
+		len = max_len;
+	    }
+	}
       status = NtReadFile (get_handle (), evt, NULL, NULL, &io, ptr,
 			   len, NULL, NULL);
+      if (!is_nonblocking ())
+	ReleaseMutex (read_mtx);
       if (evt && status == STATUS_PENDING)
 	{
 	  waitret = cygwait (evt);
@@ -313,7 +342,6 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
 ssize_t __reg3
 fhandler_pipe::raw_write (const void *ptr, size_t len)
 {
-  ssize_t ret = -1;
   size_t nbytes = 0;
   ULONG chunk;
   NTSTATUS status = STATUS_SUCCESS;
@@ -352,8 +380,36 @@ fhandler_pipe::raw_write (const void *ptr, size_t len)
       else
 	len1 = (ULONG) left;
       nbytes_now = 0;
-      status = NtWriteFile (get_handle (), evt, NULL, NULL, &io,
-			    (PVOID) ptr, len1, NULL, NULL);
+      /* NtWriteFile returns success with # of bytes written == 0 if writing
+         on a non-blocking pipe fails because the pipe buffer doesn't have
+	 sufficient space.
+
+	 POSIX requires
+	 - A write request for {PIPE_BUF} or fewer bytes shall have the
+	   following effect: if there is sufficient space available in the
+	   pipe, write() shall transfer all the data and return the number
+	   of bytes requested. Otherwise, write() shall transfer no data and
+	   return -1 with errno set to [EAGAIN].
+
+	 - A write request for more than {PIPE_BUF} bytes shall cause one
+	   of the following:
+
+	  - When at least one byte can be written, transfer what it can and
+	    return the number of bytes written. When all data previously
+	    written to the pipe is read, it shall transfer at least {PIPE_BUF}
+	    bytes.
+
+	  - When no data can be written, transfer no data, and return -1 with
+	    errno set to [EAGAIN]. */
+      while (len1 > 0)
+	{
+	  status = NtWriteFile (get_handle (), evt, NULL, NULL, &io,
+				(PVOID) ptr, len1, NULL, NULL);
+	  if (evt || !NT_SUCCESS (status) || io.Information > 0
+	      || len <= PIPE_BUF)
+	    break;
+	  len1 >>= 1;
+	}
       if (evt && status == STATUS_PENDING)
 	{
 	  waitret = cygwait (evt);
@@ -375,13 +431,11 @@ fhandler_pipe::raw_write (const void *ptr, size_t len)
       else if (NT_SUCCESS (status))
 	{
 	  nbytes_now = io.Information;
-	  /* NtWriteFile returns success with # of bytes written == 0
-	     if writing on a non-blocking pipe fails because the pipe
-	     buffer doesn't have sufficient space. */
-	  if (nbytes_now == 0)
-	    set_errno (EAGAIN);
-	  ptr = ((char *) ptr) + chunk;
+	  ptr = ((char *) ptr) + nbytes_now;
 	  nbytes += nbytes_now;
+	  /* 0 bytes returned?  EAGAIN.  See above. */
+	  if (nbytes == 0)
+	    set_errno (EAGAIN);
 	}
       else if (STATUS_PIPE_IS_CLOSED (status))
 	{
@@ -392,17 +446,23 @@ fhandler_pipe::raw_write (const void *ptr, size_t len)
 	__seterrno_from_nt_status (status);
 
       if (nbytes_now == 0)
-	len = 0;		/* Terminate loop. */
-      if (nbytes > 0)
-	ret = nbytes;
+	break;
     }
   if (evt)
     CloseHandle (evt);
-  if (status == STATUS_THREAD_SIGNALED && ret < 0)
+  if (status == STATUS_THREAD_SIGNALED && nbytes == 0)
     set_errno (EINTR);
   else if (status == STATUS_THREAD_CANCELED)
     pthread::static_cancel_self ();
-  return ret;
+  return nbytes ?: -1;
+}
+
+void
+fhandler_pipe::fixup_after_fork (HANDLE parent)
+{
+  if (read_mtx)
+    fork_fixup (parent, read_mtx, "read_mtx");
+  fhandler_base::fixup_after_fork (parent);
 }
 
 int
@@ -411,16 +471,31 @@ fhandler_pipe::dup (fhandler_base *child, int flags)
   fhandler_pipe *ftp = (fhandler_pipe *) child;
   ftp->set_popen_pid (0);
 
-  int res;
-  if (get_handle () && fhandler_base::dup (child, flags))
+  int res = 0;
+  if (fhandler_base::dup (child, flags))
     res = -1;
-  else
-    res = 0;
+  else if (read_mtx &&
+	   !DuplicateHandle (GetCurrentProcess (), read_mtx,
+			     GetCurrentProcess (), &ftp->read_mtx,
+			     0, !(flags & O_CLOEXEC), DUPLICATE_SAME_ACCESS))
+    {
+      __seterrno ();
+      ftp->close ();
+      res = -1;
+    }
 
   debug_printf ("res %d", res);
   return res;
 }
 
+int
+fhandler_pipe::close ()
+{
+  if (read_mtx)
+    NtClose (read_mtx);
+  return fhandler_base::close ();
+}
+
 #define PIPE_INTRO "\\\\.\\pipe\\cygwin-"
 
 /* Create a pipe, and return handles to the read and write ends,
@@ -608,6 +683,7 @@ fhandler_pipe::create (fhandler_pipe *fhs[2], unsigned psize, int mode)
   else if ((fhs[1] = (fhandler_pipe *) build_fh_dev (*pipew_dev)) == NULL)
     {
       delete fhs[0];
+      CloseHandle (r);
       CloseHandle (w);
     }
   else
@@ -617,7 +693,25 @@ fhandler_pipe::create (fhandler_pipe *fhs[2], unsigned psize, int mode)
 		    unique_id);
       fhs[1]->init (w, FILE_CREATE_PIPE_INSTANCE | GENERIC_WRITE, mode,
 		    unique_id);
-      res = 0;
+      /* For the read side of the pipe, add a mutex.  See raw_read for the
+	 usage. */
+      SECURITY_ATTRIBUTES sa = { .nLength = sizeof (SECURITY_ATTRIBUTES),
+				 .lpSecurityDescriptor = NULL,
+				 .bInheritHandle = !(mode & O_CLOEXEC)
+			       };
+      HANDLE mtx = CreateMutexW (&sa, FALSE, NULL);
+      if (!mtx)
+	{
+	  delete fhs[0];
+	  CloseHandle (r);
+	  delete fhs[1];
+	  CloseHandle (w);
+	}
+      else
+	{
+	  fhs[0]->set_read_mutex (mtx);
+	  res = 0;
+	}
     }
 
   debug_printf ("%R = pipe([%p, %p], %d, %y)", res, fhs[0], fhs[1], psize, mode);
@@ -658,7 +752,7 @@ nt_create (LPSECURITY_ATTRIBUTES sa_ptr, PHANDLE r, PHANDLE w,
 				 &cygheap->installation_key,
 				 GetCurrentProcessId ());
 
-  access = GENERIC_READ | FILE_WRITE_ATTRIBUTES;
+  access = GENERIC_READ | FILE_WRITE_ATTRIBUTES | SYNCHRONIZE;
 
   ULONG pipe_type = pipe_byte ? FILE_PIPE_BYTE_STREAM_TYPE
     : FILE_PIPE_MESSAGE_TYPE;
@@ -737,7 +831,7 @@ nt_create (LPSECURITY_ATTRIBUTES sa_ptr, PHANDLE r, PHANDLE w,
     {
       debug_printf ("NtOpenFile: name %S", &pipename);
 
-      access = GENERIC_WRITE | FILE_READ_ATTRIBUTES;
+      access = GENERIC_WRITE | FILE_READ_ATTRIBUTES | SYNCHRONIZE;
       status = NtOpenFile (w, access, &attr, &io, 0, 0);
       if (!NT_SUCCESS (status))
 	{
diff --git a/winsup/cygwin/flock.cc b/winsup/cygwin/flock.cc
index bd7a16d91ecd..2f12fc07e37b 100644
--- a/winsup/cygwin/flock.cc
+++ b/winsup/cygwin/flock.cc
@@ -216,22 +216,6 @@ allow_others_to_sync ()
   done = true;
 }
 
-/* Get the handle count of an object. */
-static ULONG
-get_obj_handle_count (HANDLE h)
-{
-  OBJECT_BASIC_INFORMATION obi;
-  NTSTATUS status;
-  ULONG hdl_cnt = 0;
-
-  status = NtQueryObject (h, ObjectBasicInformation, &obi, sizeof obi, NULL);
-  if (!NT_SUCCESS (status))
-    debug_printf ("NtQueryObject: %y", status);
-  else
-    hdl_cnt = obi.HandleCount;
-  return hdl_cnt;
-}
-
 /* Helper struct to construct a local OBJECT_ATTRIBUTES on the stack. */
 struct lockfattr_t
 {
diff --git a/winsup/cygwin/miscfuncs.cc b/winsup/cygwin/miscfuncs.cc
index f4c3a1c48e8e..dc36030ca572 100644
--- a/winsup/cygwin/miscfuncs.cc
+++ b/winsup/cygwin/miscfuncs.cc
@@ -18,6 +18,22 @@ details. */
 #include "tls_pbuf.h"
 #include "mmap_alloc.h"
 
+/* Get handle count of an object. */
+ULONG
+get_obj_handle_count (HANDLE h)
+{
+  OBJECT_BASIC_INFORMATION obi;
+  NTSTATUS status;
+  ULONG hdl_cnt = 0;
+
+  status = NtQueryObject (h, ObjectBasicInformation, &obi, sizeof obi, NULL);
+  if (!NT_SUCCESS (status))
+    debug_printf ("NtQueryObject: %y", status);
+  else
+    hdl_cnt = obi.HandleCount;
+  return hdl_cnt;
+}
+
 int __reg2
 check_invalid_virtual_addr (const void *s, unsigned sz)
 {
diff --git a/winsup/cygwin/miscfuncs.h b/winsup/cygwin/miscfuncs.h
index 1ff7ee0d3fde..47cef6f20c0a 100644
--- a/winsup/cygwin/miscfuncs.h
+++ b/winsup/cygwin/miscfuncs.h
@@ -98,6 +98,9 @@ transform_chars (PUNICODE_STRING upath, USHORT start_idx)
 
 PWCHAR transform_chars_af_unix (PWCHAR, const char *, __socklen_t);
 
+/* Get handle count of an object. */
+ULONG get_obj_handle_count (HANDLE h);
+
 /* Memory checking */
 int __reg2 check_invalid_virtual_addr (const void *s, unsigned sz);
 
diff --git a/winsup/cygwin/select.cc b/winsup/cygwin/select.cc
index 83e1c00e0ac7..ac2fd227eb17 100644
--- a/winsup/cygwin/select.cc
+++ b/winsup/cygwin/select.cc
@@ -612,7 +612,6 @@ pipe_data_available (int fd, fhandler_base *fh, HANDLE h, bool writing)
 	   that.  This means that a pipe could still block since you could
 	   be trying to write more to the pipe than is available in the
 	   buffer but that is the hazard of select().  */
-      fpli.WriteQuotaAvailable = fpli.OutboundQuota - fpli.ReadDataAvailable;
       if (fpli.WriteQuotaAvailable > 0)
 	{
 	  paranoid_printf ("fd %d, %s, write: size %u, avail %u", fd,

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-02 19:35                                                                                   ` Corinna Vinschen
@ 2021-09-02 20:19                                                                                     ` Ken Brown
  2021-09-03  9:12                                                                                       ` Corinna Vinschen
  2021-09-03 10:00                                                                                     ` Takashi Yano
  2021-09-08 11:32                                                                                     ` Takashi Yano
  2 siblings, 1 reply; 250+ messages in thread
From: Ken Brown @ 2021-09-02 20:19 UTC (permalink / raw)
  To: cygwin-developers

On 9/2/2021 3:35 PM, Corinna Vinschen wrote:
> On Sep  2 21:00, Corinna Vinschen wrote:
>> On Sep  2 09:01, Ken Brown wrote:
>>> On 9/2/2021 4:17 AM, Corinna Vinschen wrote:
>>>> What if the readers never request more than, say, 50 or even 25% of the
>>>> available buffer space?  Our buffer is 64K and there's no guarantee that
>>>> any read > PIPE_BUF (== 4K) is atomic anyway.  This can work without
>>>> having to check the other side of the pipe.  Something like this,
>>>> ignoring border cases:
>>>>
>>>> pipe::create()
>>>> {
>>>>      [...]
>>>>      mutex = CreateMutex();
>>>> }
>>>>
>>>> pipe::raw_read(char *buf, size_t num_requested)
>>>> {
>>>>     if (blocking)
>>>>       {
>>>>         WFSO(mutex);
>>>>         NtQueryInformationFile(FilePipeLocalInformation);
>>>>         if (!fpli.ReadDataAvailable
>>>> 	  && num_requested > fpli.InboundQuota / 4)
>>>> 	num_requested = fpli.InboundQuota / 4;
>>>>         NtReadFile(pipe, buf, num_requested);
>>>>         ReleaseMutex(mutex);
>>>>       }
>>>> }
>>>>
>>>> It's not entirely foolproof, but it should fix 99% of the cases.
>>>
>>> I like it!
>>>
>>> Do you think there's anything we can or should do to avoid a deadlock in the
>>> rare cases where this fails?  The only thing I can think of immediately is
>>> to always impose a timeout if select is called with infinite timeout on the
>>> write side of a pipe, after which we report that the pipe is write ready.
>>> After all, we've lived since 2008 with a bug that caused select to *always*
>>> report write ready.
>>
>> Indeed.  Hmm.  What timeout are you thinking of?  Seconds?  Minutes?
>>
>>> Alternatively, we could just wait and see if there's an actual use case in
>>> which someone encounters a deadlock.
>>
>> Or that.  Fixing up select isn't too hard in that case, I guess.
> 
> It's getting too late again.  I drop off for tonight, but I attached
> my POC code I have so far.  It also adds the snippets from my previous
> patch which fixes stuff Takashi found during testing.  It also fixes
> something which looks like a bug in raw_write:
> 
> -	  ptr = ((char *) ptr) + chunk;
> +	  ptr = ((char *) ptr) + nbytes_now;
> 
> Incrementing ptr by chunk bytes while only nbytes_now have been written
> looks incorrect.

Yes.  I actually copied that bug from fhandler_base_overlapped::raw_write.  It's 
amazing that no one has no one has bumped into it up to now.
> As for the reader, it makes the # of bytes to read dependent on the
> number of reader handles.  I don't know if that's such a bright idea,
> but this can be changed easily.
> 
> Anyway, this runs all my testcases successfully but they are anything
> but thorough.
> 
> Patch relativ to topic/pipe attached.  Would you both mind to take a
> scrutinizing look?

Will do.

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-02 20:19                                                                                     ` Ken Brown
@ 2021-09-03  9:12                                                                                       ` Corinna Vinschen
  2021-09-03 19:00                                                                                         ` Ken Brown
  0 siblings, 1 reply; 250+ messages in thread
From: Corinna Vinschen @ 2021-09-03  9:12 UTC (permalink / raw)
  To: cygwin-developers

On Sep  2 16:19, Ken Brown wrote:
> On 9/2/2021 3:35 PM, Corinna Vinschen wrote:
> > On Sep  2 21:00, Corinna Vinschen wrote:
> > > On Sep  2 09:01, Ken Brown wrote:
> > > > Do you think there's anything we can or should do to avoid a deadlock in the
> > > > rare cases where this fails?  The only thing I can think of immediately is
> > > > to always impose a timeout if select is called with infinite timeout on the
> > > > write side of a pipe, after which we report that the pipe is write ready.
> > > > After all, we've lived since 2008 with a bug that caused select to *always*
> > > > report write ready.
> > > 
> > > Indeed.  Hmm.  What timeout are you thinking of?  Seconds?  Minutes?
> > > 
> > > > Alternatively, we could just wait and see if there's an actual use case in
> > > > which someone encounters a deadlock.
> > > 
> > > Or that.  Fixing up select isn't too hard in that case, I guess.
> > 
> > It's getting too late again.  I drop off for tonight, but I attached
> > my POC code I have so far.  It also adds the snippets from my previous
> > patch which fixes stuff Takashi found during testing.  It also fixes
> > something which looks like a bug in raw_write:
> > 
> > -	  ptr = ((char *) ptr) + chunk;
> > +	  ptr = ((char *) ptr) + nbytes_now;
> > 
> > Incrementing ptr by chunk bytes while only nbytes_now have been written
> > looks incorrect.
> 
> Yes.  I actually copied that bug from fhandler_base_overlapped::raw_write.
> It's amazing that no one has no one has bumped into it up to now.
> > As for the reader, it makes the # of bytes to read dependent on the
> > number of reader handles.  I don't know if that's such a bright idea,
> > but this can be changed easily.
> > 
> > Anyway, this runs all my testcases successfully but they are anything
> > but thorough.
> > 
> > Patch relativ to topic/pipe attached.  Would you both mind to take a
> > scrutinizing look?
> 
> Will do.

I pushed my stuff to the topic/pipe branch split into hopefully useful
chunks.  Kick me if anything is wrong or not working.


Thanks,
Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-02 19:35                                                                                   ` Corinna Vinschen
  2021-09-02 20:19                                                                                     ` Ken Brown
@ 2021-09-03 10:00                                                                                     ` Takashi Yano
  2021-09-03 10:13                                                                                       ` Takashi Yano
  2021-09-03 10:38                                                                                       ` Takashi Yano
  2021-09-08 11:32                                                                                     ` Takashi Yano
  2 siblings, 2 replies; 250+ messages in thread
From: Takashi Yano @ 2021-09-03 10:00 UTC (permalink / raw)
  To: cygwin-developers

On Thu, 2 Sep 2021 21:35:21 +0200
Corinna Vinschen wrote:
> On Sep  2 21:00, Corinna Vinschen wrote:
> > On Sep  2 09:01, Ken Brown wrote:
> > > On 9/2/2021 4:17 AM, Corinna Vinschen wrote:
> > > > What if the readers never request more than, say, 50 or even 25% of the
> > > > available buffer space?  Our buffer is 64K and there's no guarantee that
> > > > any read > PIPE_BUF (== 4K) is atomic anyway.  This can work without
> > > > having to check the other side of the pipe.  Something like this,
> > > > ignoring border cases:
> > > > 
> > > > pipe::create()
> > > > {
> > > >     [...]
> > > >     mutex = CreateMutex();
> > > > }
> > > > 
> > > > pipe::raw_read(char *buf, size_t num_requested)
> > > > {
> > > >    if (blocking)
> > > >      {
> > > >        WFSO(mutex);
> > > >        NtQueryInformationFile(FilePipeLocalInformation);
> > > >        if (!fpli.ReadDataAvailable
> > > > 	  && num_requested > fpli.InboundQuota / 4)
> > > > 	num_requested = fpli.InboundQuota / 4;
> > > >        NtReadFile(pipe, buf, num_requested);
> > > >        ReleaseMutex(mutex);
> > > >      }
> > > > }
> > > > 
> > > > It's not entirely foolproof, but it should fix 99% of the cases.
> > > 
> > > I like it!
> > > 
> > > Do you think there's anything we can or should do to avoid a deadlock in the
> > > rare cases where this fails?  The only thing I can think of immediately is
> > > to always impose a timeout if select is called with infinite timeout on the
> > > write side of a pipe, after which we report that the pipe is write ready.
> > > After all, we've lived since 2008 with a bug that caused select to *always*
> > > report write ready.
> > 
> > Indeed.  Hmm.  What timeout are you thinking of?  Seconds?  Minutes?
> > 
> > > Alternatively, we could just wait and see if there's an actual use case in
> > > which someone encounters a deadlock.
> > 
> > Or that.  Fixing up select isn't too hard in that case, I guess.
> 
> It's getting too late again.  I drop off for tonight, but I attached
> my POC code I have so far.  It also adds the snippets from my previous
> patch which fixes stuff Takashi found during testing.  It also fixes
> something which looks like a bug in raw_write:
> 
> -	  ptr = ((char *) ptr) + chunk;
> +	  ptr = ((char *) ptr) + nbytes_now;
> 
> Incrementing ptr by chunk bytes while only nbytes_now have been written
> looks incorrect.
> 
> As for the reader, it makes the # of bytes to read dependent on the
> number of reader handles.  I don't know if that's such a bright idea,
> but this can be changed easily.
> 
> Anyway, this runs all my testcases successfully but they are anything
> but thorough.
> 
> Patch relativ to topic/pipe attached.  Would you both mind to take a
> scrutinizing look?

Thanks.

Your code seems that read() returns only the partial data even
if the pipe stil has more data. Is this by design?

This happes in both blocking and non-blocking case.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-03 10:00                                                                                     ` Takashi Yano
@ 2021-09-03 10:13                                                                                       ` Takashi Yano
  2021-09-03 11:31                                                                                         ` Corinna Vinschen
  2021-09-03 10:38                                                                                       ` Takashi Yano
  1 sibling, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-03 10:13 UTC (permalink / raw)
  To: cygwin-developers

[-- Attachment #1: Type: text/plain, Size: 3212 bytes --]

On Fri, 3 Sep 2021 19:00:46 +0900
Takashi Yano wrote:
> On Thu, 2 Sep 2021 21:35:21 +0200
> Corinna Vinschen wrote:
> > On Sep  2 21:00, Corinna Vinschen wrote:
> > > On Sep  2 09:01, Ken Brown wrote:
> > > > On 9/2/2021 4:17 AM, Corinna Vinschen wrote:
> > > > > What if the readers never request more than, say, 50 or even 25% of the
> > > > > available buffer space?  Our buffer is 64K and there's no guarantee that
> > > > > any read > PIPE_BUF (== 4K) is atomic anyway.  This can work without
> > > > > having to check the other side of the pipe.  Something like this,
> > > > > ignoring border cases:
> > > > > 
> > > > > pipe::create()
> > > > > {
> > > > >     [...]
> > > > >     mutex = CreateMutex();
> > > > > }
> > > > > 
> > > > > pipe::raw_read(char *buf, size_t num_requested)
> > > > > {
> > > > >    if (blocking)
> > > > >      {
> > > > >        WFSO(mutex);
> > > > >        NtQueryInformationFile(FilePipeLocalInformation);
> > > > >        if (!fpli.ReadDataAvailable
> > > > > 	  && num_requested > fpli.InboundQuota / 4)
> > > > > 	num_requested = fpli.InboundQuota / 4;
> > > > >        NtReadFile(pipe, buf, num_requested);
> > > > >        ReleaseMutex(mutex);
> > > > >      }
> > > > > }
> > > > > 
> > > > > It's not entirely foolproof, but it should fix 99% of the cases.
> > > > 
> > > > I like it!
> > > > 
> > > > Do you think there's anything we can or should do to avoid a deadlock in the
> > > > rare cases where this fails?  The only thing I can think of immediately is
> > > > to always impose a timeout if select is called with infinite timeout on the
> > > > write side of a pipe, after which we report that the pipe is write ready.
> > > > After all, we've lived since 2008 with a bug that caused select to *always*
> > > > report write ready.
> > > 
> > > Indeed.  Hmm.  What timeout are you thinking of?  Seconds?  Minutes?
> > > 
> > > > Alternatively, we could just wait and see if there's an actual use case in
> > > > which someone encounters a deadlock.
> > > 
> > > Or that.  Fixing up select isn't too hard in that case, I guess.
> > 
> > It's getting too late again.  I drop off for tonight, but I attached
> > my POC code I have so far.  It also adds the snippets from my previous
> > patch which fixes stuff Takashi found during testing.  It also fixes
> > something which looks like a bug in raw_write:
> > 
> > -	  ptr = ((char *) ptr) + chunk;
> > +	  ptr = ((char *) ptr) + nbytes_now;
> > 
> > Incrementing ptr by chunk bytes while only nbytes_now have been written
> > looks incorrect.
> > 
> > As for the reader, it makes the # of bytes to read dependent on the
> > number of reader handles.  I don't know if that's such a bright idea,
> > but this can be changed easily.
> > 
> > Anyway, this runs all my testcases successfully but they are anything
> > but thorough.
> > 
> > Patch relativ to topic/pipe attached.  Would you both mind to take a
> > scrutinizing look?
> 
> Thanks.
> 
> Your code seems that read() returns only the partial data even
> if the pipe stil has more data. Is this by design?
> 
> This happes in both blocking and non-blocking case.

The patch attached seems to fix the issue.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

[-- Attachment #2: pipe-corinna4-fix.patch --]
[-- Type: application/octet-stream, Size: 3483 bytes --]

diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index 7a5cefb3d..5b6b98892 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -222,6 +222,7 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
   DWORD waitret = WAIT_OBJECT_0;
   bool keep_looping = false;
   size_t orig_len = len;
+  size_t total_len = 0;
 
   if (!len)
     return;
@@ -236,29 +237,37 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
 
   do
     {
-      len = orig_len;
+      char *ptr1 = (char *) ptr + total_len;
+      len = orig_len - total_len;
       keep_looping = false;
       if (evt)
 	ResetEvent (evt);
-      if (!is_nonblocking ())
+
+      FILE_PIPE_LOCAL_INFORMATION fpli;
+      ULONG reader_count;
+      ULONG max_len = 64;
+
+      WaitForSingleObject (read_mtx, INFINITE);
+
+      /* Make sure never to request more bytes than half the pipe
+	 buffer size.  Every pending read lowers WriteQuotaAvailable
+	 on the write side and thus affects select's ability to return
+	 more or less reliable info whether a write succeeds or not.
+
+	 Let the size of the request depend on the number of readers
+	 at the time. */
+      status = NtQueryInformationFile (get_handle (), &io,
+				       &fpli, sizeof (fpli),
+				       FilePipeLocalInformation);
+      if (NT_SUCCESS (status) && fpli.ReadDataAvailable == 0)
 	{
-	  FILE_PIPE_LOCAL_INFORMATION fpli;
-	  ULONG reader_count;
-	  ULONG max_len = 64;
-
-	  WaitForSingleObject (read_mtx, INFINITE);
-
-	  /* Make sure never to request more bytes than half the pipe
-	     buffer size.  Every pending read lowers WriteQuotaAvailable
-	     on the write side and thus affects select's ability to return
-	     more or less reliable info whether a write succeeds or not.
-
-	     Let the size of the request depend on the number of readers
-	     at the time. */
-	  status = NtQueryInformationFile (get_handle (), &io,
-					   &fpli, sizeof (fpli),
-					   FilePipeLocalInformation);
-	  if (NT_SUCCESS (status) && fpli.ReadDataAvailable == 0)
+	  if (total_len != 0)
+	    {
+	      len = total_len;
+	      ReleaseMutex (read_mtx);
+	      break;
+	    }
+	  if (!is_nonblocking ())
 	    {
 	      reader_count = get_obj_handle_count (get_handle ());
 	      if (reader_count < 10)
@@ -267,10 +276,11 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
 		len = max_len;
 	    }
 	}
-      status = NtReadFile (get_handle (), evt, NULL, NULL, &io, ptr,
+
+      status = NtReadFile (get_handle (), evt, NULL, NULL, &io, ptr1,
 			   len, NULL, NULL);
-      if (!is_nonblocking ())
-	ReleaseMutex (read_mtx);
+      ReleaseMutex (read_mtx);
+
       if (evt && status == STATUS_PENDING)
 	{
 	  waitret = cygwait (evt);
@@ -291,9 +301,11 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
 	}
       else if (NT_SUCCESS (status))
 	{
-	  len = io.Information;
-	  if (len == 0)
+	  total_len += io.Information;
+	  if (total_len < orig_len)
 	    keep_looping = true;
+	  else
+	    len = total_len;
 	}
       else
 	{
@@ -308,9 +320,11 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
 	    case STATUS_MORE_ENTRIES:
 	    case STATUS_BUFFER_OVERFLOW:
 	      /* `io.Information' is supposedly valid.  */
-	      len = io.Information;
-	      if (len == 0)
+	      total_len += io.Information;
+	      if (total_len < orig_len)
 		keep_looping = true;
+	      else
+		len = total_len;
 	      break;
 	    case STATUS_PIPE_LISTENING:
 	    case STATUS_PIPE_EMPTY:

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-03 10:00                                                                                     ` Takashi Yano
  2021-09-03 10:13                                                                                       ` Takashi Yano
@ 2021-09-03 10:38                                                                                       ` Takashi Yano
  1 sibling, 0 replies; 250+ messages in thread
From: Takashi Yano @ 2021-09-03 10:38 UTC (permalink / raw)
  To: cygwin-developers

On Fri, 3 Sep 2021 19:00:46 +0900
Takashi Yano wrote:
> On Thu, 2 Sep 2021 21:35:21 +0200
> Corinna Vinschen wrote:
> > On Sep  2 21:00, Corinna Vinschen wrote:
> > > On Sep  2 09:01, Ken Brown wrote:
> > > > On 9/2/2021 4:17 AM, Corinna Vinschen wrote:
> > > > > What if the readers never request more than, say, 50 or even 25% of the
> > > > > available buffer space?  Our buffer is 64K and there's no guarantee that
> > > > > any read > PIPE_BUF (== 4K) is atomic anyway.  This can work without
> > > > > having to check the other side of the pipe.  Something like this,
> > > > > ignoring border cases:
> > > > > 
> > > > > pipe::create()
> > > > > {
> > > > >     [...]
> > > > >     mutex = CreateMutex();
> > > > > }
> > > > > 
> > > > > pipe::raw_read(char *buf, size_t num_requested)
> > > > > {
> > > > >    if (blocking)
> > > > >      {
> > > > >        WFSO(mutex);
> > > > >        NtQueryInformationFile(FilePipeLocalInformation);
> > > > >        if (!fpli.ReadDataAvailable
> > > > > 	  && num_requested > fpli.InboundQuota / 4)
> > > > > 	num_requested = fpli.InboundQuota / 4;
> > > > >        NtReadFile(pipe, buf, num_requested);
> > > > >        ReleaseMutex(mutex);
> > > > >      }
> > > > > }
> > > > > 
> > > > > It's not entirely foolproof, but it should fix 99% of the cases.
> > > > 
> > > > I like it!
> > > > 
> > > > Do you think there's anything we can or should do to avoid a deadlock in the
> > > > rare cases where this fails?  The only thing I can think of immediately is
> > > > to always impose a timeout if select is called with infinite timeout on the
> > > > write side of a pipe, after which we report that the pipe is write ready.
> > > > After all, we've lived since 2008 with a bug that caused select to *always*
> > > > report write ready.
> > > 
> > > Indeed.  Hmm.  What timeout are you thinking of?  Seconds?  Minutes?
> > > 
> > > > Alternatively, we could just wait and see if there's an actual use case in
> > > > which someone encounters a deadlock.
> > > 
> > > Or that.  Fixing up select isn't too hard in that case, I guess.
> > 
> > It's getting too late again.  I drop off for tonight, but I attached
> > my POC code I have so far.  It also adds the snippets from my previous
> > patch which fixes stuff Takashi found during testing.  It also fixes
> > something which looks like a bug in raw_write:
> > 
> > -	  ptr = ((char *) ptr) + chunk;
> > +	  ptr = ((char *) ptr) + nbytes_now;
> > 
> > Incrementing ptr by chunk bytes while only nbytes_now have been written
> > looks incorrect.
> > 
> > As for the reader, it makes the # of bytes to read dependent on the
> > number of reader handles.  I don't know if that's such a bright idea,
> > but this can be changed easily.
> > 
> > Anyway, this runs all my testcases successfully but they are anything
> > but thorough.
> > 
> > Patch relativ to topic/pipe attached.  Would you both mind to take a
> > scrutinizing look?
> 
> Thanks.
> 
> Your code seems that read() returns only the partial data even
> if the pipe stil has more data. Is this by design?
> 
> This happes in both blocking and non-blocking case.

Sorry, this may only happen if pipe is blocking mode.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-03 10:13                                                                                       ` Takashi Yano
@ 2021-09-03 11:31                                                                                         ` Corinna Vinschen
  2021-09-03 11:41                                                                                           ` Corinna Vinschen
  0 siblings, 1 reply; 250+ messages in thread
From: Corinna Vinschen @ 2021-09-03 11:31 UTC (permalink / raw)
  To: cygwin-developers

On Sep  3 19:13, Takashi Yano wrote:
> On Fri, 3 Sep 2021 19:00:46 +0900
> Takashi Yano wrote:
> > On Thu, 2 Sep 2021 21:35:21 +0200
> > Corinna Vinschen wrote:
> > > On Sep  2 21:00, Corinna Vinschen wrote:
> > > It's getting too late again.  I drop off for tonight, but I attached
> > > my POC code I have so far.  It also adds the snippets from my previous
> > > patch which fixes stuff Takashi found during testing.  It also fixes
> > > something which looks like a bug in raw_write:
> > > 
> > > -	  ptr = ((char *) ptr) + chunk;
> > > +	  ptr = ((char *) ptr) + nbytes_now;
> > > 
> > > Incrementing ptr by chunk bytes while only nbytes_now have been written
> > > looks incorrect.
> > > 
> > > As for the reader, it makes the # of bytes to read dependent on the
> > > number of reader handles.  I don't know if that's such a bright idea,
> > > but this can be changed easily.
> > > 
> > > Anyway, this runs all my testcases successfully but they are anything
> > > but thorough.
> > > 
> > > Patch relativ to topic/pipe attached.  Would you both mind to take a
> > > scrutinizing look?
> > 
> > Thanks.
> > 
> > Your code seems that read() returns only the partial data even
> > if the pipe stil has more data. Is this by design?
> > 
> > This happes in both blocking and non-blocking case.
> 
> The patch attached seems to fix the issue.

I'm sorry, but I don't see what your patch is supposed to do in the
first place.  What I see is that it now calls NtQueryInformationFile
even in the non-blocking case, which is not supposed to happen.

I'm a bit puzzled what the actual bug is.

The code changing len is only called if there's no data in the pipe.
In that case we only request a partial buffer so as not to block
the writer on select.

If there *is* data in the pipe, it will just go straight to the
NtReadFile code without changing len.

Where's the mistake?


Thanks,
Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-03 11:31                                                                                         ` Corinna Vinschen
@ 2021-09-03 11:41                                                                                           ` Corinna Vinschen
  2021-09-03 12:13                                                                                             ` Ken Brown
  2021-09-03 12:22                                                                                             ` Takashi Yano
  0 siblings, 2 replies; 250+ messages in thread
From: Corinna Vinschen @ 2021-09-03 11:41 UTC (permalink / raw)
  To: cygwin-developers

On Sep  3 13:31, Corinna Vinschen wrote:
> On Sep  3 19:13, Takashi Yano wrote:
> > On Fri, 3 Sep 2021 19:00:46 +0900
> > Takashi Yano wrote:
> > > On Thu, 2 Sep 2021 21:35:21 +0200
> > > Corinna Vinschen wrote:
> > > > On Sep  2 21:00, Corinna Vinschen wrote:
> > > > It's getting too late again.  I drop off for tonight, but I attached
> > > > my POC code I have so far.  It also adds the snippets from my previous
> > > > patch which fixes stuff Takashi found during testing.  It also fixes
> > > > something which looks like a bug in raw_write:
> > > > 
> > > > -	  ptr = ((char *) ptr) + chunk;
> > > > +	  ptr = ((char *) ptr) + nbytes_now;
> > > > 
> > > > Incrementing ptr by chunk bytes while only nbytes_now have been written
> > > > looks incorrect.
> > > > 
> > > > As for the reader, it makes the # of bytes to read dependent on the
> > > > number of reader handles.  I don't know if that's such a bright idea,
> > > > but this can be changed easily.
> > > > 
> > > > Anyway, this runs all my testcases successfully but they are anything
> > > > but thorough.
> > > > 
> > > > Patch relativ to topic/pipe attached.  Would you both mind to take a
> > > > scrutinizing look?
> > > 
> > > Thanks.
> > > 
> > > Your code seems that read() returns only the partial data even
> > > if the pipe stil has more data. Is this by design?
> > > 
> > > This happes in both blocking and non-blocking case.
> > 
> > The patch attached seems to fix the issue.
> 
> I'm sorry, but I don't see what your patch is supposed to do in the
> first place.  What I see is that it now calls NtQueryInformationFile
> even in the non-blocking case, which is not supposed to happen.
> 
> I'm a bit puzzled what the actual bug is.
> 
> The code changing len is only called if there's no data in the pipe.
> In that case we only request a partial buffer so as not to block
> the writer on select.
> 
> If there *is* data in the pipe, it will just go straight to the
> NtReadFile code without changing len.
> 
> Where's the mistake?

Oh, wait.  Do you mean, if we only request less than len bytes, but
after NtReadFile there's still data in the buffer, we should try to
deplete the buffer up to len bytes in a subsequent NtReadFile?

I thought this is unnecessary, actually, because of POSIX:

   The standard developers considered adding atomicity requirements  to  a
   pipe  or FIFO, but recognized that due to the nature of pipes and FIFOs
   there could be no guarantee of atomicity of reads of {PIPE_BUF} or  any
   other size that would be an aid to applications portability.


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-03 11:41                                                                                           ` Corinna Vinschen
@ 2021-09-03 12:13                                                                                             ` Ken Brown
  2021-09-03 15:00                                                                                               ` Corinna Vinschen
  2021-09-03 12:22                                                                                             ` Takashi Yano
  1 sibling, 1 reply; 250+ messages in thread
From: Ken Brown @ 2021-09-03 12:13 UTC (permalink / raw)
  To: cygwin-developers

On 9/3/2021 7:41 AM, Corinna Vinschen wrote:
> On Sep  3 13:31, Corinna Vinschen wrote:
>> On Sep  3 19:13, Takashi Yano wrote:
>>> On Fri, 3 Sep 2021 19:00:46 +0900
>>> Takashi Yano wrote:
>>>> On Thu, 2 Sep 2021 21:35:21 +0200
>>>> Corinna Vinschen wrote:
>>>>> On Sep  2 21:00, Corinna Vinschen wrote:
>>>>> It's getting too late again.  I drop off for tonight, but I attached
>>>>> my POC code I have so far.  It also adds the snippets from my previous
>>>>> patch which fixes stuff Takashi found during testing.  It also fixes
>>>>> something which looks like a bug in raw_write:
>>>>>
>>>>> -	  ptr = ((char *) ptr) + chunk;
>>>>> +	  ptr = ((char *) ptr) + nbytes_now;
>>>>>
>>>>> Incrementing ptr by chunk bytes while only nbytes_now have been written
>>>>> looks incorrect.
>>>>>
>>>>> As for the reader, it makes the # of bytes to read dependent on the
>>>>> number of reader handles.  I don't know if that's such a bright idea,
>>>>> but this can be changed easily.
>>>>>
>>>>> Anyway, this runs all my testcases successfully but they are anything
>>>>> but thorough.
>>>>>
>>>>> Patch relativ to topic/pipe attached.  Would you both mind to take a
>>>>> scrutinizing look?
>>>>
>>>> Thanks.
>>>>
>>>> Your code seems that read() returns only the partial data even
>>>> if the pipe stil has more data. Is this by design?
>>>>
>>>> This happes in both blocking and non-blocking case.
>>>
>>> The patch attached seems to fix the issue.
>>
>> I'm sorry, but I don't see what your patch is supposed to do in the
>> first place.  What I see is that it now calls NtQueryInformationFile
>> even in the non-blocking case, which is not supposed to happen.
>>
>> I'm a bit puzzled what the actual bug is.
>>
>> The code changing len is only called if there's no data in the pipe.
>> In that case we only request a partial buffer so as not to block
>> the writer on select.
>>
>> If there *is* data in the pipe, it will just go straight to the
>> NtReadFile code without changing len.
>>
>> Where's the mistake?
> 
> Oh, wait.  Do you mean, if we only request less than len bytes, but
> after NtReadFile there's still data in the buffer, we should try to
> deplete the buffer up to len bytes in a subsequent NtReadFile?
> 
> I thought this is unnecessary, actually, because of POSIX:
> 
>     The standard developers considered adding atomicity requirements  to  a
>     pipe  or FIFO, but recognized that due to the nature of pipes and FIFOs
>     there could be no guarantee of atomicity of reads of {PIPE_BUF} or  any
>     other size that would be an aid to applications portability.

I agree.  I think if read returns less than what was requested, it's up to the 
caller to call read again if desired.

One tiny thing I noticed: get_obj_handle_count can return 0.  So the line

   reader_count = get_obj_handle_count (get_handle ());

should be

   reader_count = get_obj_handle_count (get_handle ()) ?: 1;

Or else get_obj_handle_count should be changed to return 1 instead of 0 if 
NtQueryObject fails.

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-03 11:41                                                                                           ` Corinna Vinschen
  2021-09-03 12:13                                                                                             ` Ken Brown
@ 2021-09-03 12:22                                                                                             ` Takashi Yano
  2021-09-03 13:27                                                                                               ` Ken Brown
  2021-09-03 15:37                                                                                               ` Corinna Vinschen
  1 sibling, 2 replies; 250+ messages in thread
From: Takashi Yano @ 2021-09-03 12:22 UTC (permalink / raw)
  To: cygwin-developers

[-- Attachment #1: Type: text/plain, Size: 3507 bytes --]

On Fri, 3 Sep 2021 13:41:45 +0200
Corinna Vinschen wrote:
> On Sep  3 13:31, Corinna Vinschen wrote:
> > On Sep  3 19:13, Takashi Yano wrote:
> > > On Fri, 3 Sep 2021 19:00:46 +0900
> > > Takashi Yano wrote:
> > > > On Thu, 2 Sep 2021 21:35:21 +0200
> > > > Corinna Vinschen wrote:
> > > > > On Sep  2 21:00, Corinna Vinschen wrote:
> > > > > It's getting too late again.  I drop off for tonight, but I attached
> > > > > my POC code I have so far.  It also adds the snippets from my previous
> > > > > patch which fixes stuff Takashi found during testing.  It also fixes
> > > > > something which looks like a bug in raw_write:
> > > > > 
> > > > > -	  ptr = ((char *) ptr) + chunk;
> > > > > +	  ptr = ((char *) ptr) + nbytes_now;
> > > > > 
> > > > > Incrementing ptr by chunk bytes while only nbytes_now have been written
> > > > > looks incorrect.
> > > > > 
> > > > > As for the reader, it makes the # of bytes to read dependent on the
> > > > > number of reader handles.  I don't know if that's such a bright idea,
> > > > > but this can be changed easily.
> > > > > 
> > > > > Anyway, this runs all my testcases successfully but they are anything
> > > > > but thorough.
> > > > > 
> > > > > Patch relativ to topic/pipe attached.  Would you both mind to take a
> > > > > scrutinizing look?
> > > > 
> > > > Thanks.
> > > > 
> > > > Your code seems that read() returns only the partial data even
> > > > if the pipe stil has more data. Is this by design?
> > > > 
> > > > This happes in both blocking and non-blocking case.
> > > 
> > > The patch attached seems to fix the issue.
> > 
> > I'm sorry, but I don't see what your patch is supposed to do in the
> > first place.  What I see is that it now calls NtQueryInformationFile
> > even in the non-blocking case, which is not supposed to happen.
> > 
> > I'm a bit puzzled what the actual bug is.
> > 
> > The code changing len is only called if there's no data in the pipe.
> > In that case we only request a partial buffer so as not to block
> > the writer on select.
> > 
> > If there *is* data in the pipe, it will just go straight to the
> > NtReadFile code without changing len.
> > 
> > Where's the mistake?
> 
> Oh, wait.  Do you mean, if we only request less than len bytes, but
> after NtReadFile there's still data in the buffer, we should try to
> deplete the buffer up to len bytes in a subsequent NtReadFile?

Yes. I am sorry, my intent was not clear because I did more than
necessary in the previous patch. Please see attached patch revised.

> I thought this is unnecessary, actually, because of POSIX:
> 
>    The standard developers considered adding atomicity requirements  to  a
>    pipe  or FIFO, but recognized that due to the nature of pipes and FIFOs
>    there could be no guarantee of atomicity of reads of {PIPE_BUF} or  any
>    other size that would be an aid to applications portability.

POSIX says:
    The value returned may be less than nbyte if the number of bytes left
    in the file is less than nbyte, if the read() request was interrupted
    by a signal, or if the file is a pipe or FIFO or special file and has
                                                                      ~~~
    fewer than nbyte bytes immediately available for reading.
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

https://pubs.opengroup.org/onlinepubs/009604599/functions/read.html

If it is turned over, read() should read all data immediately available,
I think.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

[-- Attachment #2: pipe-corinna4-fix.patch --]
[-- Type: application/octet-stream, Size: 2301 bytes --]

diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index 85730d039..ef7823ae5 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -222,6 +222,7 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
   DWORD waitret = WAIT_OBJECT_0;
   bool keep_looping = false;
   size_t orig_len = len;
+  size_t total_len = 0;
 
   if (!len)
     return;
@@ -236,7 +237,8 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
 
   do
     {
-      len = orig_len;
+      char *ptr1 = (char *) ptr + total_len;
+      len = orig_len - total_len;
       keep_looping = false;
       if (evt)
 	ResetEvent (evt);
@@ -260,6 +262,12 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
 					   FilePipeLocalInformation);
 	  if (NT_SUCCESS (status) && fpli.ReadDataAvailable == 0)
 	    {
+	      if (total_len != 0)
+		{
+		  len = total_len;
+		  ReleaseMutex (read_mtx);
+		  break;
+		}
 	      reader_count = get_obj_handle_count (get_handle ());
 	      if (reader_count < 10)
 		max_len = fpli.InboundQuota / (2 * reader_count);
@@ -267,7 +275,7 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
 		len = max_len;
 	    }
 	}
-      status = NtReadFile (get_handle (), evt, NULL, NULL, &io, ptr,
+      status = NtReadFile (get_handle (), evt, NULL, NULL, &io, ptr1,
 			   len, NULL, NULL);
       if (!is_nonblocking ())
 	ReleaseMutex (read_mtx);
@@ -291,9 +299,13 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
 	}
       else if (NT_SUCCESS (status))
 	{
-	  len = io.Information;
-	  if (len == 0)
+	  if (io.Information == 0)
 	    keep_looping = true;
+	  total_len += io.Information;
+	  if (!is_nonblocking () && total_len < orig_len)
+	    keep_looping = true;
+	  else
+	    len = total_len;
 	}
       else
 	{
@@ -308,9 +320,13 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
 	    case STATUS_MORE_ENTRIES:
 	    case STATUS_BUFFER_OVERFLOW:
 	      /* `io.Information' is supposedly valid.  */
-	      len = io.Information;
-	      if (len == 0)
+	      if (io.Information == 0)
+		keep_looping = true;
+	      total_len += io.Information;
+	      if (!is_nonblocking () && total_len < orig_len)
 		keep_looping = true;
+	      else
+		len = total_len;
 	      break;
 	    case STATUS_PIPE_LISTENING:
 	    case STATUS_PIPE_EMPTY:

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-03 12:22                                                                                             ` Takashi Yano
@ 2021-09-03 13:27                                                                                               ` Ken Brown
  2021-09-03 15:37                                                                                               ` Corinna Vinschen
  1 sibling, 0 replies; 250+ messages in thread
From: Ken Brown @ 2021-09-03 13:27 UTC (permalink / raw)
  To: cygwin-developers

On 9/3/2021 8:22 AM, Takashi Yano wrote:
> On Fri, 3 Sep 2021 13:41:45 +0200
> Corinna Vinschen wrote:
>> On Sep  3 13:31, Corinna Vinschen wrote:
>>> On Sep  3 19:13, Takashi Yano wrote:
>>>> On Fri, 3 Sep 2021 19:00:46 +0900
>>>> Takashi Yano wrote:
>>>>> On Thu, 2 Sep 2021 21:35:21 +0200
>>>>> Corinna Vinschen wrote:
>>>>>> On Sep  2 21:00, Corinna Vinschen wrote:
>>>>>> It's getting too late again.  I drop off for tonight, but I attached
>>>>>> my POC code I have so far.  It also adds the snippets from my previous
>>>>>> patch which fixes stuff Takashi found during testing.  It also fixes
>>>>>> something which looks like a bug in raw_write:
>>>>>>
>>>>>> -	  ptr = ((char *) ptr) + chunk;
>>>>>> +	  ptr = ((char *) ptr) + nbytes_now;
>>>>>>
>>>>>> Incrementing ptr by chunk bytes while only nbytes_now have been written
>>>>>> looks incorrect.
>>>>>>
>>>>>> As for the reader, it makes the # of bytes to read dependent on the
>>>>>> number of reader handles.  I don't know if that's such a bright idea,
>>>>>> but this can be changed easily.
>>>>>>
>>>>>> Anyway, this runs all my testcases successfully but they are anything
>>>>>> but thorough.
>>>>>>
>>>>>> Patch relativ to topic/pipe attached.  Would you both mind to take a
>>>>>> scrutinizing look?
>>>>>
>>>>> Thanks.
>>>>>
>>>>> Your code seems that read() returns only the partial data even
>>>>> if the pipe stil has more data. Is this by design?
>>>>>
>>>>> This happes in both blocking and non-blocking case.
>>>>
>>>> The patch attached seems to fix the issue.
>>>
>>> I'm sorry, but I don't see what your patch is supposed to do in the
>>> first place.  What I see is that it now calls NtQueryInformationFile
>>> even in the non-blocking case, which is not supposed to happen.
>>>
>>> I'm a bit puzzled what the actual bug is.
>>>
>>> The code changing len is only called if there's no data in the pipe.
>>> In that case we only request a partial buffer so as not to block
>>> the writer on select.
>>>
>>> If there *is* data in the pipe, it will just go straight to the
>>> NtReadFile code without changing len.
>>>
>>> Where's the mistake?
>>
>> Oh, wait.  Do you mean, if we only request less than len bytes, but
>> after NtReadFile there's still data in the buffer, we should try to
>> deplete the buffer up to len bytes in a subsequent NtReadFile?
> 
> Yes. I am sorry, my intent was not clear because I did more than
> necessary in the previous patch. Please see attached patch revised.
> 
>> I thought this is unnecessary, actually, because of POSIX:
>>
>>     The standard developers considered adding atomicity requirements  to  a
>>     pipe  or FIFO, but recognized that due to the nature of pipes and FIFOs
>>     there could be no guarantee of atomicity of reads of {PIPE_BUF} or  any
>>     other size that would be an aid to applications portability.
> 
> POSIX says:
>      The value returned may be less than nbyte if the number of bytes left
>      in the file is less than nbyte, if the read() request was interrupted
>      by a signal, or if the file is a pipe or FIFO or special file and has
>                                                                        ~~~
>      fewer than nbyte bytes immediately available for reading.
>      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> https://pubs.opengroup.org/onlinepubs/009604599/functions/read.html
> 
> If it is turned over, read() should read all data immediately available,
> I think.

I understand the reasoning now, but I think your patch isn't quite right.  As it 
stands, if the call to NtQueryInformationFile fails but total_length != 0, 
you're trying to read again without knowing that there's data in the pipe.

Also, I think you need the following:

diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index ef7823ae5..46bb96961 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -348,8 +348,13 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
      CloseHandle (evt);
    if (status == STATUS_THREAD_SIGNALED)
      {
-      set_errno (EINTR);
-      len = (size_t) -1;
+      if (total_len == 0)
+       {
+         set_errno (EINTR);
+         len = (size_t) -1;
+       }
+      else
+       len = total_len;
      }
    else if (status == STATUS_THREAD_CANCELED)
      pthread::static_cancel_self ();

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-03 12:13                                                                                             ` Ken Brown
@ 2021-09-03 15:00                                                                                               ` Corinna Vinschen
  2021-09-03 15:14                                                                                                 ` Ken Brown
  0 siblings, 1 reply; 250+ messages in thread
From: Corinna Vinschen @ 2021-09-03 15:00 UTC (permalink / raw)
  To: cygwin-developers

On Sep  3 08:13, Ken Brown wrote:
> One tiny thing I noticed: get_obj_handle_count can return 0.  So the line
> 
>   reader_count = get_obj_handle_count (get_handle ());
> 
> should be
> 
>   reader_count = get_obj_handle_count (get_handle ()) ?: 1;
> 
> Or else get_obj_handle_count should be changed to return 1 instead of 0 if
> NtQueryObject fails.

We're in the reader with a valid read handle asking for the number of
open handles.  NtQueryObject can only fail if the handle is invalid.
I don't see how this could be the case here.


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-03 15:00                                                                                               ` Corinna Vinschen
@ 2021-09-03 15:14                                                                                                 ` Ken Brown
  2021-09-03 15:17                                                                                                   ` Corinna Vinschen
  0 siblings, 1 reply; 250+ messages in thread
From: Ken Brown @ 2021-09-03 15:14 UTC (permalink / raw)
  To: cygwin-developers

On 9/3/2021 11:00 AM, Corinna Vinschen wrote:
> On Sep  3 08:13, Ken Brown wrote:
>> One tiny thing I noticed: get_obj_handle_count can return 0.  So the line
>>
>>    reader_count = get_obj_handle_count (get_handle ());
>>
>> should be
>>
>>    reader_count = get_obj_handle_count (get_handle ()) ?: 1;
>>
>> Or else get_obj_handle_count should be changed to return 1 instead of 0 if
>> NtQueryObject fails.
> 
> We're in the reader with a valid read handle asking for the number of
> open handles.  NtQueryObject can only fail if the handle is invalid.
> I don't see how this could be the case here.

I don't either.  I was just being paranoid and wanted to avoid the 
appearance that we might divide by 0.  But if you're sure it's 
impossible, that's fine.

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-03 15:14                                                                                                 ` Ken Brown
@ 2021-09-03 15:17                                                                                                   ` Corinna Vinschen
  0 siblings, 0 replies; 250+ messages in thread
From: Corinna Vinschen @ 2021-09-03 15:17 UTC (permalink / raw)
  To: cygwin-developers

On Sep  3 11:14, Ken Brown wrote:
> On 9/3/2021 11:00 AM, Corinna Vinschen wrote:
> > On Sep  3 08:13, Ken Brown wrote:
> > > One tiny thing I noticed: get_obj_handle_count can return 0.  So the line
> > > 
> > >    reader_count = get_obj_handle_count (get_handle ());
> > > 
> > > should be
> > > 
> > >    reader_count = get_obj_handle_count (get_handle ()) ?: 1;
> > > 
> > > Or else get_obj_handle_count should be changed to return 1 instead of 0 if
> > > NtQueryObject fails.
> > 
> > We're in the reader with a valid read handle asking for the number of
> > open handles.  NtQueryObject can only fail if the handle is invalid.
> > I don't see how this could be the case here.
> 
> I don't either.  I was just being paranoid and wanted to avoid the
> appearance that we might divide by 0.

Good point.


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-03 12:22                                                                                             ` Takashi Yano
  2021-09-03 13:27                                                                                               ` Ken Brown
@ 2021-09-03 15:37                                                                                               ` Corinna Vinschen
  2021-09-04 12:02                                                                                                 ` Takashi Yano
  1 sibling, 1 reply; 250+ messages in thread
From: Corinna Vinschen @ 2021-09-03 15:37 UTC (permalink / raw)
  To: cygwin-developers

On Sep  3 21:22, Takashi Yano wrote:
> On Fri, 3 Sep 2021 13:41:45 +0200
> Corinna Vinschen wrote:
> > Oh, wait.  Do you mean, if we only request less than len bytes, but
> > after NtReadFile there's still data in the buffer, we should try to
> > deplete the buffer up to len bytes in a subsequent NtReadFile?
> 
> Yes. I am sorry, my intent was not clear because I did more than
> necessary in the previous patch. Please see attached patch revised.
> 
> > I thought this is unnecessary, actually, because of POSIX:
> > 
> >    The standard developers considered adding atomicity requirements  to  a
> >    pipe  or FIFO, but recognized that due to the nature of pipes and FIFOs
> >    there could be no guarantee of atomicity of reads of {PIPE_BUF} or  any
> >    other size that would be an aid to applications portability.
> 
> POSIX says:
>     The value returned may be less than nbyte if the number of bytes left
>     in the file is less than nbyte, if the read() request was interrupted
>     by a signal, or if the file is a pipe or FIFO or special file and has
>                                                                       ~~~
>     fewer than nbyte bytes immediately available for reading.
>     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> https://pubs.opengroup.org/onlinepubs/009604599/functions/read.html
> 
> If it is turned over, read() should read all data immediately available,
> I think.

Hmm, I see the point, but we might have another problem with that.

We can't keep the mutex while waiting on the pending read, and there
could be more than one pending read running at the time.  if so,
chances are extremly high, that the data written to the buffer gets
split like this:

   reader 1		               reader 2

   calls read(65536)                   calls read(65536)

   calls NtReadFile(16384 bytes)
                                       calls NtReadFile(16384 bytes)

writer writes 65536 bytes

   wakes up and gets 16384 bytes
                                       wakes up and gets 16384 bytes
   gets the mutex, calls
   NtReadFile(32768) which 
   returns immediately with
   32768 bytes added to the
   caller's buffer.

so the buffer returned to reader 1 is 49152 bytes, with 16384 bytes
missing in the middle of it, *without* the reader knowing about that
fact.  If reader 1 gets the first 16384 bytes, the 16384 bytes have
been read in a single call, at least, so the byte order is not
unknowingly broken on the application level.

Does that make sense?


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-03  9:12                                                                                       ` Corinna Vinschen
@ 2021-09-03 19:00                                                                                         ` Ken Brown
  2021-09-03 19:53                                                                                           ` Ken Brown
  2021-09-03 19:54                                                                                           ` Corinna Vinschen
  0 siblings, 2 replies; 250+ messages in thread
From: Ken Brown @ 2021-09-03 19:00 UTC (permalink / raw)
  To: cygwin-developers

On 9/3/2021 5:12 AM, Corinna Vinschen wrote:
> I pushed my stuff to the topic/pipe branch split into hopefully useful
> chunks.  Kick me if anything is wrong or not working.

Some of the bugs you fixed in the pipe code exist in the fifo code also.  I 
started going through them and fixing them, but then I realized that 
fhandler_pipe::raw_write and fhandler_fifo::raw_write are identical.  For ease 
of maintenance, I'm thinking we should have a single function, say 
fhandler_base::raw_write_pipe or fhandler_base::raw_write_pipe_fifo, which is 
called by both of them.

WDYT?

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-03 19:00                                                                                         ` Ken Brown
@ 2021-09-03 19:53                                                                                           ` Ken Brown
  2021-09-03 19:54                                                                                           ` Corinna Vinschen
  1 sibling, 0 replies; 250+ messages in thread
From: Ken Brown @ 2021-09-03 19:53 UTC (permalink / raw)
  To: cygwin-developers

[-- Attachment #1: Type: text/plain, Size: 691 bytes --]

On 9/3/2021 3:00 PM, Ken Brown wrote:
> On 9/3/2021 5:12 AM, Corinna Vinschen wrote:
>> I pushed my stuff to the topic/pipe branch split into hopefully useful
>> chunks.  Kick me if anything is wrong or not working.
> 
> Some of the bugs you fixed in the pipe code exist in the fifo code also.  I 
> started going through them and fixing them, but then I realized that 
> fhandler_pipe::raw_write and fhandler_fifo::raw_write are identical.  For ease 
> of maintenance, I'm thinking we should have a single function, say 
> fhandler_base::raw_write_pipe or fhandler_base::raw_write_pipe_fifo, which is 
> called by both of them.
> 
> WDYT?

Here's what that would look like (attached).

Ken

[-- Attachment #2: 0001-wip.patch --]
[-- Type: text/plain, Size: 13629 bytes --]

From 108a50006decade6dca0b91ca6f1c165bdfd20dd Mon Sep 17 00:00:00 2001
From: Ken Brown <kbrown@cornell.edu>
Date: Fri, 3 Sep 2021 15:43:17 -0400
Subject: [PATCH 1/2] wip

---
 winsup/cygwin/fhandler.cc      | 124 +++++++++++++++++++++++++++++++++
 winsup/cygwin/fhandler.h       |   6 +-
 winsup/cygwin/fhandler_fifo.cc |  98 +-------------------------
 winsup/cygwin/fhandler_pipe.cc | 121 +-------------------------------
 4 files changed, 130 insertions(+), 219 deletions(-)

diff --git a/winsup/cygwin/fhandler.cc b/winsup/cygwin/fhandler.cc
index f0c1b68f1..7d7aad216 100644
--- a/winsup/cygwin/fhandler.cc
+++ b/winsup/cygwin/fhandler.cc
@@ -280,6 +280,129 @@ fhandler_base::raw_write (const void *ptr, size_t len)
   return io.Information;
 }
 
+#define STATUS_PIPE_IS_CLOSED(status)	\
+		({ NTSTATUS _s = (status); \
+		   _s == STATUS_PIPE_CLOSING \
+		   || _s == STATUS_PIPE_BROKEN \
+		   || _s == STATUS_PIPE_EMPTY; })
+
+/* Used by fhandler_pipe::raw_write and fhandler_fifo::raw_write. */
+ssize_t __reg3
+fhandler_base::raw_write_pipe_fifo (const void *ptr, size_t len)
+{
+  size_t nbytes = 0;
+  ULONG chunk;
+  NTSTATUS status = STATUS_SUCCESS;
+  IO_STATUS_BLOCK io;
+  HANDLE evt = NULL;
+
+  if (!len)
+    return 0;
+
+  if (len <= max_atomic_write)
+    chunk = len;
+  else if (is_nonblocking ())
+    chunk = len = max_atomic_write;
+  else
+    chunk = max_atomic_write;
+
+  /* Create a wait event if the pipe or fifo is in blocking mode. */
+  if (!is_nonblocking () && !(evt = CreateEvent (NULL, false, false, NULL)))
+    {
+      __seterrno ();
+      return -1;
+    }
+
+  /* Write in chunks, accumulating a total.  If there's an error, just
+     return the accumulated total unless the first write fails, in
+     which case return -1. */
+  while (nbytes < len)
+    {
+      ULONG_PTR nbytes_now = 0;
+      size_t left = len - nbytes;
+      ULONG len1;
+      DWORD waitret = WAIT_OBJECT_0;
+
+      if (left > chunk)
+	len1 = chunk;
+      else
+	len1 = (ULONG) left;
+      /* NtWriteFile returns success with # of bytes written == 0 if writing
+         on a non-blocking pipe fails because the pipe buffer doesn't have
+	 sufficient space.
+
+	 POSIX requires
+	 - A write request for {PIPE_BUF} or fewer bytes shall have the
+	   following effect: if there is sufficient space available in the
+	   pipe, write() shall transfer all the data and return the number
+	   of bytes requested. Otherwise, write() shall transfer no data and
+	   return -1 with errno set to [EAGAIN].
+
+	 - A write request for more than {PIPE_BUF} bytes shall cause one
+	   of the following:
+
+	  - When at least one byte can be written, transfer what it can and
+	    return the number of bytes written. When all data previously
+	    written to the pipe is read, it shall transfer at least {PIPE_BUF}
+	    bytes.
+
+	  - When no data can be written, transfer no data, and return -1 with
+	    errno set to [EAGAIN]. */
+      while (len1 > 0)
+	{
+	  status = NtWriteFile (get_handle (), evt, NULL, NULL, &io,
+				(PVOID) ptr, len1, NULL, NULL);
+	  if (evt || !NT_SUCCESS (status) || io.Information > 0
+	      || len <= PIPE_BUF)
+	    break;
+	  len1 >>= 1;
+	}
+      if (evt && status == STATUS_PENDING)
+	{
+	  waitret = cygwait (evt);
+	  if (waitret == WAIT_OBJECT_0)
+	    status = io.Status;
+	}
+      if (waitret == WAIT_CANCELED)
+	status = STATUS_THREAD_CANCELED;
+      else if (waitret == WAIT_SIGNALED)
+	status = STATUS_THREAD_SIGNALED;
+      else if (isclosed ())  /* A signal handler might have closed the fd. */
+	{
+	  if (waitret == WAIT_OBJECT_0)
+	    set_errno (EBADF);
+	  else
+	    __seterrno ();
+	}
+      else if (NT_SUCCESS (status))
+	{
+	  nbytes_now = io.Information;
+	  ptr = ((char *) ptr) + nbytes_now;
+	  nbytes += nbytes_now;
+	  /* 0 bytes returned?  EAGAIN.  See above. */
+	  if (nbytes == 0)
+	    set_errno (EAGAIN);
+	}
+      else if (STATUS_PIPE_IS_CLOSED (status))
+	{
+	  set_errno (EPIPE);
+	  raise (SIGPIPE);
+	}
+      else
+	__seterrno_from_nt_status (status);
+
+      if (nbytes_now == 0)
+	break;
+    }
+  if (evt)
+    CloseHandle (evt);
+  if (status == STATUS_THREAD_SIGNALED && nbytes == 0)
+    set_errno (EINTR);
+  else if (status == STATUS_THREAD_CANCELED)
+    pthread::static_cancel_self ();
+  return nbytes ?: -1;
+}
+
 int
 fhandler_base::get_default_fmode (int flags)
 {
@@ -1464,6 +1587,7 @@ fhandler_base::fhandler_base () :
   _refcnt (0),
   openflags (0),
   unique_id (0),
+  max_atomic_write (DEFAULT_PIPEBUFSIZE),
   archetype (NULL),
   usecount (0)
 {
diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
index 032ab5fb0..ebddcca88 100644
--- a/winsup/cygwin/fhandler.h
+++ b/winsup/cygwin/fhandler.h
@@ -218,6 +218,9 @@ class fhandler_base
 
   HANDLE read_state;
 
+  /* Used by fhandler_pipe and fhandler_fifo. */
+  size_t max_atomic_write;
+
  public:
   LONG inc_refcnt () {return InterlockedIncrement (&_refcnt);}
   LONG dec_refcnt () {return InterlockedDecrement (&_refcnt);}
@@ -453,6 +456,7 @@ public:
 
   virtual void __reg3 raw_read (void *ptr, size_t& ulen);
   virtual ssize_t __reg3 raw_write (const void *ptr, size_t ulen);
+  ssize_t __reg3 raw_write_pipe_fifo (const void *ptr, size_t ulen);
 
   /* Virtual accessor functions to hide the fact
      that some fd's have two handles. */
@@ -1173,7 +1177,6 @@ class fhandler_pipe: public fhandler_base
 private:
   HANDLE read_mtx;
   pid_t popen_pid;
-  size_t max_atomic_write;
   void set_pipe_non_blocking (bool nonblocking);
 public:
   fhandler_pipe ();
@@ -1342,7 +1345,6 @@ class fhandler_fifo: public fhandler_base
   int nhandlers;                       /* Number of elements in the array. */
   af_unix_spinlock_t _fifo_client_lock;
   bool reader, writer, duplexer;
-  size_t max_atomic_write;
   fifo_reader_id_t me;
 
   HANDLE shmem_handle;
diff --git a/winsup/cygwin/fhandler_fifo.cc b/winsup/cygwin/fhandler_fifo.cc
index b55ba95e7..11b06ed08 100644
--- a/winsup/cygwin/fhandler_fifo.cc
+++ b/winsup/cygwin/fhandler_fifo.cc
@@ -108,14 +108,6 @@
 */
 
 
-/* This is only to be used for writers.  When reading,
-STATUS_PIPE_EMPTY simply means there's no data to be read. */
-#define STATUS_PIPE_IS_CLOSED(status)	\
-		({ NTSTATUS _s = (status); \
-		   _s == STATUS_PIPE_CLOSING \
-		   || _s == STATUS_PIPE_BROKEN \
-		   || _s == STATUS_PIPE_EMPTY; })
-
 #define STATUS_PIPE_NO_INSTANCE_AVAILABLE(status)	\
 		({ NTSTATUS _s = (status); \
 		   _s == STATUS_INSTANCE_NOT_AVAILABLE \
@@ -134,7 +126,6 @@ fhandler_fifo::fhandler_fifo ():
   cancel_evt (NULL), thr_sync_evt (NULL), pipe_name_buf (NULL),
   fc_handler (NULL), shandlers (0), nhandlers (0),
   reader (false), writer (false), duplexer (false),
-  max_atomic_write (DEFAULT_PIPEBUFSIZE),
   me (null_fr_id), shmem_handle (NULL), shmem (NULL),
   shared_fc_hdl (NULL), shared_fc_handler (NULL)
 {
@@ -1141,94 +1132,7 @@ fhandler_fifo::wait (HANDLE h)
 ssize_t __reg3
 fhandler_fifo::raw_write (const void *ptr, size_t len)
 {
-  ssize_t ret = -1;
-  size_t nbytes = 0;
-  ULONG chunk;
-  NTSTATUS status = STATUS_SUCCESS;
-  IO_STATUS_BLOCK io;
-  HANDLE evt = NULL;
-
-  if (!len)
-    return 0;
-
-  if (len <= max_atomic_write)
-    chunk = len;
-  else if (is_nonblocking ())
-    chunk = len = max_atomic_write;
-  else
-    chunk = max_atomic_write;
-
-  /* Create a wait event if the FIFO is in blocking mode. */
-  if (!is_nonblocking () && !(evt = CreateEvent (NULL, false, false, NULL)))
-    {
-      __seterrno ();
-      return -1;
-    }
-
-  /* Write in chunks, accumulating a total.  If there's an error, just
-     return the accumulated total unless the first write fails, in
-     which case return -1. */
-  while (nbytes < len)
-    {
-      ULONG_PTR nbytes_now = 0;
-      size_t left = len - nbytes;
-      ULONG len1;
-      DWORD waitret = WAIT_OBJECT_0;
-
-      if (left > chunk)
-	len1 = chunk;
-      else
-	len1 = (ULONG) left;
-      nbytes_now = 0;
-      status = NtWriteFile (get_handle (), evt, NULL, NULL, &io,
-			    (PVOID) ptr, len1, NULL, NULL);
-      if (evt && status == STATUS_PENDING)
-	{
-	  waitret = cygwait (evt);
-	  if (waitret == WAIT_OBJECT_0)
-	    status = io.Status;
-	}
-      if (waitret == WAIT_CANCELED)
-	status = STATUS_THREAD_CANCELED;
-      else if (waitret == WAIT_SIGNALED)
-	status = STATUS_THREAD_SIGNALED;
-      else if (isclosed ())  /* A signal handler might have closed the fd. */
-	{
-	  if (waitret == WAIT_OBJECT_0)
-	    set_errno (EBADF);
-	  else
-	    __seterrno ();
-	}
-      else if (NT_SUCCESS (status))
-	{
-	  nbytes_now = io.Information;
-	  /* NtWriteFile returns success with # of bytes written == 0
-	     if writing on a non-blocking pipe fails because the pipe
-	     buffer doesn't have sufficient space. */
-	  if (nbytes_now == 0)
-	    set_errno (EAGAIN);
-	  ptr = ((char *) ptr) + chunk;
-	  nbytes += nbytes_now;
-	}
-      else if (STATUS_PIPE_IS_CLOSED (status))
-	{
-	  set_errno (EPIPE);
-	  raise (SIGPIPE);
-	}
-      else
-	__seterrno_from_nt_status (status);
-      if (nbytes_now == 0)
-	len = 0;		/* Terminate loop. */
-      if (nbytes > 0)
-	ret = nbytes;
-    }
-  if (evt)
-    NtClose (evt);
-  if (status == STATUS_THREAD_SIGNALED && ret < 0)
-    set_errno (EINTR);
-  else if (status == STATUS_THREAD_CANCELED)
-    pthread::static_cancel_self ();
-  return ret;
+  return fhandler_base::raw_write_pipe_fifo (ptr, len);
 }
 
 /* Called from raw_read and select.cc:peek_fifo. */
diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index 81b1aed5e..88c98d41b 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -20,18 +20,9 @@ details. */
 #include "pinfo.h"
 #include "shared_info.h"
 
-/* This is only to be used for writing.  When reading,
-STATUS_PIPE_EMPTY simply means there's no data to be read. */
-#define STATUS_PIPE_IS_CLOSED(status)	\
-		({ NTSTATUS _s = (status); \
-		   _s == STATUS_PIPE_CLOSING \
-		   || _s == STATUS_PIPE_BROKEN \
-		   || _s == STATUS_PIPE_EMPTY; })
-
 fhandler_pipe::fhandler_pipe ()
   : fhandler_base (), popen_pid (0)
 {
-  max_atomic_write = DEFAULT_PIPEBUFSIZE;
   need_fork_fixup (true);
 }
 
@@ -342,117 +333,7 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
 ssize_t __reg3
 fhandler_pipe::raw_write (const void *ptr, size_t len)
 {
-  size_t nbytes = 0;
-  ULONG chunk;
-  NTSTATUS status = STATUS_SUCCESS;
-  IO_STATUS_BLOCK io;
-  HANDLE evt = NULL;
-
-  if (!len)
-    return 0;
-
-  if (len <= max_atomic_write)
-    chunk = len;
-  else if (is_nonblocking ())
-    chunk = len = max_atomic_write;
-  else
-    chunk = max_atomic_write;
-
-  /* Create a wait event if the pipe is in blocking mode. */
-  if (!is_nonblocking () && !(evt = CreateEvent (NULL, false, false, NULL)))
-    {
-      __seterrno ();
-      return -1;
-    }
-
-  /* Write in chunks, accumulating a total.  If there's an error, just
-     return the accumulated total unless the first write fails, in
-     which case return -1. */
-  while (nbytes < len)
-    {
-      ULONG_PTR nbytes_now = 0;
-      size_t left = len - nbytes;
-      ULONG len1;
-      DWORD waitret = WAIT_OBJECT_0;
-
-      if (left > chunk)
-	len1 = chunk;
-      else
-	len1 = (ULONG) left;
-      /* NtWriteFile returns success with # of bytes written == 0 if writing
-         on a non-blocking pipe fails because the pipe buffer doesn't have
-	 sufficient space.
-
-	 POSIX requires
-	 - A write request for {PIPE_BUF} or fewer bytes shall have the
-	   following effect: if there is sufficient space available in the
-	   pipe, write() shall transfer all the data and return the number
-	   of bytes requested. Otherwise, write() shall transfer no data and
-	   return -1 with errno set to [EAGAIN].
-
-	 - A write request for more than {PIPE_BUF} bytes shall cause one
-	   of the following:
-
-	  - When at least one byte can be written, transfer what it can and
-	    return the number of bytes written. When all data previously
-	    written to the pipe is read, it shall transfer at least {PIPE_BUF}
-	    bytes.
-
-	  - When no data can be written, transfer no data, and return -1 with
-	    errno set to [EAGAIN]. */
-      while (len1 > 0)
-	{
-	  status = NtWriteFile (get_handle (), evt, NULL, NULL, &io,
-				(PVOID) ptr, len1, NULL, NULL);
-	  if (evt || !NT_SUCCESS (status) || io.Information > 0
-	      || len <= PIPE_BUF)
-	    break;
-	  len1 >>= 1;
-	}
-      if (evt && status == STATUS_PENDING)
-	{
-	  waitret = cygwait (evt);
-	  if (waitret == WAIT_OBJECT_0)
-	    status = io.Status;
-	}
-      if (waitret == WAIT_CANCELED)
-	status = STATUS_THREAD_CANCELED;
-      else if (waitret == WAIT_SIGNALED)
-	status = STATUS_THREAD_SIGNALED;
-      else if (isclosed ())  /* A signal handler might have closed the fd. */
-	{
-	  if (waitret == WAIT_OBJECT_0)
-	    set_errno (EBADF);
-	  else
-	    __seterrno ();
-	}
-      else if (NT_SUCCESS (status))
-	{
-	  nbytes_now = io.Information;
-	  ptr = ((char *) ptr) + nbytes_now;
-	  nbytes += nbytes_now;
-	  /* 0 bytes returned?  EAGAIN.  See above. */
-	  if (nbytes == 0)
-	    set_errno (EAGAIN);
-	}
-      else if (STATUS_PIPE_IS_CLOSED (status))
-	{
-	  set_errno (EPIPE);
-	  raise (SIGPIPE);
-	}
-      else
-	__seterrno_from_nt_status (status);
-
-      if (nbytes_now == 0)
-	break;
-    }
-  if (evt)
-    CloseHandle (evt);
-  if (status == STATUS_THREAD_SIGNALED && nbytes == 0)
-    set_errno (EINTR);
-  else if (status == STATUS_THREAD_CANCELED)
-    pthread::static_cancel_self ();
-  return nbytes ?: -1;
+  return fhandler_base::raw_write_pipe_fifo (ptr, len);
 }
 
 void
-- 
2.33.0


^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-03 19:00                                                                                         ` Ken Brown
  2021-09-03 19:53                                                                                           ` Ken Brown
@ 2021-09-03 19:54                                                                                           ` Corinna Vinschen
  2021-09-03 20:05                                                                                             ` Ken Brown
  1 sibling, 1 reply; 250+ messages in thread
From: Corinna Vinschen @ 2021-09-03 19:54 UTC (permalink / raw)
  To: cygwin-developers

On Sep  3 15:00, Ken Brown wrote:
> On 9/3/2021 5:12 AM, Corinna Vinschen wrote:
> > I pushed my stuff to the topic/pipe branch split into hopefully useful
> > chunks.  Kick me if anything is wrong or not working.
> 
> Some of the bugs you fixed in the pipe code exist in the fifo code also.  I
> started going through them and fixing them, but then I realized that
> fhandler_pipe::raw_write and fhandler_fifo::raw_write are identical.  For
> ease of maintenance, I'm thinking we should have a single function, say
> fhandler_base::raw_write_pipe or fhandler_base::raw_write_pipe_fifo, which
> is called by both of them.
> 
> WDYT?

Returning to fhandler_base_overlapped?  :)

No, seriously, the cleanest way is probably to implement a parent class
for fhandler_pipe and fhandler_fifo, derived from fhandler_base, and
just collect the common code there.  For a start it could simply provide
raw_write and nothing else.  fhandler_pipe_fifo, perhaps?


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-03 19:54                                                                                           ` Corinna Vinschen
@ 2021-09-03 20:05                                                                                             ` Ken Brown
  0 siblings, 0 replies; 250+ messages in thread
From: Ken Brown @ 2021-09-03 20:05 UTC (permalink / raw)
  To: cygwin-developers

On 9/3/2021 3:54 PM, Corinna Vinschen wrote:
> On Sep  3 15:00, Ken Brown wrote:
>> On 9/3/2021 5:12 AM, Corinna Vinschen wrote:
>>> I pushed my stuff to the topic/pipe branch split into hopefully useful
>>> chunks.  Kick me if anything is wrong or not working.
>>
>> Some of the bugs you fixed in the pipe code exist in the fifo code also.  I
>> started going through them and fixing them, but then I realized that
>> fhandler_pipe::raw_write and fhandler_fifo::raw_write are identical.  For
>> ease of maintenance, I'm thinking we should have a single function, say
>> fhandler_base::raw_write_pipe or fhandler_base::raw_write_pipe_fifo, which
>> is called by both of them.
>>
>> WDYT?
> 
> Returning to fhandler_base_overlapped?  :)

Touché.

> No, seriously, the cleanest way is probably to implement a parent class
> for fhandler_pipe and fhandler_fifo, derived from fhandler_base, and
> just collect the common code there.  For a start it could simply provide
> raw_write and nothing else.  fhandler_pipe_fifo, perhaps?

Sounds good.  Thanks.  Since the new class has only raw_write for now, I think I 
could just define that function in fhandler_pipe.cc rather than a new file 
fhandler_pipe_fifo.cc.  But I could also do the latter if you prefer.

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-03 15:37                                                                                               ` Corinna Vinschen
@ 2021-09-04 12:02                                                                                                 ` Takashi Yano
  2021-09-04 12:37                                                                                                   ` Takashi Yano
  0 siblings, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-04 12:02 UTC (permalink / raw)
  To: cygwin-developers

Hi Corinna, Ken,

On Fri, 3 Sep 2021 09:27:37 -0400
Ken Brown wrote:
> On 9/3/2021 8:22 AM, Takashi Yano wrote:
> > POSIX says:
> >      The value returned may be less than nbyte if the number of bytes left
> >      in the file is less than nbyte, if the read() request was interrupted
> >      by a signal, or if the file is a pipe or FIFO or special file and has
> >                                                                        ~~~
> >      fewer than nbyte bytes immediately available for reading.
> >      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > 
> > https://pubs.opengroup.org/onlinepubs/009604599/functions/read.html
> > 
> > If it is turned over, read() should read all data immediately available,
> > I think.
> 
> I understand the reasoning now, but I think your patch isn't quite right.  As it 
> stands, if the call to NtQueryInformationFile fails but total_length != 0, 
> you're trying to read again without knowing that there's data in the pipe.
> 
> Also, I think you need the following:
> 
> diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
> index ef7823ae5..46bb96961 100644
> --- a/winsup/cygwin/fhandler_pipe.cc
> +++ b/winsup/cygwin/fhandler_pipe.cc
> @@ -348,8 +348,13 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
>       CloseHandle (evt);
>     if (status == STATUS_THREAD_SIGNALED)
>       {
> -      set_errno (EINTR);
> -      len = (size_t) -1;
> +      if (total_len == 0)
> +       {
> +         set_errno (EINTR);
> +         len = (size_t) -1;
> +       }
> +      else
> +       len = total_len;
>       }
>     else if (status == STATUS_THREAD_CANCELED)
>       pthread::static_cancel_self ();

Thanks for your advice. I fixed the issue and attached new patch.

On Fri, 3 Sep 2021 17:37:13 +0200
Corinna Vinschen wrote:
> Hmm, I see the point, but we might have another problem with that.
> 
> We can't keep the mutex while waiting on the pending read, and there
> could be more than one pending read running at the time.  if so,
> chances are extremly high, that the data written to the buffer gets
> split like this:
> 
>    reader 1		               reader 2
> 
>    calls read(65536)                   calls read(65536)
> 
>    calls NtReadFile(16384 bytes)
>                                        calls NtReadFile(16384 bytes)
> 
> writer writes 65536 bytes
> 
>    wakes up and gets 16384 bytes
>                                        wakes up and gets 16384 bytes
>    gets the mutex, calls
>    NtReadFile(32768) which 
>    returns immediately with
>    32768 bytes added to the
>    caller's buffer.
> 
> so the buffer returned to reader 1 is 49152 bytes, with 16384 bytes
> missing in the middle of it, *without* the reader knowing about that
> fact.  If reader 1 gets the first 16384 bytes, the 16384 bytes have
> been read in a single call, at least, so the byte order is not
> unknowingly broken on the application level.
> 
> Does that make sense?

Why can't we keep the mutex while waiting on the pending read?
If we can keep the mutex, the issue above mentioned does not
happen, right?

What about the patch attached? This keeps the mutex while read()
but I do not see any defects so far.


-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-04 12:02                                                                                                 ` Takashi Yano
@ 2021-09-04 12:37                                                                                                   ` Takashi Yano
  2021-09-04 14:04                                                                                                     ` Ken Brown
  0 siblings, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-04 12:37 UTC (permalink / raw)
  To: cygwin-developers

[-- Attachment #1: Type: text/plain, Size: 3518 bytes --]

On Sat, 4 Sep 2021 21:02:58 +0900
Takashi Yano wrote:
> Hi Corinna, Ken,
> 
> On Fri, 3 Sep 2021 09:27:37 -0400
> Ken Brown wrote:
> > On 9/3/2021 8:22 AM, Takashi Yano wrote:
> > > POSIX says:
> > >      The value returned may be less than nbyte if the number of bytes left
> > >      in the file is less than nbyte, if the read() request was interrupted
> > >      by a signal, or if the file is a pipe or FIFO or special file and has
> > >                                                                        ~~~
> > >      fewer than nbyte bytes immediately available for reading.
> > >      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > > 
> > > https://pubs.opengroup.org/onlinepubs/009604599/functions/read.html
> > > 
> > > If it is turned over, read() should read all data immediately available,
> > > I think.
> > 
> > I understand the reasoning now, but I think your patch isn't quite right.  As it 
> > stands, if the call to NtQueryInformationFile fails but total_length != 0, 
> > you're trying to read again without knowing that there's data in the pipe.
> > 
> > Also, I think you need the following:
> > 
> > diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
> > index ef7823ae5..46bb96961 100644
> > --- a/winsup/cygwin/fhandler_pipe.cc
> > +++ b/winsup/cygwin/fhandler_pipe.cc
> > @@ -348,8 +348,13 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
> >       CloseHandle (evt);
> >     if (status == STATUS_THREAD_SIGNALED)
> >       {
> > -      set_errno (EINTR);
> > -      len = (size_t) -1;
> > +      if (total_len == 0)
> > +       {
> > +         set_errno (EINTR);
> > +         len = (size_t) -1;
> > +       }
> > +      else
> > +       len = total_len;
> >       }
> >     else if (status == STATUS_THREAD_CANCELED)
> >       pthread::static_cancel_self ();
> 
> Thanks for your advice. I fixed the issue and attached new patch.
> 
> On Fri, 3 Sep 2021 17:37:13 +0200
> Corinna Vinschen wrote:
> > Hmm, I see the point, but we might have another problem with that.
> > 
> > We can't keep the mutex while waiting on the pending read, and there
> > could be more than one pending read running at the time.  if so,
> > chances are extremly high, that the data written to the buffer gets
> > split like this:
> > 
> >    reader 1		               reader 2
> > 
> >    calls read(65536)                   calls read(65536)
> > 
> >    calls NtReadFile(16384 bytes)
> >                                        calls NtReadFile(16384 bytes)
> > 
> > writer writes 65536 bytes
> > 
> >    wakes up and gets 16384 bytes
> >                                        wakes up and gets 16384 bytes
> >    gets the mutex, calls
> >    NtReadFile(32768) which 
> >    returns immediately with
> >    32768 bytes added to the
> >    caller's buffer.
> > 
> > so the buffer returned to reader 1 is 49152 bytes, with 16384 bytes
> > missing in the middle of it, *without* the reader knowing about that
> > fact.  If reader 1 gets the first 16384 bytes, the 16384 bytes have
> > been read in a single call, at least, so the byte order is not
> > unknowingly broken on the application level.
> > 
> > Does that make sense?
> 
> Why can't we keep the mutex while waiting on the pending read?
> If we can keep the mutex, the issue above mentioned does not
> happen, right?
> 
> What about the patch attached? This keeps the mutex while read()
> but I do not see any defects so far.

Sorry, I forgot to attach the patch.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

[-- Attachment #2: pipe-corinna4-fix.patch --]
[-- Type: application/octet-stream, Size: 3540 bytes --]

diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index 85730d039..57354712d 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -222,6 +222,7 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
   DWORD waitret = WAIT_OBJECT_0;
   bool keep_looping = false;
   size_t orig_len = len;
+  size_t total_len = 0;
 
   if (!len)
     return;
@@ -234,9 +235,17 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
       return;
     }
 
+  DWORD timeout = is_nonblocking () ? 0 : INFINITE;
+  if (WAIT_TIMEOUT == WaitForSingleObject (read_mtx, timeout))
+    {
+      set_errno (EAGAIN);
+      len = (size_t) -1;
+      return;
+    }
   do
     {
-      len = orig_len;
+      char *ptr1 = (char *) ptr + total_len;
+      len = orig_len - total_len;
       keep_looping = false;
       if (evt)
 	ResetEvent (evt);
@@ -246,7 +255,6 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
 	  ULONG reader_count;
 	  ULONG max_len = 64;
 
-	  WaitForSingleObject (read_mtx, INFINITE);
 
 	  /* Make sure never to request more bytes than half the pipe
 	     buffer size.  Every pending read lowers WriteQuotaAvailable
@@ -260,17 +268,25 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
 					   FilePipeLocalInformation);
 	  if (NT_SUCCESS (status) && fpli.ReadDataAvailable == 0)
 	    {
+	      if (total_len != 0)
+		{
+		  len = total_len;
+		  break;
+		}
 	      reader_count = get_obj_handle_count (get_handle ());
 	      if (reader_count < 10)
 		max_len = fpli.InboundQuota / (2 * reader_count);
 	      if (len > max_len)
 		len = max_len;
 	    }
+	  if (!NT_SUCCESS (status) && total_len != 0)
+	    {
+	      len = total_len;
+	      break;
+	    }
 	}
-      status = NtReadFile (get_handle (), evt, NULL, NULL, &io, ptr,
+      status = NtReadFile (get_handle (), evt, NULL, NULL, &io, ptr1,
 			   len, NULL, NULL);
-      if (!is_nonblocking ())
-	ReleaseMutex (read_mtx);
       if (evt && status == STATUS_PENDING)
 	{
 	  waitret = cygwait (evt);
@@ -291,9 +307,11 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
 	}
       else if (NT_SUCCESS (status))
 	{
-	  len = io.Information;
-	  if (len == 0)
+	  total_len += io.Information;
+	  if (total_len < orig_len)
 	    keep_looping = true;
+	  else
+	    len = total_len;
 	}
       else
 	{
@@ -303,17 +321,24 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
 	    case STATUS_END_OF_FILE:
 	    case STATUS_PIPE_BROKEN:
 	      /* This is really EOF.  */
-	      len = 0;
+	      len = total_len;
 	      break;
 	    case STATUS_MORE_ENTRIES:
 	    case STATUS_BUFFER_OVERFLOW:
 	      /* `io.Information' is supposedly valid.  */
-	      len = io.Information;
-	      if (len == 0)
+	      total_len += io.Information;
+	      if (total_len < orig_len)
 		keep_looping = true;
+	      else
+		len = total_len;
 	      break;
 	    case STATUS_PIPE_LISTENING:
 	    case STATUS_PIPE_EMPTY:
+	      if (total_len != 0)
+		{
+		  len = total_len;
+		  break;
+		}
 	      if (is_nonblocking ())
 		{
 		  set_errno (EAGAIN);
@@ -328,12 +353,18 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
 	    }
 	}
     } while (keep_looping);
+  ReleaseMutex (read_mtx);
   if (evt)
     CloseHandle (evt);
   if (status == STATUS_THREAD_SIGNALED)
     {
-      set_errno (EINTR);
-      len = (size_t) -1;
+      if (total_len == 0)
+	{
+	  set_errno (EINTR);
+	  len = (size_t) -1;
+	}
+      else
+	len = total_len;
     }
   else if (status == STATUS_THREAD_CANCELED)
     pthread::static_cancel_self ();

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-04 12:37                                                                                                   ` Takashi Yano
@ 2021-09-04 14:04                                                                                                     ` Ken Brown
  2021-09-04 23:15                                                                                                       ` Takashi Yano
  0 siblings, 1 reply; 250+ messages in thread
From: Ken Brown @ 2021-09-04 14:04 UTC (permalink / raw)
  To: cygwin-developers

On 9/4/2021 8:37 AM, Takashi Yano wrote:
> On Sat, 4 Sep 2021 21:02:58 +0900
> Takashi Yano wrote:
>> Hi Corinna, Ken,
>>
>> On Fri, 3 Sep 2021 09:27:37 -0400
>> Ken Brown wrote:
>>> On 9/3/2021 8:22 AM, Takashi Yano wrote:
>>>> POSIX says:
>>>>       The value returned may be less than nbyte if the number of bytes left
>>>>       in the file is less than nbyte, if the read() request was interrupted
>>>>       by a signal, or if the file is a pipe or FIFO or special file and has
>>>>                                                                         ~~~
>>>>       fewer than nbyte bytes immediately available for reading.
>>>>       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>
>>>> https://pubs.opengroup.org/onlinepubs/009604599/functions/read.html
>>>>
>>>> If it is turned over, read() should read all data immediately available,
>>>> I think.
>>>
>>> I understand the reasoning now, but I think your patch isn't quite right.  As it
>>> stands, if the call to NtQueryInformationFile fails but total_length != 0,
>>> you're trying to read again without knowing that there's data in the pipe.
>>>
>>> Also, I think you need the following:
>>>
>>> diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
>>> index ef7823ae5..46bb96961 100644
>>> --- a/winsup/cygwin/fhandler_pipe.cc
>>> +++ b/winsup/cygwin/fhandler_pipe.cc
>>> @@ -348,8 +348,13 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
>>>        CloseHandle (evt);
>>>      if (status == STATUS_THREAD_SIGNALED)
>>>        {
>>> -      set_errno (EINTR);
>>> -      len = (size_t) -1;
>>> +      if (total_len == 0)
>>> +       {
>>> +         set_errno (EINTR);
>>> +         len = (size_t) -1;
>>> +       }
>>> +      else
>>> +       len = total_len;
>>>        }
>>>      else if (status == STATUS_THREAD_CANCELED)
>>>        pthread::static_cancel_self ();
>>
>> Thanks for your advice. I fixed the issue and attached new patch.
>>
>> On Fri, 3 Sep 2021 17:37:13 +0200
>> Corinna Vinschen wrote:
>>> Hmm, I see the point, but we might have another problem with that.
>>>
>>> We can't keep the mutex while waiting on the pending read, and there
>>> could be more than one pending read running at the time.  if so,
>>> chances are extremly high, that the data written to the buffer gets
>>> split like this:
>>>
>>>     reader 1		               reader 2
>>>
>>>     calls read(65536)                   calls read(65536)
>>>
>>>     calls NtReadFile(16384 bytes)
>>>                                         calls NtReadFile(16384 bytes)
>>>
>>> writer writes 65536 bytes
>>>
>>>     wakes up and gets 16384 bytes
>>>                                         wakes up and gets 16384 bytes
>>>     gets the mutex, calls
>>>     NtReadFile(32768) which
>>>     returns immediately with
>>>     32768 bytes added to the
>>>     caller's buffer.
>>>
>>> so the buffer returned to reader 1 is 49152 bytes, with 16384 bytes
>>> missing in the middle of it, *without* the reader knowing about that
>>> fact.  If reader 1 gets the first 16384 bytes, the 16384 bytes have
>>> been read in a single call, at least, so the byte order is not
>>> unknowingly broken on the application level.
>>>
>>> Does that make sense?
>>
>> Why can't we keep the mutex while waiting on the pending read?
>> If we can keep the mutex, the issue above mentioned does not
>> happen, right?
>>
>> What about the patch attached? This keeps the mutex while read()
>> but I do not see any defects so far.

LGTM.

If Corinna agrees, I have a couple of suggestions.

1. With this patch, we can no longer have more than one pending ReadFile.  So 
there's no longer a need to count read handles, and the problem with select is 
completely fixed as long as the number of bytes requested is less than the pipe 
buffer size.

2. raw_read is now reading in chunks, like raw_write.  For readability of the 
code, I think it would be better to make the two functions as similar as 
possible.  For example, you could replace the do/while loop by a 
while(total_len<orig_len) loop.  And you could even use similar names for the 
variables, e.g., nbytes instead of total_len, or vice versa.

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-04 14:04                                                                                                     ` Ken Brown
@ 2021-09-04 23:15                                                                                                       ` Takashi Yano
  2021-09-05 13:40                                                                                                         ` Takashi Yano
  0 siblings, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-04 23:15 UTC (permalink / raw)
  To: cygwin-developers

[-- Attachment #1: Type: text/plain, Size: 4559 bytes --]

Hi Ken,

On Sat, 4 Sep 2021 10:04:12 -0400
Ken Brown wrote:
> On 9/4/2021 8:37 AM, Takashi Yano wrote:
> > On Sat, 4 Sep 2021 21:02:58 +0900
> > Takashi Yano wrote:
> >> Hi Corinna, Ken,
> >>
> >> On Fri, 3 Sep 2021 09:27:37 -0400
> >> Ken Brown wrote:
> >>> On 9/3/2021 8:22 AM, Takashi Yano wrote:
> >>>> POSIX says:
> >>>>       The value returned may be less than nbyte if the number of bytes left
> >>>>       in the file is less than nbyte, if the read() request was interrupted
> >>>>       by a signal, or if the file is a pipe or FIFO or special file and has
> >>>>                                                                         ~~~
> >>>>       fewer than nbyte bytes immediately available for reading.
> >>>>       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >>>>
> >>>> https://pubs.opengroup.org/onlinepubs/009604599/functions/read.html
> >>>>
> >>>> If it is turned over, read() should read all data immediately available,
> >>>> I think.
> >>>
> >>> I understand the reasoning now, but I think your patch isn't quite right.  As it
> >>> stands, if the call to NtQueryInformationFile fails but total_length != 0,
> >>> you're trying to read again without knowing that there's data in the pipe.
> >>>
> >>> Also, I think you need the following:
> >>>
> >>> diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
> >>> index ef7823ae5..46bb96961 100644
> >>> --- a/winsup/cygwin/fhandler_pipe.cc
> >>> +++ b/winsup/cygwin/fhandler_pipe.cc
> >>> @@ -348,8 +348,13 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
> >>>        CloseHandle (evt);
> >>>      if (status == STATUS_THREAD_SIGNALED)
> >>>        {
> >>> -      set_errno (EINTR);
> >>> -      len = (size_t) -1;
> >>> +      if (total_len == 0)
> >>> +       {
> >>> +         set_errno (EINTR);
> >>> +         len = (size_t) -1;
> >>> +       }
> >>> +      else
> >>> +       len = total_len;
> >>>        }
> >>>      else if (status == STATUS_THREAD_CANCELED)
> >>>        pthread::static_cancel_self ();
> >>
> >> Thanks for your advice. I fixed the issue and attached new patch.
> >>
> >> On Fri, 3 Sep 2021 17:37:13 +0200
> >> Corinna Vinschen wrote:
> >>> Hmm, I see the point, but we might have another problem with that.
> >>>
> >>> We can't keep the mutex while waiting on the pending read, and there
> >>> could be more than one pending read running at the time.  if so,
> >>> chances are extremly high, that the data written to the buffer gets
> >>> split like this:
> >>>
> >>>     reader 1		               reader 2
> >>>
> >>>     calls read(65536)                   calls read(65536)
> >>>
> >>>     calls NtReadFile(16384 bytes)
> >>>                                         calls NtReadFile(16384 bytes)
> >>>
> >>> writer writes 65536 bytes
> >>>
> >>>     wakes up and gets 16384 bytes
> >>>                                         wakes up and gets 16384 bytes
> >>>     gets the mutex, calls
> >>>     NtReadFile(32768) which
> >>>     returns immediately with
> >>>     32768 bytes added to the
> >>>     caller's buffer.
> >>>
> >>> so the buffer returned to reader 1 is 49152 bytes, with 16384 bytes
> >>> missing in the middle of it, *without* the reader knowing about that
> >>> fact.  If reader 1 gets the first 16384 bytes, the 16384 bytes have
> >>> been read in a single call, at least, so the byte order is not
> >>> unknowingly broken on the application level.
> >>>
> >>> Does that make sense?
> >>
> >> Why can't we keep the mutex while waiting on the pending read?
> >> If we can keep the mutex, the issue above mentioned does not
> >> happen, right?
> >>
> >> What about the patch attached? This keeps the mutex while read()
> >> but I do not see any defects so far.
> 
> LGTM.
> 
> If Corinna agrees, I have a couple of suggestions.
> 
> 1. With this patch, we can no longer have more than one pending ReadFile.  So 
> there's no longer a need to count read handles, and the problem with select is 
> completely fixed as long as the number of bytes requested is less than the pipe 
> buffer size.
> 
> 2. raw_read is now reading in chunks, like raw_write.  For readability of the 
> code, I think it would be better to make the two functions as similar as 
> possible.  For example, you could replace the do/while loop by a 
> while(total_len<orig_len) loop.  And you could even use similar names for the 
> variables, e.g., nbytes instead of total_len, or vice versa.

Thanks for the suggestion. I have rebuilt the patch.
Please see the patch attached.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

[-- Attachment #2: 0001-Cygwin-pipe-Stop-counting-reader-and-read-all-availa.patch --]
[-- Type: application/octet-stream, Size: 5079 bytes --]

From 338c1ed9e260d5d456354ea5985002d1915168b5 Mon Sep 17 00:00:00 2001
From: Takashi Yano <takashi.yano@nifty.ne.jp>
Date: Sun, 5 Sep 2021 06:10:29 +0900
Subject: [PATCH] Cygwin: pipe: Stop counting reader and read all available
 data.

- By guarding read with read_mtx, no more than one ReadFile can
  be called simultaneously. So couting read handles is no longer
  necessary.
- Make raw_read code as similar as possible to raw_write code.
---
 winsup/cygwin/fhandler_pipe.cc | 84 +++++++++++++++++++---------------
 1 file changed, 46 insertions(+), 38 deletions(-)

diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index 85730d039..544e5872d 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -216,12 +216,10 @@ fhandler_pipe::get_proc_fd_name (char *buf)
 void __reg3
 fhandler_pipe::raw_read (void *ptr, size_t& len)
 {
-  NTSTATUS status;
+  size_t nbytes = 0;
+  NTSTATUS status = STATUS_SUCCESS;
   IO_STATUS_BLOCK io;
   HANDLE evt = NULL;
-  DWORD waitret = WAIT_OBJECT_0;
-  bool keep_looping = false;
-  size_t orig_len = len;
 
   if (!len)
     return;
@@ -234,43 +232,47 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
       return;
     }
 
-  do
+  DWORD timeout = is_nonblocking () ? 0 : INFINITE;
+  if (WAIT_TIMEOUT == WaitForSingleObject (read_mtx, timeout))
+    {
+      set_errno (EAGAIN);
+      len = (size_t) -1;
+      return;
+    }
+  while (nbytes < len)
     {
-      len = orig_len;
-      keep_looping = false;
+      ULONG_PTR nbytes_now = 0;
+      size_t left = len - nbytes;
+      ULONG len1 = (ULONG) left;
+      DWORD waitret = WAIT_OBJECT_0;
+
       if (evt)
 	ResetEvent (evt);
       if (!is_nonblocking ())
 	{
 	  FILE_PIPE_LOCAL_INFORMATION fpli;
-	  ULONG reader_count;
-	  ULONG max_len = 64;
-
-	  WaitForSingleObject (read_mtx, INFINITE);
 
 	  /* Make sure never to request more bytes than half the pipe
-	     buffer size.  Every pending read lowers WriteQuotaAvailable
-	     on the write side and thus affects select's ability to return
-	     more or less reliable info whether a write succeeds or not.
-
-	     Let the size of the request depend on the number of readers
-	     at the time. */
+	     buffer size. Pending read lowers WriteQuotaAvailable on
+	     the write side and thus affects select's ability to return
+	     more or less reliable info whether a write succeeds or not. */
+	  ULONG chunk = max_atomic_write / 2;
 	  status = NtQueryInformationFile (get_handle (), &io,
 					   &fpli, sizeof (fpli),
 					   FilePipeLocalInformation);
 	  if (NT_SUCCESS (status) && fpli.ReadDataAvailable == 0)
 	    {
-	      reader_count = get_obj_handle_count (get_handle ());
-	      if (reader_count < 10)
-		max_len = fpli.InboundQuota / (2 * reader_count);
-	      if (len > max_len)
-		len = max_len;
+	      if (nbytes != 0)
+		break;
+	      chunk = fpli.InboundQuota / 2;
 	    }
+	  if (!NT_SUCCESS (status) && nbytes != 0)
+	    break;
+
+	  len1 = (left < chunk) ? (ULONG) left : chunk;
 	}
       status = NtReadFile (get_handle (), evt, NULL, NULL, &io, ptr,
-			   len, NULL, NULL);
-      if (!is_nonblocking ())
-	ReleaseMutex (read_mtx);
+			   len1, NULL, NULL);
       if (evt && status == STATUS_PENDING)
 	{
 	  waitret = cygwait (evt);
@@ -287,13 +289,13 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
 	    set_errno (EBADF);
 	  else
 	    __seterrno ();
-	  len = (size_t) -1;
+	  nbytes = (size_t) -1;
 	}
       else if (NT_SUCCESS (status))
 	{
-	  len = io.Information;
-	  if (len == 0)
-	    keep_looping = true;
+	  nbytes_now = io.Information;
+	  ptr = ((char *) ptr) + nbytes_now;
+	  nbytes += nbytes_now;
 	}
       else
 	{
@@ -303,40 +305,46 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
 	    case STATUS_END_OF_FILE:
 	    case STATUS_PIPE_BROKEN:
 	      /* This is really EOF.  */
-	      len = 0;
 	      break;
 	    case STATUS_MORE_ENTRIES:
 	    case STATUS_BUFFER_OVERFLOW:
 	      /* `io.Information' is supposedly valid.  */
-	      len = io.Information;
-	      if (len == 0)
-		keep_looping = true;
+	      nbytes_now = io.Information;
+	      ptr = ((char *) ptr) + nbytes_now;
+	      nbytes += nbytes_now;
 	      break;
 	    case STATUS_PIPE_LISTENING:
 	    case STATUS_PIPE_EMPTY:
+	      if (nbytes != 0)
+		break;
 	      if (is_nonblocking ())
 		{
 		  set_errno (EAGAIN);
-		  len = (size_t) -1;
+		  nbytes = (size_t) -1;
 		  break;
 		}
 	      fallthrough;
 	    default:
 	      __seterrno_from_nt_status (status);
-	      len = (size_t) -1;
+	      nbytes = (size_t) -1;
 	      break;
 	    }
 	}
-    } while (keep_looping);
+
+      if (nbytes_now == 0)
+	break;
+    }
+  ReleaseMutex (read_mtx);
   if (evt)
     CloseHandle (evt);
-  if (status == STATUS_THREAD_SIGNALED)
+  if (status == STATUS_THREAD_SIGNALED && nbytes == 0)
     {
       set_errno (EINTR);
-      len = (size_t) -1;
+      nbytes = (size_t) -1;
     }
   else if (status == STATUS_THREAD_CANCELED)
     pthread::static_cancel_self ();
+  len = nbytes;
 }
 
 ssize_t __reg3
-- 
2.33.0


^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-04 23:15                                                                                                       ` Takashi Yano
@ 2021-09-05 13:40                                                                                                         ` Takashi Yano
  2021-09-05 13:50                                                                                                           ` Takashi Yano
  0 siblings, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-05 13:40 UTC (permalink / raw)
  To: cygwin-developers

[-- Attachment #1: Type: text/plain, Size: 4991 bytes --]

On Sun, 5 Sep 2021 08:15:23 +0900
Takashi Yano wrote:
> Hi Ken,
> 
> On Sat, 4 Sep 2021 10:04:12 -0400
> Ken Brown wrote:
> > On 9/4/2021 8:37 AM, Takashi Yano wrote:
> > > On Sat, 4 Sep 2021 21:02:58 +0900
> > > Takashi Yano wrote:
> > >> Hi Corinna, Ken,
> > >>
> > >> On Fri, 3 Sep 2021 09:27:37 -0400
> > >> Ken Brown wrote:
> > >>> On 9/3/2021 8:22 AM, Takashi Yano wrote:
> > >>>> POSIX says:
> > >>>>       The value returned may be less than nbyte if the number of bytes left
> > >>>>       in the file is less than nbyte, if the read() request was interrupted
> > >>>>       by a signal, or if the file is a pipe or FIFO or special file and has
> > >>>>                                                                         ~~~
> > >>>>       fewer than nbyte bytes immediately available for reading.
> > >>>>       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > >>>>
> > >>>> https://pubs.opengroup.org/onlinepubs/009604599/functions/read.html
> > >>>>
> > >>>> If it is turned over, read() should read all data immediately available,
> > >>>> I think.
> > >>>
> > >>> I understand the reasoning now, but I think your patch isn't quite right.  As it
> > >>> stands, if the call to NtQueryInformationFile fails but total_length != 0,
> > >>> you're trying to read again without knowing that there's data in the pipe.
> > >>>
> > >>> Also, I think you need the following:
> > >>>
> > >>> diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
> > >>> index ef7823ae5..46bb96961 100644
> > >>> --- a/winsup/cygwin/fhandler_pipe.cc
> > >>> +++ b/winsup/cygwin/fhandler_pipe.cc
> > >>> @@ -348,8 +348,13 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
> > >>>        CloseHandle (evt);
> > >>>      if (status == STATUS_THREAD_SIGNALED)
> > >>>        {
> > >>> -      set_errno (EINTR);
> > >>> -      len = (size_t) -1;
> > >>> +      if (total_len == 0)
> > >>> +       {
> > >>> +         set_errno (EINTR);
> > >>> +         len = (size_t) -1;
> > >>> +       }
> > >>> +      else
> > >>> +       len = total_len;
> > >>>        }
> > >>>      else if (status == STATUS_THREAD_CANCELED)
> > >>>        pthread::static_cancel_self ();
> > >>
> > >> Thanks for your advice. I fixed the issue and attached new patch.
> > >>
> > >> On Fri, 3 Sep 2021 17:37:13 +0200
> > >> Corinna Vinschen wrote:
> > >>> Hmm, I see the point, but we might have another problem with that.
> > >>>
> > >>> We can't keep the mutex while waiting on the pending read, and there
> > >>> could be more than one pending read running at the time.  if so,
> > >>> chances are extremly high, that the data written to the buffer gets
> > >>> split like this:
> > >>>
> > >>>     reader 1		               reader 2
> > >>>
> > >>>     calls read(65536)                   calls read(65536)
> > >>>
> > >>>     calls NtReadFile(16384 bytes)
> > >>>                                         calls NtReadFile(16384 bytes)
> > >>>
> > >>> writer writes 65536 bytes
> > >>>
> > >>>     wakes up and gets 16384 bytes
> > >>>                                         wakes up and gets 16384 bytes
> > >>>     gets the mutex, calls
> > >>>     NtReadFile(32768) which
> > >>>     returns immediately with
> > >>>     32768 bytes added to the
> > >>>     caller's buffer.
> > >>>
> > >>> so the buffer returned to reader 1 is 49152 bytes, with 16384 bytes
> > >>> missing in the middle of it, *without* the reader knowing about that
> > >>> fact.  If reader 1 gets the first 16384 bytes, the 16384 bytes have
> > >>> been read in a single call, at least, so the byte order is not
> > >>> unknowingly broken on the application level.
> > >>>
> > >>> Does that make sense?
> > >>
> > >> Why can't we keep the mutex while waiting on the pending read?
> > >> If we can keep the mutex, the issue above mentioned does not
> > >> happen, right?
> > >>
> > >> What about the patch attached? This keeps the mutex while read()
> > >> but I do not see any defects so far.
> > 
> > LGTM.
> > 
> > If Corinna agrees, I have a couple of suggestions.
> > 
> > 1. With this patch, we can no longer have more than one pending ReadFile.  So 
> > there's no longer a need to count read handles, and the problem with select is 
> > completely fixed as long as the number of bytes requested is less than the pipe 
> > buffer size.
> > 
> > 2. raw_read is now reading in chunks, like raw_write.  For readability of the 
> > code, I think it would be better to make the two functions as similar as 
> > possible.  For example, you could replace the do/while loop by a 
> > while(total_len<orig_len) loop.  And you could even use similar names for the 
> > variables, e.g., nbytes instead of total_len, or vice versa.
> 
> Thanks for the suggestion. I have rebuilt the patch.
> Please see the patch attached.

This patch seems to fail to adopt to current git head of topic/pipe
branch. I rebuilt the patch to fit current top/pipe.

Please see the patch attached.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

[-- Attachment #2: 0001-Cygwin-pipe-Stop-counting-reader-and-read-all-availa.patch --]
[-- Type: application/octet-stream, Size: 5024 bytes --]

From 8bd38b3748adb1aa0037fb0683d20d680ce553c4 Mon Sep 17 00:00:00 2001
From: Takashi Yano <takashi.yano@nifty.ne.jp>
Date: Sun, 5 Sep 2021 22:35:28 +0900
Subject: [PATCH] Cygwin: pipe: Stop counting reader and read all available
 data.

- By guarding read with read_mtx, no more than one ReadFile can
  be called simultaneously. So couting read handles is no longer
  necessary.
- Make raw_read code as similar as possible to raw_write code.
---
 winsup/cygwin/fhandler_pipe.cc | 82 +++++++++++++++++++---------------
 1 file changed, 45 insertions(+), 37 deletions(-)

diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index 68974eb80..c094515b3 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -221,12 +221,10 @@ fhandler_pipe::get_proc_fd_name (char *buf)
 void __reg3
 fhandler_pipe::raw_read (void *ptr, size_t& len)
 {
-  NTSTATUS status;
+  size_t nbytes = 0;
+  NTSTATUS status = STATUS_SUCCESS;
   IO_STATUS_BLOCK io;
   HANDLE evt = NULL;
-  DWORD waitret = WAIT_OBJECT_0;
-  bool keep_looping = false;
-  size_t orig_len = len;
 
   if (!len)
     return;
@@ -239,43 +237,47 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
       return;
     }
 
-  do
+  DWORD timeout = is_nonblocking () ? 0 : INFINITE;
+  if (WAIT_TIMEOUT == WaitForSingleObject (read_mtx, timeout))
     {
-      len = orig_len;
-      keep_looping = false;
+      set_errno (EAGAIN);
+      len = (size_t) -1;
+      return;
+    }
+  while (nbytes < len)
+    {
+      ULONG_PTR nbytes_now = 0;
+      size_t left = len - nbytes;
+      ULONG len1 = (ULONG) left;
+      DWORD waitret = WAIT_OBJECT_0;
+
       if (evt)
 	ResetEvent (evt);
       if (!is_nonblocking ())
 	{
 	  FILE_PIPE_LOCAL_INFORMATION fpli;
-	  ULONG reader_count;
-	  ULONG max_len = 64;
-
-	  WaitForSingleObject (read_mtx, INFINITE);
 
 	  /* If the pipe is empty, don't request more bytes than half the
-	     pipe buffer size.  Every pending read lowers WriteQuotaAvailable
+	     pipe buffer size. pending read lowers WriteQuotaAvailable
 	     on the write side and thus affects select's ability to return
-	     more or less reliable info whether a write succeeds or not.
-
-	     Let the size of the request depend on the number of readers
-	     at the time. */
+	     more or less reliable info whether a write succeeds or not. */
+	  ULONG chunk = max_atomic_write / 2;
 	  status = NtQueryInformationFile (get_handle (), &io,
 					   &fpli, sizeof (fpli),
 					   FilePipeLocalInformation);
 	  if (NT_SUCCESS (status) && fpli.ReadDataAvailable == 0)
 	    {
-	      reader_count = get_obj_handle_count (get_handle ());
-	      if (reader_count < 10)
-		max_len = fpli.InboundQuota / (2 * reader_count);
-	      if (len > max_len)
-		len = max_len;
+	      if (nbytes != 0)
+		break;
+	      chunk = fpli.InboundQuota / 2;
 	    }
+	  if (!NT_SUCCESS (status) && nbytes != 0)
+	    break;
+
+	  len1 = (left < chunk) ? (ULONG) left : chunk;
 	}
       status = NtReadFile (get_handle (), evt, NULL, NULL, &io, ptr,
-			   len, NULL, NULL);
-      if (!is_nonblocking ())
-	ReleaseMutex (read_mtx);
+			   len1, NULL, NULL);
       if (evt && status == STATUS_PENDING)
 	{
 	  waitret = cygwait (evt);
@@ -292,13 +294,13 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
 	    set_errno (EBADF);
 	  else
 	    __seterrno ();
-	  len = (size_t) -1;
+	  nbytes = (size_t) -1;
 	}
       else if (NT_SUCCESS (status))
 	{
-	  len = io.Information;
-	  if (len == 0)
-	    keep_looping = true;
+	  nbytes_now = io.Information;
+	  ptr = ((char *) ptr) + nbytes_now;
+	  nbytes += nbytes_now;
 	}
       else
 	{
@@ -308,40 +310,46 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
 	    case STATUS_END_OF_FILE:
 	    case STATUS_PIPE_BROKEN:
 	      /* This is really EOF.  */
-	      len = 0;
 	      break;
 	    case STATUS_MORE_ENTRIES:
 	    case STATUS_BUFFER_OVERFLOW:
 	      /* `io.Information' is supposedly valid.  */
-	      len = io.Information;
-	      if (len == 0)
-		keep_looping = true;
+	      nbytes_now = io.Information;
+	      ptr = ((char *) ptr) + nbytes_now;
+	      nbytes += nbytes_now;
 	      break;
 	    case STATUS_PIPE_LISTENING:
 	    case STATUS_PIPE_EMPTY:
+	      if (nbytes != 0)
+		break;
 	      if (is_nonblocking ())
 		{
 		  set_errno (EAGAIN);
-		  len = (size_t) -1;
+		  nbytes = (size_t) -1;
 		  break;
 		}
 	      fallthrough;
 	    default:
 	      __seterrno_from_nt_status (status);
-	      len = (size_t) -1;
+	      nbytes = (size_t) -1;
 	      break;
 	    }
 	}
-    } while (keep_looping);
+
+      if (nbytes_now == 0)
+	break;
+    }
+  ReleaseMutex (read_mtx);
   if (evt)
     CloseHandle (evt);
-  if (status == STATUS_THREAD_SIGNALED)
+  if (status == STATUS_THREAD_SIGNALED && nbytes == 0)
     {
       set_errno (EINTR);
-      len = (size_t) -1;
+      nbytes = (size_t) -1;
     }
   else if (status == STATUS_THREAD_CANCELED)
     pthread::static_cancel_self ();
+  len = nbytes;
 }
 
 ssize_t __reg3
-- 
2.33.0


^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-05 13:40                                                                                                         ` Takashi Yano
@ 2021-09-05 13:50                                                                                                           ` Takashi Yano
  2021-09-05 18:47                                                                                                             ` Ken Brown
  0 siblings, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-05 13:50 UTC (permalink / raw)
  To: cygwin-developers

On Sun, 5 Sep 2021 22:40:59 +0900
Takashi Yano wrote:
> On Sun, 5 Sep 2021 08:15:23 +0900
> Takashi Yano wrote:
> > Hi Ken,
> > 
> > On Sat, 4 Sep 2021 10:04:12 -0400
> > Ken Brown wrote:
> > > On 9/4/2021 8:37 AM, Takashi Yano wrote:
> > > > On Sat, 4 Sep 2021 21:02:58 +0900
> > > > Takashi Yano wrote:
> > > >> Hi Corinna, Ken,
> > > >>
> > > >> On Fri, 3 Sep 2021 09:27:37 -0400
> > > >> Ken Brown wrote:
> > > >>> On 9/3/2021 8:22 AM, Takashi Yano wrote:
> > > >>>> POSIX says:
> > > >>>>       The value returned may be less than nbyte if the number of bytes left
> > > >>>>       in the file is less than nbyte, if the read() request was interrupted
> > > >>>>       by a signal, or if the file is a pipe or FIFO or special file and has
> > > >>>>                                                                         ~~~
> > > >>>>       fewer than nbyte bytes immediately available for reading.
> > > >>>>       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > > >>>>
> > > >>>> https://pubs.opengroup.org/onlinepubs/009604599/functions/read.html
> > > >>>>
> > > >>>> If it is turned over, read() should read all data immediately available,
> > > >>>> I think.
> > > >>>
> > > >>> I understand the reasoning now, but I think your patch isn't quite right.  As it
> > > >>> stands, if the call to NtQueryInformationFile fails but total_length != 0,
> > > >>> you're trying to read again without knowing that there's data in the pipe.
> > > >>>
> > > >>> Also, I think you need the following:
> > > >>>
> > > >>> diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
> > > >>> index ef7823ae5..46bb96961 100644
> > > >>> --- a/winsup/cygwin/fhandler_pipe.cc
> > > >>> +++ b/winsup/cygwin/fhandler_pipe.cc
> > > >>> @@ -348,8 +348,13 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
> > > >>>        CloseHandle (evt);
> > > >>>      if (status == STATUS_THREAD_SIGNALED)
> > > >>>        {
> > > >>> -      set_errno (EINTR);
> > > >>> -      len = (size_t) -1;
> > > >>> +      if (total_len == 0)
> > > >>> +       {
> > > >>> +         set_errno (EINTR);
> > > >>> +         len = (size_t) -1;
> > > >>> +       }
> > > >>> +      else
> > > >>> +       len = total_len;
> > > >>>        }
> > > >>>      else if (status == STATUS_THREAD_CANCELED)
> > > >>>        pthread::static_cancel_self ();
> > > >>
> > > >> Thanks for your advice. I fixed the issue and attached new patch.
> > > >>
> > > >> On Fri, 3 Sep 2021 17:37:13 +0200
> > > >> Corinna Vinschen wrote:
> > > >>> Hmm, I see the point, but we might have another problem with that.
> > > >>>
> > > >>> We can't keep the mutex while waiting on the pending read, and there
> > > >>> could be more than one pending read running at the time.  if so,
> > > >>> chances are extremly high, that the data written to the buffer gets
> > > >>> split like this:
> > > >>>
> > > >>>     reader 1		               reader 2
> > > >>>
> > > >>>     calls read(65536)                   calls read(65536)
> > > >>>
> > > >>>     calls NtReadFile(16384 bytes)
> > > >>>                                         calls NtReadFile(16384 bytes)
> > > >>>
> > > >>> writer writes 65536 bytes
> > > >>>
> > > >>>     wakes up and gets 16384 bytes
> > > >>>                                         wakes up and gets 16384 bytes
> > > >>>     gets the mutex, calls
> > > >>>     NtReadFile(32768) which
> > > >>>     returns immediately with
> > > >>>     32768 bytes added to the
> > > >>>     caller's buffer.
> > > >>>
> > > >>> so the buffer returned to reader 1 is 49152 bytes, with 16384 bytes
> > > >>> missing in the middle of it, *without* the reader knowing about that
> > > >>> fact.  If reader 1 gets the first 16384 bytes, the 16384 bytes have
> > > >>> been read in a single call, at least, so the byte order is not
> > > >>> unknowingly broken on the application level.
> > > >>>
> > > >>> Does that make sense?
> > > >>
> > > >> Why can't we keep the mutex while waiting on the pending read?
> > > >> If we can keep the mutex, the issue above mentioned does not
> > > >> happen, right?
> > > >>
> > > >> What about the patch attached? This keeps the mutex while read()
> > > >> but I do not see any defects so far.
> > > 
> > > LGTM.
> > > 
> > > If Corinna agrees, I have a couple of suggestions.
> > > 
> > > 1. With this patch, we can no longer have more than one pending ReadFile.  So 
> > > there's no longer a need to count read handles, and the problem with select is 
> > > completely fixed as long as the number of bytes requested is less than the pipe 
> > > buffer size.
> > > 
> > > 2. raw_read is now reading in chunks, like raw_write.  For readability of the 
> > > code, I think it would be better to make the two functions as similar as 
> > > possible.  For example, you could replace the do/while loop by a 
> > > while(total_len<orig_len) loop.  And you could even use similar names for the 
> > > variables, e.g., nbytes instead of total_len, or vice versa.
> > 
> > Thanks for the suggestion. I have rebuilt the patch.
> > Please see the patch attached.
> 
> This patch seems to fail to adopt to current git head of topic/pipe
> branch. I rebuilt the patch to fit current top/pipe.
> 
> Please see the patch attached.

Small typo.

-	     pipe buffer size.  Every pending read lowers WriteQuotaAvailable
+	     pipe buffer size. pending read lowers WriteQuotaAvailable

should be:

-	     pipe buffer size.  Every pending read lowers WriteQuotaAvailable
+	     pipe buffer size. Pending read lowers WriteQuotaAvailable

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-05 13:50                                                                                                           ` Takashi Yano
@ 2021-09-05 18:47                                                                                                             ` Ken Brown
  2021-09-05 19:42                                                                                                               ` Takashi Yano
  2021-09-05 20:09                                                                                                               ` Takashi Yano
  0 siblings, 2 replies; 250+ messages in thread
From: Ken Brown @ 2021-09-05 18:47 UTC (permalink / raw)
  To: cygwin-developers

Hi Takashi,

On 9/5/2021 9:50 AM, Takashi Yano wrote:
> On Sun, 5 Sep 2021 22:40:59 +0900
> Takashi Yano wrote:
>> On Sun, 5 Sep 2021 08:15:23 +0900
>> Takashi Yano wrote:
>>> Hi Ken,
>>>
>>> On Sat, 4 Sep 2021 10:04:12 -0400
>>> Ken Brown wrote:
>>>> On 9/4/2021 8:37 AM, Takashi Yano wrote:
>>>>> On Sat, 4 Sep 2021 21:02:58 +0900
>>>>> Takashi Yano wrote:
>>>>>> Hi Corinna, Ken,
>>>>>>
>>>>>> On Fri, 3 Sep 2021 09:27:37 -0400
>>>>>> Ken Brown wrote:
>>>>>>> On 9/3/2021 8:22 AM, Takashi Yano wrote:
>>>>>>>> POSIX says:
>>>>>>>>        The value returned may be less than nbyte if the number of bytes left
>>>>>>>>        in the file is less than nbyte, if the read() request was interrupted
>>>>>>>>        by a signal, or if the file is a pipe or FIFO or special file and has
>>>>>>>>                                                                          ~~~
>>>>>>>>        fewer than nbyte bytes immediately available for reading.
>>>>>>>>        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>>>>>
>>>>>>>> https://pubs.opengroup.org/onlinepubs/009604599/functions/read.html
>>>>>>>>
>>>>>>>> If it is turned over, read() should read all data immediately available,
>>>>>>>> I think.
>>>>>>>
>>>>>>> I understand the reasoning now, but I think your patch isn't quite right.  As it
>>>>>>> stands, if the call to NtQueryInformationFile fails but total_length != 0,
>>>>>>> you're trying to read again without knowing that there's data in the pipe.
>>>>>>>
>>>>>>> Also, I think you need the following:
>>>>>>>
>>>>>>> diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
>>>>>>> index ef7823ae5..46bb96961 100644
>>>>>>> --- a/winsup/cygwin/fhandler_pipe.cc
>>>>>>> +++ b/winsup/cygwin/fhandler_pipe.cc
>>>>>>> @@ -348,8 +348,13 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
>>>>>>>         CloseHandle (evt);
>>>>>>>       if (status == STATUS_THREAD_SIGNALED)
>>>>>>>         {
>>>>>>> -      set_errno (EINTR);
>>>>>>> -      len = (size_t) -1;
>>>>>>> +      if (total_len == 0)
>>>>>>> +       {
>>>>>>> +         set_errno (EINTR);
>>>>>>> +         len = (size_t) -1;
>>>>>>> +       }
>>>>>>> +      else
>>>>>>> +       len = total_len;
>>>>>>>         }
>>>>>>>       else if (status == STATUS_THREAD_CANCELED)
>>>>>>>         pthread::static_cancel_self ();
>>>>>>
>>>>>> Thanks for your advice. I fixed the issue and attached new patch.
>>>>>>
>>>>>> On Fri, 3 Sep 2021 17:37:13 +0200
>>>>>> Corinna Vinschen wrote:
>>>>>>> Hmm, I see the point, but we might have another problem with that.
>>>>>>>
>>>>>>> We can't keep the mutex while waiting on the pending read, and there
>>>>>>> could be more than one pending read running at the time.  if so,
>>>>>>> chances are extremly high, that the data written to the buffer gets
>>>>>>> split like this:
>>>>>>>
>>>>>>>      reader 1		               reader 2
>>>>>>>
>>>>>>>      calls read(65536)                   calls read(65536)
>>>>>>>
>>>>>>>      calls NtReadFile(16384 bytes)
>>>>>>>                                          calls NtReadFile(16384 bytes)
>>>>>>>
>>>>>>> writer writes 65536 bytes
>>>>>>>
>>>>>>>      wakes up and gets 16384 bytes
>>>>>>>                                          wakes up and gets 16384 bytes
>>>>>>>      gets the mutex, calls
>>>>>>>      NtReadFile(32768) which
>>>>>>>      returns immediately with
>>>>>>>      32768 bytes added to the
>>>>>>>      caller's buffer.
>>>>>>>
>>>>>>> so the buffer returned to reader 1 is 49152 bytes, with 16384 bytes
>>>>>>> missing in the middle of it, *without* the reader knowing about that
>>>>>>> fact.  If reader 1 gets the first 16384 bytes, the 16384 bytes have
>>>>>>> been read in a single call, at least, so the byte order is not
>>>>>>> unknowingly broken on the application level.
>>>>>>>
>>>>>>> Does that make sense?
>>>>>>
>>>>>> Why can't we keep the mutex while waiting on the pending read?
>>>>>> If we can keep the mutex, the issue above mentioned does not
>>>>>> happen, right?
>>>>>>
>>>>>> What about the patch attached? This keeps the mutex while read()
>>>>>> but I do not see any defects so far.
>>>>
>>>> LGTM.
>>>>
>>>> If Corinna agrees, I have a couple of suggestions.
>>>>
>>>> 1. With this patch, we can no longer have more than one pending ReadFile.  So
>>>> there's no longer a need to count read handles, and the problem with select is
>>>> completely fixed as long as the number of bytes requested is less than the pipe
>>>> buffer size.
>>>>
>>>> 2. raw_read is now reading in chunks, like raw_write.  For readability of the
>>>> code, I think it would be better to make the two functions as similar as
>>>> possible.  For example, you could replace the do/while loop by a
>>>> while(total_len<orig_len) loop.  And you could even use similar names for the
>>>> variables, e.g., nbytes instead of total_len, or vice versa.
>>>
>>> Thanks for the suggestion. I have rebuilt the patch.
>>> Please see the patch attached.
>>
>> This patch seems to fail to adopt to current git head of topic/pipe
>> branch. I rebuilt the patch to fit current top/pipe.
>>
>> Please see the patch attached.
> 
> Small typo.
> 
> -	     pipe buffer size.  Every pending read lowers WriteQuotaAvailable
> +	     pipe buffer size. pending read lowers WriteQuotaAvailable
> 
> should be:
> 
> -	     pipe buffer size.  Every pending read lowers WriteQuotaAvailable
> +	     pipe buffer size. Pending read lowers WriteQuotaAvailable

The patch looks great to me.  Two minor nits:

1. The patch doesn't apply cleanly.  Could you rebase it against the current 
HEAD of topic/pipe?

2. There's no need for chunk to be less than the number of bytes requested if we 
know there's data in the pipe.  So maybe something like this (untested) would be 
better:

           ULONG chunk;
           status = NtQueryInformationFile (get_handle (), &io,
                                            &fpli, sizeof (fpli),
                                            FilePipeLocalInformation);
           if (NT_SUCCESS (status))
             {
               if (fpli.ReadDataAvailable > 0)
                 chunk = left;
               else if (nbytes != 0)
                 break;
               else
                 chunk = fpli.InboundQuota / 2;
             }
           else if (nbytes != 0)
             break;
           else
             chunk = max_atomic_write / 2;

           if (chunk < left)
             len1 = chunk;

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-05 18:47                                                                                                             ` Ken Brown
@ 2021-09-05 19:42                                                                                                               ` Takashi Yano
  2021-09-05 20:09                                                                                                               ` Takashi Yano
  1 sibling, 0 replies; 250+ messages in thread
From: Takashi Yano @ 2021-09-05 19:42 UTC (permalink / raw)
  To: cygwin-developers

On Sun, 5 Sep 2021 14:47:26 -0400
Ken Brown wrote:
> Hi Takashi,
> 
> On 9/5/2021 9:50 AM, Takashi Yano wrote:
> > On Sun, 5 Sep 2021 22:40:59 +0900
> > Takashi Yano wrote:
> >> On Sun, 5 Sep 2021 08:15:23 +0900
> >> Takashi Yano wrote:
> >>> Hi Ken,
> >>>
> >>> On Sat, 4 Sep 2021 10:04:12 -0400
> >>> Ken Brown wrote:
> >>>> On 9/4/2021 8:37 AM, Takashi Yano wrote:
> >>>>> On Sat, 4 Sep 2021 21:02:58 +0900
> >>>>> Takashi Yano wrote:
> >>>>>> Hi Corinna, Ken,
> >>>>>>
> >>>>>> On Fri, 3 Sep 2021 09:27:37 -0400
> >>>>>> Ken Brown wrote:
> >>>>>>> On 9/3/2021 8:22 AM, Takashi Yano wrote:
> >>>>>>>> POSIX says:
> >>>>>>>>        The value returned may be less than nbyte if the number of bytes left
> >>>>>>>>        in the file is less than nbyte, if the read() request was interrupted
> >>>>>>>>        by a signal, or if the file is a pipe or FIFO or special file and has
> >>>>>>>>                                                                          ~~~
> >>>>>>>>        fewer than nbyte bytes immediately available for reading.
> >>>>>>>>        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >>>>>>>>
> >>>>>>>> https://pubs.opengroup.org/onlinepubs/009604599/functions/read.html
> >>>>>>>>
> >>>>>>>> If it is turned over, read() should read all data immediately available,
> >>>>>>>> I think.
> >>>>>>>
> >>>>>>> I understand the reasoning now, but I think your patch isn't quite right.  As it
> >>>>>>> stands, if the call to NtQueryInformationFile fails but total_length != 0,
> >>>>>>> you're trying to read again without knowing that there's data in the pipe.
> >>>>>>>
> >>>>>>> Also, I think you need the following:
> >>>>>>>
> >>>>>>> diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
> >>>>>>> index ef7823ae5..46bb96961 100644
> >>>>>>> --- a/winsup/cygwin/fhandler_pipe.cc
> >>>>>>> +++ b/winsup/cygwin/fhandler_pipe.cc
> >>>>>>> @@ -348,8 +348,13 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
> >>>>>>>         CloseHandle (evt);
> >>>>>>>       if (status == STATUS_THREAD_SIGNALED)
> >>>>>>>         {
> >>>>>>> -      set_errno (EINTR);
> >>>>>>> -      len = (size_t) -1;
> >>>>>>> +      if (total_len == 0)
> >>>>>>> +       {
> >>>>>>> +         set_errno (EINTR);
> >>>>>>> +         len = (size_t) -1;
> >>>>>>> +       }
> >>>>>>> +      else
> >>>>>>> +       len = total_len;
> >>>>>>>         }
> >>>>>>>       else if (status == STATUS_THREAD_CANCELED)
> >>>>>>>         pthread::static_cancel_self ();
> >>>>>>
> >>>>>> Thanks for your advice. I fixed the issue and attached new patch.
> >>>>>>
> >>>>>> On Fri, 3 Sep 2021 17:37:13 +0200
> >>>>>> Corinna Vinschen wrote:
> >>>>>>> Hmm, I see the point, but we might have another problem with that.
> >>>>>>>
> >>>>>>> We can't keep the mutex while waiting on the pending read, and there
> >>>>>>> could be more than one pending read running at the time.  if so,
> >>>>>>> chances are extremly high, that the data written to the buffer gets
> >>>>>>> split like this:
> >>>>>>>
> >>>>>>>      reader 1		               reader 2
> >>>>>>>
> >>>>>>>      calls read(65536)                   calls read(65536)
> >>>>>>>
> >>>>>>>      calls NtReadFile(16384 bytes)
> >>>>>>>                                          calls NtReadFile(16384 bytes)
> >>>>>>>
> >>>>>>> writer writes 65536 bytes
> >>>>>>>
> >>>>>>>      wakes up and gets 16384 bytes
> >>>>>>>                                          wakes up and gets 16384 bytes
> >>>>>>>      gets the mutex, calls
> >>>>>>>      NtReadFile(32768) which
> >>>>>>>      returns immediately with
> >>>>>>>      32768 bytes added to the
> >>>>>>>      caller's buffer.
> >>>>>>>
> >>>>>>> so the buffer returned to reader 1 is 49152 bytes, with 16384 bytes
> >>>>>>> missing in the middle of it, *without* the reader knowing about that
> >>>>>>> fact.  If reader 1 gets the first 16384 bytes, the 16384 bytes have
> >>>>>>> been read in a single call, at least, so the byte order is not
> >>>>>>> unknowingly broken on the application level.
> >>>>>>>
> >>>>>>> Does that make sense?
> >>>>>>
> >>>>>> Why can't we keep the mutex while waiting on the pending read?
> >>>>>> If we can keep the mutex, the issue above mentioned does not
> >>>>>> happen, right?
> >>>>>>
> >>>>>> What about the patch attached? This keeps the mutex while read()
> >>>>>> but I do not see any defects so far.
> >>>>
> >>>> LGTM.
> >>>>
> >>>> If Corinna agrees, I have a couple of suggestions.
> >>>>
> >>>> 1. With this patch, we can no longer have more than one pending ReadFile.  So
> >>>> there's no longer a need to count read handles, and the problem with select is
> >>>> completely fixed as long as the number of bytes requested is less than the pipe
> >>>> buffer size.
> >>>>
> >>>> 2. raw_read is now reading in chunks, like raw_write.  For readability of the
> >>>> code, I think it would be better to make the two functions as similar as
> >>>> possible.  For example, you could replace the do/while loop by a
> >>>> while(total_len<orig_len) loop.  And you could even use similar names for the
> >>>> variables, e.g., nbytes instead of total_len, or vice versa.
> >>>
> >>> Thanks for the suggestion. I have rebuilt the patch.
> >>> Please see the patch attached.
> >>
> >> This patch seems to fail to adopt to current git head of topic/pipe
> >> branch. I rebuilt the patch to fit current top/pipe.
> >>
> >> Please see the patch attached.
> > 
> > Small typo.
> > 
> > -	     pipe buffer size.  Every pending read lowers WriteQuotaAvailable
> > +	     pipe buffer size. pending read lowers WriteQuotaAvailable
> > 
> > should be:
> > 
> > -	     pipe buffer size.  Every pending read lowers WriteQuotaAvailable
> > +	     pipe buffer size. Pending read lowers WriteQuotaAvailable
> 
> The patch looks great to me.  Two minor nits:
> 
> 1. The patch doesn't apply cleanly.  Could you rebase it against the current 
> HEAD of topic/pipe?
> 
> 2. There's no need for chunk to be less than the number of bytes requested if we 
> know there's data in the pipe.  So maybe something like this (untested) would be 
> better:
> 
>            ULONG chunk;
>            status = NtQueryInformationFile (get_handle (), &io,
>                                             &fpli, sizeof (fpli),
>                                             FilePipeLocalInformation);
>            if (NT_SUCCESS (status))
>              {
>                if (fpli.ReadDataAvailable > 0)
>                  chunk = left;
>                else if (nbytes != 0)
>                  break;
>                else
>                  chunk = fpli.InboundQuota / 2;
>              }
>            else if (nbytes != 0)
>              break;
>            else
>              chunk = max_atomic_write / 2;
> 
>            if (chunk < left)
>              len1 = chunk;

Thanks for the advice.

As for 1., is not the current git head 866a62037e235d558584e821a11d60d848e06234?

In my environment, patch can apply cleanly by git am.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-05 18:47                                                                                                             ` Ken Brown
  2021-09-05 19:42                                                                                                               ` Takashi Yano
@ 2021-09-05 20:09                                                                                                               ` Takashi Yano
  2021-09-05 20:27                                                                                                                 ` Ken Brown
  2021-09-06  8:13                                                                                                                 ` Corinna Vinschen
  1 sibling, 2 replies; 250+ messages in thread
From: Takashi Yano @ 2021-09-05 20:09 UTC (permalink / raw)
  To: cygwin-developers

[-- Attachment #1: Type: text/plain, Size: 6900 bytes --]

Hi Ken,

On Sun, 5 Sep 2021 14:47:26 -0400
Ken Brown wrote:
> Hi Takashi,
> 
> On 9/5/2021 9:50 AM, Takashi Yano wrote:
> > On Sun, 5 Sep 2021 22:40:59 +0900
> > Takashi Yano wrote:
> >> On Sun, 5 Sep 2021 08:15:23 +0900
> >> Takashi Yano wrote:
> >>> Hi Ken,
> >>>
> >>> On Sat, 4 Sep 2021 10:04:12 -0400
> >>> Ken Brown wrote:
> >>>> On 9/4/2021 8:37 AM, Takashi Yano wrote:
> >>>>> On Sat, 4 Sep 2021 21:02:58 +0900
> >>>>> Takashi Yano wrote:
> >>>>>> Hi Corinna, Ken,
> >>>>>>
> >>>>>> On Fri, 3 Sep 2021 09:27:37 -0400
> >>>>>> Ken Brown wrote:
> >>>>>>> On 9/3/2021 8:22 AM, Takashi Yano wrote:
> >>>>>>>> POSIX says:
> >>>>>>>>        The value returned may be less than nbyte if the number of bytes left
> >>>>>>>>        in the file is less than nbyte, if the read() request was interrupted
> >>>>>>>>        by a signal, or if the file is a pipe or FIFO or special file and has
> >>>>>>>>                                                                          ~~~
> >>>>>>>>        fewer than nbyte bytes immediately available for reading.
> >>>>>>>>        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >>>>>>>>
> >>>>>>>> https://pubs.opengroup.org/onlinepubs/009604599/functions/read.html
> >>>>>>>>
> >>>>>>>> If it is turned over, read() should read all data immediately available,
> >>>>>>>> I think.
> >>>>>>>
> >>>>>>> I understand the reasoning now, but I think your patch isn't quite right.  As it
> >>>>>>> stands, if the call to NtQueryInformationFile fails but total_length != 0,
> >>>>>>> you're trying to read again without knowing that there's data in the pipe.
> >>>>>>>
> >>>>>>> Also, I think you need the following:
> >>>>>>>
> >>>>>>> diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
> >>>>>>> index ef7823ae5..46bb96961 100644
> >>>>>>> --- a/winsup/cygwin/fhandler_pipe.cc
> >>>>>>> +++ b/winsup/cygwin/fhandler_pipe.cc
> >>>>>>> @@ -348,8 +348,13 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
> >>>>>>>         CloseHandle (evt);
> >>>>>>>       if (status == STATUS_THREAD_SIGNALED)
> >>>>>>>         {
> >>>>>>> -      set_errno (EINTR);
> >>>>>>> -      len = (size_t) -1;
> >>>>>>> +      if (total_len == 0)
> >>>>>>> +       {
> >>>>>>> +         set_errno (EINTR);
> >>>>>>> +         len = (size_t) -1;
> >>>>>>> +       }
> >>>>>>> +      else
> >>>>>>> +       len = total_len;
> >>>>>>>         }
> >>>>>>>       else if (status == STATUS_THREAD_CANCELED)
> >>>>>>>         pthread::static_cancel_self ();
> >>>>>>
> >>>>>> Thanks for your advice. I fixed the issue and attached new patch.
> >>>>>>
> >>>>>> On Fri, 3 Sep 2021 17:37:13 +0200
> >>>>>> Corinna Vinschen wrote:
> >>>>>>> Hmm, I see the point, but we might have another problem with that.
> >>>>>>>
> >>>>>>> We can't keep the mutex while waiting on the pending read, and there
> >>>>>>> could be more than one pending read running at the time.  if so,
> >>>>>>> chances are extremly high, that the data written to the buffer gets
> >>>>>>> split like this:
> >>>>>>>
> >>>>>>>      reader 1		               reader 2
> >>>>>>>
> >>>>>>>      calls read(65536)                   calls read(65536)
> >>>>>>>
> >>>>>>>      calls NtReadFile(16384 bytes)
> >>>>>>>                                          calls NtReadFile(16384 bytes)
> >>>>>>>
> >>>>>>> writer writes 65536 bytes
> >>>>>>>
> >>>>>>>      wakes up and gets 16384 bytes
> >>>>>>>                                          wakes up and gets 16384 bytes
> >>>>>>>      gets the mutex, calls
> >>>>>>>      NtReadFile(32768) which
> >>>>>>>      returns immediately with
> >>>>>>>      32768 bytes added to the
> >>>>>>>      caller's buffer.
> >>>>>>>
> >>>>>>> so the buffer returned to reader 1 is 49152 bytes, with 16384 bytes
> >>>>>>> missing in the middle of it, *without* the reader knowing about that
> >>>>>>> fact.  If reader 1 gets the first 16384 bytes, the 16384 bytes have
> >>>>>>> been read in a single call, at least, so the byte order is not
> >>>>>>> unknowingly broken on the application level.
> >>>>>>>
> >>>>>>> Does that make sense?
> >>>>>>
> >>>>>> Why can't we keep the mutex while waiting on the pending read?
> >>>>>> If we can keep the mutex, the issue above mentioned does not
> >>>>>> happen, right?
> >>>>>>
> >>>>>> What about the patch attached? This keeps the mutex while read()
> >>>>>> but I do not see any defects so far.
> >>>>
> >>>> LGTM.
> >>>>
> >>>> If Corinna agrees, I have a couple of suggestions.
> >>>>
> >>>> 1. With this patch, we can no longer have more than one pending ReadFile.  So
> >>>> there's no longer a need to count read handles, and the problem with select is
> >>>> completely fixed as long as the number of bytes requested is less than the pipe
> >>>> buffer size.
> >>>>
> >>>> 2. raw_read is now reading in chunks, like raw_write.  For readability of the
> >>>> code, I think it would be better to make the two functions as similar as
> >>>> possible.  For example, you could replace the do/while loop by a
> >>>> while(total_len<orig_len) loop.  And you could even use similar names for the
> >>>> variables, e.g., nbytes instead of total_len, or vice versa.
> >>>
> >>> Thanks for the suggestion. I have rebuilt the patch.
> >>> Please see the patch attached.
> >>
> >> This patch seems to fail to adopt to current git head of topic/pipe
> >> branch. I rebuilt the patch to fit current top/pipe.
> >>
> >> Please see the patch attached.
> > 
> > Small typo.
> > 
> > -	     pipe buffer size.  Every pending read lowers WriteQuotaAvailable
> > +	     pipe buffer size. pending read lowers WriteQuotaAvailable
> > 
> > should be:
> > 
> > -	     pipe buffer size.  Every pending read lowers WriteQuotaAvailable
> > +	     pipe buffer size. Pending read lowers WriteQuotaAvailable
> 
> The patch looks great to me.  Two minor nits:
> 
> 1. The patch doesn't apply cleanly.  Could you rebase it against the current 
> HEAD of topic/pipe?
> 
> 2. There's no need for chunk to be less than the number of bytes requested if we 
> know there's data in the pipe.  So maybe something like this (untested) would be 
> better:
> 
>            ULONG chunk;
>            status = NtQueryInformationFile (get_handle (), &io,
>                                             &fpli, sizeof (fpli),
>                                             FilePipeLocalInformation);
>            if (NT_SUCCESS (status))
>              {
>                if (fpli.ReadDataAvailable > 0)
>                  chunk = left;
>                else if (nbytes != 0)
>                  break;
>                else
>                  chunk = fpli.InboundQuota / 2;
>              }
>            else if (nbytes != 0)
>              break;
>            else
>              chunk = max_atomic_write / 2;
> 
>            if (chunk < left)
>              len1 = chunk;

Could you please try attached new patch?

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

[-- Attachment #2: 0001-Cygwin-pipe-Stop-counting-reader-and-read-all-availa.patch --]
[-- Type: application/octet-stream, Size: 5165 bytes --]

From d83d0314a83467d4f1778ad835b12a36c9dcb83b Mon Sep 17 00:00:00 2001
From: Takashi Yano <takashi.yano@nifty.ne.jp>
Date: Mon, 6 Sep 2021 04:58:58 +0900
Subject: [PATCH] Cygwin: pipe: Stop counting reader and read all available
 data.

- By guarding read with read_mtx, no more than one ReadFile can
  be called simultaneously. So couting read handles is no longer
  necessary.
- Make raw_read code as similar as possible to raw_write code.
---
 winsup/cygwin/fhandler_pipe.cc | 90 +++++++++++++++++++---------------
 1 file changed, 51 insertions(+), 39 deletions(-)

diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index 68974eb80..1a74551c6 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -221,12 +221,10 @@ fhandler_pipe::get_proc_fd_name (char *buf)
 void __reg3
 fhandler_pipe::raw_read (void *ptr, size_t& len)
 {
-  NTSTATUS status;
+  size_t nbytes = 0;
+  NTSTATUS status = STATUS_SUCCESS;
   IO_STATUS_BLOCK io;
   HANDLE evt = NULL;
-  DWORD waitret = WAIT_OBJECT_0;
-  bool keep_looping = false;
-  size_t orig_len = len;
 
   if (!len)
     return;
@@ -239,43 +237,51 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
       return;
     }
 
-  do
+  DWORD timeout = is_nonblocking () ? 0 : INFINITE;
+  if (WAIT_TIMEOUT == WaitForSingleObject (read_mtx, timeout))
     {
-      len = orig_len;
-      keep_looping = false;
+      set_errno (EAGAIN);
+      len = (size_t) -1;
+      return;
+    }
+  while (nbytes < len)
+    {
+      ULONG_PTR nbytes_now = 0;
+      size_t left = len - nbytes;
+      ULONG len1 = (ULONG) left;
+      DWORD waitret = WAIT_OBJECT_0;
+
       if (evt)
 	ResetEvent (evt);
       if (!is_nonblocking ())
 	{
 	  FILE_PIPE_LOCAL_INFORMATION fpli;
-	  ULONG reader_count;
-	  ULONG max_len = 64;
-
-	  WaitForSingleObject (read_mtx, INFINITE);
 
 	  /* If the pipe is empty, don't request more bytes than half the
-	     pipe buffer size.  Every pending read lowers WriteQuotaAvailable
-	     on the write side and thus affects select's ability to return
-	     more or less reliable info whether a write succeeds or not.
-
-	     Let the size of the request depend on the number of readers
-	     at the time. */
+	     pipe buffer size. Pending read lowers WriteQuotaAvailable on
+	     the write side and thus affects select's ability to return
+	     more or less reliable info whether a write succeeds or not. */
+	  ULONG chunk = max_atomic_write / 2;
 	  status = NtQueryInformationFile (get_handle (), &io,
 					   &fpli, sizeof (fpli),
 					   FilePipeLocalInformation);
-	  if (NT_SUCCESS (status) && fpli.ReadDataAvailable == 0)
+	  if (NT_SUCCESS (status))
 	    {
-	      reader_count = get_obj_handle_count (get_handle ());
-	      if (reader_count < 10)
-		max_len = fpli.InboundQuota / (2 * reader_count);
-	      if (len > max_len)
-		len = max_len;
+	      if (fpli.ReadDataAvailable > 0)
+		chunk = left;
+	      else if (nbytes != 0)
+		break;
+	      else
+		chunk = fpli.InboundQuota / 2;
 	    }
+	  else if (nbytes != 0)
+	    break;
+
+	  if (len1 > chunk)
+	    len1 = chunk;
 	}
       status = NtReadFile (get_handle (), evt, NULL, NULL, &io, ptr,
-			   len, NULL, NULL);
-      if (!is_nonblocking ())
-	ReleaseMutex (read_mtx);
+			   len1, NULL, NULL);
       if (evt && status == STATUS_PENDING)
 	{
 	  waitret = cygwait (evt);
@@ -292,13 +298,13 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
 	    set_errno (EBADF);
 	  else
 	    __seterrno ();
-	  len = (size_t) -1;
+	  nbytes = (size_t) -1;
 	}
       else if (NT_SUCCESS (status))
 	{
-	  len = io.Information;
-	  if (len == 0)
-	    keep_looping = true;
+	  nbytes_now = io.Information;
+	  ptr = ((char *) ptr) + nbytes_now;
+	  nbytes += nbytes_now;
 	}
       else
 	{
@@ -308,40 +314,46 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
 	    case STATUS_END_OF_FILE:
 	    case STATUS_PIPE_BROKEN:
 	      /* This is really EOF.  */
-	      len = 0;
 	      break;
 	    case STATUS_MORE_ENTRIES:
 	    case STATUS_BUFFER_OVERFLOW:
 	      /* `io.Information' is supposedly valid.  */
-	      len = io.Information;
-	      if (len == 0)
-		keep_looping = true;
+	      nbytes_now = io.Information;
+	      ptr = ((char *) ptr) + nbytes_now;
+	      nbytes += nbytes_now;
 	      break;
 	    case STATUS_PIPE_LISTENING:
 	    case STATUS_PIPE_EMPTY:
+	      if (nbytes != 0)
+		break;
 	      if (is_nonblocking ())
 		{
 		  set_errno (EAGAIN);
-		  len = (size_t) -1;
+		  nbytes = (size_t) -1;
 		  break;
 		}
 	      fallthrough;
 	    default:
 	      __seterrno_from_nt_status (status);
-	      len = (size_t) -1;
+	      nbytes = (size_t) -1;
 	      break;
 	    }
 	}
-    } while (keep_looping);
+
+      if (nbytes_now == 0)
+	break;
+    }
+  ReleaseMutex (read_mtx);
   if (evt)
     CloseHandle (evt);
-  if (status == STATUS_THREAD_SIGNALED)
+  if (status == STATUS_THREAD_SIGNALED && nbytes == 0)
     {
       set_errno (EINTR);
-      len = (size_t) -1;
+      nbytes = (size_t) -1;
     }
   else if (status == STATUS_THREAD_CANCELED)
     pthread::static_cancel_self ();
+  len = nbytes;
 }
 
 ssize_t __reg3
-- 
2.33.0


^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-05 20:09                                                                                                               ` Takashi Yano
@ 2021-09-05 20:27                                                                                                                 ` Ken Brown
  2021-09-06  8:13                                                                                                                 ` Corinna Vinschen
  1 sibling, 0 replies; 250+ messages in thread
From: Ken Brown @ 2021-09-05 20:27 UTC (permalink / raw)
  To: cygwin-developers

On 9/5/2021 4:09 PM, Takashi Yano wrote:
> Hi Ken,
> 
> On Sun, 5 Sep 2021 14:47:26 -0400
> Ken Brown wrote:
>> Hi Takashi,
>>
>> On 9/5/2021 9:50 AM, Takashi Yano wrote:
>>> On Sun, 5 Sep 2021 22:40:59 +0900
>>> Takashi Yano wrote:
>>>> On Sun, 5 Sep 2021 08:15:23 +0900
>>>> Takashi Yano wrote:
>>>>> Hi Ken,
>>>>>
>>>>> On Sat, 4 Sep 2021 10:04:12 -0400
>>>>> Ken Brown wrote:
>>>>>> On 9/4/2021 8:37 AM, Takashi Yano wrote:
>>>>>>> On Sat, 4 Sep 2021 21:02:58 +0900
>>>>>>> Takashi Yano wrote:
>>>>>>>> Hi Corinna, Ken,
>>>>>>>>
>>>>>>>> On Fri, 3 Sep 2021 09:27:37 -0400
>>>>>>>> Ken Brown wrote:
>>>>>>>>> On 9/3/2021 8:22 AM, Takashi Yano wrote:
>>>>>>>>>> POSIX says:
>>>>>>>>>>         The value returned may be less than nbyte if the number of bytes left
>>>>>>>>>>         in the file is less than nbyte, if the read() request was interrupted
>>>>>>>>>>         by a signal, or if the file is a pipe or FIFO or special file and has
>>>>>>>>>>                                                                           ~~~
>>>>>>>>>>         fewer than nbyte bytes immediately available for reading.
>>>>>>>>>>         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>>>>>>>
>>>>>>>>>> https://pubs.opengroup.org/onlinepubs/009604599/functions/read.html
>>>>>>>>>>
>>>>>>>>>> If it is turned over, read() should read all data immediately available,
>>>>>>>>>> I think.
>>>>>>>>>
>>>>>>>>> I understand the reasoning now, but I think your patch isn't quite right.  As it
>>>>>>>>> stands, if the call to NtQueryInformationFile fails but total_length != 0,
>>>>>>>>> you're trying to read again without knowing that there's data in the pipe.
>>>>>>>>>
>>>>>>>>> Also, I think you need the following:
>>>>>>>>>
>>>>>>>>> diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
>>>>>>>>> index ef7823ae5..46bb96961 100644
>>>>>>>>> --- a/winsup/cygwin/fhandler_pipe.cc
>>>>>>>>> +++ b/winsup/cygwin/fhandler_pipe.cc
>>>>>>>>> @@ -348,8 +348,13 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
>>>>>>>>>          CloseHandle (evt);
>>>>>>>>>        if (status == STATUS_THREAD_SIGNALED)
>>>>>>>>>          {
>>>>>>>>> -      set_errno (EINTR);
>>>>>>>>> -      len = (size_t) -1;
>>>>>>>>> +      if (total_len == 0)
>>>>>>>>> +       {
>>>>>>>>> +         set_errno (EINTR);
>>>>>>>>> +         len = (size_t) -1;
>>>>>>>>> +       }
>>>>>>>>> +      else
>>>>>>>>> +       len = total_len;
>>>>>>>>>          }
>>>>>>>>>        else if (status == STATUS_THREAD_CANCELED)
>>>>>>>>>          pthread::static_cancel_self ();
>>>>>>>>
>>>>>>>> Thanks for your advice. I fixed the issue and attached new patch.
>>>>>>>>
>>>>>>>> On Fri, 3 Sep 2021 17:37:13 +0200
>>>>>>>> Corinna Vinschen wrote:
>>>>>>>>> Hmm, I see the point, but we might have another problem with that.
>>>>>>>>>
>>>>>>>>> We can't keep the mutex while waiting on the pending read, and there
>>>>>>>>> could be more than one pending read running at the time.  if so,
>>>>>>>>> chances are extremly high, that the data written to the buffer gets
>>>>>>>>> split like this:
>>>>>>>>>
>>>>>>>>>       reader 1		               reader 2
>>>>>>>>>
>>>>>>>>>       calls read(65536)                   calls read(65536)
>>>>>>>>>
>>>>>>>>>       calls NtReadFile(16384 bytes)
>>>>>>>>>                                           calls NtReadFile(16384 bytes)
>>>>>>>>>
>>>>>>>>> writer writes 65536 bytes
>>>>>>>>>
>>>>>>>>>       wakes up and gets 16384 bytes
>>>>>>>>>                                           wakes up and gets 16384 bytes
>>>>>>>>>       gets the mutex, calls
>>>>>>>>>       NtReadFile(32768) which
>>>>>>>>>       returns immediately with
>>>>>>>>>       32768 bytes added to the
>>>>>>>>>       caller's buffer.
>>>>>>>>>
>>>>>>>>> so the buffer returned to reader 1 is 49152 bytes, with 16384 bytes
>>>>>>>>> missing in the middle of it, *without* the reader knowing about that
>>>>>>>>> fact.  If reader 1 gets the first 16384 bytes, the 16384 bytes have
>>>>>>>>> been read in a single call, at least, so the byte order is not
>>>>>>>>> unknowingly broken on the application level.
>>>>>>>>>
>>>>>>>>> Does that make sense?
>>>>>>>>
>>>>>>>> Why can't we keep the mutex while waiting on the pending read?
>>>>>>>> If we can keep the mutex, the issue above mentioned does not
>>>>>>>> happen, right?
>>>>>>>>
>>>>>>>> What about the patch attached? This keeps the mutex while read()
>>>>>>>> but I do not see any defects so far.
>>>>>>
>>>>>> LGTM.
>>>>>>
>>>>>> If Corinna agrees, I have a couple of suggestions.
>>>>>>
>>>>>> 1. With this patch, we can no longer have more than one pending ReadFile.  So
>>>>>> there's no longer a need to count read handles, and the problem with select is
>>>>>> completely fixed as long as the number of bytes requested is less than the pipe
>>>>>> buffer size.
>>>>>>
>>>>>> 2. raw_read is now reading in chunks, like raw_write.  For readability of the
>>>>>> code, I think it would be better to make the two functions as similar as
>>>>>> possible.  For example, you could replace the do/while loop by a
>>>>>> while(total_len<orig_len) loop.  And you could even use similar names for the
>>>>>> variables, e.g., nbytes instead of total_len, or vice versa.
>>>>>
>>>>> Thanks for the suggestion. I have rebuilt the patch.
>>>>> Please see the patch attached.
>>>>
>>>> This patch seems to fail to adopt to current git head of topic/pipe
>>>> branch. I rebuilt the patch to fit current top/pipe.
>>>>
>>>> Please see the patch attached.
>>>
>>> Small typo.
>>>
>>> -	     pipe buffer size.  Every pending read lowers WriteQuotaAvailable
>>> +	     pipe buffer size. pending read lowers WriteQuotaAvailable
>>>
>>> should be:
>>>
>>> -	     pipe buffer size.  Every pending read lowers WriteQuotaAvailable
>>> +	     pipe buffer size. Pending read lowers WriteQuotaAvailable
>>
>> The patch looks great to me.  Two minor nits:
>>
>> 1. The patch doesn't apply cleanly.  Could you rebase it against the current
>> HEAD of topic/pipe?
>>
>> 2. There's no need for chunk to be less than the number of bytes requested if we
>> know there's data in the pipe.  So maybe something like this (untested) would be
>> better:
>>
>>             ULONG chunk;
>>             status = NtQueryInformationFile (get_handle (), &io,
>>                                              &fpli, sizeof (fpli),
>>                                              FilePipeLocalInformation);
>>             if (NT_SUCCESS (status))
>>               {
>>                 if (fpli.ReadDataAvailable > 0)
>>                   chunk = left;
>>                 else if (nbytes != 0)
>>                   break;
>>                 else
>>                   chunk = fpli.InboundQuota / 2;
>>               }
>>             else if (nbytes != 0)
>>               break;
>>             else
>>               chunk = max_atomic_write / 2;
>>
>>             if (chunk < left)
>>               len1 = chunk;
> 
> Could you please try attached new patch?

LGTM.  And it applies cleanly.  Maybe I did something wrong when I thought it 
didn't apply.

Thanks.

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-05 20:09                                                                                                               ` Takashi Yano
  2021-09-05 20:27                                                                                                                 ` Ken Brown
@ 2021-09-06  8:13                                                                                                                 ` Corinna Vinschen
  2021-09-06 11:16                                                                                                                   ` Takashi Yano
  1 sibling, 1 reply; 250+ messages in thread
From: Corinna Vinschen @ 2021-09-06  8:13 UTC (permalink / raw)
  To: cygwin-developers

On Sep  6 05:09, Takashi Yano wrote:
> On Sun, 5 Sep 2021 14:47:26 -0400
> Ken Brown wrote:
> > On 9/5/2021 9:50 AM, Takashi Yano wrote:
> > > On Sun, 5 Sep 2021 22:40:59 +0900
> > > Takashi Yano wrote:
> > >> On Sun, 5 Sep 2021 08:15:23 +0900
> > >> Takashi Yano wrote:
> > >>> On Sat, 4 Sep 2021 10:04:12 -0400
> > >>> Ken Brown wrote:
> > >>>> On 9/4/2021 8:37 AM, Takashi Yano wrote:
> > >>>>> On Sat, 4 Sep 2021 21:02:58 +0900
> > >>>>> Takashi Yano wrote:
> > >>>>>> On Fri, 3 Sep 2021 09:27:37 -0400
> > >>>>>> Ken Brown wrote:
> > >>>>>>> On 9/3/2021 8:22 AM, Takashi Yano wrote:
> > >>>>>>>> POSIX says:
> > >>>>>>>>        The value returned may be less than nbyte if the number of bytes left
> > >>>>>>>>        in the file is less than nbyte, if the read() request was interrupted
> > >>>>>>>>        by a signal, or if the file is a pipe or FIFO or special file and has
> > >>>>>>>>                                                                          ~~~
> > >>>>>>>>        fewer than nbyte bytes immediately available for reading.
> > >>>>>>>>        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > >>>>>>>>
> > >>>>>>>> https://pubs.opengroup.org/onlinepubs/009604599/functions/read.html
> > >>>>>>>>
> > >>>>>>>> If it is turned over, read() should read all data immediately available,
> > >>>>>>>> I think.
> > >>>>>>> [...]
> > >>>>>> Corinna Vinschen wrote:
> > >>>>>>> We can't keep the mutex while waiting on the pending read, and there
> > >>>>>>> could be more than one pending read running at the time.  if so,
> > >>>>>>> chances are extremly high, that the data written to the buffer gets
> > >>>>>>> split like this:
> > >>>>>>> [...]
> > >>>>> Takashi Yano wrote:
> > >>>>>> Why can't we keep the mutex while waiting on the pending read?
> > >>>>>> If we can keep the mutex, the issue above mentioned does not
> > >>>>>> happen, right?
> [...]
> @@ -239,43 +237,51 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
>        return;
>      }
>  
> -  do
> +  DWORD timeout = is_nonblocking () ? 0 : INFINITE;
> +  if (WAIT_TIMEOUT == WaitForSingleObject (read_mtx, timeout))

My code was originally not supposed to serialise the readers.  The
mutex block should be short lived and only create an atomic block
for the two calls NtQueryInformationFile and NtReadFile.

If you have multiple readers, all but one of them will hang in this
WFSO.  They will block here without a chance to kill or Ctrl-C them
and thread cancellation won't work.

To fix that you have to use cygwait and handle signals and thread
cancellation the same way as in the below code following the NtReadFile.

>  	  /* If the pipe is empty, don't request more bytes than half the
> -	     pipe buffer size.  Every pending read lowers WriteQuotaAvailable
> -	     on the write side and thus affects select's ability to return
> -	     more or less reliable info whether a write succeeds or not.
> -
> -	     Let the size of the request depend on the number of readers
> -	     at the time. */
> +	     pipe buffer size. Pending read lowers WriteQuotaAvailable on
> +	     the write side and thus affects select's ability to return
> +	     more or less reliable info whether a write succeeds or not. */
> +	  ULONG chunk = max_atomic_write / 2;
>  	  status = NtQueryInformationFile (get_handle (), &io,
>  					   &fpli, sizeof (fpli),
>  					   FilePipeLocalInformation);
> -	  if (NT_SUCCESS (status) && fpli.ReadDataAvailable == 0)

If the readers are serialized anyway, why fetch only half the remaining
buffer size?  In that case fetching fpli.InboundQuota - 1 is as good
as fetching just the half of it, isn't it?


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-06  8:13                                                                                                                 ` Corinna Vinschen
@ 2021-09-06 11:16                                                                                                                   ` Takashi Yano
  2021-09-06 12:49                                                                                                                     ` Corinna Vinschen
  0 siblings, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-06 11:16 UTC (permalink / raw)
  To: cygwin-developers

[-- Attachment #1: Type: text/plain, Size: 4027 bytes --]

Hi Corinna,

On Mon, 6 Sep 2021 10:13:10 +0200
Corinna Vinschen wrote:
> On Sep  6 05:09, Takashi Yano wrote:
> > On Sun, 5 Sep 2021 14:47:26 -0400
> > Ken Brown wrote:
> > > On 9/5/2021 9:50 AM, Takashi Yano wrote:
> > > > On Sun, 5 Sep 2021 22:40:59 +0900
> > > > Takashi Yano wrote:
> > > >> On Sun, 5 Sep 2021 08:15:23 +0900
> > > >> Takashi Yano wrote:
> > > >>> On Sat, 4 Sep 2021 10:04:12 -0400
> > > >>> Ken Brown wrote:
> > > >>>> On 9/4/2021 8:37 AM, Takashi Yano wrote:
> > > >>>>> On Sat, 4 Sep 2021 21:02:58 +0900
> > > >>>>> Takashi Yano wrote:
> > > >>>>>> On Fri, 3 Sep 2021 09:27:37 -0400
> > > >>>>>> Ken Brown wrote:
> > > >>>>>>> On 9/3/2021 8:22 AM, Takashi Yano wrote:
> > > >>>>>>>> POSIX says:
> > > >>>>>>>>        The value returned may be less than nbyte if the number of bytes left
> > > >>>>>>>>        in the file is less than nbyte, if the read() request was interrupted
> > > >>>>>>>>        by a signal, or if the file is a pipe or FIFO or special file and has
> > > >>>>>>>>                                                                          ~~~
> > > >>>>>>>>        fewer than nbyte bytes immediately available for reading.
> > > >>>>>>>>        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > > >>>>>>>>
> > > >>>>>>>> https://pubs.opengroup.org/onlinepubs/009604599/functions/read.html
> > > >>>>>>>>
> > > >>>>>>>> If it is turned over, read() should read all data immediately available,
> > > >>>>>>>> I think.
> > > >>>>>>> [...]
> > > >>>>>> Corinna Vinschen wrote:
> > > >>>>>>> We can't keep the mutex while waiting on the pending read, and there
> > > >>>>>>> could be more than one pending read running at the time.  if so,
> > > >>>>>>> chances are extremly high, that the data written to the buffer gets
> > > >>>>>>> split like this:
> > > >>>>>>> [...]
> > > >>>>> Takashi Yano wrote:
> > > >>>>>> Why can't we keep the mutex while waiting on the pending read?
> > > >>>>>> If we can keep the mutex, the issue above mentioned does not
> > > >>>>>> happen, right?
> > [...]
> > @@ -239,43 +237,51 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
> >        return;
> >      }
> >  
> > -  do
> > +  DWORD timeout = is_nonblocking () ? 0 : INFINITE;
> > +  if (WAIT_TIMEOUT == WaitForSingleObject (read_mtx, timeout))
> 
> My code was originally not supposed to serialise the readers.  The
> mutex block should be short lived and only create an atomic block
> for the two calls NtQueryInformationFile and NtReadFile.
> 
> If you have multiple readers, all but one of them will hang in this
> WFSO.  They will block here without a chance to kill or Ctrl-C them
> and thread cancellation won't work.
> 
> To fix that you have to use cygwait and handle signals and thread
> cancellation the same way as in the below code following the NtReadFile.

OK. Thanks for the advice. Then what about the patch attached?

> >  	  /* If the pipe is empty, don't request more bytes than half the
> > -	     pipe buffer size.  Every pending read lowers WriteQuotaAvailable
> > -	     on the write side and thus affects select's ability to return
> > -	     more or less reliable info whether a write succeeds or not.
> > -
> > -	     Let the size of the request depend on the number of readers
> > -	     at the time. */
> > +	     pipe buffer size. Pending read lowers WriteQuotaAvailable on
> > +	     the write side and thus affects select's ability to return
> > +	     more or less reliable info whether a write succeeds or not. */
> > +	  ULONG chunk = max_atomic_write / 2;
> >  	  status = NtQueryInformationFile (get_handle (), &io,
> >  					   &fpli, sizeof (fpli),
> >  					   FilePipeLocalInformation);
> > -	  if (NT_SUCCESS (status) && fpli.ReadDataAvailable == 0)
> 
> If the readers are serialized anyway, why fetch only half the remaining
> buffer size?  In that case fetching fpli.InboundQuota - 1 is as good
> as fetching just the half of it, isn't it?

It sounds reasonable. Adopted in the attached patch.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

[-- Attachment #2: 0001-Cygwin-pipe-Stop-counting-reader-and-read-all-availa.patch --]
[-- Type: application/octet-stream, Size: 5370 bytes --]

From 0bb2ea9e552f8cb788c097519174145cd368b45b Mon Sep 17 00:00:00 2001
From: Takashi Yano <takashi.yano@nifty.ne.jp>
Date: Mon, 6 Sep 2021 20:12:16 +0900
Subject: [PATCH] Cygwin: pipe: Stop counting reader and read all available
 data.

- By guarding read with read_mtx, no more than one ReadFile can
  be called simultaneously. So couting read handles is no longer
  necessary.
- Make raw_read code as similar as possible to raw_write code.
---
 winsup/cygwin/fhandler_pipe.cc | 100 ++++++++++++++++++++-------------
 1 file changed, 60 insertions(+), 40 deletions(-)

diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index 68974eb80..fe0bf0ca2 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -221,12 +221,10 @@ fhandler_pipe::get_proc_fd_name (char *buf)
 void __reg3
 fhandler_pipe::raw_read (void *ptr, size_t& len)
 {
-  NTSTATUS status;
+  size_t nbytes = 0;
+  NTSTATUS status = STATUS_SUCCESS;
   IO_STATUS_BLOCK io;
   HANDLE evt = NULL;
-  DWORD waitret = WAIT_OBJECT_0;
-  bool keep_looping = false;
-  size_t orig_len = len;
 
   if (!len)
     return;
@@ -239,43 +237,59 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
       return;
     }
 
-  do
+  DWORD timeout = is_nonblocking () ? 0 : INFINITE;
+  DWORD waitret = cygwait (read_mtx, timeout);
+  switch (waitret)
     {
-      len = orig_len;
-      keep_looping = false;
+    case WAIT_OBJECT_0:
+      break;
+    case WAIT_TIMEOUT:
+      set_errno (EAGAIN);
+      len = (size_t) -1;
+      return;
+    default:
+      set_errno (EINTR);
+      len = (size_t) -1;
+      return;
+    }
+  while (nbytes < len)
+    {
+      ULONG_PTR nbytes_now = 0;
+      size_t left = len - nbytes;
+      ULONG len1 = (ULONG) left;
+      waitret = WAIT_OBJECT_0;
+
       if (evt)
 	ResetEvent (evt);
       if (!is_nonblocking ())
 	{
 	  FILE_PIPE_LOCAL_INFORMATION fpli;
-	  ULONG reader_count;
-	  ULONG max_len = 64;
-
-	  WaitForSingleObject (read_mtx, INFINITE);
 
-	  /* If the pipe is empty, don't request more bytes than half the
-	     pipe buffer size.  Every pending read lowers WriteQuotaAvailable
-	     on the write side and thus affects select's ability to return
-	     more or less reliable info whether a write succeeds or not.
-
-	     Let the size of the request depend on the number of readers
-	     at the time. */
+	  /* If the pipe is empty, don't request more bytes than pipe
+	     buffer size - 1. Pending read lowers WriteQuotaAvailable on
+	     the write side and thus affects select's ability to return
+	     more or less reliable info whether a write succeeds or not. */
+	  ULONG chunk = max_atomic_write - 1;
 	  status = NtQueryInformationFile (get_handle (), &io,
 					   &fpli, sizeof (fpli),
 					   FilePipeLocalInformation);
-	  if (NT_SUCCESS (status) && fpli.ReadDataAvailable == 0)
+	  if (NT_SUCCESS (status))
 	    {
-	      reader_count = get_obj_handle_count (get_handle ());
-	      if (reader_count < 10)
-		max_len = fpli.InboundQuota / (2 * reader_count);
-	      if (len > max_len)
-		len = max_len;
+	      if (fpli.ReadDataAvailable > 0)
+		chunk = left;
+	      else if (nbytes != 0)
+		break;
+	      else
+		chunk = fpli.InboundQuota - 1;
 	    }
+	  else if (nbytes != 0)
+	    break;
+
+	  if (len1 > chunk)
+	    len1 = chunk;
 	}
       status = NtReadFile (get_handle (), evt, NULL, NULL, &io, ptr,
-			   len, NULL, NULL);
-      if (!is_nonblocking ())
-	ReleaseMutex (read_mtx);
+			   len1, NULL, NULL);
       if (evt && status == STATUS_PENDING)
 	{
 	  waitret = cygwait (evt);
@@ -292,13 +306,13 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
 	    set_errno (EBADF);
 	  else
 	    __seterrno ();
-	  len = (size_t) -1;
+	  nbytes = (size_t) -1;
 	}
       else if (NT_SUCCESS (status))
 	{
-	  len = io.Information;
-	  if (len == 0)
-	    keep_looping = true;
+	  nbytes_now = io.Information;
+	  ptr = ((char *) ptr) + nbytes_now;
+	  nbytes += nbytes_now;
 	}
       else
 	{
@@ -308,40 +322,46 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
 	    case STATUS_END_OF_FILE:
 	    case STATUS_PIPE_BROKEN:
 	      /* This is really EOF.  */
-	      len = 0;
 	      break;
 	    case STATUS_MORE_ENTRIES:
 	    case STATUS_BUFFER_OVERFLOW:
 	      /* `io.Information' is supposedly valid.  */
-	      len = io.Information;
-	      if (len == 0)
-		keep_looping = true;
+	      nbytes_now = io.Information;
+	      ptr = ((char *) ptr) + nbytes_now;
+	      nbytes += nbytes_now;
 	      break;
 	    case STATUS_PIPE_LISTENING:
 	    case STATUS_PIPE_EMPTY:
+	      if (nbytes != 0)
+		break;
 	      if (is_nonblocking ())
 		{
 		  set_errno (EAGAIN);
-		  len = (size_t) -1;
+		  nbytes = (size_t) -1;
 		  break;
 		}
 	      fallthrough;
 	    default:
 	      __seterrno_from_nt_status (status);
-	      len = (size_t) -1;
+	      nbytes = (size_t) -1;
 	      break;
 	    }
 	}
-    } while (keep_looping);
+
+      if (nbytes_now == 0)
+	break;
+    }
+  ReleaseMutex (read_mtx);
   if (evt)
     CloseHandle (evt);
-  if (status == STATUS_THREAD_SIGNALED)
+  if (status == STATUS_THREAD_SIGNALED && nbytes == 0)
     {
       set_errno (EINTR);
-      len = (size_t) -1;
+      nbytes = (size_t) -1;
     }
   else if (status == STATUS_THREAD_CANCELED)
     pthread::static_cancel_self ();
+  len = nbytes;
 }
 
 ssize_t __reg3
-- 
2.33.0


^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-06 11:16                                                                                                                   ` Takashi Yano
@ 2021-09-06 12:49                                                                                                                     ` Corinna Vinschen
  2021-09-06 13:16                                                                                                                       ` Takashi Yano
  2021-09-07 16:14                                                                                                                       ` Ken Brown
  0 siblings, 2 replies; 250+ messages in thread
From: Corinna Vinschen @ 2021-09-06 12:49 UTC (permalink / raw)
  To: cygwin-developers

On Sep  6 20:16, Takashi Yano wrote:
> Hi Corinna,
> 
> On Mon, 6 Sep 2021 10:13:10 +0200
> Corinna Vinschen wrote:
> > If you have multiple readers, all but one of them will hang in this
> > WFSO.  They will block here without a chance to kill or Ctrl-C them
> > and thread cancellation won't work.
> > 
> > To fix that you have to use cygwait and handle signals and thread
> > cancellation the same way as in the below code following the NtReadFile.
> 
> OK. Thanks for the advice. Then what about the patch attached?
> 
> > >  	  /* If the pipe is empty, don't request more bytes than half the
> > > -	     pipe buffer size.  Every pending read lowers WriteQuotaAvailable
> > > -	     on the write side and thus affects select's ability to return
> > > -	     more or less reliable info whether a write succeeds or not.
> > > -
> > > -	     Let the size of the request depend on the number of readers
> > > -	     at the time. */
> > > +	     pipe buffer size. Pending read lowers WriteQuotaAvailable on
> > > +	     the write side and thus affects select's ability to return
> > > +	     more or less reliable info whether a write succeeds or not. */
> > > +	  ULONG chunk = max_atomic_write / 2;
> > >  	  status = NtQueryInformationFile (get_handle (), &io,
> > >  					   &fpli, sizeof (fpli),
> > >  					   FilePipeLocalInformation);
> > > -	  if (NT_SUCCESS (status) && fpli.ReadDataAvailable == 0)
> > 
> > If the readers are serialized anyway, why fetch only half the remaining
> > buffer size?  In that case fetching fpli.InboundQuota - 1 is as good
> > as fetching just the half of it, isn't it?
> 
> It sounds reasonable. Adopted in the attached patch.

Patch looks ok.

I added one more patches:

- It occured to me that the code is lacking a CancelIo in case cygwait
  is waiting for NtReadFile/NtWriteFile.  Actually, calling cygwait
  without "mask" parameter will result in cygwait performing the thread
  cancellation by itself, but cancelling a thread does not cancel the
  async IO started by that thread.  So I fixed the cygwait calls in
  raw_read/raw_write to return to the caller and then call CancelIo
  before cancelling the thread.

I planned to push one more patch:

- Drop max_atomic_write, it's == DEFAULT_PIPEBUFSIZE anyway

But then some things were coming to mind, which we still have to discuss.

- I think setting chunk to DEFAULT_PIPEBUFSIZE - 1 in the read case and
  DEFAULT_PIPEBUFSIZE in the write case by default is dangerous.
  Assuming the pipe has been created by a non-Cygwin process, the values
  may be way too high.

  Suggestion: Actually set max_atomic_write to something useful.
  Set max_atomic_write to DEFAULT_PIPEBUFSIZE in fhandler_pipe::create.
  In case of stdio handles inherited from non-Cygwin processes, fetch
  the pipe buffer size via NtQueryInformationFile in
  dtable::init_std_file_from_handle().  Better, in a matching
  fhandler_pipe method called from init_std_file_from_handle().

- What about calling select for writing on pipes read by non-Cygwin
  processes?  In that case, we still can't rely on WriteQuotaAvailable,
  just as before.

  I have a vague idea that we might want to count readers in that case,
  but I have to think about it some more.


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-06 12:49                                                                                                                     ` Corinna Vinschen
@ 2021-09-06 13:16                                                                                                                       ` Takashi Yano
  2021-09-06 16:08                                                                                                                         ` Corinna Vinschen
  2021-09-07 16:14                                                                                                                       ` Ken Brown
  1 sibling, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-06 13:16 UTC (permalink / raw)
  To: cygwin-developers

On Mon, 6 Sep 2021 14:49:55 +0200
Corinna Vinschen wrote:
> On Sep  6 20:16, Takashi Yano wrote:
> > Hi Corinna,
> > 
> > On Mon, 6 Sep 2021 10:13:10 +0200
> > Corinna Vinschen wrote:
> > > If you have multiple readers, all but one of them will hang in this
> > > WFSO.  They will block here without a chance to kill or Ctrl-C them
> > > and thread cancellation won't work.
> > > 
> > > To fix that you have to use cygwait and handle signals and thread
> > > cancellation the same way as in the below code following the NtReadFile.
> > 
> > OK. Thanks for the advice. Then what about the patch attached?
> > 
> > > >  	  /* If the pipe is empty, don't request more bytes than half the
> > > > -	     pipe buffer size.  Every pending read lowers WriteQuotaAvailable
> > > > -	     on the write side and thus affects select's ability to return
> > > > -	     more or less reliable info whether a write succeeds or not.
> > > > -
> > > > -	     Let the size of the request depend on the number of readers
> > > > -	     at the time. */
> > > > +	     pipe buffer size. Pending read lowers WriteQuotaAvailable on
> > > > +	     the write side and thus affects select's ability to return
> > > > +	     more or less reliable info whether a write succeeds or not. */
> > > > +	  ULONG chunk = max_atomic_write / 2;
> > > >  	  status = NtQueryInformationFile (get_handle (), &io,
> > > >  					   &fpli, sizeof (fpli),
> > > >  					   FilePipeLocalInformation);
> > > > -	  if (NT_SUCCESS (status) && fpli.ReadDataAvailable == 0)
> > > 
> > > If the readers are serialized anyway, why fetch only half the remaining
> > > buffer size?  In that case fetching fpli.InboundQuota - 1 is as good
> > > as fetching just the half of it, isn't it?
> > 
> > It sounds reasonable. Adopted in the attached patch.
> 
> Patch looks ok.
> 
> I added one more patches:
> 
> - It occured to me that the code is lacking a CancelIo in case cygwait
>   is waiting for NtReadFile/NtWriteFile.  Actually, calling cygwait
>   without "mask" parameter will result in cygwait performing the thread
>   cancellation by itself, but cancelling a thread does not cancel the
>   async IO started by that thread.  So I fixed the cygwait calls in
>   raw_read/raw_write to return to the caller and then call CancelIo
>   before cancelling the thread.
> 
> I planned to push one more patch:
> 
> - Drop max_atomic_write, it's == DEFAULT_PIPEBUFSIZE anyway
> 
> But then some things were coming to mind, which we still have to discuss.
> 
> - I think setting chunk to DEFAULT_PIPEBUFSIZE - 1 in the read case and
>   DEFAULT_PIPEBUFSIZE in the write case by default is dangerous.
>   Assuming the pipe has been created by a non-Cygwin process, the values
>   may be way too high.
> 
>   Suggestion: Actually set max_atomic_write to something useful.
>   Set max_atomic_write to DEFAULT_PIPEBUFSIZE in fhandler_pipe::create.
>   In case of stdio handles inherited from non-Cygwin processes, fetch
>   the pipe buffer size via NtQueryInformationFile in
>   dtable::init_std_file_from_handle().  Better, in a matching
>   fhandler_pipe method called from init_std_file_from_handle().
> 
> - What about calling select for writing on pipes read by non-Cygwin
>   processes?  In that case, we still can't rely on WriteQuotaAvailable,
>   just as before.
> 
>   I have a vague idea that we might want to count readers in that case,
>   but I have to think about it some more.

Current git head seems to have some bug. With and without my patch,
sftp get for large file causes error:

[yano@Express5800-S70 ~]$ sftp 192.168.0.133
yano@192.168.0.133's password:
Connected to 192.168.0.133.
sftp> get test.dat
Fetching /home/yano/test.dat to test.dat
test.dat                                       13%   66MB  66.4MB/s   00:06 ETAReceived message too long 1728053256
Ensure the remote shell produces no output for non-interactive sessions.
[yano@Express5800-S70 ~]$ sftp 192.168.0.133
yano@192.168.0.133's password:
Connected to 192.168.0.133.
sftp> get test.dat
Fetching /home/yano/test.dat to test.dat
test.dat                                       22%  111MB 110.6MB/s   00:03 ETAdo_download: parse: incomplete message
[yano@Express5800-S70 ~]$



-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-06 13:16                                                                                                                       ` Takashi Yano
@ 2021-09-06 16:08                                                                                                                         ` Corinna Vinschen
  2021-09-06 23:39                                                                                                                           ` Takashi Yano
  0 siblings, 1 reply; 250+ messages in thread
From: Corinna Vinschen @ 2021-09-06 16:08 UTC (permalink / raw)
  To: cygwin-developers

On Sep  6 22:16, Takashi Yano wrote:
> Current git head seems to have some bug. With and without my patch,
> sftp get for large file causes error:
> 
> [yano@Express5800-S70 ~]$ sftp 192.168.0.133
> yano@192.168.0.133's password:
> Connected to 192.168.0.133.
> sftp> get test.dat
> Fetching /home/yano/test.dat to test.dat
> test.dat                                       13%   66MB  66.4MB/s   00:06 ETAReceived message too long 1728053256
> Ensure the remote shell produces no output for non-interactive sessions.
> [yano@Express5800-S70 ~]$ sftp 192.168.0.133
> yano@192.168.0.133's password:
> Connected to 192.168.0.133.
> sftp> get test.dat
> Fetching /home/yano/test.dat to test.dat
> test.dat                                       22%  111MB 110.6MB/s   00:03 ETAdo_download: parse: incomplete message
> [yano@Express5800-S70 ~]$

I bisected this down to commit 296bd3e78b52, but I'm at a loss in
terms of the cause of the problem, ATM.


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-06 16:08                                                                                                                         ` Corinna Vinschen
@ 2021-09-06 23:39                                                                                                                           ` Takashi Yano
  2021-09-07  9:14                                                                                                                             ` Corinna Vinschen
  0 siblings, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-06 23:39 UTC (permalink / raw)
  To: cygwin-developers

On Mon, 6 Sep 2021 18:08:54 +0200
Corinna Vinschen wrote:
> On Sep  6 22:16, Takashi Yano wrote:
> > Current git head seems to have some bug. With and without my patch,
> > sftp get for large file causes error:
> > 
> > [yano@Express5800-S70 ~]$ sftp 192.168.0.133
> > yano@192.168.0.133's password:
> > Connected to 192.168.0.133.
> > sftp> get test.dat
> > Fetching /home/yano/test.dat to test.dat
> > test.dat                                       13%   66MB  66.4MB/s   00:06 ETAReceived message too long 1728053256
> > Ensure the remote shell produces no output for non-interactive sessions.
> > [yano@Express5800-S70 ~]$ sftp 192.168.0.133
> > yano@192.168.0.133's password:
> > Connected to 192.168.0.133.
> > sftp> get test.dat
> > Fetching /home/yano/test.dat to test.dat
> > test.dat                                       22%  111MB 110.6MB/s   00:03 ETAdo_download: parse: incomplete message
> > [yano@Express5800-S70 ~]$
> 
> I bisected this down to commit 296bd3e78b52, but I'm at a loss in
> terms of the cause of the problem, ATM.

Thanks for bisecting this.

I am not sure this is the correct thing, however, found the following
patch solves the issue.

diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index 76ce895e2..83efb8296 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -292,7 +292,7 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
                           len1, NULL, NULL);
       if (evt && status == STATUS_PENDING)
        {
-         waitret = cygwait (evt, INFINITE, cw_cancel | cw_sig);
+         waitret = cygwait (evt, INFINITE, cw_cancel | cw_sig_restart);
          if (waitret == WAIT_OBJECT_0)
            status = io.Status;
        }
@@ -442,7 +442,7 @@ fhandler_pipe_fifo::raw_write (const void *ptr, size_t len)
        }
       if (evt && status == STATUS_PENDING)
        {
-         waitret = cygwait (evt, INFINITE, cw_cancel | cw_sig);
+         waitret = cygwait (evt, INFINITE, cw_cancel | cw_sig_restart);
          if (waitret == WAIT_OBJECT_0)
            status = io.Status;
        }

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
       [not found]           ` <20210827202440.47706fc2fc07c5e9a1bc0047@nifty.ne.jp>
       [not found]             ` <4f2cb5f3-ce9c-c617-f65f-841a5eca096e@cornell.edu>
@ 2021-09-07  3:26             ` Takashi Yano
  2021-09-07 10:50               ` Takashi Yano
  2021-09-09  3:41               ` Takashi Yano
  1 sibling, 2 replies; 250+ messages in thread
From: Takashi Yano @ 2021-09-07  3:26 UTC (permalink / raw)
  To: cygwin-developers

[-- Attachment #1: Type: text/plain, Size: 1549 bytes --]

On Fri, 27 Aug 2021 20:24:40 +0900
Takashi Yano wrote:
> Hi Ken,
> 
> Thanks much! I tested topic/pipe branch.
> 
> [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:.
> yano@linux-server's password:
> test.dat                                      100%  100MB  95.9MB/s   00:01
> [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat .
> yano@linux-server's password:
> test.dat                                      100%  100MB   8.0MB/s   00:12
> 
> yano@linux-server:~$ scp yano@cygwin-PC:test.dat .
> yano@cygwin-PC's password:
> test.dat                                      100%  100MB 109.7MB/s   00:00
> yano@linux-server:~$ scp test.dat yano@cygwin-PC:.
> yano@cygwin-PC's password:
> test.dat                                      100%  100MB  31.4MB/s   00:03
> 
> As shown above, outgoing transfer-rate has been improved upto near
> theoretical limit. However, incoming transfer-rate is not improved
> much.
> 
> I digged further and found the first patch attached solves the issue
> as follows.
> 
> [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat .
> yano@linux-server's password:
> test.dat                                      100%  100MB 112.8MB/s   00:00
> 
> yano@linux-server2:~$ scp test.dat yano@cygwin-PC:.
> yano@cygwin-PC's password:
> test.dat                                      100%  100MB 102.5MB/s   00:00

With this patch (2e36ae2e), I found a problem that mintty gets into
high load if several keys are typed quickly.

Therefore, I would like to propose a patch attached.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

[-- Attachment #2: 0001-Cygwin-select-Introduce-select_evt-event-for-pipe.patch --]
[-- Type: application/octet-stream, Size: 6936 bytes --]

From a455ae9a0ed871e5f1e9ab5cf89ffdcbe34a49db Mon Sep 17 00:00:00 2001
From: Takashi Yano <takashi.yano@nifty.ne.jp>
Date: Tue, 7 Sep 2021 09:02:55 +0900
Subject: [PATCH] Cygwin: select: Introduce select_evt event for pipe.

- This patch reverts "Cygwin: select: Improve select/poll response",
  and introduces select_evt event which notifies pipe status change.
---
 winsup/cygwin/fhandler.cc      |  1 +
 winsup/cygwin/fhandler.h       |  3 +++
 winsup/cygwin/fhandler_pipe.cc | 28 +++++++++++++++++++++++
 winsup/cygwin/select.cc        | 41 +++++++++-------------------------
 4 files changed, 42 insertions(+), 31 deletions(-)

diff --git a/winsup/cygwin/fhandler.cc b/winsup/cygwin/fhandler.cc
index f0c1b68f1..265e8ee59 100644
--- a/winsup/cygwin/fhandler.cc
+++ b/winsup/cygwin/fhandler.cc
@@ -1464,6 +1464,7 @@ fhandler_base::fhandler_base () :
   _refcnt (0),
   openflags (0),
   unique_id (0),
+  select_evt (NULL),
   archetype (NULL),
   usecount (0)
 {
diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
index bb7eb09ce..9022aa09c 100644
--- a/winsup/cygwin/fhandler.h
+++ b/winsup/cygwin/fhandler.h
@@ -217,6 +217,7 @@ class fhandler_base
   void set_ino (ino_t i) { ino = i; }
 
   HANDLE read_state;
+  HANDLE select_evt;
 
  public:
   LONG inc_refcnt () {return InterlockedIncrement (&_refcnt);}
@@ -520,6 +521,8 @@ public:
     fh->copy_from (this);
     return fh;
   }
+
+  HANDLE get_select_evt () { return select_evt; }
 };
 
 struct wsa_event
diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index 83efb8296..7cce4564c 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -367,6 +367,9 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
       CancelIo (get_handle ());
       pthread::static_cancel_self ();
     }
+  if (select_evt && nbytes)
+    for (ULONG i = 0; i < get_obj_handle_count (select_evt); i++)
+      SetEvent (select_evt);
   len = nbytes;
 }
 
@@ -489,6 +492,9 @@ fhandler_pipe_fifo::raw_write (const void *ptr, size_t len)
       CancelIo (get_handle ());
       pthread::static_cancel_self ();
     }
+  if (select_evt && nbytes)
+    for (ULONG i = 0; i < get_obj_handle_count (select_evt); i++)
+      SetEvent (select_evt);
   return nbytes ?: -1;
 }
 
@@ -497,6 +503,8 @@ fhandler_pipe::fixup_after_fork (HANDLE parent)
 {
   if (read_mtx)
     fork_fixup (parent, read_mtx, "read_mtx");
+  if (select_evt)
+    fork_fixup (parent, select_evt, "select_evt");
   fhandler_base::fixup_after_fork (parent);
 }
 
@@ -518,6 +526,15 @@ fhandler_pipe::dup (fhandler_base *child, int flags)
       ftp->close ();
       res = -1;
     }
+  else if (select_evt &&
+	   !DuplicateHandle (GetCurrentProcess (), select_evt,
+			    GetCurrentProcess (), &ftp->select_evt,
+			    0, !(flags & O_CLOEXEC), DUPLICATE_SAME_ACCESS))
+    {
+      __seterrno ();
+      ftp->close ();
+      res = -1;
+    }
 
   debug_printf ("res %d", res);
   return res;
@@ -528,6 +545,12 @@ fhandler_pipe::close ()
 {
   if (read_mtx)
     CloseHandle (read_mtx);
+  if (select_evt)
+    {
+      for (ULONG i = 0; i < get_obj_handle_count (select_evt); i++)
+	SetEvent (select_evt);
+      CloseHandle (select_evt);
+    }
   return fhandler_base::close ();
 }
 
@@ -747,6 +770,11 @@ fhandler_pipe::create (fhandler_pipe *fhs[2], unsigned psize, int mode)
 	  fhs[0]->set_read_mutex (mtx);
 	  res = 0;
 	}
+      fhs[0]->select_evt = CreateEvent (&sa, FALSE, FALSE, NULL);
+      if (fhs[0]->select_evt)
+	DuplicateHandle (GetCurrentProcess (), fhs[0]->select_evt,
+			 GetCurrentProcess (), &fhs[1]->select_evt,
+			 0, 1, DUPLICATE_SAME_ACCESS);
     }
 
   debug_printf ("%R = pipe([%p, %p], %d, %y)", res, fhs[0], fhs[1], psize, mode);
diff --git a/winsup/cygwin/select.cc b/winsup/cygwin/select.cc
index ac2fd227e..19efe9e95 100644
--- a/winsup/cygwin/select.cc
+++ b/winsup/cygwin/select.cc
@@ -734,7 +734,6 @@ thread_pipe (void *arg)
   select_pipe_info *pi = (select_pipe_info *) arg;
   DWORD sleep_time = 0;
   bool looping = true;
-  DWORD t0 = GetTickCount ();
 
   while (looping)
     {
@@ -754,12 +753,7 @@ thread_pipe (void *arg)
 	break;
       cygwait (pi->bye, sleep_time >> 3);
       if (sleep_time < 80)
-	{
-	  DWORD t1 = GetTickCount ();
-	  if (t0 != t1)
-	    ++sleep_time;
-	  t0 = t1;
-	}
+	++sleep_time;
       if (pi->stop_thread)
 	break;
     }
@@ -776,7 +770,9 @@ start_thread_pipe (select_record *me, select_stuff *stuff)
     {
       pi->start = &stuff->start;
       pi->stop_thread = false;
-      pi->bye = CreateEvent (&sec_none_nih, TRUE, FALSE, NULL);
+      pi->bye = me->fh->get_select_evt ();
+      if (pi->bye == NULL)
+	pi->bye = CreateEvent (&sec_none_nih, TRUE, FALSE, NULL);
       pi->thread = new cygthread (thread_pipe, pi, "pipesel");
       me->h = *pi->thread;
       if (!me->h)
@@ -786,7 +782,7 @@ start_thread_pipe (select_record *me, select_stuff *stuff)
 }
 
 static void
-pipe_cleanup (select_record *, select_stuff *stuff)
+pipe_cleanup (select_record *me, select_stuff *stuff)
 {
   select_pipe_info *pi = (select_pipe_info *) stuff->device_specific_pipe;
   if (!pi)
@@ -796,7 +792,8 @@ pipe_cleanup (select_record *, select_stuff *stuff)
       pi->stop_thread = true;
       SetEvent (pi->bye);
       pi->thread->detach ();
-      CloseHandle (pi->bye);
+      if (me->fh->get_select_evt () == NULL)
+	CloseHandle (pi->bye);
     }
   delete pi;
   stuff->device_specific_pipe = NULL;
@@ -935,7 +932,6 @@ thread_fifo (void *arg)
   select_fifo_info *pi = (select_fifo_info *) arg;
   DWORD sleep_time = 0;
   bool looping = true;
-  DWORD t0 = GetTickCount ();
 
   while (looping)
     {
@@ -955,12 +951,7 @@ thread_fifo (void *arg)
 	break;
       cygwait (pi->bye, sleep_time >> 3);
       if (sleep_time < 80)
-	{
-	  DWORD t1 = GetTickCount ();
-	  if (t0 != t1)
-	    ++sleep_time;
-	  t0 = t1;
-	}
+	++sleep_time;
       if (pi->stop_thread)
 	break;
     }
@@ -1136,7 +1127,6 @@ thread_console (void *arg)
   select_console_info *ci = (select_console_info *) arg;
   DWORD sleep_time = 0;
   bool looping = true;
-  DWORD t0 = GetTickCount ();
 
   while (looping)
     {
@@ -1156,12 +1146,7 @@ thread_console (void *arg)
 	break;
       cygwait (ci->bye, sleep_time >> 3);
       if (sleep_time < 80)
-	{
-	  DWORD t1 = GetTickCount ();
-	  if (t0 != t1)
-	    ++sleep_time;
-	  t0 = t1;
-	}
+	++sleep_time;
       if (ci->stop_thread)
 	break;
     }
@@ -1381,7 +1366,6 @@ thread_pty_slave (void *arg)
   select_pipe_info *pi = (select_pipe_info *) arg;
   DWORD sleep_time = 0;
   bool looping = true;
-  DWORD t0 = GetTickCount ();
 
   while (looping)
     {
@@ -1401,12 +1385,7 @@ thread_pty_slave (void *arg)
 	break;
       cygwait (pi->bye, sleep_time >> 3);
       if (sleep_time < 80)
-	{
-	  DWORD t1 = GetTickCount ();
-	  if (t0 != t1)
-	    ++sleep_time;
-	  t0 = t1;
-	}
+	++sleep_time;
       if (pi->stop_thread)
 	break;
     }
-- 
2.33.0


^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-06 23:39                                                                                                                           ` Takashi Yano
@ 2021-09-07  9:14                                                                                                                             ` Corinna Vinschen
  2021-09-07 11:03                                                                                                                               ` Takashi Yano
  0 siblings, 1 reply; 250+ messages in thread
From: Corinna Vinschen @ 2021-09-07  9:14 UTC (permalink / raw)
  To: cygwin-developers

On Sep  7 08:39, Takashi Yano wrote:
> On Mon, 6 Sep 2021 18:08:54 +0200
> Corinna Vinschen wrote:
> > On Sep  6 22:16, Takashi Yano wrote:
> > > Current git head seems to have some bug. With and without my patch,
> > > sftp get for large file causes error:
> > > 
> > > [yano@Express5800-S70 ~]$ sftp 192.168.0.133
> > > yano@192.168.0.133's password:
> > > Connected to 192.168.0.133.
> > > sftp> get test.dat
> > > Fetching /home/yano/test.dat to test.dat
> > > test.dat                                       13%   66MB  66.4MB/s   00:06 ETAReceived message too long 1728053256
> > > Ensure the remote shell produces no output for non-interactive sessions.
> > > [yano@Express5800-S70 ~]$ sftp 192.168.0.133
> > > yano@192.168.0.133's password:
> > > Connected to 192.168.0.133.
> > > sftp> get test.dat
> > > Fetching /home/yano/test.dat to test.dat
> > > test.dat                                       22%  111MB 110.6MB/s   00:03 ETAdo_download: parse: incomplete message
> > > [yano@Express5800-S70 ~]$
> > 
> > I bisected this down to commit 296bd3e78b52, but I'm at a loss in
> > terms of the cause of the problem, ATM.
> 
> Thanks for bisecting this.
> 
> I am not sure this is the correct thing, however, found the following
> patch solves the issue.

Thanks for the patch!  It's not correct as such, because it enables
SA_RESTART behaviour unconditionally, but it gave me the right hint.

The underlying problem is that in case of a signal, the CancelIo call
was missing.  The signal was processed, but the IO was still ongoing
and so data was read or written without the application's knowledge.

Actually we can always call CancelIo.  It doesn't break the information
in the IO_STATUS_BLOCK if the IO was already finished.  It just sets
io.Status to STATUS_CANCELLED and io.Information to the number of bytes
processed if it really canceled the ongoing IO.

I pushed a matching patch.


Thanks again!
Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-07  3:26             ` Takashi Yano
@ 2021-09-07 10:50               ` Takashi Yano
  2021-09-08  0:07                 ` Takashi Yano
  2021-09-09  3:41               ` Takashi Yano
  1 sibling, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-07 10:50 UTC (permalink / raw)
  To: cygwin-developers

[-- Attachment #1: Type: text/plain, Size: 1753 bytes --]

On Tue, 7 Sep 2021 12:26:31 +0900
Takashi Yano wrote:
> On Fri, 27 Aug 2021 20:24:40 +0900
> Takashi Yano wrote:
> > Hi Ken,
> > 
> > Thanks much! I tested topic/pipe branch.
> > 
> > [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:.
> > yano@linux-server's password:
> > test.dat                                      100%  100MB  95.9MB/s   00:01
> > [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat .
> > yano@linux-server's password:
> > test.dat                                      100%  100MB   8.0MB/s   00:12
> > 
> > yano@linux-server:~$ scp yano@cygwin-PC:test.dat .
> > yano@cygwin-PC's password:
> > test.dat                                      100%  100MB 109.7MB/s   00:00
> > yano@linux-server:~$ scp test.dat yano@cygwin-PC:.
> > yano@cygwin-PC's password:
> > test.dat                                      100%  100MB  31.4MB/s   00:03
> > 
> > As shown above, outgoing transfer-rate has been improved upto near
> > theoretical limit. However, incoming transfer-rate is not improved
> > much.
> > 
> > I digged further and found the first patch attached solves the issue
> > as follows.
> > 
> > [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat .
> > yano@linux-server's password:
> > test.dat                                      100%  100MB 112.8MB/s   00:00
> > 
> > yano@linux-server2:~$ scp test.dat yano@cygwin-PC:.
> > yano@cygwin-PC's password:
> > test.dat                                      100%  100MB 102.5MB/s   00:00
> 
> With this patch (2e36ae2e), I found a problem that mintty gets into
> high load if several keys are typed quickly.
> 
> Therefore, I would like to propose a patch attached.

I revised this patch to fit the current git head of topic/pipe branch.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

[-- Attachment #2: 0001-Cygwin-select-Introduce-select_evt-event-for-pipe.patch --]
[-- Type: application/octet-stream, Size: 6975 bytes --]

From 0a6901492631e5abb0dedf8fb70159fd7205c9e8 Mon Sep 17 00:00:00 2001
From: Takashi Yano <takashi.yano@nifty.ne.jp>
Date: Tue, 7 Sep 2021 19:38:58 +0900
Subject: [PATCH] Cygwin: select: Introduce select_evt event for pipe.

- This patch reverts "Cygwin: select: Improve select/poll response",
  and introduces select_evt event which notifies pipe status change.
---
 winsup/cygwin/fhandler.cc      |  1 +
 winsup/cygwin/fhandler.h       |  3 +++
 winsup/cygwin/fhandler_pipe.cc | 28 +++++++++++++++++++++++
 winsup/cygwin/select.cc        | 41 +++++++++-------------------------
 4 files changed, 42 insertions(+), 31 deletions(-)

diff --git a/winsup/cygwin/fhandler.cc b/winsup/cygwin/fhandler.cc
index f0c1b68f1..265e8ee59 100644
--- a/winsup/cygwin/fhandler.cc
+++ b/winsup/cygwin/fhandler.cc
@@ -1464,6 +1464,7 @@ fhandler_base::fhandler_base () :
   _refcnt (0),
   openflags (0),
   unique_id (0),
+  select_evt (NULL),
   archetype (NULL),
   usecount (0)
 {
diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
index bb7eb09ce..9022aa09c 100644
--- a/winsup/cygwin/fhandler.h
+++ b/winsup/cygwin/fhandler.h
@@ -217,6 +217,7 @@ class fhandler_base
   void set_ino (ino_t i) { ino = i; }
 
   HANDLE read_state;
+  HANDLE select_evt;
 
  public:
   LONG inc_refcnt () {return InterlockedIncrement (&_refcnt);}
@@ -520,6 +521,8 @@ public:
     fh->copy_from (this);
     return fh;
   }
+
+  HANDLE get_select_evt () { return select_evt; }
 };
 
 struct wsa_event
diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index 4bd807a09..d995acbeb 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -376,6 +376,9 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
     }
   else if (status == STATUS_THREAD_CANCELED)
     pthread::static_cancel_self ();
+  if (select_evt && nbytes)
+    for (ULONG i = 0; i < get_obj_handle_count (select_evt); i++)
+      SetEvent (select_evt);
   len = nbytes;
 }
 
@@ -507,6 +510,9 @@ fhandler_pipe_fifo::raw_write (const void *ptr, size_t len)
     set_errno (EINTR);
   else if (status == STATUS_THREAD_CANCELED)
     pthread::static_cancel_self ();
+  if (select_evt && nbytes)
+    for (ULONG i = 0; i < get_obj_handle_count (select_evt); i++)
+      SetEvent (select_evt);
   return nbytes ?: -1;
 }
 
@@ -515,6 +521,8 @@ fhandler_pipe::fixup_after_fork (HANDLE parent)
 {
   if (read_mtx)
     fork_fixup (parent, read_mtx, "read_mtx");
+  if (select_evt)
+    fork_fixup (parent, select_evt, "select_evt");
   fhandler_base::fixup_after_fork (parent);
 }
 
@@ -536,6 +544,15 @@ fhandler_pipe::dup (fhandler_base *child, int flags)
       ftp->close ();
       res = -1;
     }
+  else if (select_evt &&
+	   !DuplicateHandle (GetCurrentProcess (), select_evt,
+			    GetCurrentProcess (), &ftp->select_evt,
+			    0, !(flags & O_CLOEXEC), DUPLICATE_SAME_ACCESS))
+    {
+      __seterrno ();
+      ftp->close ();
+      res = -1;
+    }
 
   debug_printf ("res %d", res);
   return res;
@@ -546,6 +563,12 @@ fhandler_pipe::close ()
 {
   if (read_mtx)
     CloseHandle (read_mtx);
+  if (select_evt)
+    {
+      for (ULONG i = 0; i < get_obj_handle_count (select_evt); i++)
+	SetEvent (select_evt);
+      CloseHandle (select_evt);
+    }
   return fhandler_base::close ();
 }
 
@@ -765,6 +788,11 @@ fhandler_pipe::create (fhandler_pipe *fhs[2], unsigned psize, int mode)
 	  fhs[0]->set_read_mutex (mtx);
 	  res = 0;
 	}
+      fhs[0]->select_evt = CreateEvent (&sa, FALSE, FALSE, NULL);
+      if (fhs[0]->select_evt)
+	DuplicateHandle (GetCurrentProcess (), fhs[0]->select_evt,
+			 GetCurrentProcess (), &fhs[1]->select_evt,
+			 0, 1, DUPLICATE_SAME_ACCESS);
     }
 
   debug_printf ("%R = pipe([%p, %p], %d, %y)", res, fhs[0], fhs[1], psize, mode);
diff --git a/winsup/cygwin/select.cc b/winsup/cygwin/select.cc
index ac2fd227e..19efe9e95 100644
--- a/winsup/cygwin/select.cc
+++ b/winsup/cygwin/select.cc
@@ -734,7 +734,6 @@ thread_pipe (void *arg)
   select_pipe_info *pi = (select_pipe_info *) arg;
   DWORD sleep_time = 0;
   bool looping = true;
-  DWORD t0 = GetTickCount ();
 
   while (looping)
     {
@@ -754,12 +753,7 @@ thread_pipe (void *arg)
 	break;
       cygwait (pi->bye, sleep_time >> 3);
       if (sleep_time < 80)
-	{
-	  DWORD t1 = GetTickCount ();
-	  if (t0 != t1)
-	    ++sleep_time;
-	  t0 = t1;
-	}
+	++sleep_time;
       if (pi->stop_thread)
 	break;
     }
@@ -776,7 +770,9 @@ start_thread_pipe (select_record *me, select_stuff *stuff)
     {
       pi->start = &stuff->start;
       pi->stop_thread = false;
-      pi->bye = CreateEvent (&sec_none_nih, TRUE, FALSE, NULL);
+      pi->bye = me->fh->get_select_evt ();
+      if (pi->bye == NULL)
+	pi->bye = CreateEvent (&sec_none_nih, TRUE, FALSE, NULL);
       pi->thread = new cygthread (thread_pipe, pi, "pipesel");
       me->h = *pi->thread;
       if (!me->h)
@@ -786,7 +782,7 @@ start_thread_pipe (select_record *me, select_stuff *stuff)
 }
 
 static void
-pipe_cleanup (select_record *, select_stuff *stuff)
+pipe_cleanup (select_record *me, select_stuff *stuff)
 {
   select_pipe_info *pi = (select_pipe_info *) stuff->device_specific_pipe;
   if (!pi)
@@ -796,7 +792,8 @@ pipe_cleanup (select_record *, select_stuff *stuff)
       pi->stop_thread = true;
       SetEvent (pi->bye);
       pi->thread->detach ();
-      CloseHandle (pi->bye);
+      if (me->fh->get_select_evt () == NULL)
+	CloseHandle (pi->bye);
     }
   delete pi;
   stuff->device_specific_pipe = NULL;
@@ -935,7 +932,6 @@ thread_fifo (void *arg)
   select_fifo_info *pi = (select_fifo_info *) arg;
   DWORD sleep_time = 0;
   bool looping = true;
-  DWORD t0 = GetTickCount ();
 
   while (looping)
     {
@@ -955,12 +951,7 @@ thread_fifo (void *arg)
 	break;
       cygwait (pi->bye, sleep_time >> 3);
       if (sleep_time < 80)
-	{
-	  DWORD t1 = GetTickCount ();
-	  if (t0 != t1)
-	    ++sleep_time;
-	  t0 = t1;
-	}
+	++sleep_time;
       if (pi->stop_thread)
 	break;
     }
@@ -1136,7 +1127,6 @@ thread_console (void *arg)
   select_console_info *ci = (select_console_info *) arg;
   DWORD sleep_time = 0;
   bool looping = true;
-  DWORD t0 = GetTickCount ();
 
   while (looping)
     {
@@ -1156,12 +1146,7 @@ thread_console (void *arg)
 	break;
       cygwait (ci->bye, sleep_time >> 3);
       if (sleep_time < 80)
-	{
-	  DWORD t1 = GetTickCount ();
-	  if (t0 != t1)
-	    ++sleep_time;
-	  t0 = t1;
-	}
+	++sleep_time;
       if (ci->stop_thread)
 	break;
     }
@@ -1381,7 +1366,6 @@ thread_pty_slave (void *arg)
   select_pipe_info *pi = (select_pipe_info *) arg;
   DWORD sleep_time = 0;
   bool looping = true;
-  DWORD t0 = GetTickCount ();
 
   while (looping)
     {
@@ -1401,12 +1385,7 @@ thread_pty_slave (void *arg)
 	break;
       cygwait (pi->bye, sleep_time >> 3);
       if (sleep_time < 80)
-	{
-	  DWORD t1 = GetTickCount ();
-	  if (t0 != t1)
-	    ++sleep_time;
-	  t0 = t1;
-	}
+	++sleep_time;
       if (pi->stop_thread)
 	break;
     }
-- 
2.33.0


^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-07  9:14                                                                                                                             ` Corinna Vinschen
@ 2021-09-07 11:03                                                                                                                               ` Takashi Yano
  0 siblings, 0 replies; 250+ messages in thread
From: Takashi Yano @ 2021-09-07 11:03 UTC (permalink / raw)
  To: cygwin-developers

On Tue, 7 Sep 2021 11:14:28 +0200
Corinna Vinschen wrote:
> On Sep  7 08:39, Takashi Yano wrote:
> > On Mon, 6 Sep 2021 18:08:54 +0200
> > Corinna Vinschen wrote:
> > > On Sep  6 22:16, Takashi Yano wrote:
> > > > Current git head seems to have some bug. With and without my patch,
> > > > sftp get for large file causes error:
> > > > 
> > > > [yano@Express5800-S70 ~]$ sftp 192.168.0.133
> > > > yano@192.168.0.133's password:
> > > > Connected to 192.168.0.133.
> > > > sftp> get test.dat
> > > > Fetching /home/yano/test.dat to test.dat
> > > > test.dat                                       13%   66MB  66.4MB/s   00:06 ETAReceived message too long 1728053256
> > > > Ensure the remote shell produces no output for non-interactive sessions.
> > > > [yano@Express5800-S70 ~]$ sftp 192.168.0.133
> > > > yano@192.168.0.133's password:
> > > > Connected to 192.168.0.133.
> > > > sftp> get test.dat
> > > > Fetching /home/yano/test.dat to test.dat
> > > > test.dat                                       22%  111MB 110.6MB/s   00:03 ETAdo_download: parse: incomplete message
> > > > [yano@Express5800-S70 ~]$
> > > 
> > > I bisected this down to commit 296bd3e78b52, but I'm at a loss in
> > > terms of the cause of the problem, ATM.
> > 
> > Thanks for bisecting this.
> > 
> > I am not sure this is the correct thing, however, found the following
> > patch solves the issue.
> 
> Thanks for the patch!  It's not correct as such, because it enables
> SA_RESTART behaviour unconditionally, but it gave me the right hint.
> 
> The underlying problem is that in case of a signal, the CancelIo call
> was missing.  The signal was processed, but the IO was still ongoing
> and so data was read or written without the application's knowledge.
> 
> Actually we can always call CancelIo.  It doesn't break the information
> in the IO_STATUS_BLOCK if the IO was already finished.  It just sets
> io.Status to STATUS_CANCELLED and io.Information to the number of bytes
> processed if it really canceled the ongoing IO.
> 
> I pushed a matching patch.

I have confirmed that your patch solves the issue. Thanks!


-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-06 12:49                                                                                                                     ` Corinna Vinschen
  2021-09-06 13:16                                                                                                                       ` Takashi Yano
@ 2021-09-07 16:14                                                                                                                       ` Ken Brown
  2021-09-07 18:26                                                                                                                         ` Corinna Vinschen
  1 sibling, 1 reply; 250+ messages in thread
From: Ken Brown @ 2021-09-07 16:14 UTC (permalink / raw)
  To: cygwin-developers

[-- Attachment #1: Type: text/plain, Size: 1353 bytes --]

On 9/6/2021 8:49 AM, Corinna Vinschen wrote:
> - I think setting chunk to DEFAULT_PIPEBUFSIZE - 1 in the read case and
>    DEFAULT_PIPEBUFSIZE in the write case by default is dangerous.
>    Assuming the pipe has been created by a non-Cygwin process, the values
>    may be way too high.
> 
>    Suggestion: Actually set max_atomic_write to something useful.
>    Set max_atomic_write to DEFAULT_PIPEBUFSIZE in fhandler_pipe::create.
>    In case of stdio handles inherited from non-Cygwin processes, fetch
>    the pipe buffer size via NtQueryInformationFile in
>    dtable::init_std_file_from_handle().  Better, in a matching
>    fhandler_pipe method called from init_std_file_from_handle().

How about something like the attached (untested)?

> - What about calling select for writing on pipes read by non-Cygwin
>    processes?  In that case, we still can't rely on WriteQuotaAvailable,
>    just as before.
> 
>    I have a vague idea that we might want to count readers in that case,
>    but I have to think about it some more.

Even if we count readers, we have no way of knowing whether a pending read has 
reduced WriteQuotaAvailable to 0.  Maybe this is a case where we should impose 
some artificial timeout, after which we report write ready.  Falsely reporting 
write ready in this corner case seems better than risking a deadlock.

Ken

[-- Attachment #2: pipe_buf.diff --]
[-- Type: text/plain, Size: 3225 bytes --]

diff --git a/winsup/cygwin/dtable.cc b/winsup/cygwin/dtable.cc
index 8085e656e..a638a5995 100644
--- a/winsup/cygwin/dtable.cc
+++ b/winsup/cygwin/dtable.cc
@@ -406,6 +406,11 @@ dtable::init_std_file_from_handle (int fd, HANDLE handle)
 	}
       if (!fh->init (handle, access, bin))
 	api_fatal ("couldn't initialize fd %d for %s", fd, fh->get_name ());
+      if (fh->ispipe ())
+	{
+	  fhandler_pipe *fhp = (fhandler_pipe *) fh;
+	  fhp->set_pipe_buf_size ();
+	}
 
       fh->open_setup (openflags);
       fh->usecount = 0;
diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
index bb7eb09ce..7aed089eb 100644
--- a/winsup/cygwin/fhandler.h
+++ b/winsup/cygwin/fhandler.h
@@ -1172,7 +1172,7 @@ class fhandler_socket_unix : public fhandler_socket
 class fhandler_pipe_fifo: public fhandler_base
 {
  protected:
-  size_t max_atomic_write;
+  size_t pipe_buf_size;
 
  public:
   fhandler_pipe_fifo ();
@@ -1192,6 +1192,7 @@ public:
 
   bool ispipe() const { return true; }
   void set_read_mutex (HANDLE mtx) { read_mtx = mtx; }
+  void set_pipe_buf_size ();
 
   void set_popen_pid (pid_t pid) {popen_pid = pid;}
   pid_t get_popen_pid () const {return popen_pid;}
diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index 4bd807a09..c75476040 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -29,7 +29,7 @@ STATUS_PIPE_EMPTY simply means there's no data to be read. */
 		   || _s == STATUS_PIPE_EMPTY; })
 
 fhandler_pipe_fifo::fhandler_pipe_fifo ()
-  : fhandler_base (), max_atomic_write (DEFAULT_PIPEBUFSIZE)
+  : fhandler_base (), pipe_buf_size (DEFAULT_PIPEBUFSIZE)
 {
 }
 
@@ -269,7 +269,7 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
 	     buffer size - 1. Pending read lowers WriteQuotaAvailable on
 	     the write side and thus affects select's ability to return
 	     more or less reliable info whether a write succeeds or not. */
-	  ULONG chunk = max_atomic_write - 1;
+	  ULONG chunk = pipe_buf_size - 1;
 	  status = NtQueryInformationFile (get_handle (), &io,
 					   &fpli, sizeof (fpli),
 					   FilePipeLocalInformation);
@@ -391,12 +391,12 @@ fhandler_pipe_fifo::raw_write (const void *ptr, size_t len)
   if (!len)
     return 0;
 
-  if (len <= max_atomic_write)
+  if (len <= pipe_buf_size)
     chunk = len;
   else if (is_nonblocking ())
-    chunk = len = max_atomic_write;
+    chunk = len = pipe_buf_size;
   else
-    chunk = max_atomic_write;
+    chunk = pipe_buf_size;
 
   /* Create a wait event if the pipe or fifo is in blocking mode. */
   if (!is_nonblocking () && !(evt = CreateEvent (NULL, false, false, NULL)))
@@ -894,6 +894,21 @@ nt_create (LPSECURITY_ATTRIBUTES sa_ptr, HANDLE &r, HANDLE &w,
   return 0;
 }
 
+/* Called by dtable::init_std_file_from_handle for stdio handles
+   inherited from non-Cygwin processes. */
+void
+fhandler_pipe::set_pipe_buf_size ()
+{
+  NTSTATUS status;
+  IO_STATUS_BLOCK io;
+  FILE_PIPE_LOCAL_INFORMATION fpli;
+
+  status = NtQueryInformationFile (get_handle (), &io, &fpli, sizeof fpli,
+				   FilePipeLocalInformation);
+  if (NT_SUCCESS (status))
+    pipe_buf_size = fpli.InboundQuota;
+}
+
 int
 fhandler_pipe::ioctl (unsigned int cmd, void *p)
 {

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-07 16:14                                                                                                                       ` Ken Brown
@ 2021-09-07 18:26                                                                                                                         ` Corinna Vinschen
  0 siblings, 0 replies; 250+ messages in thread
From: Corinna Vinschen @ 2021-09-07 18:26 UTC (permalink / raw)
  To: cygwin-developers

On Sep  7 12:14, Ken Brown wrote:
> On 9/6/2021 8:49 AM, Corinna Vinschen wrote:
> > - I think setting chunk to DEFAULT_PIPEBUFSIZE - 1 in the read case and
> >    DEFAULT_PIPEBUFSIZE in the write case by default is dangerous.
> >    Assuming the pipe has been created by a non-Cygwin process, the values
> >    may be way too high.
> > 
> >    Suggestion: Actually set max_atomic_write to something useful.
> >    Set max_atomic_write to DEFAULT_PIPEBUFSIZE in fhandler_pipe::create.
> >    In case of stdio handles inherited from non-Cygwin processes, fetch
> >    the pipe buffer size via NtQueryInformationFile in
> >    dtable::init_std_file_from_handle().  Better, in a matching
> >    fhandler_pipe method called from init_std_file_from_handle().
> 
> How about something like the attached (untested)?

LGTM.  I like the name change!

> > - What about calling select for writing on pipes read by non-Cygwin
> >    processes?  In that case, we still can't rely on WriteQuotaAvailable,
> >    just as before.
> > 
> >    I have a vague idea that we might want to count readers in that case,
> >    but I have to think about it some more.
> 
> Even if we count readers, we have no way of knowing whether a pending read
> has reduced WriteQuotaAvailable to 0.  Maybe this is a case where we should
> impose some artificial timeout, after which we report write ready.  Falsely
> reporting write ready in this corner case seems better than risking a
> deadlock.

Yeah, it's an almost hopeless case.  A timeout may be a way out.


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-07 10:50               ` Takashi Yano
@ 2021-09-08  0:07                 ` Takashi Yano
  2021-09-08  4:11                   ` Takashi Yano
  0 siblings, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-08  0:07 UTC (permalink / raw)
  To: cygwin-developers

On Tue, 7 Sep 2021 19:50:23 +0900
Takashi Yano wrote:

> @@ -796,7 +792,8 @@ pipe_cleanup (select_record *, select_stuff *stuff)
>        pi->stop_thread = true;
>        SetEvent (pi->bye);
         ~~~~~~~~~~~~~~~~~~~
This is not correct. SetEvent() wakes-up one of thread_pipe()s,
but it may be other thread than one which should be stopped.

>        pi->thread->detach ();
> -      CloseHandle (pi->bye);
> +      if (me->fh->get_select_evt () == NULL)
> +	CloseHandle (pi->bye);
>      }
>    delete pi;
>    stuff->device_specific_pipe = NULL;

I think it also should be
> +    for (ULONG i = 0; i < get_obj_handle_count (select_evt); i++)
> +      SetEvent (select_evt);

Actually I want to use PulseEvent() here if it is not **UNRELIABLE**.
https://docs.microsoft.com/en-us/windows-hardware/drivers/devtest/28648-pulseevent-is-an-unreliable-function

Does using semaphore object instead of event, and releasing
resources equal to the number of handles make sense?

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-08  0:07                 ` Takashi Yano
@ 2021-09-08  4:11                   ` Takashi Yano
  2021-09-08  9:01                     ` Takashi Yano
  2021-09-08  9:01                     ` Corinna Vinschen
  0 siblings, 2 replies; 250+ messages in thread
From: Takashi Yano @ 2021-09-08  4:11 UTC (permalink / raw)
  To: cygwin-developers

On Wed, 8 Sep 2021 09:07:48 +0900
Takashi Yano wrote:
> On Tue, 7 Sep 2021 19:50:23 +0900
> Takashi Yano wrote:
> 
> > @@ -796,7 +792,8 @@ pipe_cleanup (select_record *, select_stuff *stuff)
> >        pi->stop_thread = true;
> >        SetEvent (pi->bye);
>          ~~~~~~~~~~~~~~~~~~~
> This is not correct. SetEvent() wakes-up one of thread_pipe()s,
> but it may be other thread than one which should be stopped.
> 
> >        pi->thread->detach ();
> > -      CloseHandle (pi->bye);
> > +      if (me->fh->get_select_evt () == NULL)
> > +	CloseHandle (pi->bye);
> >      }
> >    delete pi;
> >    stuff->device_specific_pipe = NULL;
> 
> I think it also should be
> > +    for (ULONG i = 0; i < get_obj_handle_count (select_evt); i++)
> > +      SetEvent (select_evt);
> 
> Actually I want to use PulseEvent() here if it is not **UNRELIABLE**.
> https://docs.microsoft.com/en-us/windows-hardware/drivers/devtest/28648-pulseevent-is-an-unreliable-function
> 
> Does using semaphore object instead of event, and releasing
> resources equal to the number of handles make sense?

No it does not. One thread may consume semaphore multiple times....

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-08  4:11                   ` Takashi Yano
@ 2021-09-08  9:01                     ` Takashi Yano
  2021-09-08  9:01                     ` Corinna Vinschen
  1 sibling, 0 replies; 250+ messages in thread
From: Takashi Yano @ 2021-09-08  9:01 UTC (permalink / raw)
  To: cygwin-developers

[-- Attachment #1: Type: text/plain, Size: 1570 bytes --]

On Wed, 8 Sep 2021 13:11:41 +0900
Takashi Yano wrote:
> On Wed, 8 Sep 2021 09:07:48 +0900
> Takashi Yano wrote:
> > On Tue, 7 Sep 2021 19:50:23 +0900
> > Takashi Yano wrote:
> > 
> > > @@ -796,7 +792,8 @@ pipe_cleanup (select_record *, select_stuff *stuff)
> > >        pi->stop_thread = true;
> > >        SetEvent (pi->bye);
> >          ~~~~~~~~~~~~~~~~~~~
> > This is not correct. SetEvent() wakes-up one of thread_pipe()s,
> > but it may be other thread than one which should be stopped.
> > 
> > >        pi->thread->detach ();
> > > -      CloseHandle (pi->bye);
> > > +      if (me->fh->get_select_evt () == NULL)
> > > +	CloseHandle (pi->bye);
> > >      }
> > >    delete pi;
> > >    stuff->device_specific_pipe = NULL;
> > 
> > I think it also should be
> > > +    for (ULONG i = 0; i < get_obj_handle_count (select_evt); i++)
> > > +      SetEvent (select_evt);
> > 
> > Actually I want to use PulseEvent() here if it is not **UNRELIABLE**.
> > https://docs.microsoft.com/en-us/windows-hardware/drivers/devtest/28648-pulseevent-is-an-unreliable-function
> > 
> > Does using semaphore object instead of event, and releasing
> > resources equal to the number of handles make sense?
> 
> No it does not. One thread may consume semaphore multiple times....

I wrote a simple test program which counts wake-up 1000 times in 100
threads, and confirmed that semaphore method works as expected as well
as PulseEvent(). All 100000 (100*1000) wake-up were counted without
miscount.

Therefore, patch attached should work.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

[-- Attachment #2: 0001-Cygwin-select-Introduce-select_sem-semaphore-for-pip.patch --]
[-- Type: application/octet-stream, Size: 6742 bytes --]

From aaf0d8ba7ff8385884968a3664c58d05931b09b6 Mon Sep 17 00:00:00 2001
From: Takashi Yano <takashi.yano@nifty.ne.jp>
Date: Wed, 8 Sep 2021 17:18:35 +0900
Subject: [PATCH] Cygwin: select: Introduce select_sem semaphore for pipe.

- This patch reverts "Cygwin: select: Improve select/poll response",
  and introduces select_sem semaphore which notifies pipe status
  change.
---
 winsup/cygwin/fhandler.cc      |  1 +
 winsup/cygwin/fhandler.h       |  3 +++
 winsup/cygwin/fhandler_pipe.cc | 25 ++++++++++++++++++++
 winsup/cygwin/select.cc        | 42 ++++++++++------------------------
 4 files changed, 41 insertions(+), 30 deletions(-)

diff --git a/winsup/cygwin/fhandler.cc b/winsup/cygwin/fhandler.cc
index f0c1b68f1..39fe2640a 100644
--- a/winsup/cygwin/fhandler.cc
+++ b/winsup/cygwin/fhandler.cc
@@ -1464,6 +1464,7 @@ fhandler_base::fhandler_base () :
   _refcnt (0),
   openflags (0),
   unique_id (0),
+  select_sem (NULL),
   archetype (NULL),
   usecount (0)
 {
diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
index 7aed089eb..d309be2f7 100644
--- a/winsup/cygwin/fhandler.h
+++ b/winsup/cygwin/fhandler.h
@@ -217,6 +217,7 @@ class fhandler_base
   void set_ino (ino_t i) { ino = i; }
 
   HANDLE read_state;
+  HANDLE select_sem;
 
  public:
   LONG inc_refcnt () {return InterlockedIncrement (&_refcnt);}
@@ -520,6 +521,8 @@ public:
     fh->copy_from (this);
     return fh;
   }
+
+  HANDLE get_select_sem () { return select_sem; }
 };
 
 struct wsa_event
diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index c75476040..503d5c888 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -376,6 +376,8 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
     }
   else if (status == STATUS_THREAD_CANCELED)
     pthread::static_cancel_self ();
+  if (select_sem && nbytes)
+    ReleaseSemaphore (select_sem, get_obj_handle_count (select_sem), NULL);
   len = nbytes;
 }
 
@@ -507,6 +509,8 @@ fhandler_pipe_fifo::raw_write (const void *ptr, size_t len)
     set_errno (EINTR);
   else if (status == STATUS_THREAD_CANCELED)
     pthread::static_cancel_self ();
+  if (select_sem && nbytes)
+    ReleaseSemaphore (select_sem, get_obj_handle_count (select_sem), NULL);
   return nbytes ?: -1;
 }
 
@@ -515,6 +519,8 @@ fhandler_pipe::fixup_after_fork (HANDLE parent)
 {
   if (read_mtx)
     fork_fixup (parent, read_mtx, "read_mtx");
+  if (select_sem)
+    fork_fixup (parent, select_sem, "select_sem");
   fhandler_base::fixup_after_fork (parent);
 }
 
@@ -536,6 +542,15 @@ fhandler_pipe::dup (fhandler_base *child, int flags)
       ftp->close ();
       res = -1;
     }
+  else if (select_sem &&
+	   !DuplicateHandle (GetCurrentProcess (), select_sem,
+			    GetCurrentProcess (), &ftp->select_sem,
+			    0, !(flags & O_CLOEXEC), DUPLICATE_SAME_ACCESS))
+    {
+      __seterrno ();
+      ftp->close ();
+      res = -1;
+    }
 
   debug_printf ("res %d", res);
   return res;
@@ -546,6 +561,11 @@ fhandler_pipe::close ()
 {
   if (read_mtx)
     CloseHandle (read_mtx);
+  if (select_sem)
+    {
+      ReleaseSemaphore (select_sem, get_obj_handle_count (select_sem), NULL);
+      CloseHandle (select_sem);
+    }
   return fhandler_base::close ();
 }
 
@@ -765,6 +785,11 @@ fhandler_pipe::create (fhandler_pipe *fhs[2], unsigned psize, int mode)
 	  fhs[0]->set_read_mutex (mtx);
 	  res = 0;
 	}
+      fhs[0]->select_sem = CreateSemaphore (&sa, 0, INT32_MAX, NULL);
+      if (fhs[0]->select_sem)
+	DuplicateHandle (GetCurrentProcess (), fhs[0]->select_sem,
+			 GetCurrentProcess (), &fhs[1]->select_sem,
+			 0, 1, DUPLICATE_SAME_ACCESS);
     }
 
   debug_printf ("%R = pipe([%p, %p], %d, %y)", res, fhs[0], fhs[1], psize, mode);
diff --git a/winsup/cygwin/select.cc b/winsup/cygwin/select.cc
index 5e338e43f..e9e71b269 100644
--- a/winsup/cygwin/select.cc
+++ b/winsup/cygwin/select.cc
@@ -726,7 +726,6 @@ thread_pipe (void *arg)
   select_pipe_info *pi = (select_pipe_info *) arg;
   DWORD sleep_time = 0;
   bool looping = true;
-  DWORD t0 = GetTickCount ();
 
   while (looping)
     {
@@ -746,12 +745,7 @@ thread_pipe (void *arg)
 	break;
       cygwait (pi->bye, sleep_time >> 3);
       if (sleep_time < 80)
-	{
-	  DWORD t1 = GetTickCount ();
-	  if (t0 != t1)
-	    ++sleep_time;
-	  t0 = t1;
-	}
+	++sleep_time;
       if (pi->stop_thread)
 	break;
     }
@@ -768,7 +762,13 @@ start_thread_pipe (select_record *me, select_stuff *stuff)
     {
       pi->start = &stuff->start;
       pi->stop_thread = false;
-      pi->bye = CreateEvent (&sec_none_nih, TRUE, FALSE, NULL);
+      pi->bye = me->fh->get_select_sem ();
+      if (pi->bye)
+	DuplicateHandle (GetCurrentProcess (), pi->bye,
+			 GetCurrentProcess (), &pi->bye,
+			 0, 0, DUPLICATE_SAME_ACCESS);
+      else
+	pi->bye = CreateSemaphore (&sec_none_nih, 0, INT32_MAX, NULL);
       pi->thread = new cygthread (thread_pipe, pi, "pipesel");
       me->h = *pi->thread;
       if (!me->h)
@@ -786,7 +786,7 @@ pipe_cleanup (select_record *, select_stuff *stuff)
   if (pi->thread)
     {
       pi->stop_thread = true;
-      SetEvent (pi->bye);
+      ReleaseSemaphore (pi->bye, get_obj_handle_count (pi->bye), NULL);
       pi->thread->detach ();
       CloseHandle (pi->bye);
     }
@@ -927,7 +927,6 @@ thread_fifo (void *arg)
   select_fifo_info *pi = (select_fifo_info *) arg;
   DWORD sleep_time = 0;
   bool looping = true;
-  DWORD t0 = GetTickCount ();
 
   while (looping)
     {
@@ -947,12 +946,7 @@ thread_fifo (void *arg)
 	break;
       cygwait (pi->bye, sleep_time >> 3);
       if (sleep_time < 80)
-	{
-	  DWORD t1 = GetTickCount ();
-	  if (t0 != t1)
-	    ++sleep_time;
-	  t0 = t1;
-	}
+	++sleep_time;
       if (pi->stop_thread)
 	break;
     }
@@ -1128,7 +1122,6 @@ thread_console (void *arg)
   select_console_info *ci = (select_console_info *) arg;
   DWORD sleep_time = 0;
   bool looping = true;
-  DWORD t0 = GetTickCount ();
 
   while (looping)
     {
@@ -1148,12 +1141,7 @@ thread_console (void *arg)
 	break;
       cygwait (ci->bye, sleep_time >> 3);
       if (sleep_time < 80)
-	{
-	  DWORD t1 = GetTickCount ();
-	  if (t0 != t1)
-	    ++sleep_time;
-	  t0 = t1;
-	}
+	++sleep_time;
       if (ci->stop_thread)
 	break;
     }
@@ -1373,7 +1361,6 @@ thread_pty_slave (void *arg)
   select_pipe_info *pi = (select_pipe_info *) arg;
   DWORD sleep_time = 0;
   bool looping = true;
-  DWORD t0 = GetTickCount ();
 
   while (looping)
     {
@@ -1393,12 +1380,7 @@ thread_pty_slave (void *arg)
 	break;
       cygwait (pi->bye, sleep_time >> 3);
       if (sleep_time < 80)
-	{
-	  DWORD t1 = GetTickCount ();
-	  if (t0 != t1)
-	    ++sleep_time;
-	  t0 = t1;
-	}
+	++sleep_time;
       if (pi->stop_thread)
 	break;
     }
-- 
2.33.0


^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-08  4:11                   ` Takashi Yano
  2021-09-08  9:01                     ` Takashi Yano
@ 2021-09-08  9:01                     ` Corinna Vinschen
  2021-09-08  9:26                       ` Corinna Vinschen
  2021-09-08  9:37                       ` Takashi Yano
  1 sibling, 2 replies; 250+ messages in thread
From: Corinna Vinschen @ 2021-09-08  9:01 UTC (permalink / raw)
  To: cygwin-developers

On Sep  8 13:11, Takashi Yano wrote:
> On Wed, 8 Sep 2021 09:07:48 +0900
> Takashi Yano wrote:
> > On Tue, 7 Sep 2021 19:50:23 +0900
> > Takashi Yano wrote:
> > 
> > > @@ -796,7 +792,8 @@ pipe_cleanup (select_record *, select_stuff *stuff)
> > >        pi->stop_thread = true;
> > >        SetEvent (pi->bye);
> >          ~~~~~~~~~~~~~~~~~~~
> > This is not correct. SetEvent() wakes-up one of thread_pipe()s,
> > but it may be other thread than one which should be stopped.
> > 
> > >        pi->thread->detach ();
> > > -      CloseHandle (pi->bye);
> > > +      if (me->fh->get_select_evt () == NULL)
> > > +	CloseHandle (pi->bye);
> > >      }
> > >    delete pi;
> > >    stuff->device_specific_pipe = NULL;
> > 
> > I think it also should be
> > > +    for (ULONG i = 0; i < get_obj_handle_count (select_evt); i++)
> > > +      SetEvent (select_evt);
> > 
> > Actually I want to use PulseEvent() here if it is not **UNRELIABLE**.
> > https://docs.microsoft.com/en-us/windows-hardware/drivers/devtest/28648-pulseevent-is-an-unreliable-function
> > 
> > Does using semaphore object instead of event, and releasing
> > resources equal to the number of handles make sense?
> 
> No it does not. One thread may consume semaphore multiple times....

What exactly is the problem in the code which results in high CPU
load?  Can you explain this a bit?  Maybe we need an entirely
different approach to avoid that.


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-08  9:01                     ` Corinna Vinschen
@ 2021-09-08  9:26                       ` Corinna Vinschen
  2021-09-08  9:45                         ` Takashi Yano
  2021-09-08  9:37                       ` Takashi Yano
  1 sibling, 1 reply; 250+ messages in thread
From: Corinna Vinschen @ 2021-09-08  9:26 UTC (permalink / raw)
  To: cygwin-developers

On Sep  8 11:01, Corinna Vinschen wrote:
> On Sep  8 13:11, Takashi Yano wrote:
> > On Wed, 8 Sep 2021 09:07:48 +0900
> > Takashi Yano wrote:
> > > On Tue, 7 Sep 2021 19:50:23 +0900
> > > Takashi Yano wrote:
> > > 
> > > > @@ -796,7 +792,8 @@ pipe_cleanup (select_record *, select_stuff *stuff)
> > > >        pi->stop_thread = true;
> > > >        SetEvent (pi->bye);
> > >          ~~~~~~~~~~~~~~~~~~~
> > > This is not correct. SetEvent() wakes-up one of thread_pipe()s,
> > > but it may be other thread than one which should be stopped.
> > > 
> > > >        pi->thread->detach ();
> > > > -      CloseHandle (pi->bye);
> > > > +      if (me->fh->get_select_evt () == NULL)
> > > > +	CloseHandle (pi->bye);
> > > >      }
> > > >    delete pi;
> > > >    stuff->device_specific_pipe = NULL;
> > > 
> > > I think it also should be
> > > > +    for (ULONG i = 0; i < get_obj_handle_count (select_evt); i++)
> > > > +      SetEvent (select_evt);
> > > 
> > > Actually I want to use PulseEvent() here if it is not **UNRELIABLE**.
> > > https://docs.microsoft.com/en-us/windows-hardware/drivers/devtest/28648-pulseevent-is-an-unreliable-function
> > > 
> > > Does using semaphore object instead of event, and releasing
> > > resources equal to the number of handles make sense?
> > 
> > No it does not. One thread may consume semaphore multiple times....
> 
> What exactly is the problem in the code which results in high CPU
> load?  Can you explain this a bit?  Maybe we need an entirely
> different approach to avoid that.

I saw your new patch, but I don't see the problem.  I typed a lot of
keys in mintty quickly and what happens is that the load of mintty
goes up to 9% on a 4 CPU system, but only temporarily while typing.
How do you reproduce the problem?


Thanks,
Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-08  9:01                     ` Corinna Vinschen
  2021-09-08  9:26                       ` Corinna Vinschen
@ 2021-09-08  9:37                       ` Takashi Yano
  1 sibling, 0 replies; 250+ messages in thread
From: Takashi Yano @ 2021-09-08  9:37 UTC (permalink / raw)
  To: cygwin-developers

On Wed, 8 Sep 2021 11:01:52 +0200
Corinna Vinschen wrote:
> On Sep  8 13:11, Takashi Yano wrote:
> > On Wed, 8 Sep 2021 09:07:48 +0900
> > Takashi Yano wrote:
> > > On Tue, 7 Sep 2021 19:50:23 +0900
> > > Takashi Yano wrote:
> > > 
> > > > @@ -796,7 +792,8 @@ pipe_cleanup (select_record *, select_stuff *stuff)
> > > >        pi->stop_thread = true;
> > > >        SetEvent (pi->bye);
> > >          ~~~~~~~~~~~~~~~~~~~
> > > This is not correct. SetEvent() wakes-up one of thread_pipe()s,
> > > but it may be other thread than one which should be stopped.
> > > 
> > > >        pi->thread->detach ();
> > > > -      CloseHandle (pi->bye);
> > > > +      if (me->fh->get_select_evt () == NULL)
> > > > +	CloseHandle (pi->bye);
> > > >      }
> > > >    delete pi;
> > > >    stuff->device_specific_pipe = NULL;
> > > 
> > > I think it also should be
> > > > +    for (ULONG i = 0; i < get_obj_handle_count (select_evt); i++)
> > > > +      SetEvent (select_evt);
> > > 
> > > Actually I want to use PulseEvent() here if it is not **UNRELIABLE**.
> > > https://docs.microsoft.com/en-us/windows-hardware/drivers/devtest/28648-pulseevent-is-an-unreliable-function
> > > 
> > > Does using semaphore object instead of event, and releasing
> > > resources equal to the number of handles make sense?
> > 
> > No it does not. One thread may consume semaphore multiple times....
> 
> What exactly is the problem in the code which results in high CPU
> load?  Can you explain this a bit?  Maybe we need an entirely
> different approach to avoid that.

The thread_pipe() code in the current git head of master is like:

while (looping)
  {
    ...
    if (peek_pipe ())
      looping = false;
    ...
    if (!looping)
      break;
    cygwait (pi->bye, sleep_time >> 3);
    if (sleep_time < 80)
      ++sleep_time;
    if (pi->stop_thread)
      break;
  }
returnn 0;

With this code, the first 8 loops calls cygwait() with
cygwait (pi->bye, 0);
and after that
cygwait (pi->bye, nonzero value);

cygwait() with nonzero timeout value causes usually sleeps at least
15msec (because of the resolution of the timer).

Looping 8 times is just a moment for the CPU.

After the 8 loops, thread_pipe() responds slowly to select() call
because thread notices the status change of pipe after the timeout.

To avoid this, commit dccde0dc changes the code as follows.

diff --git a/winsup/cygwin/select.cc b/winsup/cygwin/select.cc
index 8ad982c12..83e1c00e0 100644
--- a/winsup/cygwin/select.cc
+++ b/winsup/cygwin/select.cc
@@ -735,6 +735,7 @@ thread_pipe (void *arg)
   select_pipe_info *pi = (select_pipe_info *) arg;
   DWORD sleep_time = 0;
   bool looping = true;
+  DWORD t0 = GetTickCount ();

   while (looping)
     {
@@ -754,7 +755,12 @@ thread_pipe (void *arg)
        break;
       cygwait (pi->bye, sleep_time >> 3);
       if (sleep_time < 80)
-       ++sleep_time;
+       {
+         DWORD t1 = GetTickCount ();
+         if (t0 != t1)
+           ++sleep_time;
+         t0 = t1;
+       }
       if (pi->stop_thread)
        break;
     }

In this code, I expected t0 != t1 happens at most every 1msec,
and after 8msec, timeout for cygwait() becomes nonzero. During
this 8msec, CPU gets high load because looping is without sleep.

However, in fact, t0 != t1 happens every 15msec because the
resolution of timer is low. Then, CPU gets high load during
first 120msec after starting thread_pipe(). So if you type
keys quickly, the loop is always performs without sleep.

Therefore, I proposed new patch which use event or semaphore
to notify read()/write()/close() of the pipe instead of just
sleeping a while.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-08  9:26                       ` Corinna Vinschen
@ 2021-09-08  9:45                         ` Takashi Yano
  2021-09-08 10:04                           ` Corinna Vinschen
  0 siblings, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-08  9:45 UTC (permalink / raw)
  To: cygwin-developers

On Wed, 8 Sep 2021 11:26:21 +0200
Corinna Vinschen wrote:
> On Sep  8 11:01, Corinna Vinschen wrote:
> > On Sep  8 13:11, Takashi Yano wrote:
> > > On Wed, 8 Sep 2021 09:07:48 +0900
> > > Takashi Yano wrote:
> > > > On Tue, 7 Sep 2021 19:50:23 +0900
> > > > Takashi Yano wrote:
> > > > 
> > > > > @@ -796,7 +792,8 @@ pipe_cleanup (select_record *, select_stuff *stuff)
> > > > >        pi->stop_thread = true;
> > > > >        SetEvent (pi->bye);
> > > >          ~~~~~~~~~~~~~~~~~~~
> > > > This is not correct. SetEvent() wakes-up one of thread_pipe()s,
> > > > but it may be other thread than one which should be stopped.
> > > > 
> > > > >        pi->thread->detach ();
> > > > > -      CloseHandle (pi->bye);
> > > > > +      if (me->fh->get_select_evt () == NULL)
> > > > > +	CloseHandle (pi->bye);
> > > > >      }
> > > > >    delete pi;
> > > > >    stuff->device_specific_pipe = NULL;
> > > > 
> > > > I think it also should be
> > > > > +    for (ULONG i = 0; i < get_obj_handle_count (select_evt); i++)
> > > > > +      SetEvent (select_evt);
> > > > 
> > > > Actually I want to use PulseEvent() here if it is not **UNRELIABLE**.
> > > > https://docs.microsoft.com/en-us/windows-hardware/drivers/devtest/28648-pulseevent-is-an-unreliable-function
> > > > 
> > > > Does using semaphore object instead of event, and releasing
> > > > resources equal to the number of handles make sense?
> > > 
> > > No it does not. One thread may consume semaphore multiple times....
> > 
> > What exactly is the problem in the code which results in high CPU
> > load?  Can you explain this a bit?  Maybe we need an entirely
> > different approach to avoid that.
> 
> I saw your new patch, but I don't see the problem.  I typed a lot of
> keys in mintty quickly and what happens is that the load of mintty
> goes up to 9% on a 4 CPU system, but only temporarily while typing.
> How do you reproduce the problem?

Did you apply the patch
0001-Cygwin-select-Introduce-select_evt-event-for-pipe.patch
or
0001-Cygwin-select-Introduce-select_sem-semaphore-for-pip.patch
?

With these patch, the problem does not occur. The problem occurs
with the commit dccde0dc.

With my 4 core 8 thread CPU, CPU loads goes up to 12-13 % if
I type keys using key repeat (30cps) after the commit dccde0dc.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-08  9:45                         ` Takashi Yano
@ 2021-09-08 10:04                           ` Corinna Vinschen
  2021-09-08 10:45                             ` Takashi Yano
  0 siblings, 1 reply; 250+ messages in thread
From: Corinna Vinschen @ 2021-09-08 10:04 UTC (permalink / raw)
  To: cygwin-developers

On Sep  8 18:45, Takashi Yano wrote:
> On Wed, 8 Sep 2021 11:26:21 +0200
> Corinna Vinschen wrote:
> > On Sep  8 11:01, Corinna Vinschen wrote:
> > > What exactly is the problem in the code which results in high CPU
> > > load?  Can you explain this a bit?  Maybe we need an entirely
> > > different approach to avoid that.
> > 
> > I saw your new patch, but I don't see the problem.  I typed a lot of
> > keys in mintty quickly and what happens is that the load of mintty
> > goes up to 9% on a 4 CPU system, but only temporarily while typing.
> > How do you reproduce the problem?
> 
> Did you apply the patch
> 0001-Cygwin-select-Introduce-select_evt-event-for-pipe.patch
> or
> 0001-Cygwin-select-Introduce-select_sem-semaphore-for-pip.patch
> ?
> 
> With these patch, the problem does not occur. The problem occurs
> with the commit dccde0dc.

dccde0dc is 23bb19efcc45 in topic/pipe ATM (after force push) so, yes,
I'm running this with topic/pipe HEAD including this patch.

> With my 4 core 8 thread CPU, CPU loads goes up to 12-13 % if
> I type keys using key repeat (30cps) after the commit dccde0dc.

Oh, wow!  As I wrote above, before applying "Cygwin: select: Introduce
select_sem semaphore for pipe." I only saw a 9% load.  After applying
the patch I saw the same load.

I don't know what I did differently, but after reverting your semaphore
patch I now see loads of up to 30%.  So, never mind, apparently I tested
wrongly before.  Your patch reduces the load tremendously.

Just one question.  Would you mind to split your patch into two parts,
one being just the revert of your "Improve select/poll response." patch
and one introducing select_sem?


Thanks,
Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-08 10:04                           ` Corinna Vinschen
@ 2021-09-08 10:45                             ` Takashi Yano
  2021-09-08 10:51                               ` Corinna Vinschen
  0 siblings, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-08 10:45 UTC (permalink / raw)
  To: cygwin-developers

[-- Attachment #1: Type: text/plain, Size: 1918 bytes --]

Hi Corinna,

On Wed, 8 Sep 2021 12:04:21 +0200
Corinna Vinschen wrote:
> On Sep  8 18:45, Takashi Yano wrote:
> > On Wed, 8 Sep 2021 11:26:21 +0200
> > Corinna Vinschen wrote:
> > > On Sep  8 11:01, Corinna Vinschen wrote:
> > > > What exactly is the problem in the code which results in high CPU
> > > > load?  Can you explain this a bit?  Maybe we need an entirely
> > > > different approach to avoid that.
> > > 
> > > I saw your new patch, but I don't see the problem.  I typed a lot of
> > > keys in mintty quickly and what happens is that the load of mintty
> > > goes up to 9% on a 4 CPU system, but only temporarily while typing.
> > > How do you reproduce the problem?
> > 
> > Did you apply the patch
> > 0001-Cygwin-select-Introduce-select_evt-event-for-pipe.patch
> > or
> > 0001-Cygwin-select-Introduce-select_sem-semaphore-for-pip.patch
> > ?
> > 
> > With these patch, the problem does not occur. The problem occurs
> > with the commit dccde0dc.
> 
> dccde0dc is 23bb19efcc45 in topic/pipe ATM (after force push) so, yes,
> I'm running this with topic/pipe HEAD including this patch.
> 
> > With my 4 core 8 thread CPU, CPU loads goes up to 12-13 % if
> > I type keys using key repeat (30cps) after the commit dccde0dc.
> 
> Oh, wow!  As I wrote above, before applying "Cygwin: select: Introduce
> select_sem semaphore for pipe." I only saw a 9% load.  After applying
> the patch I saw the same load.
> 
> I don't know what I did differently, but after reverting your semaphore
> patch I now see loads of up to 30%.  So, never mind, apparently I tested
> wrongly before.  Your patch reduces the load tremendously.

Thanks for testing again.

> Just one question.  Would you mind to split your patch into two parts,
> one being just the revert of your "Improve select/poll response." patch
> and one introducing select_sem?

I split the patch as you advised.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

[-- Attachment #2: 0001-Revert-Cygwin-select-Improve-select-poll-response.patch --]
[-- Type: application/octet-stream, Size: 2525 bytes --]

From bc9fde7a1e4f0c80c4a58c7ed3e9b83c1f4f24d2 Mon Sep 17 00:00:00 2001
From: Takashi Yano <takashi.yano@nifty.ne.jp>
Date: Wed, 8 Sep 2021 19:22:40 +0900
Subject: [PATCH 1/2] Revert "Cygwin: select: Improve select/poll response."

... because this commit (23bb19ef) causes high CPU load.
---
 winsup/cygwin/select.cc | 32 ++++----------------------------
 1 file changed, 4 insertions(+), 28 deletions(-)

diff --git a/winsup/cygwin/select.cc b/winsup/cygwin/select.cc
index 5e338e43f..c85ce748c 100644
--- a/winsup/cygwin/select.cc
+++ b/winsup/cygwin/select.cc
@@ -726,7 +726,6 @@ thread_pipe (void *arg)
   select_pipe_info *pi = (select_pipe_info *) arg;
   DWORD sleep_time = 0;
   bool looping = true;
-  DWORD t0 = GetTickCount ();
 
   while (looping)
     {
@@ -746,12 +745,7 @@ thread_pipe (void *arg)
 	break;
       cygwait (pi->bye, sleep_time >> 3);
       if (sleep_time < 80)
-	{
-	  DWORD t1 = GetTickCount ();
-	  if (t0 != t1)
-	    ++sleep_time;
-	  t0 = t1;
-	}
+	++sleep_time;
       if (pi->stop_thread)
 	break;
     }
@@ -927,7 +921,6 @@ thread_fifo (void *arg)
   select_fifo_info *pi = (select_fifo_info *) arg;
   DWORD sleep_time = 0;
   bool looping = true;
-  DWORD t0 = GetTickCount ();
 
   while (looping)
     {
@@ -947,12 +940,7 @@ thread_fifo (void *arg)
 	break;
       cygwait (pi->bye, sleep_time >> 3);
       if (sleep_time < 80)
-	{
-	  DWORD t1 = GetTickCount ();
-	  if (t0 != t1)
-	    ++sleep_time;
-	  t0 = t1;
-	}
+	++sleep_time;
       if (pi->stop_thread)
 	break;
     }
@@ -1128,7 +1116,6 @@ thread_console (void *arg)
   select_console_info *ci = (select_console_info *) arg;
   DWORD sleep_time = 0;
   bool looping = true;
-  DWORD t0 = GetTickCount ();
 
   while (looping)
     {
@@ -1148,12 +1135,7 @@ thread_console (void *arg)
 	break;
       cygwait (ci->bye, sleep_time >> 3);
       if (sleep_time < 80)
-	{
-	  DWORD t1 = GetTickCount ();
-	  if (t0 != t1)
-	    ++sleep_time;
-	  t0 = t1;
-	}
+	++sleep_time;
       if (ci->stop_thread)
 	break;
     }
@@ -1373,7 +1355,6 @@ thread_pty_slave (void *arg)
   select_pipe_info *pi = (select_pipe_info *) arg;
   DWORD sleep_time = 0;
   bool looping = true;
-  DWORD t0 = GetTickCount ();
 
   while (looping)
     {
@@ -1393,12 +1374,7 @@ thread_pty_slave (void *arg)
 	break;
       cygwait (pi->bye, sleep_time >> 3);
       if (sleep_time < 80)
-	{
-	  DWORD t1 = GetTickCount ();
-	  if (t0 != t1)
-	    ++sleep_time;
-	  t0 = t1;
-	}
+	++sleep_time;
       if (pi->stop_thread)
 	break;
     }
-- 
2.33.0


[-- Attachment #3: 0002-Cygwin-select-Introduce-select_sem-semaphore-for-pip.patch --]
[-- Type: application/octet-stream, Size: 4714 bytes --]

From 3b35c69225f82554b186264c6c35b4b1dd874cbd Mon Sep 17 00:00:00 2001
From: Takashi Yano <takashi.yano@nifty.ne.jp>
Date: Wed, 8 Sep 2021 17:18:35 +0900
Subject: [PATCH 2/2] Cygwin: select: Introduce select_sem semaphore for pipe.

- This patch introduces select_sem semaphore which notifies pipe status
  change.
---
 winsup/cygwin/fhandler.cc      |  1 +
 winsup/cygwin/fhandler.h       |  3 +++
 winsup/cygwin/fhandler_pipe.cc | 25 +++++++++++++++++++++++++
 winsup/cygwin/select.cc        | 10 ++++++++--
 4 files changed, 37 insertions(+), 2 deletions(-)

diff --git a/winsup/cygwin/fhandler.cc b/winsup/cygwin/fhandler.cc
index f0c1b68f1..39fe2640a 100644
--- a/winsup/cygwin/fhandler.cc
+++ b/winsup/cygwin/fhandler.cc
@@ -1464,6 +1464,7 @@ fhandler_base::fhandler_base () :
   _refcnt (0),
   openflags (0),
   unique_id (0),
+  select_sem (NULL),
   archetype (NULL),
   usecount (0)
 {
diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
index 7aed089eb..d309be2f7 100644
--- a/winsup/cygwin/fhandler.h
+++ b/winsup/cygwin/fhandler.h
@@ -217,6 +217,7 @@ class fhandler_base
   void set_ino (ino_t i) { ino = i; }
 
   HANDLE read_state;
+  HANDLE select_sem;
 
  public:
   LONG inc_refcnt () {return InterlockedIncrement (&_refcnt);}
@@ -520,6 +521,8 @@ public:
     fh->copy_from (this);
     return fh;
   }
+
+  HANDLE get_select_sem () { return select_sem; }
 };
 
 struct wsa_event
diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index c75476040..ea8ea41dd 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -376,6 +376,8 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
     }
   else if (status == STATUS_THREAD_CANCELED)
     pthread::static_cancel_self ();
+  if (select_sem && nbytes)
+    ReleaseSemaphore (select_sem, get_obj_handle_count (select_sem), NULL);
   len = nbytes;
 }
 
@@ -507,6 +509,8 @@ fhandler_pipe_fifo::raw_write (const void *ptr, size_t len)
     set_errno (EINTR);
   else if (status == STATUS_THREAD_CANCELED)
     pthread::static_cancel_self ();
+  if (select_sem && nbytes)
+    ReleaseSemaphore (select_sem, get_obj_handle_count (select_sem), NULL);
   return nbytes ?: -1;
 }
 
@@ -515,6 +519,8 @@ fhandler_pipe::fixup_after_fork (HANDLE parent)
 {
   if (read_mtx)
     fork_fixup (parent, read_mtx, "read_mtx");
+  if (select_sem)
+    fork_fixup (parent, select_sem, "select_sem");
   fhandler_base::fixup_after_fork (parent);
 }
 
@@ -536,6 +542,15 @@ fhandler_pipe::dup (fhandler_base *child, int flags)
       ftp->close ();
       res = -1;
     }
+  else if (select_sem &&
+	   !DuplicateHandle (GetCurrentProcess (), select_sem,
+			    GetCurrentProcess (), &ftp->select_sem,
+			    0, !(flags & O_CLOEXEC), DUPLICATE_SAME_ACCESS))
+    {
+      __seterrno ();
+      ftp->close ();
+      res = -1;
+    }
 
   debug_printf ("res %d", res);
   return res;
@@ -546,6 +561,11 @@ fhandler_pipe::close ()
 {
   if (read_mtx)
     CloseHandle (read_mtx);
+  if (select_sem)
+    {
+      ReleaseSemaphore (select_sem, get_obj_handle_count (select_sem), NULL);
+      CloseHandle (select_sem);
+    }
   return fhandler_base::close ();
 }
 
@@ -765,6 +785,11 @@ fhandler_pipe::create (fhandler_pipe *fhs[2], unsigned psize, int mode)
 	  fhs[0]->set_read_mutex (mtx);
 	  res = 0;
 	}
+      fhs[0]->select_sem = CreateSemaphore (&sa, 0, INT32_MAX, NULL);
+      if (fhs[0]->select_sem)
+	DuplicateHandle (GetCurrentProcess (), fhs[0]->select_sem,
+			 GetCurrentProcess (), &fhs[1]->select_sem,
+			 0, 1, DUPLICATE_SAME_ACCESS);
     }
 
   debug_printf ("%R = pipe([%p, %p], %d, %y)", res, fhs[0], fhs[1], psize, mode);
diff --git a/winsup/cygwin/select.cc b/winsup/cygwin/select.cc
index c85ce748c..e9e71b269 100644
--- a/winsup/cygwin/select.cc
+++ b/winsup/cygwin/select.cc
@@ -762,7 +762,13 @@ start_thread_pipe (select_record *me, select_stuff *stuff)
     {
       pi->start = &stuff->start;
       pi->stop_thread = false;
-      pi->bye = CreateEvent (&sec_none_nih, TRUE, FALSE, NULL);
+      pi->bye = me->fh->get_select_sem ();
+      if (pi->bye)
+	DuplicateHandle (GetCurrentProcess (), pi->bye,
+			 GetCurrentProcess (), &pi->bye,
+			 0, 0, DUPLICATE_SAME_ACCESS);
+      else
+	pi->bye = CreateSemaphore (&sec_none_nih, 0, INT32_MAX, NULL);
       pi->thread = new cygthread (thread_pipe, pi, "pipesel");
       me->h = *pi->thread;
       if (!me->h)
@@ -780,7 +786,7 @@ pipe_cleanup (select_record *, select_stuff *stuff)
   if (pi->thread)
     {
       pi->stop_thread = true;
-      SetEvent (pi->bye);
+      ReleaseSemaphore (pi->bye, get_obj_handle_count (pi->bye), NULL);
       pi->thread->detach ();
       CloseHandle (pi->bye);
     }
-- 
2.33.0


^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-08 10:45                             ` Takashi Yano
@ 2021-09-08 10:51                               ` Corinna Vinschen
  2021-09-09  3:21                                 ` Takashi Yano
  0 siblings, 1 reply; 250+ messages in thread
From: Corinna Vinschen @ 2021-09-08 10:51 UTC (permalink / raw)
  To: cygwin-developers

On Sep  8 19:45, Takashi Yano wrote:
> Hi Corinna,
> 
> On Wed, 8 Sep 2021 12:04:21 +0200
> Corinna Vinschen wrote:
> > On Sep  8 18:45, Takashi Yano wrote:
> > > On Wed, 8 Sep 2021 11:26:21 +0200
> > > Corinna Vinschen wrote:
> > > > On Sep  8 11:01, Corinna Vinschen wrote:
> > > > > What exactly is the problem in the code which results in high CPU
> > > > > load?  Can you explain this a bit?  Maybe we need an entirely
> > > > > different approach to avoid that.
> > > > 
> > > > I saw your new patch, but I don't see the problem.  I typed a lot of
> > > > keys in mintty quickly and what happens is that the load of mintty
> > > > goes up to 9% on a 4 CPU system, but only temporarily while typing.
> > > > How do you reproduce the problem?
> > > 
> > > Did you apply the patch
> > > 0001-Cygwin-select-Introduce-select_evt-event-for-pipe.patch
> > > or
> > > 0001-Cygwin-select-Introduce-select_sem-semaphore-for-pip.patch
> > > ?
> > > 
> > > With these patch, the problem does not occur. The problem occurs
> > > with the commit dccde0dc.
> > 
> > dccde0dc is 23bb19efcc45 in topic/pipe ATM (after force push) so, yes,
> > I'm running this with topic/pipe HEAD including this patch.
> > 
> > > With my 4 core 8 thread CPU, CPU loads goes up to 12-13 % if
> > > I type keys using key repeat (30cps) after the commit dccde0dc.
> > 
> > Oh, wow!  As I wrote above, before applying "Cygwin: select: Introduce
> > select_sem semaphore for pipe." I only saw a 9% load.  After applying
> > the patch I saw the same load.
> > 
> > I don't know what I did differently, but after reverting your semaphore
> > patch I now see loads of up to 30%.  So, never mind, apparently I tested
> > wrongly before.  Your patch reduces the load tremendously.
> 
> Thanks for testing again.
> 
> > Just one question.  Would you mind to split your patch into two parts,
> > one being just the revert of your "Improve select/poll response." patch
> > and one introducing select_sem?
> 
> I split the patch as you advised.

Pushed.


Thanks,
Corinna




^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-02 19:35                                                                                   ` Corinna Vinschen
  2021-09-02 20:19                                                                                     ` Ken Brown
  2021-09-03 10:00                                                                                     ` Takashi Yano
@ 2021-09-08 11:32                                                                                     ` Takashi Yano
  2021-09-08 11:55                                                                                       ` Corinna Vinschen
  2 siblings, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-08 11:32 UTC (permalink / raw)
  To: cygwin-developers

Hi Corinna,

On Thu, 2 Sep 2021 21:35:21 +0200
Corinna Vinschen wrote:
> On Sep  2 21:00, Corinna Vinschen wrote:
> > On Sep  2 09:01, Ken Brown wrote:
> > > On 9/2/2021 4:17 AM, Corinna Vinschen wrote:
> > > > What if the readers never request more than, say, 50 or even 25% of the
> > > > available buffer space?  Our buffer is 64K and there's no guarantee that
> > > > any read > PIPE_BUF (== 4K) is atomic anyway.  This can work without
> > > > having to check the other side of the pipe.  Something like this,
> > > > ignoring border cases:
> > > > 
> > > > pipe::create()
> > > > {
> > > >     [...]
> > > >     mutex = CreateMutex();
> > > > }
> > > > 
> > > > pipe::raw_read(char *buf, size_t num_requested)
> > > > {
> > > >    if (blocking)
> > > >      {
> > > >        WFSO(mutex);
> > > >        NtQueryInformationFile(FilePipeLocalInformation);
> > > >        if (!fpli.ReadDataAvailable
> > > > 	  && num_requested > fpli.InboundQuota / 4)
> > > > 	num_requested = fpli.InboundQuota / 4;
> > > >        NtReadFile(pipe, buf, num_requested);
> > > >        ReleaseMutex(mutex);
> > > >      }
> > > > }
> > > > 
> > > > It's not entirely foolproof, but it should fix 99% of the cases.
> > > 
> > > I like it!
> > > 
> > > Do you think there's anything we can or should do to avoid a deadlock in the
> > > rare cases where this fails?  The only thing I can think of immediately is
> > > to always impose a timeout if select is called with infinite timeout on the
> > > write side of a pipe, after which we report that the pipe is write ready.
> > > After all, we've lived since 2008 with a bug that caused select to *always*
> > > report write ready.
> > 
> > Indeed.  Hmm.  What timeout are you thinking of?  Seconds?  Minutes?
> > 
> > > Alternatively, we could just wait and see if there's an actual use case in
> > > which someone encounters a deadlock.
> > 
> > Or that.  Fixing up select isn't too hard in that case, I guess.
> 
> It's getting too late again.  I drop off for tonight, but I attached
> my POC code I have so far.  It also adds the snippets from my previous
> patch which fixes stuff Takashi found during testing.  It also fixes
> something which looks like a bug in raw_write:
> 
> -	  ptr = ((char *) ptr) + chunk;
> +	  ptr = ((char *) ptr) + nbytes_now;
> 
> Incrementing ptr by chunk bytes while only nbytes_now have been written
> looks incorrect.
> 
> As for the reader, it makes the # of bytes to read dependent on the
> number of reader handles.  I don't know if that's such a bright idea,
> but this can be changed easily.
> 
> Anyway, this runs all my testcases successfully but they are anything
> but thorough.
> 
> Patch relativ to topic/pipe attached.  Would you both mind to take a
> scrutinizing look?

Sorry for replying to old post.

As for this patch, read_mtx was introduced. This handle is initialized
only for read pipe. However, this seems to be NULL even without
initialization in write pipe. I wonder why initializing read_mtx in
the constructor is not necessary.

How do you guarantee that read_mtx is NULL on the write pipe?

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-08 11:32                                                                                     ` Takashi Yano
@ 2021-09-08 11:55                                                                                       ` Corinna Vinschen
  2021-09-08 12:33                                                                                         ` Takashi Yano
  2021-09-08 17:43                                                                                         ` Ken Brown
  0 siblings, 2 replies; 250+ messages in thread
From: Corinna Vinschen @ 2021-09-08 11:55 UTC (permalink / raw)
  To: cygwin-developers

On Sep  8 20:32, Takashi Yano wrote:
> Hi Corinna,
> 
> On Thu, 2 Sep 2021 21:35:21 +0200
> Corinna Vinschen wrote:
> > It's getting too late again.  I drop off for tonight, but I attached
> > my POC code I have so far.  It also adds the snippets from my previous
> > patch which fixes stuff Takashi found during testing.
> > [...]
> > Patch relativ to topic/pipe attached.  Would you both mind to take a
> > scrutinizing look?
> 
> Sorry for replying to old post.
> 
> As for this patch, read_mtx was introduced. This handle is initialized
> only for read pipe. However, this seems to be NULL even without
> initialization in write pipe. I wonder why initializing read_mtx in
> the constructor is not necessary.
> 
> How do you guarantee that read_mtx is NULL on the write pipe?

fhandlers are always calloc'ed on the cygheap, so you don't have
to initialize anything to NULL.  It doesn't hurt, of course, but
it's certainly not required.


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-08 11:55                                                                                       ` Corinna Vinschen
@ 2021-09-08 12:33                                                                                         ` Takashi Yano
  2021-09-08 17:43                                                                                         ` Ken Brown
  1 sibling, 0 replies; 250+ messages in thread
From: Takashi Yano @ 2021-09-08 12:33 UTC (permalink / raw)
  To: cygwin-developers

On Wed, 8 Sep 2021 13:55:41 +0200
Corinna Vinschen wrote:
> On Sep  8 20:32, Takashi Yano wrote:
> > Hi Corinna,
> > 
> > On Thu, 2 Sep 2021 21:35:21 +0200
> > Corinna Vinschen wrote:
> > > It's getting too late again.  I drop off for tonight, but I attached
> > > my POC code I have so far.  It also adds the snippets from my previous
> > > patch which fixes stuff Takashi found during testing.
> > > [...]
> > > Patch relativ to topic/pipe attached.  Would you both mind to take a
> > > scrutinizing look?
> > 
> > Sorry for replying to old post.
> > 
> > As for this patch, read_mtx was introduced. This handle is initialized
> > only for read pipe. However, this seems to be NULL even without
> > initialization in write pipe. I wonder why initializing read_mtx in
> > the constructor is not necessary.
> > 
> > How do you guarantee that read_mtx is NULL on the write pipe?
> 
> fhandlers are always calloc'ed on the cygheap, so you don't have
> to initialize anything to NULL.  It doesn't hurt, of course, but
> it's certainly not required.

Thanks for the answer. I understand.


-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-08 11:55                                                                                       ` Corinna Vinschen
  2021-09-08 12:33                                                                                         ` Takashi Yano
@ 2021-09-08 17:43                                                                                         ` Ken Brown
  2021-09-08 18:28                                                                                           ` Corinna Vinschen
  1 sibling, 1 reply; 250+ messages in thread
From: Ken Brown @ 2021-09-08 17:43 UTC (permalink / raw)
  To: cygwin-developers

On 9/8/2021 7:55 AM, Corinna Vinschen wrote:
> On Sep  8 20:32, Takashi Yano wrote:
>> As for this patch, read_mtx was introduced. This handle is initialized
>> only for read pipe. However, this seems to be NULL even without
>> initialization in write pipe. I wonder why initializing read_mtx in
>> the constructor is not necessary.
>>
>> How do you guarantee that read_mtx is NULL on the write pipe?
> 
> fhandlers are always calloc'ed on the cygheap, so you don't have
> to initialize anything to NULL.  It doesn't hurt, of course, but
> it's certainly not required.

Takashi and I both asked this question, and you had to answer it twice.  It's 
likely that future readers of the code will ask it again.  Would you be amenable 
to a code cleanup patch that answers it once and for all?  I would suggest (a) 
removing all 0/NULL initializers from fhandler constructors and (b) adding 
comments explaining why they're not needed.

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-08 17:43                                                                                         ` Ken Brown
@ 2021-09-08 18:28                                                                                           ` Corinna Vinschen
  0 siblings, 0 replies; 250+ messages in thread
From: Corinna Vinschen @ 2021-09-08 18:28 UTC (permalink / raw)
  To: cygwin-developers

On Sep  8 13:43, Ken Brown wrote:
> On 9/8/2021 7:55 AM, Corinna Vinschen wrote:
> > On Sep  8 20:32, Takashi Yano wrote:
> > > As for this patch, read_mtx was introduced. This handle is initialized
> > > only for read pipe. However, this seems to be NULL even without
> > > initialization in write pipe. I wonder why initializing read_mtx in
> > > the constructor is not necessary.
> > > 
> > > How do you guarantee that read_mtx is NULL on the write pipe?
> > 
> > fhandlers are always calloc'ed on the cygheap, so you don't have
> > to initialize anything to NULL.  It doesn't hurt, of course, but
> > it's certainly not required.
> 
> Takashi and I both asked this question, and you had to answer it twice.
> It's likely that future readers of the code will ask it again.

Yes, but ... it's not hidden knowledge.  You're using the function
build_fh_dev.  Looking into dtable.cc shows build_fh_dev calls
build_fh_pc which in turn calls fh_alloc.  fh_alloc uses the cnew macro,
defined in the same file, which resolves into

  void* ptr = (void*) ccalloc (HEAP_FHANDLER, 1, sizeof (fhandler_pipe));
  new (ptr) fhandler_pipe (...);

So the constructor is called on a memory slot on the cygheap, allocated
with ccalloc, the cygheap equivalent to calloc on the user heap.

> Would you be
> amenable to a code cleanup patch that answers it once and for all?  I would
> suggest (a) removing all 0/NULL initializers from fhandler constructors and
> (b) adding comments explaining why they're not needed.

a) ok

b) Commenting on the same issue in every single fhandler_* file appears a
   bit excessive, IMHO.  Only one comment in fhandler.h should suffice.


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-08 10:51                               ` Corinna Vinschen
@ 2021-09-09  3:21                                 ` Takashi Yano
  2021-09-09  9:37                                   ` Corinna Vinschen
  0 siblings, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-09  3:21 UTC (permalink / raw)
  To: cygwin-developers

[-- Attachment #1: Type: text/plain, Size: 2284 bytes --]

On Wed, 8 Sep 2021 12:51:31 +0200
Corinna Vinschen wrote:
> On Sep  8 19:45, Takashi Yano wrote:
> > Hi Corinna,
> > 
> > On Wed, 8 Sep 2021 12:04:21 +0200
> > Corinna Vinschen wrote:
> > > On Sep  8 18:45, Takashi Yano wrote:
> > > > On Wed, 8 Sep 2021 11:26:21 +0200
> > > > Corinna Vinschen wrote:
> > > > > On Sep  8 11:01, Corinna Vinschen wrote:
> > > > > > What exactly is the problem in the code which results in high CPU
> > > > > > load?  Can you explain this a bit?  Maybe we need an entirely
> > > > > > different approach to avoid that.
> > > > > 
> > > > > I saw your new patch, but I don't see the problem.  I typed a lot of
> > > > > keys in mintty quickly and what happens is that the load of mintty
> > > > > goes up to 9% on a 4 CPU system, but only temporarily while typing.
> > > > > How do you reproduce the problem?
> > > > 
> > > > Did you apply the patch
> > > > 0001-Cygwin-select-Introduce-select_evt-event-for-pipe.patch
> > > > or
> > > > 0001-Cygwin-select-Introduce-select_sem-semaphore-for-pip.patch
> > > > ?
> > > > 
> > > > With these patch, the problem does not occur. The problem occurs
> > > > with the commit dccde0dc.
> > > 
> > > dccde0dc is 23bb19efcc45 in topic/pipe ATM (after force push) so, yes,
> > > I'm running this with topic/pipe HEAD including this patch.
> > > 
> > > > With my 4 core 8 thread CPU, CPU loads goes up to 12-13 % if
> > > > I type keys using key repeat (30cps) after the commit dccde0dc.
> > > 
> > > Oh, wow!  As I wrote above, before applying "Cygwin: select: Introduce
> > > select_sem semaphore for pipe." I only saw a 9% load.  After applying
> > > the patch I saw the same load.
> > > 
> > > I don't know what I did differently, but after reverting your semaphore
> > > patch I now see loads of up to 30%.  So, never mind, apparently I tested
> > > wrongly before.  Your patch reduces the load tremendously.
> > 
> > Thanks for testing again.
> > 
> > > Just one question.  Would you mind to split your patch into two parts,
> > > one being just the revert of your "Improve select/poll response." patch
> > > and one introducing select_sem?
> > 
> > I split the patch as you advised.
> 
> Pushed.

Timing of select_sem notification is fixed by the patch attached.


-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

[-- Attachment #2: 0001-Cygwin-pipe-Fix-notification-timing-of-select_sem.patch --]
[-- Type: application/octet-stream, Size: 2381 bytes --]

From 56ff64eadfb539a18f4b7d5ffd3355dfb23391ab Mon Sep 17 00:00:00 2001
From: Takashi Yano <takashi.yano@nifty.ne.jp>
Date: Thu, 9 Sep 2021 12:08:39 +0900
Subject: [PATCH] Cygwin: pipe: Fix notification timing of select_sem.

- Make select_sem notify even when read/write partially.
---
 winsup/cygwin/fhandler_pipe.cc | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)

diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index ea8ea41dd..488e14be8 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -311,6 +311,9 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
 	    {
 	      status = STATUS_THREAD_SIGNALED;
 	      nbytes += io.Information;
+	      if (select_sem && io.Information > 0)
+		ReleaseSemaphore (select_sem,
+				  get_obj_handle_count (select_sem), NULL);
 	      break;
 	    }
 	  status = io.Status;
@@ -365,6 +368,8 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
 
       if (nbytes_now == 0)
 	break;
+      else if (select_sem)
+	ReleaseSemaphore (select_sem, get_obj_handle_count (select_sem), NULL);
     }
   ReleaseMutex (read_mtx);
   if (evt)
@@ -376,8 +381,6 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
     }
   else if (status == STATUS_THREAD_CANCELED)
     pthread::static_cancel_self ();
-  if (select_sem && nbytes)
-    ReleaseSemaphore (select_sem, get_obj_handle_count (select_sem), NULL);
   len = nbytes;
 }
 
@@ -472,6 +475,9 @@ fhandler_pipe_fifo::raw_write (const void *ptr, size_t len)
 	    {
 	      status = STATUS_THREAD_SIGNALED;
 	      nbytes += io.Information;
+	      if (select_sem && io.Information > 0)
+		ReleaseSemaphore (select_sem,
+				  get_obj_handle_count (select_sem), NULL);
 	      break;
 	    }
 	  status = io.Status;
@@ -502,6 +508,8 @@ fhandler_pipe_fifo::raw_write (const void *ptr, size_t len)
 
       if (nbytes_now == 0)
 	break;
+      else if (select_sem)
+	ReleaseSemaphore (select_sem, get_obj_handle_count (select_sem), NULL);
     }
   if (evt)
     CloseHandle (evt);
@@ -509,8 +517,6 @@ fhandler_pipe_fifo::raw_write (const void *ptr, size_t len)
     set_errno (EINTR);
   else if (status == STATUS_THREAD_CANCELED)
     pthread::static_cancel_self ();
-  if (select_sem && nbytes)
-    ReleaseSemaphore (select_sem, get_obj_handle_count (select_sem), NULL);
   return nbytes ?: -1;
 }
 
-- 
2.33.0


^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-07  3:26             ` Takashi Yano
  2021-09-07 10:50               ` Takashi Yano
@ 2021-09-09  3:41               ` Takashi Yano
  2021-09-09  8:05                 ` Takashi Yano
  1 sibling, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-09  3:41 UTC (permalink / raw)
  To: cygwin-developers

[-- Attachment #1: Type: text/plain, Size: 3704 bytes --]

Hi Ken,

On Tue, 7 Sep 2021 12:26:31 +0900
Takashi Yano wrote:
> On Fri, 27 Aug 2021 20:24:40 +0900
> Takashi Yano wrote:
> > Hi Ken,
> > 
> > Thanks much! I tested topic/pipe branch.
> > 
> > [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:.
> > yano@linux-server's password:
> > test.dat                                      100%  100MB  95.9MB/s   00:01
> > [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat .
> > yano@linux-server's password:
> > test.dat                                      100%  100MB   8.0MB/s   00:12
> > 
> > yano@linux-server:~$ scp yano@cygwin-PC:test.dat .
> > yano@cygwin-PC's password:
> > test.dat                                      100%  100MB 109.7MB/s   00:00
> > yano@linux-server:~$ scp test.dat yano@cygwin-PC:.
> > yano@cygwin-PC's password:
> > test.dat                                      100%  100MB  31.4MB/s   00:03
> > 
> > As shown above, outgoing transfer-rate has been improved upto near
> > theoretical limit. However, incoming transfer-rate is not improved
> > much.
> > 
> > I digged further and found the first patch attached solves the issue
> > as follows.
> > 
> > [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat .
> > yano@linux-server's password:
> > test.dat                                      100%  100MB 112.8MB/s   00:00
> > 
> > yano@linux-server2:~$ scp test.dat yano@cygwin-PC:.
> > yano@cygwin-PC's password:
> > test.dat                                      100%  100MB 102.5MB/s   00:00
> 
> With this patch (2e36ae2e), I found a problem that mintty gets into
> high load if several keys are typed quickly.
> 
> Therefore, I would like to propose a patch attached.

I found the fifo has same issue as pipe that throughput is
slowing down occasionally.

Please try simple test case attached (fifo_test.c).
Argument of fifo_test means:
0: with select, blocking I/O
1: with select, non-blocking I/O
2: without select, blocking I/O
3: without select, non-blocking I/O

In my environment, the results are as follows.

[yano@Express5800-S70 ~/pipe_test]$ ./fifo_test 0

Total: 100MB in 4.593770 second, 21.768613MB/s
[yano@Express5800-S70 ~/pipe_test]$ ./fifo_test 1

Total: 100MB in 0.564711 second, 177.081603MB/s
[yano@Express5800-S70 ~/pipe_test]$ ./fifo_test 2

Total: 100MB in 0.514730 second, 194.276724MB/s
[yano@Express5800-S70 ~/pipe_test]$ ./fifo_test 3
wwwwwwwwwwwwwwwwwwwwwwwwwwwwwrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr
Total: 100MB in 0.038880 second, 2572.003230MB/s

Therefore, I would like to propose patch attached, which utilizes
select_sem just as pipe.

With the patch attached, the throughput of the fifo is improved
much as follows.

[yano@Express5800-S70 ~/pipe_test]$ ./fifo_test 0

Total: 100MB in 0.076400 second, 1308.895384MB/s
[yano@Express5800-S70 ~/pipe_test]$ ./fifo_test 1

Total: 100MB in 0.064198 second, 1557.683351MB/s
[yano@Express5800-S70 ~/pipe_test]$ ./fifo_test 2

Total: 100MB in 0.021803 second, 4586.503754MB/s
[yano@Express5800-S70 ~/pipe_test]$ ./fifo_test 3
wwwwwwrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr
Total: 100MB in 0.021747 second, 4598.271969MB/s


-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

[-- Attachment #2: fifo_test.c --]
[-- Type: text/x-csrc, Size: 2228 bytes --]

#include <unistd.h>
#include <stdlib.h>
#include <sys/wait.h>
#include <stdio.h>
#include <time.h>
#include <sys/select.h>
#include <fcntl.h>
#include <errno.h>

#define FIFO_NAME "/tmp/fifo1"
#define BLKSIZ 65536
#define NBLK (100*(1<<20)/BLKSIZ)


int main(int argc, char *argv[])
{
	pid_t pid;
	int nonblock = 0;
	int skip_select = 0;

	if (argc >= 2 && (atoi(argv[1]) & 1)) nonblock = 1;
	if (argc >= 2 && (atoi(argv[1]) & 2)) skip_select = 1;

	mkfifo("/tmp/fifo1", 0644);

	if (!(pid = fork ())) {
		int fifo;
		char buf[BLKSIZ] = {0,};
		int i;
		fifo = open(FIFO_NAME, O_WRONLY);
		if (nonblock) {
			int flags;
			flags = fcntl(fifo, F_GETFL);
			flags |= O_NONBLOCK;
			fcntl(fifo, F_SETFL, flags);
		}
		fd_set wfds;
		for (i=0; i<NBLK; i++) {
			int total = 0;
			while (total < sizeof(buf)) {
				FD_ZERO(&wfds);
				FD_SET(fifo, &wfds);
				if (skip_select || select(fifo+1, NULL, &wfds, NULL, NULL) > 0
						&& FD_ISSET(fifo, &wfds)) {
					errno = 0;
					ssize_t len = write(fifo, buf + total, sizeof(buf) - total);
					if (len <= 0 && errno == EAGAIN) printf("w", i);
					if (len > 0) total += len;
				}
			}
		}
		close(fifo);
	} else {
		int fifo;
		char buf[BLKSIZ] = {0,};
		int total = 0;
		fd_set rfds;
		struct timespec tv0, tv1;
		double elasped;
		fifo = open(FIFO_NAME, O_RDONLY);
		if (nonblock) {
			int flags;
			flags = fcntl(fifo, F_GETFL);
			flags |= O_NONBLOCK;
			fcntl(fifo, F_SETFL, flags);
		}
		clock_gettime(CLOCK_MONOTONIC, &tv0);
		for (;;) {
			FD_ZERO(&rfds);
			FD_SET(fifo, &rfds);
			if (skip_select || select(fifo+1, &rfds, NULL, NULL, NULL) > 0
					&& FD_ISSET(fifo, &rfds)) {
				errno = 0;
				ssize_t len = read(fifo, buf, sizeof(buf));
				if (len <= 0 && errno == EAGAIN) printf("r");
				else if(len <= 0) break;
				else total += len;
			}
		}
		clock_gettime(CLOCK_MONOTONIC, &tv1);
		close(fifo);
		elasped = (tv1.tv_sec - tv0.tv_sec) + (tv1.tv_nsec - tv0.tv_nsec)*1e-9;
		printf("\nTotal: %dMB in %f second, %fMB/s\n",
			total/(1<<20), elasped, total/(1<<20)/elasped);
		if (total != NBLK*BLKSIZ) {
			printf("Error: %d/%d (%f)\n",
				total, NBLK*BLKSIZ, (double)(total-NBLK*BLKSIZ)/BLKSIZ);
		}
		waitpid(pid, NULL, 0);
		unlink(FIFO_NAME);
	}

	return 0;
}


[-- Attachment #3: 0001-Cygwin-fifo-Utilize-select_sem-for-fifo-as-well-as-p.patch --]
[-- Type: application/octet-stream, Size: 4455 bytes --]

From beeae61f97145d795c4479b0d46cd3b75d6256c4 Mon Sep 17 00:00:00 2001
From: Takashi Yano <takashi.yano@nifty.ne.jp>
Date: Thu, 9 Sep 2021 11:53:11 +0900
Subject: [PATCH] Cygwin: fifo: Utilize select_sem for fifo as well as pipe.

---
 winsup/cygwin/fhandler_fifo.cc | 37 +++++++++++++++++++++++++++++++++-
 winsup/cygwin/select.cc        | 10 +++++++--
 2 files changed, 44 insertions(+), 3 deletions(-)

diff --git a/winsup/cygwin/fhandler_fifo.cc b/winsup/cygwin/fhandler_fifo.cc
index 6709fb974..c40573783 100644
--- a/winsup/cygwin/fhandler_fifo.cc
+++ b/winsup/cygwin/fhandler_fifo.cc
@@ -1047,6 +1047,12 @@ writer_shmem:
   ResetEvent (writer_opening);
   nwriters_unlock ();
 success:
+  if (!select_sem)
+    {
+      char name[MAX_PATH];
+      __small_sprintf(name, "semaphore-%W", get_pipe_name ()->Buffer);
+      select_sem = CreateSemaphore (&sec_none, 0, INT32_MAX, name);
+    }
   return 1;
 err_close_reader:
   saved_errno = get_errno ();
@@ -1235,6 +1241,10 @@ fhandler_fifo::raw_read (void *in_ptr, size_t& len)
 		  len = io.Information;
 		  fifo_client_unlock ();
 		  reading_unlock ();
+		  if (select_sem)
+		    ReleaseSemaphore (select_sem,
+				      get_obj_handle_count (select_sem),
+				      NULL);
 		  return;
 		}
 	      break;
@@ -1273,6 +1283,10 @@ fhandler_fifo::raw_read (void *in_ptr, size_t& len)
 		    fc_handler[i].last_read = true;
 		    fifo_client_unlock ();
 		    reading_unlock ();
+		    if (select_sem)
+		      ReleaseSemaphore (select_sem,
+					get_obj_handle_count (select_sem),
+					NULL);
 		    return;
 		  }
 		break;
@@ -1312,6 +1326,10 @@ fhandler_fifo::raw_read (void *in_ptr, size_t& len)
 		    fc_handler[i].last_read = true;
 		    fifo_client_unlock ();
 		    reading_unlock ();
+		    if (select_sem)
+		      ReleaseSemaphore (select_sem,
+					get_obj_handle_count (select_sem),
+					NULL);
 		    return;
 		  }
 		break;
@@ -1345,7 +1363,7 @@ maybe_retry:
       else
 	{
 	  /* Allow interruption and don't hog the CPU. */
-	  DWORD waitret = cygwait (NULL, 1, cw_cancel | cw_sig_eintr);
+	  DWORD waitret = cygwait (select_sem, 1, cw_cancel | cw_sig_eintr);
 	  if (waitret == WAIT_CANCELED)
 	    pthread::static_cancel_self ();
 	  else if (waitret == WAIT_SIGNALED)
@@ -1569,6 +1587,11 @@ fhandler_fifo::close ()
     NtClose (write_ready);
   if (writer_opening)
     NtClose (writer_opening);
+  if (select_sem)
+    {
+      ReleaseSemaphore (select_sem, get_obj_handle_count (select_sem), NULL);
+      NtClose (select_sem);
+    }
   if (nohandle ())
     return 0;
   else
@@ -1683,7 +1706,17 @@ fhandler_fifo::dup (fhandler_base *child, int flags)
     }
   if (writer)
     inc_nwriters ();
+  if (select_sem &&
+      !DuplicateHandle (GetCurrentProcess (), select_sem,
+			GetCurrentProcess (), &fhf->select_sem,
+			0, !(flags & O_CLOEXEC), DUPLICATE_SAME_ACCESS))
+    {
+      __seterrno ();
+      goto err_close_thr_sync_evt;
+    }
   return 0;
+err_close_thr_sync_evt:
+  NtClose (fhf->thr_sync_evt);
 err_close_cancel_evt:
   NtClose (fhf->cancel_evt);
 err_close_update_needed_evt:
@@ -1743,6 +1776,8 @@ fhandler_fifo::fixup_after_fork (HANDLE parent)
       me.winpid = GetCurrentProcessId ();
       new cygthread (fifo_reader_thread, this, "fifo_reader", thr_sync_evt);
     }
+  if (select_sem)
+    fork_fixup (parent, select_sem, "select_sem");
   if (writer)
     inc_nwriters ();
 }
diff --git a/winsup/cygwin/select.cc b/winsup/cygwin/select.cc
index e9e71b269..5e583434c 100644
--- a/winsup/cygwin/select.cc
+++ b/winsup/cygwin/select.cc
@@ -963,7 +963,13 @@ start_thread_fifo (select_record *me, select_stuff *stuff)
     {
       pi->start = &stuff->start;
       pi->stop_thread = false;
-      pi->bye = CreateEvent (&sec_none_nih, TRUE, FALSE, NULL);
+      pi->bye = me->fh->get_select_sem ();
+      if (pi->bye)
+	DuplicateHandle (GetCurrentProcess (), pi->bye,
+			 GetCurrentProcess (), &pi->bye,
+			 0, 0, DUPLICATE_SAME_ACCESS);
+      else
+	pi->bye = CreateSemaphore (&sec_none_nih, 0, INT32_MAX, NULL);
       pi->thread = new cygthread (thread_fifo, pi, "fifosel");
       me->h = *pi->thread;
       if (!me->h)
@@ -981,7 +987,7 @@ fifo_cleanup (select_record *, select_stuff *stuff)
   if (pi->thread)
     {
       pi->stop_thread = true;
-      SetEvent (pi->bye);
+      ReleaseSemaphore (pi->bye, get_obj_handle_count (pi->bye), NULL);
       pi->thread->detach ();
       CloseHandle (pi->bye);
     }
-- 
2.33.0


^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-09  3:41               ` Takashi Yano
@ 2021-09-09  8:05                 ` Takashi Yano
  2021-09-09 12:19                   ` Takashi Yano
  0 siblings, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-09  8:05 UTC (permalink / raw)
  To: cygwin-developers

On Thu, 9 Sep 2021 12:41:15 +0900
Takashi Yano wrote:
> diff --git a/winsup/cygwin/fhandler_fifo.cc b/winsup/cygwin/fhandler_fifo.cc
> index 6709fb974..c40573783 100644
> --- a/winsup/cygwin/fhandler_fifo.cc
> +++ b/winsup/cygwin/fhandler_fifo.cc
> @@ -1047,6 +1047,12 @@ writer_shmem:
>    ResetEvent (writer_opening);
>    nwriters_unlock ();
>  success:
> +  if (!select_sem)
> +    {
> +      char name[MAX_PATH];
> +      __small_sprintf(name, "semaphore-%W", get_pipe_name ()->Buffer);
> +      select_sem = CreateSemaphore (&sec_none, 0, INT32_MAX, name);
> +    }
>    return 1;
>  err_close_reader:
>    saved_errno = get_errno ();

Should this be:
> +      select_sem = CreateSemaphore (sa_buf, 0, INT32_MAX, name);
?

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-09  3:21                                 ` Takashi Yano
@ 2021-09-09  9:37                                   ` Corinna Vinschen
  2021-09-09 10:55                                     ` Takashi Yano
  0 siblings, 1 reply; 250+ messages in thread
From: Corinna Vinschen @ 2021-09-09  9:37 UTC (permalink / raw)
  To: cygwin-developers

[-- Attachment #1: Type: text/plain, Size: 1200 bytes --]

On Sep  9 12:21, Takashi Yano wrote:
> On Wed, 8 Sep 2021 12:51:31 +0200
> Corinna Vinschen wrote:
> > > > Just one question.  Would you mind to split your patch into two parts,
> > > > one being just the revert of your "Improve select/poll response." patch
> > > > and one introducing select_sem?
> > > 
> > > I split the patch as you advised.
> > 
> > Pushed.
> 
> Timing of select_sem notification is fixed by the patch attached.

If I'm not entirely off-track, I think this isn't quite right yet.
I assume you want to release the semaphore in all cases some bytes
have been read or written, right?

If so, this should cover the STATUS_THREAD_CANCELED and
STATUS_BUFFER_OVERFLOW cases as well.  But then again, the
ReleaseSemaphore calls are a bit spread out over the calls.

I took the liberty to create a followup patch to the attached one.  It
merges all cases potentially reading or writing some bytes into a single
if branch, so only a single ReleaseSemaphore should be required.  I
dropped STATUS_MORE_ENTRIES because it's not an error, subsumed under
NT_SUCCESS, and IIUC, never emitted by the underlying code.

I attached the patch to this mail, can you please check it?


Thanks,
Corinna

[-- Attachment #2: 0001-Cygwin-pipes-always-signal-select_sem-if-any-bytes-a.patch --]
[-- Type: text/patch, Size: 4540 bytes --]

From 11bea88b788aabb762b4e972a7f3fb2423fb32d8 Mon Sep 17 00:00:00 2001
From: Corinna Vinschen <corinna@vinschen.de>
Date: Thu, 9 Sep 2021 11:35:54 +0200
Subject: [PATCH] Cygwin: pipes: always signal select_sem if any bytes are read
 or written

Fold all code branches potentially having read or written data into
a single if branch, so signalling select_sem catches all cases.

Signed-off-by: Corinna Vinschen <corinna@vinschen.de>
---
 winsup/cygwin/fhandler_pipe.cc | 64 ++++++++++++----------------------
 1 file changed, 23 insertions(+), 41 deletions(-)

diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index 488e14be8c2b..6994a5dceb84 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -303,20 +303,11 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
 	    waitret = WAIT_OBJECT_0;
 
 	  if (waitret == WAIT_CANCELED)
-	    {
-	      status = STATUS_THREAD_CANCELED;
-	      break;
-	    }
+	    status = STATUS_THREAD_CANCELED;
 	  else if (waitret == WAIT_SIGNALED)
-	    {
-	      status = STATUS_THREAD_SIGNALED;
-	      nbytes += io.Information;
-	      if (select_sem && io.Information > 0)
-		ReleaseSemaphore (select_sem,
-				  get_obj_handle_count (select_sem), NULL);
-	      break;
-	    }
-	  status = io.Status;
+	    status = STATUS_THREAD_SIGNALED;
+	  else
+	    status = io.Status;
 	}
       if (isclosed ())  /* A signal handler might have closed the fd. */
 	{
@@ -326,11 +317,17 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
 	    __seterrno ();
 	  nbytes = (size_t) -1;
 	}
-      else if (NT_SUCCESS (status))
+      else if (NT_SUCCESS (status)
+	       || status == STATUS_BUFFER_OVERFLOW
+	       || status == STATUS_THREAD_CANCELED
+	       || status == STATUS_THREAD_SIGNALED)
 	{
 	  nbytes_now = io.Information;
 	  ptr = ((char *) ptr) + nbytes_now;
 	  nbytes += nbytes_now;
+	  if (select_sem && nbytes_now > 0)
+	    ReleaseSemaphore (select_sem,
+			      get_obj_handle_count (select_sem), NULL);
 	}
       else
 	{
@@ -341,13 +338,6 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
 	    case STATUS_PIPE_BROKEN:
 	      /* This is really EOF.  */
 	      break;
-	    case STATUS_MORE_ENTRIES:
-	    case STATUS_BUFFER_OVERFLOW:
-	      /* `io.Information' is supposedly valid.  */
-	      nbytes_now = io.Information;
-	      ptr = ((char *) ptr) + nbytes_now;
-	      nbytes += nbytes_now;
-	      break;
 	    case STATUS_PIPE_LISTENING:
 	    case STATUS_PIPE_EMPTY:
 	      if (nbytes != 0)
@@ -366,10 +356,8 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
 	    }
 	}
 
-      if (nbytes_now == 0)
+      if (nbytes_now == 0 || status == STATUS_BUFFER_OVERFLOW)
 	break;
-      else if (select_sem)
-	ReleaseSemaphore (select_sem, get_obj_handle_count (select_sem), NULL);
     }
   ReleaseMutex (read_mtx);
   if (evt)
@@ -467,20 +455,11 @@ fhandler_pipe_fifo::raw_write (const void *ptr, size_t len)
 	    waitret = WAIT_OBJECT_0;
 
 	  if (waitret == WAIT_CANCELED)
-	    {
-	      status = STATUS_THREAD_CANCELED;
-	      break;
-	    }
+	    status = STATUS_THREAD_CANCELED;
 	  else if (waitret == WAIT_SIGNALED)
-	    {
-	      status = STATUS_THREAD_SIGNALED;
-	      nbytes += io.Information;
-	      if (select_sem && io.Information > 0)
-		ReleaseSemaphore (select_sem,
-				  get_obj_handle_count (select_sem), NULL);
-	      break;
-	    }
-	  status = io.Status;
+	    status = STATUS_THREAD_SIGNALED;
+	  else
+	    status = io.Status;
 	}
       if (isclosed ())  /* A signal handler might have closed the fd. */
 	{
@@ -489,13 +468,18 @@ fhandler_pipe_fifo::raw_write (const void *ptr, size_t len)
 	  else
 	    __seterrno ();
 	}
-      else if (NT_SUCCESS (status))
+      else if (NT_SUCCESS (status)
+	       || status == STATUS_THREAD_CANCELED
+	       || status == STATUS_THREAD_SIGNALED)
 	{
 	  nbytes_now = io.Information;
 	  ptr = ((char *) ptr) + nbytes_now;
 	  nbytes += nbytes_now;
+	  if (select_sem && nbytes_now > 0)
+	    ReleaseSemaphore (select_sem,
+			      get_obj_handle_count (select_sem), NULL);
 	  /* 0 bytes returned?  EAGAIN.  See above. */
-	  if (nbytes == 0)
+	  if (NT_SUCCESS (status) && nbytes == 0)
 	    set_errno (EAGAIN);
 	}
       else if (STATUS_PIPE_IS_CLOSED (status))
@@ -508,8 +492,6 @@ fhandler_pipe_fifo::raw_write (const void *ptr, size_t len)
 
       if (nbytes_now == 0)
 	break;
-      else if (select_sem)
-	ReleaseSemaphore (select_sem, get_obj_handle_count (select_sem), NULL);
     }
   if (evt)
     CloseHandle (evt);
-- 
2.31.1


^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-09  9:37                                   ` Corinna Vinschen
@ 2021-09-09 10:55                                     ` Takashi Yano
  2021-09-09 11:41                                       ` Corinna Vinschen
  0 siblings, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-09 10:55 UTC (permalink / raw)
  To: cygwin-developers

Hi Corinna,

On Thu, 9 Sep 2021 11:37:06 +0200
Corinna Vinschen wrote:
> On Sep  9 12:21, Takashi Yano wrote:
> > On Wed, 8 Sep 2021 12:51:31 +0200
> > Corinna Vinschen wrote:
> > > > > Just one question.  Would you mind to split your patch into two parts,
> > > > > one being just the revert of your "Improve select/poll response." patch
> > > > > and one introducing select_sem?
> > > > 
> > > > I split the patch as you advised.
> > > 
> > > Pushed.
> > 
> > Timing of select_sem notification is fixed by the patch attached.
> 
> If I'm not entirely off-track, I think this isn't quite right yet.
> I assume you want to release the semaphore in all cases some bytes
> have been read or written, right?

Right.

> If so, this should cover the STATUS_THREAD_CANCELED and
> STATUS_BUFFER_OVERFLOW cases as well.  But then again, the
> ReleaseSemaphore calls are a bit spread out over the calls.
> 
> I took the liberty to create a followup patch to the attached one.  It
> merges all cases potentially reading or writing some bytes into a single
> if branch, so only a single ReleaseSemaphore should be required.  I
> dropped STATUS_MORE_ENTRIES because it's not an error, subsumed under
> NT_SUCCESS, and IIUC, never emitted by the underlying code.
> 
> I attached the patch to this mail, can you please check it?

Thanks for the patch. LGTM. I also confirmed that your patch
solves the problem which I wanted to fix.

Thanks again.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-09 10:55                                     ` Takashi Yano
@ 2021-09-09 11:41                                       ` Corinna Vinschen
  0 siblings, 0 replies; 250+ messages in thread
From: Corinna Vinschen @ 2021-09-09 11:41 UTC (permalink / raw)
  To: cygwin-developers

On Sep  9 19:55, Takashi Yano wrote:
> Hi Corinna,
> 
> On Thu, 9 Sep 2021 11:37:06 +0200
> Corinna Vinschen wrote:
> > On Sep  9 12:21, Takashi Yano wrote:
> > > On Wed, 8 Sep 2021 12:51:31 +0200
> > > Corinna Vinschen wrote:
> > > > > > Just one question.  Would you mind to split your patch into two parts,
> > > > > > one being just the revert of your "Improve select/poll response." patch
> > > > > > and one introducing select_sem?
> > > > > 
> > > > > I split the patch as you advised.
> > > > 
> > > > Pushed.
> > > 
> > > Timing of select_sem notification is fixed by the patch attached.
> > 
> > If I'm not entirely off-track, I think this isn't quite right yet.
> > I assume you want to release the semaphore in all cases some bytes
> > have been read or written, right?
> 
> Right.
> 
> > If so, this should cover the STATUS_THREAD_CANCELED and
> > STATUS_BUFFER_OVERFLOW cases as well.  But then again, the
> > ReleaseSemaphore calls are a bit spread out over the calls.
> > 
> > I took the liberty to create a followup patch to the attached one.  It
> > merges all cases potentially reading or writing some bytes into a single
> > if branch, so only a single ReleaseSemaphore should be required.  I
> > dropped STATUS_MORE_ENTRIES because it's not an error, subsumed under
> > NT_SUCCESS, and IIUC, never emitted by the underlying code.
> > 
> > I attached the patch to this mail, can you please check it?
> 
> Thanks for the patch. LGTM. I also confirmed that your patch
> solves the problem which I wanted to fix.

Great, I pushed both patches.


Thanks,
Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-09  8:05                 ` Takashi Yano
@ 2021-09-09 12:19                   ` Takashi Yano
  2021-09-09 12:42                     ` Takashi Yano
  0 siblings, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-09 12:19 UTC (permalink / raw)
  To: cygwin-developers

[-- Attachment #1: Type: text/plain, Size: 901 bytes --]

On Thu, 9 Sep 2021 17:05:49 +0900
Takashi Yano wrote:
> On Thu, 9 Sep 2021 12:41:15 +0900
> Takashi Yano wrote:
> > diff --git a/winsup/cygwin/fhandler_fifo.cc b/winsup/cygwin/fhandler_fifo.cc
> > index 6709fb974..c40573783 100644
> > --- a/winsup/cygwin/fhandler_fifo.cc
> > +++ b/winsup/cygwin/fhandler_fifo.cc
> > @@ -1047,6 +1047,12 @@ writer_shmem:
> >    ResetEvent (writer_opening);
> >    nwriters_unlock ();
> >  success:
> > +  if (!select_sem)
> > +    {
> > +      char name[MAX_PATH];
> > +      __small_sprintf(name, "semaphore-%W", get_pipe_name ()->Buffer);
> > +      select_sem = CreateSemaphore (&sec_none, 0, INT32_MAX, name);
> > +    }
> >    return 1;
> >  err_close_reader:
> >    saved_errno = get_errno ();
> 
> Should this be:
> > +      select_sem = CreateSemaphore (sa_buf, 0, INT32_MAX, name);
> ?

I revised the patch a bit.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

[-- Attachment #2: 0001-Cygwin-fifo-Utilize-select_sem-for-fifo-as-well-as-p.patch --]
[-- Type: application/octet-stream, Size: 4434 bytes --]

From b7be5fb11bac6258e1ac01eccfcfcf2401da1225 Mon Sep 17 00:00:00 2001
From: Takashi Yano <takashi.yano@nifty.ne.jp>
Date: Thu, 9 Sep 2021 17:16:50 +0900
Subject: [PATCH] Cygwin: fifo: Utilize select_sem for fifo as well as pipe.

---
 winsup/cygwin/fhandler_fifo.cc | 36 +++++++++++++++++++++++++++++++++-
 winsup/cygwin/select.cc        | 10 ++++++++--
 2 files changed, 43 insertions(+), 3 deletions(-)

diff --git a/winsup/cygwin/fhandler_fifo.cc b/winsup/cygwin/fhandler_fifo.cc
index 6709fb974..a6fcc38f4 100644
--- a/winsup/cygwin/fhandler_fifo.cc
+++ b/winsup/cygwin/fhandler_fifo.cc
@@ -1047,6 +1047,11 @@ writer_shmem:
   ResetEvent (writer_opening);
   nwriters_unlock ();
 success:
+  if (!select_sem)
+    {
+      __small_sprintf (npbuf, "semaphore.%08x.%016X", get_dev (), get_ino ());
+      select_sem = CreateSemaphore (sa_buf, 0, INT32_MAX, npbuf);
+    }
   return 1;
 err_close_reader:
   saved_errno = get_errno ();
@@ -1235,6 +1240,10 @@ fhandler_fifo::raw_read (void *in_ptr, size_t& len)
 		  len = io.Information;
 		  fifo_client_unlock ();
 		  reading_unlock ();
+		  if (select_sem)
+		    ReleaseSemaphore (select_sem,
+				      get_obj_handle_count (select_sem),
+				      NULL);
 		  return;
 		}
 	      break;
@@ -1273,6 +1282,10 @@ fhandler_fifo::raw_read (void *in_ptr, size_t& len)
 		    fc_handler[i].last_read = true;
 		    fifo_client_unlock ();
 		    reading_unlock ();
+		    if (select_sem)
+		      ReleaseSemaphore (select_sem,
+					get_obj_handle_count (select_sem),
+					NULL);
 		    return;
 		  }
 		break;
@@ -1312,6 +1325,10 @@ fhandler_fifo::raw_read (void *in_ptr, size_t& len)
 		    fc_handler[i].last_read = true;
 		    fifo_client_unlock ();
 		    reading_unlock ();
+		    if (select_sem)
+		      ReleaseSemaphore (select_sem,
+					get_obj_handle_count (select_sem),
+					NULL);
 		    return;
 		  }
 		break;
@@ -1345,7 +1362,7 @@ maybe_retry:
       else
 	{
 	  /* Allow interruption and don't hog the CPU. */
-	  DWORD waitret = cygwait (NULL, 1, cw_cancel | cw_sig_eintr);
+	  DWORD waitret = cygwait (select_sem, 1, cw_cancel | cw_sig_eintr);
 	  if (waitret == WAIT_CANCELED)
 	    pthread::static_cancel_self ();
 	  else if (waitret == WAIT_SIGNALED)
@@ -1569,6 +1586,11 @@ fhandler_fifo::close ()
     NtClose (write_ready);
   if (writer_opening)
     NtClose (writer_opening);
+  if (select_sem)
+    {
+      ReleaseSemaphore (select_sem, get_obj_handle_count (select_sem), NULL);
+      NtClose (select_sem);
+    }
   if (nohandle ())
     return 0;
   else
@@ -1683,7 +1705,17 @@ fhandler_fifo::dup (fhandler_base *child, int flags)
     }
   if (writer)
     inc_nwriters ();
+  if (select_sem &&
+      !DuplicateHandle (GetCurrentProcess (), select_sem,
+			GetCurrentProcess (), &fhf->select_sem,
+			0, !(flags & O_CLOEXEC), DUPLICATE_SAME_ACCESS))
+    {
+      __seterrno ();
+      goto err_close_thr_sync_evt;
+    }
   return 0;
+err_close_thr_sync_evt:
+  NtClose (fhf->thr_sync_evt);
 err_close_cancel_evt:
   NtClose (fhf->cancel_evt);
 err_close_update_needed_evt:
@@ -1743,6 +1775,8 @@ fhandler_fifo::fixup_after_fork (HANDLE parent)
       me.winpid = GetCurrentProcessId ();
       new cygthread (fifo_reader_thread, this, "fifo_reader", thr_sync_evt);
     }
+  if (select_sem)
+    fork_fixup (parent, select_sem, "select_sem");
   if (writer)
     inc_nwriters ();
 }
diff --git a/winsup/cygwin/select.cc b/winsup/cygwin/select.cc
index e9e71b269..5e583434c 100644
--- a/winsup/cygwin/select.cc
+++ b/winsup/cygwin/select.cc
@@ -963,7 +963,13 @@ start_thread_fifo (select_record *me, select_stuff *stuff)
     {
       pi->start = &stuff->start;
       pi->stop_thread = false;
-      pi->bye = CreateEvent (&sec_none_nih, TRUE, FALSE, NULL);
+      pi->bye = me->fh->get_select_sem ();
+      if (pi->bye)
+	DuplicateHandle (GetCurrentProcess (), pi->bye,
+			 GetCurrentProcess (), &pi->bye,
+			 0, 0, DUPLICATE_SAME_ACCESS);
+      else
+	pi->bye = CreateSemaphore (&sec_none_nih, 0, INT32_MAX, NULL);
       pi->thread = new cygthread (thread_fifo, pi, "fifosel");
       me->h = *pi->thread;
       if (!me->h)
@@ -981,7 +987,7 @@ fifo_cleanup (select_record *, select_stuff *stuff)
   if (pi->thread)
     {
       pi->stop_thread = true;
-      SetEvent (pi->bye);
+      ReleaseSemaphore (pi->bye, get_obj_handle_count (pi->bye), NULL);
       pi->thread->detach ();
       CloseHandle (pi->bye);
     }
-- 
2.33.0


^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-09 12:19                   ` Takashi Yano
@ 2021-09-09 12:42                     ` Takashi Yano
  2021-09-09 21:53                       ` Takashi Yano
  2021-09-10 10:57                       ` Ken Brown
  0 siblings, 2 replies; 250+ messages in thread
From: Takashi Yano @ 2021-09-09 12:42 UTC (permalink / raw)
  To: cygwin-developers

[-- Attachment #1: Type: text/plain, Size: 1044 bytes --]

On Thu, 9 Sep 2021 21:19:40 +0900
Takashi Yano wrote:
> On Thu, 9 Sep 2021 17:05:49 +0900
> Takashi Yano wrote:
> > On Thu, 9 Sep 2021 12:41:15 +0900
> > Takashi Yano wrote:
> > > diff --git a/winsup/cygwin/fhandler_fifo.cc b/winsup/cygwin/fhandler_fifo.cc
> > > index 6709fb974..c40573783 100644
> > > --- a/winsup/cygwin/fhandler_fifo.cc
> > > +++ b/winsup/cygwin/fhandler_fifo.cc
> > > @@ -1047,6 +1047,12 @@ writer_shmem:
> > >    ResetEvent (writer_opening);
> > >    nwriters_unlock ();
> > >  success:
> > > +  if (!select_sem)
> > > +    {
> > > +      char name[MAX_PATH];
> > > +      __small_sprintf(name, "semaphore-%W", get_pipe_name ()->Buffer);
> > > +      select_sem = CreateSemaphore (&sec_none, 0, INT32_MAX, name);
> > > +    }
> > >    return 1;
> > >  err_close_reader:
> > >    saved_errno = get_errno ();
> > 
> > Should this be:
> > > +      select_sem = CreateSemaphore (sa_buf, 0, INT32_MAX, name);
> > ?
> 
> I revised the patch a bit.

Sorry, I revised the patch again.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

[-- Attachment #2: 0001-Cygwin-fifo-Utilize-select_sem-for-fifo-as-well-as-p.patch --]
[-- Type: application/octet-stream, Size: 5062 bytes --]

From ba92a1ffb1c9b9bdacc363444f27fd328137fefd Mon Sep 17 00:00:00 2001
From: Takashi Yano <takashi.yano@nifty.ne.jp>
Date: Thu, 9 Sep 2021 21:40:37 +0900
Subject: [PATCH] Cygwin: fifo: Utilize select_sem for fifo as well as pipe.

---
 winsup/cygwin/fhandler_fifo.cc | 38 ++++++++++++++++++++++++++++++++--
 winsup/cygwin/select.cc        | 10 +++++++--
 2 files changed, 44 insertions(+), 4 deletions(-)

diff --git a/winsup/cygwin/fhandler_fifo.cc b/winsup/cygwin/fhandler_fifo.cc
index 6709fb974..6de9a229b 100644
--- a/winsup/cygwin/fhandler_fifo.cc
+++ b/winsup/cygwin/fhandler_fifo.cc
@@ -1047,6 +1047,11 @@ writer_shmem:
   ResetEvent (writer_opening);
   nwriters_unlock ();
 success:
+  if (!select_sem)
+    {
+      __small_sprintf (npbuf, "semaphore.%08x.%016X", get_dev (), get_ino ());
+      select_sem = CreateSemaphore (sa_buf, 0, INT32_MAX, npbuf);
+    }
   return 1;
 err_close_reader:
   saved_errno = get_errno ();
@@ -1235,6 +1240,10 @@ fhandler_fifo::raw_read (void *in_ptr, size_t& len)
 		  len = io.Information;
 		  fifo_client_unlock ();
 		  reading_unlock ();
+		  if (select_sem)
+		    ReleaseSemaphore (select_sem,
+				      get_obj_handle_count (select_sem),
+				      NULL);
 		  return;
 		}
 	      break;
@@ -1273,6 +1282,10 @@ fhandler_fifo::raw_read (void *in_ptr, size_t& len)
 		    fc_handler[i].last_read = true;
 		    fifo_client_unlock ();
 		    reading_unlock ();
+		    if (select_sem)
+		      ReleaseSemaphore (select_sem,
+					get_obj_handle_count (select_sem),
+					NULL);
 		    return;
 		  }
 		break;
@@ -1312,6 +1325,10 @@ fhandler_fifo::raw_read (void *in_ptr, size_t& len)
 		    fc_handler[i].last_read = true;
 		    fifo_client_unlock ();
 		    reading_unlock ();
+		    if (select_sem)
+		      ReleaseSemaphore (select_sem,
+					get_obj_handle_count (select_sem),
+					NULL);
 		    return;
 		  }
 		break;
@@ -1345,7 +1362,7 @@ maybe_retry:
       else
 	{
 	  /* Allow interruption and don't hog the CPU. */
-	  DWORD waitret = cygwait (NULL, 1, cw_cancel | cw_sig_eintr);
+	  DWORD waitret = cygwait (select_sem, 1, cw_cancel | cw_sig_eintr);
 	  if (waitret == WAIT_CANCELED)
 	    pthread::static_cancel_self ();
 	  else if (waitret == WAIT_SIGNALED)
@@ -1569,6 +1586,11 @@ fhandler_fifo::close ()
     NtClose (write_ready);
   if (writer_opening)
     NtClose (writer_opening);
+  if (select_sem)
+    {
+      ReleaseSemaphore (select_sem, get_obj_handle_count (select_sem), NULL);
+      NtClose (select_sem);
+    }
   if (nohandle ())
     return 0;
   else
@@ -1632,6 +1654,14 @@ fhandler_fifo::dup (fhandler_base *child, int flags)
       __seterrno ();
       goto err_close_writer_opening;
     }
+  if (select_sem &&
+      !DuplicateHandle (GetCurrentProcess (), select_sem,
+			GetCurrentProcess (), &fhf->select_sem,
+			0, !(flags & O_CLOEXEC), DUPLICATE_SAME_ACCESS))
+    {
+      __seterrno ();
+      goto err_close_shmem;
+    }
   if (fhf->reopen_shmem () < 0)
     goto err_close_shmem_handle;
   if (reader)
@@ -1648,7 +1678,7 @@ fhandler_fifo::dup (fhandler_base *child, int flags)
 			    0, !(flags & O_CLOEXEC), DUPLICATE_SAME_ACCESS))
 	{
 	  __seterrno ();
-	  goto err_close_shmem;
+	  goto err_close_select_sem;
 	}
       if (fhf->reopen_shared_fc_handler () < 0)
 	goto err_close_shared_fc_hdl;
@@ -1696,6 +1726,8 @@ err_close_shared_fc_handler:
   NtUnmapViewOfSection (GetCurrentProcess (), fhf->shared_fc_handler);
 err_close_shared_fc_hdl:
   NtClose (fhf->shared_fc_hdl);
+err_close_select_sem:
+  NtClose (fhf->select_sem);
 err_close_shmem:
   NtUnmapViewOfSection (GetCurrentProcess (), fhf->shmem);
 err_close_shmem_handle:
@@ -1720,6 +1752,8 @@ fhandler_fifo::fixup_after_fork (HANDLE parent)
   fork_fixup (parent, shmem_handle, "shmem_handle");
   if (reopen_shmem () < 0)
     api_fatal ("Can't reopen shared memory during fork, %E");
+  if (select_sem)
+    fork_fixup (parent, select_sem, "select_sem");
   if (reader)
     {
       /* Make sure the child starts unlocked. */
diff --git a/winsup/cygwin/select.cc b/winsup/cygwin/select.cc
index e9e71b269..5e583434c 100644
--- a/winsup/cygwin/select.cc
+++ b/winsup/cygwin/select.cc
@@ -963,7 +963,13 @@ start_thread_fifo (select_record *me, select_stuff *stuff)
     {
       pi->start = &stuff->start;
       pi->stop_thread = false;
-      pi->bye = CreateEvent (&sec_none_nih, TRUE, FALSE, NULL);
+      pi->bye = me->fh->get_select_sem ();
+      if (pi->bye)
+	DuplicateHandle (GetCurrentProcess (), pi->bye,
+			 GetCurrentProcess (), &pi->bye,
+			 0, 0, DUPLICATE_SAME_ACCESS);
+      else
+	pi->bye = CreateSemaphore (&sec_none_nih, 0, INT32_MAX, NULL);
       pi->thread = new cygthread (thread_fifo, pi, "fifosel");
       me->h = *pi->thread;
       if (!me->h)
@@ -981,7 +987,7 @@ fifo_cleanup (select_record *, select_stuff *stuff)
   if (pi->thread)
     {
       pi->stop_thread = true;
-      SetEvent (pi->bye);
+      ReleaseSemaphore (pi->bye, get_obj_handle_count (pi->bye), NULL);
       pi->thread->detach ();
       CloseHandle (pi->bye);
     }
-- 
2.33.0


^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-09 12:42                     ` Takashi Yano
@ 2021-09-09 21:53                       ` Takashi Yano
  2021-09-10  3:41                         ` Takashi Yano
  2021-09-10 10:57                       ` Ken Brown
  1 sibling, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-09 21:53 UTC (permalink / raw)
  To: cygwin-developers

[-- Attachment #1: Type: text/plain, Size: 1207 bytes --]

On Thu, 9 Sep 2021 21:42:46 +0900
Takashi Yano wrote:
> On Thu, 9 Sep 2021 21:19:40 +0900
> Takashi Yano wrote:
> > On Thu, 9 Sep 2021 17:05:49 +0900
> > Takashi Yano wrote:
> > > On Thu, 9 Sep 2021 12:41:15 +0900
> > > Takashi Yano wrote:
> > > > diff --git a/winsup/cygwin/fhandler_fifo.cc b/winsup/cygwin/fhandler_fifo.cc
> > > > index 6709fb974..c40573783 100644
> > > > --- a/winsup/cygwin/fhandler_fifo.cc
> > > > +++ b/winsup/cygwin/fhandler_fifo.cc
> > > > @@ -1047,6 +1047,12 @@ writer_shmem:
> > > >    ResetEvent (writer_opening);
> > > >    nwriters_unlock ();
> > > >  success:
> > > > +  if (!select_sem)
> > > > +    {
> > > > +      char name[MAX_PATH];
> > > > +      __small_sprintf(name, "semaphore-%W", get_pipe_name ()->Buffer);
> > > > +      select_sem = CreateSemaphore (&sec_none, 0, INT32_MAX, name);
> > > > +    }
> > > >    return 1;
> > > >  err_close_reader:
> > > >    saved_errno = get_errno ();
> > > 
> > > Should this be:
> > > > +      select_sem = CreateSemaphore (sa_buf, 0, INT32_MAX, name);
> > > ?
> > 
> > I revised the patch a bit.
> 
> Sorry, I revised the patch again.

The patch still has a mistake. Revised again.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

[-- Attachment #2: 0001-Cygwin-fifo-Utilize-select_sem-for-fifo-as-well-as-p.patch --]
[-- Type: application/octet-stream, Size: 5060 bytes --]

From 69e8fea5c34d746faa9a268ec3a3d00d92498127 Mon Sep 17 00:00:00 2001
From: Takashi Yano <takashi.yano@nifty.ne.jp>
Date: Fri, 10 Sep 2021 06:46:40 +0900
Subject: [PATCH] Cygwin: fifo: Utilize select_sem for fifo as well as pipe.

---
 winsup/cygwin/fhandler_fifo.cc | 38 ++++++++++++++++++++++++++++++++--
 winsup/cygwin/select.cc        | 10 +++++++--
 2 files changed, 44 insertions(+), 4 deletions(-)

diff --git a/winsup/cygwin/fhandler_fifo.cc b/winsup/cygwin/fhandler_fifo.cc
index 6709fb974..1e605d697 100644
--- a/winsup/cygwin/fhandler_fifo.cc
+++ b/winsup/cygwin/fhandler_fifo.cc
@@ -1047,6 +1047,11 @@ writer_shmem:
   ResetEvent (writer_opening);
   nwriters_unlock ();
 success:
+  if (!select_sem)
+    {
+      __small_sprintf (npbuf, "semaphore.%08x.%016X", get_dev (), get_ino ());
+      select_sem = CreateSemaphore (sa_buf, 0, INT32_MAX, npbuf);
+    }
   return 1;
 err_close_reader:
   saved_errno = get_errno ();
@@ -1235,6 +1240,10 @@ fhandler_fifo::raw_read (void *in_ptr, size_t& len)
 		  len = io.Information;
 		  fifo_client_unlock ();
 		  reading_unlock ();
+		  if (select_sem)
+		    ReleaseSemaphore (select_sem,
+				      get_obj_handle_count (select_sem),
+				      NULL);
 		  return;
 		}
 	      break;
@@ -1273,6 +1282,10 @@ fhandler_fifo::raw_read (void *in_ptr, size_t& len)
 		    fc_handler[i].last_read = true;
 		    fifo_client_unlock ();
 		    reading_unlock ();
+		    if (select_sem)
+		      ReleaseSemaphore (select_sem,
+					get_obj_handle_count (select_sem),
+					NULL);
 		    return;
 		  }
 		break;
@@ -1312,6 +1325,10 @@ fhandler_fifo::raw_read (void *in_ptr, size_t& len)
 		    fc_handler[i].last_read = true;
 		    fifo_client_unlock ();
 		    reading_unlock ();
+		    if (select_sem)
+		      ReleaseSemaphore (select_sem,
+					get_obj_handle_count (select_sem),
+					NULL);
 		    return;
 		  }
 		break;
@@ -1345,7 +1362,7 @@ maybe_retry:
       else
 	{
 	  /* Allow interruption and don't hog the CPU. */
-	  DWORD waitret = cygwait (NULL, 1, cw_cancel | cw_sig_eintr);
+	  DWORD waitret = cygwait (select_sem, 1, cw_cancel | cw_sig_eintr);
 	  if (waitret == WAIT_CANCELED)
 	    pthread::static_cancel_self ();
 	  else if (waitret == WAIT_SIGNALED)
@@ -1569,6 +1586,11 @@ fhandler_fifo::close ()
     NtClose (write_ready);
   if (writer_opening)
     NtClose (writer_opening);
+  if (select_sem)
+    {
+      ReleaseSemaphore (select_sem, get_obj_handle_count (select_sem), NULL);
+      NtClose (select_sem);
+    }
   if (nohandle ())
     return 0;
   else
@@ -1634,6 +1656,14 @@ fhandler_fifo::dup (fhandler_base *child, int flags)
     }
   if (fhf->reopen_shmem () < 0)
     goto err_close_shmem_handle;
+  if (select_sem &&
+      !DuplicateHandle (GetCurrentProcess (), select_sem,
+			GetCurrentProcess (), &fhf->select_sem,
+			0, !(flags & O_CLOEXEC), DUPLICATE_SAME_ACCESS))
+    {
+      __seterrno ();
+      goto err_close_shmem;
+    }
   if (reader)
     {
       /* Make sure the child starts unlocked. */
@@ -1648,7 +1678,7 @@ fhandler_fifo::dup (fhandler_base *child, int flags)
 			    0, !(flags & O_CLOEXEC), DUPLICATE_SAME_ACCESS))
 	{
 	  __seterrno ();
-	  goto err_close_shmem;
+	  goto err_close_select_sem;
 	}
       if (fhf->reopen_shared_fc_handler () < 0)
 	goto err_close_shared_fc_hdl;
@@ -1696,6 +1726,8 @@ err_close_shared_fc_handler:
   NtUnmapViewOfSection (GetCurrentProcess (), fhf->shared_fc_handler);
 err_close_shared_fc_hdl:
   NtClose (fhf->shared_fc_hdl);
+err_close_select_sem:
+  NtClose (fhf->select_sem);
 err_close_shmem:
   NtUnmapViewOfSection (GetCurrentProcess (), fhf->shmem);
 err_close_shmem_handle:
@@ -1720,6 +1752,8 @@ fhandler_fifo::fixup_after_fork (HANDLE parent)
   fork_fixup (parent, shmem_handle, "shmem_handle");
   if (reopen_shmem () < 0)
     api_fatal ("Can't reopen shared memory during fork, %E");
+  if (select_sem)
+    fork_fixup (parent, select_sem, "select_sem");
   if (reader)
     {
       /* Make sure the child starts unlocked. */
diff --git a/winsup/cygwin/select.cc b/winsup/cygwin/select.cc
index e9e71b269..5e583434c 100644
--- a/winsup/cygwin/select.cc
+++ b/winsup/cygwin/select.cc
@@ -963,7 +963,13 @@ start_thread_fifo (select_record *me, select_stuff *stuff)
     {
       pi->start = &stuff->start;
       pi->stop_thread = false;
-      pi->bye = CreateEvent (&sec_none_nih, TRUE, FALSE, NULL);
+      pi->bye = me->fh->get_select_sem ();
+      if (pi->bye)
+	DuplicateHandle (GetCurrentProcess (), pi->bye,
+			 GetCurrentProcess (), &pi->bye,
+			 0, 0, DUPLICATE_SAME_ACCESS);
+      else
+	pi->bye = CreateSemaphore (&sec_none_nih, 0, INT32_MAX, NULL);
       pi->thread = new cygthread (thread_fifo, pi, "fifosel");
       me->h = *pi->thread;
       if (!me->h)
@@ -981,7 +987,7 @@ fifo_cleanup (select_record *, select_stuff *stuff)
   if (pi->thread)
     {
       pi->stop_thread = true;
-      SetEvent (pi->bye);
+      ReleaseSemaphore (pi->bye, get_obj_handle_count (pi->bye), NULL);
       pi->thread->detach ();
       CloseHandle (pi->bye);
     }
-- 
2.33.0


^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-09 21:53                       ` Takashi Yano
@ 2021-09-10  3:41                         ` Takashi Yano
  0 siblings, 0 replies; 250+ messages in thread
From: Takashi Yano @ 2021-09-10  3:41 UTC (permalink / raw)
  To: cygwin-developers

[-- Attachment #1: Type: text/plain, Size: 1407 bytes --]

On Fri, 10 Sep 2021 06:53:34 +0900
Takashi Yano wrote:
> On Thu, 9 Sep 2021 21:42:46 +0900
> Takashi Yano wrote:
> > On Thu, 9 Sep 2021 21:19:40 +0900
> > Takashi Yano wrote:
> > > On Thu, 9 Sep 2021 17:05:49 +0900
> > > Takashi Yano wrote:
> > > > On Thu, 9 Sep 2021 12:41:15 +0900
> > > > Takashi Yano wrote:
> > > > > diff --git a/winsup/cygwin/fhandler_fifo.cc b/winsup/cygwin/fhandler_fifo.cc
> > > > > index 6709fb974..c40573783 100644
> > > > > --- a/winsup/cygwin/fhandler_fifo.cc
> > > > > +++ b/winsup/cygwin/fhandler_fifo.cc
> > > > > @@ -1047,6 +1047,12 @@ writer_shmem:
> > > > >    ResetEvent (writer_opening);
> > > > >    nwriters_unlock ();
> > > > >  success:
> > > > > +  if (!select_sem)
> > > > > +    {
> > > > > +      char name[MAX_PATH];
> > > > > +      __small_sprintf(name, "semaphore-%W", get_pipe_name ()->Buffer);
> > > > > +      select_sem = CreateSemaphore (&sec_none, 0, INT32_MAX, name);
> > > > > +    }
> > > > >    return 1;
> > > > >  err_close_reader:
> > > > >    saved_errno = get_errno ();
> > > > 
> > > > Should this be:
> > > > > +      select_sem = CreateSemaphore (sa_buf, 0, INT32_MAX, name);
> > > > ?
> > > 
> > > I revised the patch a bit.
> > 
> > Sorry, I revised the patch again.
> 
> The patch still has a mistake. Revised again.

Partially commonize the codes regarding ReleaseSemaphore() in raw_read().

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

[-- Attachment #2: 0001-Cygwin-fifo-Utilize-select_sem-for-fifo-as-well-as-p.patch --]
[-- Type: application/octet-stream, Size: 5271 bytes --]

From 73e83d320bb7de7f5a68918e03ea377e6edf2d12 Mon Sep 17 00:00:00 2001
From: Takashi Yano <takashi.yano@nifty.ne.jp>
Date: Fri, 10 Sep 2021 08:43:59 +0900
Subject: [PATCH] Cygwin: fifo: Utilize select_sem for fifo as well as pipe.

---
 winsup/cygwin/fhandler_fifo.cc | 44 +++++++++++++++++++++++++---------
 winsup/cygwin/select.cc        | 10 ++++++--
 2 files changed, 41 insertions(+), 13 deletions(-)

diff --git a/winsup/cygwin/fhandler_fifo.cc b/winsup/cygwin/fhandler_fifo.cc
index 6709fb974..aa89fa7ae 100644
--- a/winsup/cygwin/fhandler_fifo.cc
+++ b/winsup/cygwin/fhandler_fifo.cc
@@ -1047,6 +1047,11 @@ writer_shmem:
   ResetEvent (writer_opening);
   nwriters_unlock ();
 success:
+  if (!select_sem)
+    {
+      __small_sprintf (npbuf, "semaphore.%08x.%016X", get_dev (), get_ino ());
+      select_sem = CreateSemaphore (sa_buf, 0, INT32_MAX, npbuf);
+    }
   return 1;
 err_close_reader:
   saved_errno = get_errno ();
@@ -1233,9 +1238,7 @@ fhandler_fifo::raw_read (void *in_ptr, size_t& len)
 	      if (io.Information > 0)
 		{
 		  len = io.Information;
-		  fifo_client_unlock ();
-		  reading_unlock ();
-		  return;
+		  goto out;
 		}
 	      break;
 	    case STATUS_PIPE_EMPTY:
@@ -1271,9 +1274,7 @@ fhandler_fifo::raw_read (void *in_ptr, size_t& len)
 		    if (j < nhandlers)
 		      fc_handler[j].last_read = false;
 		    fc_handler[i].last_read = true;
-		    fifo_client_unlock ();
-		    reading_unlock ();
-		    return;
+		    goto out;
 		  }
 		break;
 	      case STATUS_PIPE_EMPTY:
@@ -1310,9 +1311,7 @@ fhandler_fifo::raw_read (void *in_ptr, size_t& len)
 		    if (j < nhandlers)
 		      fc_handler[j].last_read = false;
 		    fc_handler[i].last_read = true;
-		    fifo_client_unlock ();
-		    reading_unlock ();
-		    return;
+		    goto out;
 		  }
 		break;
 	      case STATUS_PIPE_EMPTY:
@@ -1345,7 +1344,7 @@ maybe_retry:
       else
 	{
 	  /* Allow interruption and don't hog the CPU. */
-	  DWORD waitret = cygwait (NULL, 1, cw_cancel | cw_sig_eintr);
+	  DWORD waitret = cygwait (select_sem, 1, cw_cancel | cw_sig_eintr);
 	  if (waitret == WAIT_CANCELED)
 	    pthread::static_cancel_self ();
 	  else if (waitret == WAIT_SIGNALED)
@@ -1368,6 +1367,12 @@ maybe_retry:
     }
 errout:
   len = (size_t) -1;
+  return;
+out:
+  fifo_client_unlock ();
+  reading_unlock ();
+  if (select_sem)
+    ReleaseSemaphore (select_sem, get_obj_handle_count (select_sem), NULL);
 }
 
 int __reg2
@@ -1569,6 +1574,11 @@ fhandler_fifo::close ()
     NtClose (write_ready);
   if (writer_opening)
     NtClose (writer_opening);
+  if (select_sem)
+    {
+      ReleaseSemaphore (select_sem, get_obj_handle_count (select_sem), NULL);
+      NtClose (select_sem);
+    }
   if (nohandle ())
     return 0;
   else
@@ -1634,6 +1644,14 @@ fhandler_fifo::dup (fhandler_base *child, int flags)
     }
   if (fhf->reopen_shmem () < 0)
     goto err_close_shmem_handle;
+  if (select_sem &&
+      !DuplicateHandle (GetCurrentProcess (), select_sem,
+			GetCurrentProcess (), &fhf->select_sem,
+			0, !(flags & O_CLOEXEC), DUPLICATE_SAME_ACCESS))
+    {
+      __seterrno ();
+      goto err_close_shmem;
+    }
   if (reader)
     {
       /* Make sure the child starts unlocked. */
@@ -1648,7 +1666,7 @@ fhandler_fifo::dup (fhandler_base *child, int flags)
 			    0, !(flags & O_CLOEXEC), DUPLICATE_SAME_ACCESS))
 	{
 	  __seterrno ();
-	  goto err_close_shmem;
+	  goto err_close_select_sem;
 	}
       if (fhf->reopen_shared_fc_handler () < 0)
 	goto err_close_shared_fc_hdl;
@@ -1696,6 +1714,8 @@ err_close_shared_fc_handler:
   NtUnmapViewOfSection (GetCurrentProcess (), fhf->shared_fc_handler);
 err_close_shared_fc_hdl:
   NtClose (fhf->shared_fc_hdl);
+err_close_select_sem:
+  NtClose (fhf->select_sem);
 err_close_shmem:
   NtUnmapViewOfSection (GetCurrentProcess (), fhf->shmem);
 err_close_shmem_handle:
@@ -1720,6 +1740,8 @@ fhandler_fifo::fixup_after_fork (HANDLE parent)
   fork_fixup (parent, shmem_handle, "shmem_handle");
   if (reopen_shmem () < 0)
     api_fatal ("Can't reopen shared memory during fork, %E");
+  if (select_sem)
+    fork_fixup (parent, select_sem, "select_sem");
   if (reader)
     {
       /* Make sure the child starts unlocked. */
diff --git a/winsup/cygwin/select.cc b/winsup/cygwin/select.cc
index e9e71b269..5e583434c 100644
--- a/winsup/cygwin/select.cc
+++ b/winsup/cygwin/select.cc
@@ -963,7 +963,13 @@ start_thread_fifo (select_record *me, select_stuff *stuff)
     {
       pi->start = &stuff->start;
       pi->stop_thread = false;
-      pi->bye = CreateEvent (&sec_none_nih, TRUE, FALSE, NULL);
+      pi->bye = me->fh->get_select_sem ();
+      if (pi->bye)
+	DuplicateHandle (GetCurrentProcess (), pi->bye,
+			 GetCurrentProcess (), &pi->bye,
+			 0, 0, DUPLICATE_SAME_ACCESS);
+      else
+	pi->bye = CreateSemaphore (&sec_none_nih, 0, INT32_MAX, NULL);
       pi->thread = new cygthread (thread_fifo, pi, "fifosel");
       me->h = *pi->thread;
       if (!me->h)
@@ -981,7 +987,7 @@ fifo_cleanup (select_record *, select_stuff *stuff)
   if (pi->thread)
     {
       pi->stop_thread = true;
-      SetEvent (pi->bye);
+      ReleaseSemaphore (pi->bye, get_obj_handle_count (pi->bye), NULL);
       pi->thread->detach ();
       CloseHandle (pi->bye);
     }
-- 
2.33.0


^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-09 12:42                     ` Takashi Yano
  2021-09-09 21:53                       ` Takashi Yano
@ 2021-09-10 10:57                       ` Ken Brown
  2021-09-10 15:17                         ` Ken Brown
  1 sibling, 1 reply; 250+ messages in thread
From: Ken Brown @ 2021-09-10 10:57 UTC (permalink / raw)
  To: cygwin-developers

Hi Takashi,

On 9/9/2021 8:42 AM, Takashi Yano wrote:
> On Thu, 9 Sep 2021 21:19:40 +0900
> Takashi Yano wrote:
>> On Thu, 9 Sep 2021 17:05:49 +0900
>> Takashi Yano wrote:
>>> On Thu, 9 Sep 2021 12:41:15 +0900
>>> Takashi Yano wrote:
>>>> diff --git a/winsup/cygwin/fhandler_fifo.cc b/winsup/cygwin/fhandler_fifo.cc
>>>> index 6709fb974..c40573783 100644
>>>> --- a/winsup/cygwin/fhandler_fifo.cc
>>>> +++ b/winsup/cygwin/fhandler_fifo.cc
>>>> @@ -1047,6 +1047,12 @@ writer_shmem:
>>>>     ResetEvent (writer_opening);
>>>>     nwriters_unlock ();
>>>>   success:
>>>> +  if (!select_sem)
>>>> +    {
>>>> +      char name[MAX_PATH];
>>>> +      __small_sprintf(name, "semaphore-%W", get_pipe_name ()->Buffer);
>>>> +      select_sem = CreateSemaphore (&sec_none, 0, INT32_MAX, name);
>>>> +    }
>>>>     return 1;
>>>>   err_close_reader:
>>>>     saved_errno = get_errno ();
>>>
>>> Should this be:
>>>> +      select_sem = CreateSemaphore (sa_buf, 0, INT32_MAX, name);
>>> ?
>>
>> I revised the patch a bit.
> 
> Sorry, I revised the patch again.

Thanks!  This is an amazing speed-up.  Here's what I see on my system:

Using the HEAD of topic/pipe:

$ ./fifo_test 0

Total: 100MB in 7.646128 second, 13.078515MB/s

$ ./fifo_test 1

Total: 100MB in 5.472798 second, 18.272189MB/s

$ ./fifo_test 2

Total: 100MB in 0.191965 second, 520.928837MB/s

$ ./fifo_test 3
wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr
Total: 100MB in 0.025944 second, 3854.440894MB/s

After applying your patch:

$ ./fifo_test 0

Total: 100MB in 0.074328 second, 1345.391630MB/s

$ ./fifo_test 1

Total: 100MB in 0.062126 second, 1609.632038MB/s

$ ./fifo_test 2

Total: 100MB in 0.013286 second, 7527.003124MB/s

$ ./fifo_test 3
wwwwrrrrrrrrrr
Total: 100MB in 0.014044 second, 7120.326396MB/s

I need to study your patch a little more, but then I'll push it if I don't see 
any problems.

Thanks again.  This is great.

Ken

P.S. I wrote this yesterday before you sent further revisions, but I forgot to 
send it.  I'll recheck the latest version shortly.

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-10 10:57                       ` Ken Brown
@ 2021-09-10 15:17                         ` Ken Brown
  2021-09-10 15:26                           ` Corinna Vinschen
  2021-09-10 22:57                           ` Takashi Yano
  0 siblings, 2 replies; 250+ messages in thread
From: Ken Brown @ 2021-09-10 15:17 UTC (permalink / raw)
  To: cygwin-developers

On 9/10/2021 6:57 AM, Ken Brown wrote:
> Hi Takashi,
> 
> On 9/9/2021 8:42 AM, Takashi Yano wrote:
>> On Thu, 9 Sep 2021 21:19:40 +0900
>> Takashi Yano wrote:
>>> On Thu, 9 Sep 2021 17:05:49 +0900
>>> Takashi Yano wrote:
>>>> On Thu, 9 Sep 2021 12:41:15 +0900
>>>> Takashi Yano wrote:
>>>>> diff --git a/winsup/cygwin/fhandler_fifo.cc b/winsup/cygwin/fhandler_fifo.cc
>>>>> index 6709fb974..c40573783 100644
>>>>> --- a/winsup/cygwin/fhandler_fifo.cc
>>>>> +++ b/winsup/cygwin/fhandler_fifo.cc
>>>>> @@ -1047,6 +1047,12 @@ writer_shmem:
>>>>>     ResetEvent (writer_opening);
>>>>>     nwriters_unlock ();
>>>>>   success:
>>>>> +  if (!select_sem)
>>>>> +    {
>>>>> +      char name[MAX_PATH];
>>>>> +      __small_sprintf(name, "semaphore-%W", get_pipe_name ()->Buffer);
>>>>> +      select_sem = CreateSemaphore (&sec_none, 0, INT32_MAX, name);
>>>>> +    }
>>>>>     return 1;
>>>>>   err_close_reader:
>>>>>     saved_errno = get_errno ();
>>>>
>>>> Should this be:
>>>>> +      select_sem = CreateSemaphore (sa_buf, 0, INT32_MAX, name);
>>>> ?
>>>
>>> I revised the patch a bit.
>>
>> Sorry, I revised the patch again.
> 
> Thanks!  This is an amazing speed-up.  Here's what I see on my system:
> 
> Using the HEAD of topic/pipe:
> 
> $ ./fifo_test 0
> 
> Total: 100MB in 7.646128 second, 13.078515MB/s
> 
> $ ./fifo_test 1
> 
> Total: 100MB in 5.472798 second, 18.272189MB/s
> 
> $ ./fifo_test 2
> 
> Total: 100MB in 0.191965 second, 520.928837MB/s
> 
> $ ./fifo_test 3
> wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr 
> 
> Total: 100MB in 0.025944 second, 3854.440894MB/s
> 
> After applying your patch:
> 
> $ ./fifo_test 0
> 
> Total: 100MB in 0.074328 second, 1345.391630MB/s
> 
> $ ./fifo_test 1
> 
> Total: 100MB in 0.062126 second, 1609.632038MB/s
> 
> $ ./fifo_test 2
> 
> Total: 100MB in 0.013286 second, 7527.003124MB/s
> 
> $ ./fifo_test 3
> wwwwrrrrrrrrrr
> Total: 100MB in 0.014044 second, 7120.326396MB/s
> 
> I need to study your patch a little more, but then I'll push it if I don't see 
> any problems.
> 
> Thanks again.  This is great.
> 
> Ken
> 
> P.S. I wrote this yesterday before you sent further revisions, but I forgot to 
> send it.  I'll recheck the latest version shortly.

I've rerun your test with the latest version, and the test results are similar. 
  I've also run a suite of fifo tests that I've accumulated, and they all pass 
also, so I pushed your patch.

I think we're in pretty good shape now.  The only detail remaining, AFAIK, is 
how to best avoid a deadlock if the pipe has been created by a non-Cygwin 
process.  I've proposed a timeout, but maybe there's a better idea.

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-10 15:17                         ` Ken Brown
@ 2021-09-10 15:26                           ` Corinna Vinschen
  2021-09-10 22:57                           ` Takashi Yano
  1 sibling, 0 replies; 250+ messages in thread
From: Corinna Vinschen @ 2021-09-10 15:26 UTC (permalink / raw)
  To: cygwin-developers

On Sep 10 11:17, Ken Brown wrote:
> I've rerun your test with the latest version, and the test results are
> similar.  I've also run a suite of fifo tests that I've accumulated, and
> they all pass also, so I pushed your patch.
> 
> I think we're in pretty good shape now.  The only detail remaining, AFAIK,
> is how to best avoid a deadlock if the pipe has been created by a non-Cygwin
> process.  I've proposed a timeout, but maybe there's a better idea.

Not from my side.  Every time I think I have an idea it's another
sort of chicken-egg problem...


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-10 15:17                         ` Ken Brown
  2021-09-10 15:26                           ` Corinna Vinschen
@ 2021-09-10 22:57                           ` Takashi Yano
  2021-09-11  2:17                             ` Ken Brown
  1 sibling, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-10 22:57 UTC (permalink / raw)
  To: cygwin-developers

On Fri, 10 Sep 2021 11:17:58 -0400
Ken Brown wrote:
> I've rerun your test with the latest version, and the test results are similar. 
>   I've also run a suite of fifo tests that I've accumulated, and they all pass 
> also, so I pushed your patch.
> 
> I think we're in pretty good shape now.  The only detail remaining, AFAIK, is 
> how to best avoid a deadlock if the pipe has been created by a non-Cygwin 
> process.  I've proposed a timeout, but maybe there's a better idea.

I am not pretty sure what is the problem, but is not the following
patch enough?

diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
index d309be2f7..13fba9a14 100644
--- a/winsup/cygwin/fhandler.h
+++ b/winsup/cygwin/fhandler.h
@@ -1205,6 +1205,7 @@ public:
   select_record *select_except (select_stuff *);
   char *get_proc_fd_name (char *buf);
   int open (int flags, mode_t mode = 0);
+  void open_setup (int flags);
   void fixup_after_fork (HANDLE);
   int dup (fhandler_base *child, int);
   int close ();
diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index 6994a5dce..d84e6ad84 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -191,6 +191,17 @@ out:
   return 0;
 }

+void
+fhandler_pipe::open_setup (int flags)
+{
+  fhandler_base::open_setup (flags);
+  if (get_dev () == FH_PIPER && !read_mtx)
+    {
+      SECURITY_ATTRIBUTES *sa = sec_none_cloexec (flags);
+      read_mtx = CreateMutexW (sa, FALSE, NULL);
+    }
+}
+
 off_t
 fhandler_pipe::lseek (off_t offset, int whence)
 {


AFAIK, another problem remaining is:

On Mon, 6 Sep 2021 14:49:55 +0200
Corinna Vinschen wrote:
> - What about calling select for writing on pipes read by non-Cygwin
>   processes?  In that case, we still can't rely on WriteQuotaAvailable,
>   just as before.


-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-10 22:57                           ` Takashi Yano
@ 2021-09-11  2:17                             ` Ken Brown
  2021-09-11  2:35                               ` Takashi Yano
  0 siblings, 1 reply; 250+ messages in thread
From: Ken Brown @ 2021-09-11  2:17 UTC (permalink / raw)
  To: cygwin-developers

On 9/10/2021 6:57 PM, Takashi Yano wrote:
> On Fri, 10 Sep 2021 11:17:58 -0400
> Ken Brown wrote:
>> I've rerun your test with the latest version, and the test results are similar.
>>    I've also run a suite of fifo tests that I've accumulated, and they all pass
>> also, so I pushed your patch.
>>
>> I think we're in pretty good shape now.  The only detail remaining, AFAIK, is
>> how to best avoid a deadlock if the pipe has been created by a non-Cygwin
>> process.  I've proposed a timeout, but maybe there's a better idea.
> 
> I am not pretty sure what is the problem, but is not the following
> patch enough?
> 
> diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
> index d309be2f7..13fba9a14 100644
> --- a/winsup/cygwin/fhandler.h
> +++ b/winsup/cygwin/fhandler.h
> @@ -1205,6 +1205,7 @@ public:
>     select_record *select_except (select_stuff *);
>     char *get_proc_fd_name (char *buf);
>     int open (int flags, mode_t mode = 0);
> +  void open_setup (int flags);
>     void fixup_after_fork (HANDLE);
>     int dup (fhandler_base *child, int);
>     int close ();
> diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
> index 6994a5dce..d84e6ad84 100644
> --- a/winsup/cygwin/fhandler_pipe.cc
> +++ b/winsup/cygwin/fhandler_pipe.cc
> @@ -191,6 +191,17 @@ out:
>     return 0;
>   }
> 
> +void
> +fhandler_pipe::open_setup (int flags)
> +{
> +  fhandler_base::open_setup (flags);
> +  if (get_dev () == FH_PIPER && !read_mtx)
> +    {
> +      SECURITY_ATTRIBUTES *sa = sec_none_cloexec (flags);
> +      read_mtx = CreateMutexW (sa, FALSE, NULL);
> +    }
> +}
> +
>   off_t
>   fhandler_pipe::lseek (off_t offset, int whence)
>   {
> 
> 
> AFAIK, another problem remaining is:
> 
> On Mon, 6 Sep 2021 14:49:55 +0200
> Corinna Vinschen wrote:
>> - What about calling select for writing on pipes read by non-Cygwin
>>    processes?  In that case, we still can't rely on WriteQuotaAvailable,
>>    just as before.

This is the problem I was talking about.  In this case the non-Cygwin process 
might have a large pending read, so that the Cygwin process calling select on 
the write side will see WriteQuotaAvailable == 0.  This could lead to a deadlock 
with the Cygwin process waiting for write ready while the non-Cygwin process is 
blocked trying to read.

My suggestion is that we impose a timeout in this situation, after which select 
reports write ready.

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-11  2:17                             ` Ken Brown
@ 2021-09-11  2:35                               ` Takashi Yano
  2021-09-11 13:12                                 ` Ken Brown
  2021-09-13  9:07                                 ` Corinna Vinschen
  0 siblings, 2 replies; 250+ messages in thread
From: Takashi Yano @ 2021-09-11  2:35 UTC (permalink / raw)
  To: cygwin-developers

On Fri, 10 Sep 2021 22:17:21 -0400
Ken Brown wrote:
> On 9/10/2021 6:57 PM, Takashi Yano wrote:
> > On Fri, 10 Sep 2021 11:17:58 -0400
> > Ken Brown wrote:
> >> I've rerun your test with the latest version, and the test results are similar.
> >>    I've also run a suite of fifo tests that I've accumulated, and they all pass
> >> also, so I pushed your patch.
> >>
> >> I think we're in pretty good shape now.  The only detail remaining, AFAIK, is
> >> how to best avoid a deadlock if the pipe has been created by a non-Cygwin
> >> process.  I've proposed a timeout, but maybe there's a better idea.
> > 
> > I am not pretty sure what is the problem, but is not the following
> > patch enough?
> > 
> > diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
> > index d309be2f7..13fba9a14 100644
> > --- a/winsup/cygwin/fhandler.h
> > +++ b/winsup/cygwin/fhandler.h
> > @@ -1205,6 +1205,7 @@ public:
> >     select_record *select_except (select_stuff *);
> >     char *get_proc_fd_name (char *buf);
> >     int open (int flags, mode_t mode = 0);
> > +  void open_setup (int flags);
> >     void fixup_after_fork (HANDLE);
> >     int dup (fhandler_base *child, int);
> >     int close ();
> > diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
> > index 6994a5dce..d84e6ad84 100644
> > --- a/winsup/cygwin/fhandler_pipe.cc
> > +++ b/winsup/cygwin/fhandler_pipe.cc
> > @@ -191,6 +191,17 @@ out:
> >     return 0;
> >   }
> > 
> > +void
> > +fhandler_pipe::open_setup (int flags)
> > +{
> > +  fhandler_base::open_setup (flags);
> > +  if (get_dev () == FH_PIPER && !read_mtx)
> > +    {
> > +      SECURITY_ATTRIBUTES *sa = sec_none_cloexec (flags);
> > +      read_mtx = CreateMutexW (sa, FALSE, NULL);
> > +    }
> > +}
> > +
> >   off_t
> >   fhandler_pipe::lseek (off_t offset, int whence)
> >   {
> > 
> > 
> > AFAIK, another problem remaining is:
> > 
> > On Mon, 6 Sep 2021 14:49:55 +0200
> > Corinna Vinschen wrote:
> >> - What about calling select for writing on pipes read by non-Cygwin
> >>    processes?  In that case, we still can't rely on WriteQuotaAvailable,
> >>    just as before.
> 
> This is the problem I was talking about.  In this case the non-Cygwin process 
> might have a large pending read, so that the Cygwin process calling select on 
> the write side will see WriteQuotaAvailable == 0.  This could lead to a deadlock 
> with the Cygwin process waiting for write ready while the non-Cygwin process is 
> blocked trying to read.

Then, the above patch is for another issue.
The problem happes when:
1) Start command prompt.
2) Run 'echo AAAAAAAAAAAA | \cygwin64\bin\cat
This causes hang up in cat. In this case, pipe is created by cmd.exe.
Therefore, read_mtx is not created.
 
> My suggestion is that we impose a timeout in this situation, after which select 
> reports write ready.

Keeping read handle in write pipe (Corinna's query_hdl) causes problem
that write side cannot detect close on read side.
Is it possible to open read handle temporally when pipe_data_available()
is called?

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-11  2:35                               ` Takashi Yano
@ 2021-09-11 13:12                                 ` Ken Brown
  2021-09-12  6:23                                   ` Takashi Yano
  2021-09-12  8:48                                   ` Takashi Yano
  2021-09-13  9:07                                 ` Corinna Vinschen
  1 sibling, 2 replies; 250+ messages in thread
From: Ken Brown @ 2021-09-11 13:12 UTC (permalink / raw)
  To: cygwin-developers

On 9/10/2021 10:35 PM, Takashi Yano wrote:
> On Fri, 10 Sep 2021 22:17:21 -0400
> Ken Brown wrote:
>> On 9/10/2021 6:57 PM, Takashi Yano wrote:
>>> On Fri, 10 Sep 2021 11:17:58 -0400
>>> Ken Brown wrote:
>>>> I've rerun your test with the latest version, and the test results are similar.
>>>>     I've also run a suite of fifo tests that I've accumulated, and they all pass
>>>> also, so I pushed your patch.
>>>>
>>>> I think we're in pretty good shape now.  The only detail remaining, AFAIK, is
>>>> how to best avoid a deadlock if the pipe has been created by a non-Cygwin
>>>> process.  I've proposed a timeout, but maybe there's a better idea.
>>>
>>> I am not pretty sure what is the problem, but is not the following
>>> patch enough?
>>>
>>> diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
>>> index d309be2f7..13fba9a14 100644
>>> --- a/winsup/cygwin/fhandler.h
>>> +++ b/winsup/cygwin/fhandler.h
>>> @@ -1205,6 +1205,7 @@ public:
>>>      select_record *select_except (select_stuff *);
>>>      char *get_proc_fd_name (char *buf);
>>>      int open (int flags, mode_t mode = 0);
>>> +  void open_setup (int flags);
>>>      void fixup_after_fork (HANDLE);
>>>      int dup (fhandler_base *child, int);
>>>      int close ();
>>> diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
>>> index 6994a5dce..d84e6ad84 100644
>>> --- a/winsup/cygwin/fhandler_pipe.cc
>>> +++ b/winsup/cygwin/fhandler_pipe.cc
>>> @@ -191,6 +191,17 @@ out:
>>>      return 0;
>>>    }
>>>
>>> +void
>>> +fhandler_pipe::open_setup (int flags)
>>> +{
>>> +  fhandler_base::open_setup (flags);
>>> +  if (get_dev () == FH_PIPER && !read_mtx)
>>> +    {
>>> +      SECURITY_ATTRIBUTES *sa = sec_none_cloexec (flags);
>>> +      read_mtx = CreateMutexW (sa, FALSE, NULL);
>>> +    }
>>> +}
>>> +
>>>    off_t
>>>    fhandler_pipe::lseek (off_t offset, int whence)
>>>    {
>>>
>>>
>>> AFAIK, another problem remaining is:
>>>
>>> On Mon, 6 Sep 2021 14:49:55 +0200
>>> Corinna Vinschen wrote:
>>>> - What about calling select for writing on pipes read by non-Cygwin
>>>>     processes?  In that case, we still can't rely on WriteQuotaAvailable,
>>>>     just as before.
>>
>> This is the problem I was talking about.  In this case the non-Cygwin process
>> might have a large pending read, so that the Cygwin process calling select on
>> the write side will see WriteQuotaAvailable == 0.  This could lead to a deadlock
>> with the Cygwin process waiting for write ready while the non-Cygwin process is
>> blocked trying to read.
> 
> Then, the above patch is for another issue.
> The problem happes when:
> 1) Start command prompt.
> 2) Run 'echo AAAAAAAAAAAA | \cygwin64\bin\cat
> This causes hang up in cat. In this case, pipe is created by cmd.exe.
> Therefore, read_mtx is not created.

Confirmed, and your patch fixes it.  Maybe you should check for error in the 
call to CreateMutexW and print a debug message in that case.

>> My suggestion is that we impose a timeout in this situation, after which select
>> reports write ready.
> 
> Keeping read handle in write pipe (Corinna's query_hdl) causes problem
> that write side cannot detect close on read side.
> Is it possible to open read handle temporally when pipe_data_available()
> is called?

That would be nice, but I have no idea how you could do that.

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-11 13:12                                 ` Ken Brown
@ 2021-09-12  6:23                                   ` Takashi Yano
  2021-09-12 14:39                                     ` Ken Brown
  2021-09-12  8:48                                   ` Takashi Yano
  1 sibling, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-12  6:23 UTC (permalink / raw)
  To: cygwin-developers

[-- Attachment #1: Type: text/plain, Size: 3229 bytes --]

On Sat, 11 Sep 2021 09:12:02 -0400
Ken Brown wrote:
> On 9/10/2021 10:35 PM, Takashi Yano wrote:
> > On Fri, 10 Sep 2021 22:17:21 -0400
> > Ken Brown wrote:
> >> On 9/10/2021 6:57 PM, Takashi Yano wrote:
> >>> On Fri, 10 Sep 2021 11:17:58 -0400
> >>> Ken Brown wrote:
> >>>> I've rerun your test with the latest version, and the test results are similar.
> >>>>     I've also run a suite of fifo tests that I've accumulated, and they all pass
> >>>> also, so I pushed your patch.
> >>>>
> >>>> I think we're in pretty good shape now.  The only detail remaining, AFAIK, is
> >>>> how to best avoid a deadlock if the pipe has been created by a non-Cygwin
> >>>> process.  I've proposed a timeout, but maybe there's a better idea.
> >>>
> >>> I am not pretty sure what is the problem, but is not the following
> >>> patch enough?
> >>>
> >>> diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
> >>> index d309be2f7..13fba9a14 100644
> >>> --- a/winsup/cygwin/fhandler.h
> >>> +++ b/winsup/cygwin/fhandler.h
> >>> @@ -1205,6 +1205,7 @@ public:
> >>>      select_record *select_except (select_stuff *);
> >>>      char *get_proc_fd_name (char *buf);
> >>>      int open (int flags, mode_t mode = 0);
> >>> +  void open_setup (int flags);
> >>>      void fixup_after_fork (HANDLE);
> >>>      int dup (fhandler_base *child, int);
> >>>      int close ();
> >>> diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
> >>> index 6994a5dce..d84e6ad84 100644
> >>> --- a/winsup/cygwin/fhandler_pipe.cc
> >>> +++ b/winsup/cygwin/fhandler_pipe.cc
> >>> @@ -191,6 +191,17 @@ out:
> >>>      return 0;
> >>>    }
> >>>
> >>> +void
> >>> +fhandler_pipe::open_setup (int flags)
> >>> +{
> >>> +  fhandler_base::open_setup (flags);
> >>> +  if (get_dev () == FH_PIPER && !read_mtx)
> >>> +    {
> >>> +      SECURITY_ATTRIBUTES *sa = sec_none_cloexec (flags);
> >>> +      read_mtx = CreateMutexW (sa, FALSE, NULL);
> >>> +    }
> >>> +}
> >>> +
> >>>    off_t
> >>>    fhandler_pipe::lseek (off_t offset, int whence)
> >>>    {
> >>>
> >>>
> >>> AFAIK, another problem remaining is:
> >>>
> >>> On Mon, 6 Sep 2021 14:49:55 +0200
> >>> Corinna Vinschen wrote:
> >>>> - What about calling select for writing on pipes read by non-Cygwin
> >>>>     processes?  In that case, we still can't rely on WriteQuotaAvailable,
> >>>>     just as before.
> >>
> >> This is the problem I was talking about.  In this case the non-Cygwin process
> >> might have a large pending read, so that the Cygwin process calling select on
> >> the write side will see WriteQuotaAvailable == 0.  This could lead to a deadlock
> >> with the Cygwin process waiting for write ready while the non-Cygwin process is
> >> blocked trying to read.
> > 
> > Then, the above patch is for another issue.
> > The problem happes when:
> > 1) Start command prompt.
> > 2) Run 'echo AAAAAAAAAAAA | \cygwin64\bin\cat
> > This causes hang up in cat. In this case, pipe is created by cmd.exe.
> > Therefore, read_mtx is not created.
> 
> Confirmed, and your patch fixes it.  Maybe you should check for error in the 
> call to CreateMutexW and print a debug message in that case.

I added the debug message.


-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

[-- Attachment #2: 0001-Cygwin-pipe-Fix-deadlock-if-pipe-is-created-by-non-c.patch --]
[-- Type: application/octet-stream, Size: 1437 bytes --]

From 1c929f7d3633d99c0040e51552082829b17f550d Mon Sep 17 00:00:00 2001
From: Takashi Yano <takashi.yano@nifty.ne.jp>
Date: Sun, 12 Sep 2021 15:06:05 +0900
Subject: [PATCH] Cygwin: pipe: Fix deadlock if pipe is created by non-cygwin
 app.

---
 winsup/cygwin/fhandler.h       |  1 +
 winsup/cygwin/fhandler_pipe.cc | 13 +++++++++++++
 2 files changed, 14 insertions(+)

diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
index d309be2f7..13fba9a14 100644
--- a/winsup/cygwin/fhandler.h
+++ b/winsup/cygwin/fhandler.h
@@ -1205,6 +1205,7 @@ public:
   select_record *select_except (select_stuff *);
   char *get_proc_fd_name (char *buf);
   int open (int flags, mode_t mode = 0);
+  void open_setup (int flags);
   void fixup_after_fork (HANDLE);
   int dup (fhandler_base *child, int);
   int close ();
diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index 6994a5dce..9b4255cfd 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -191,6 +191,19 @@ out:
   return 0;
 }
 
+void
+fhandler_pipe::open_setup (int flags)
+{
+  fhandler_base::open_setup (flags);
+  if (get_dev () == FH_PIPER && !read_mtx)
+    {
+      SECURITY_ATTRIBUTES *sa = sec_none_cloexec (flags);
+      read_mtx = CreateMutex (sa, FALSE, NULL);
+      if (!read_mtx)
+	debug_printf ("CreateMutex failed: %E");
+    }
+}
+
 off_t
 fhandler_pipe::lseek (off_t offset, int whence)
 {
-- 
2.33.0


^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-11 13:12                                 ` Ken Brown
  2021-09-12  6:23                                   ` Takashi Yano
@ 2021-09-12  8:48                                   ` Takashi Yano
  2021-09-12 11:04                                     ` Takashi Yano
  1 sibling, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-12  8:48 UTC (permalink / raw)
  To: cygwin-developers

[-- Attachment #1: Type: text/plain, Size: 3968 bytes --]

On Sat, 11 Sep 2021 09:12:02 -0400
Ken Brown wrote:
> On 9/10/2021 10:35 PM, Takashi Yano wrote:
> > On Fri, 10 Sep 2021 22:17:21 -0400
> > Ken Brown wrote:
> >> On 9/10/2021 6:57 PM, Takashi Yano wrote:
> >>> On Fri, 10 Sep 2021 11:17:58 -0400
> >>> Ken Brown wrote:
> >>>> I've rerun your test with the latest version, and the test results are similar.
> >>>>     I've also run a suite of fifo tests that I've accumulated, and they all pass
> >>>> also, so I pushed your patch.
> >>>>
> >>>> I think we're in pretty good shape now.  The only detail remaining, AFAIK, is
> >>>> how to best avoid a deadlock if the pipe has been created by a non-Cygwin
> >>>> process.  I've proposed a timeout, but maybe there's a better idea.
> >>>
> >>> I am not pretty sure what is the problem, but is not the following
> >>> patch enough?
> >>>
> >>> diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
> >>> index d309be2f7..13fba9a14 100644
> >>> --- a/winsup/cygwin/fhandler.h
> >>> +++ b/winsup/cygwin/fhandler.h
> >>> @@ -1205,6 +1205,7 @@ public:
> >>>      select_record *select_except (select_stuff *);
> >>>      char *get_proc_fd_name (char *buf);
> >>>      int open (int flags, mode_t mode = 0);
> >>> +  void open_setup (int flags);
> >>>      void fixup_after_fork (HANDLE);
> >>>      int dup (fhandler_base *child, int);
> >>>      int close ();
> >>> diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
> >>> index 6994a5dce..d84e6ad84 100644
> >>> --- a/winsup/cygwin/fhandler_pipe.cc
> >>> +++ b/winsup/cygwin/fhandler_pipe.cc
> >>> @@ -191,6 +191,17 @@ out:
> >>>      return 0;
> >>>    }
> >>>
> >>> +void
> >>> +fhandler_pipe::open_setup (int flags)
> >>> +{
> >>> +  fhandler_base::open_setup (flags);
> >>> +  if (get_dev () == FH_PIPER && !read_mtx)
> >>> +    {
> >>> +      SECURITY_ATTRIBUTES *sa = sec_none_cloexec (flags);
> >>> +      read_mtx = CreateMutexW (sa, FALSE, NULL);
> >>> +    }
> >>> +}
> >>> +
> >>>    off_t
> >>>    fhandler_pipe::lseek (off_t offset, int whence)
> >>>    {
> >>>
> >>>
> >>> AFAIK, another problem remaining is:
> >>>
> >>> On Mon, 6 Sep 2021 14:49:55 +0200
> >>> Corinna Vinschen wrote:
> >>>> - What about calling select for writing on pipes read by non-Cygwin
> >>>>     processes?  In that case, we still can't rely on WriteQuotaAvailable,
> >>>>     just as before.
> >>
> >> This is the problem I was talking about.  In this case the non-Cygwin process
> >> might have a large pending read, so that the Cygwin process calling select on
> >> the write side will see WriteQuotaAvailable == 0.  This could lead to a deadlock
> >> with the Cygwin process waiting for write ready while the non-Cygwin process is
> >> blocked trying to read.
> > 
> > Then, the above patch is for another issue.
> > The problem happes when:
> > 1) Start command prompt.
> > 2) Run 'echo AAAAAAAAAAAA | \cygwin64\bin\cat
> > This causes hang up in cat. In this case, pipe is created by cmd.exe.
> > Therefore, read_mtx is not created.
> 
> Confirmed, and your patch fixes it.  Maybe you should check for error in the 
> call to CreateMutexW and print a debug message in that case.
> 
> >> My suggestion is that we impose a timeout in this situation, after which select
> >> reports write ready.
> > 
> > Keeping read handle in write pipe (Corinna's query_hdl) causes problem
> > that write side cannot detect close on read side.
> > Is it possible to open read handle temporally when pipe_data_available()
> > is called?
> 
> That would be nice, but I have no idea how you could do that.

Hmm. Then, what about PoC code attached? This returns to Corinna's
query_hdl, and counts read/write handles to detect closing reader side.

If the number of read handles is equal to number of write handles,
only the pairs of write handle and query_hdl are alive. So, read pipe
supposed to be closed.

This patch depends another patch I posted a few hours ago.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

[-- Attachment #2: 0001-Cygwin-pipe-Return-to-query_hdl-strategy-with-counti.patch --]
[-- Type: application/octet-stream, Size: 9622 bytes --]

From 050a1e204dd0753d3498b7f2e84dff5e6c7704d9 Mon Sep 17 00:00:00 2001
From: Takashi Yano <takashi.yano@nifty.ne.jp>
Date: Sun, 12 Sep 2021 17:42:03 +0900
Subject: [PATCH] Cygwin: pipe: Return to query_hdl strategy with counting r/w
 handles.

---
 winsup/cygwin/fhandler.h       |  5 ++
 winsup/cygwin/fhandler_pipe.cc | 91 +++++++++++++++++++++++-----------
 winsup/cygwin/select.cc        | 50 ++++++++++++++++---
 winsup/cygwin/spawn.cc         |  2 +
 4 files changed, 112 insertions(+), 36 deletions(-)

diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
index 13fba9a14..f09af2c37 100644
--- a/winsup/cygwin/fhandler.h
+++ b/winsup/cygwin/fhandler.h
@@ -1176,10 +1176,15 @@ class fhandler_pipe_fifo: public fhandler_base
 {
  protected:
   size_t pipe_buf_size;
+  HANDLE query_hdl;
 
  public:
   fhandler_pipe_fifo ();
 
+  HANDLE get_query_handle () const { return query_hdl; }
+  void close_query_handle () { CloseHandle (query_hdl); query_hdl = NULL; }
+  bool reader_closed ();
+
   ssize_t __reg3 raw_write (const void *ptr, size_t len);
 
 };
diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index 9b4255cfd..2818421ec 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -56,6 +56,8 @@ fhandler_pipe::set_pipe_non_blocking (bool nonblocking)
   fpi.ReadMode = FILE_PIPE_BYTE_STREAM_MODE;
   fpi.CompletionMode = nonblocking ? FILE_PIPE_COMPLETE_OPERATION
     : FILE_PIPE_QUEUE_OPERATION;
+  if (query_hdl)
+    fpi.CompletionMode = FILE_PIPE_COMPLETE_OPERATION;
   status = NtSetInformationFile (get_handle (), &io, &fpi, sizeof fpi,
 				 FilePipeInformation);
   if (!NT_SUCCESS (status))
@@ -202,6 +204,8 @@ fhandler_pipe::open_setup (int flags)
       if (!read_mtx)
 	debug_printf ("CreateMutex failed: %E");
     }
+  if (get_dev () == FH_PIPEW && !query_hdl)
+    set_pipe_non_blocking (is_nonblocking ());
 }
 
 off_t
@@ -268,39 +272,11 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
   while (nbytes < len)
     {
       ULONG_PTR nbytes_now = 0;
-      size_t left = len - nbytes;
-      ULONG len1 = (ULONG) left;
+      ULONG len1 = (ULONG) (len - nbytes);
       waitret = WAIT_OBJECT_0;
 
       if (evt)
 	ResetEvent (evt);
-      if (!is_nonblocking ())
-	{
-	  FILE_PIPE_LOCAL_INFORMATION fpli;
-
-	  /* If the pipe is empty, don't request more bytes than pipe
-	     buffer size - 1. Pending read lowers WriteQuotaAvailable on
-	     the write side and thus affects select's ability to return
-	     more or less reliable info whether a write succeeds or not. */
-	  ULONG chunk = pipe_buf_size - 1;
-	  status = NtQueryInformationFile (get_handle (), &io,
-					   &fpli, sizeof (fpli),
-					   FilePipeLocalInformation);
-	  if (NT_SUCCESS (status))
-	    {
-	      if (fpli.ReadDataAvailable > 0)
-		chunk = left;
-	      else if (nbytes != 0)
-		break;
-	      else
-		chunk = fpli.InboundQuota - 1;
-	    }
-	  else if (nbytes != 0)
-	    break;
-
-	  if (len1 > chunk)
-	    len1 = chunk;
-	}
       status = NtReadFile (get_handle (), evt, NULL, NULL, &io, ptr,
 			   len1, NULL, NULL);
       if (evt && status == STATUS_PENDING)
@@ -385,6 +361,16 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
   len = nbytes;
 }
 
+bool
+fhandler_pipe_fifo::reader_closed ()
+{
+  if (!query_hdl)
+    return false;
+  int n_reader = get_obj_handle_count (query_hdl);
+  int n_writer = get_obj_handle_count (get_handle ());
+  return n_reader == n_writer;
+}
+
 ssize_t __reg3
 fhandler_pipe_fifo::raw_write (const void *ptr, size_t len)
 {
@@ -493,7 +479,20 @@ fhandler_pipe_fifo::raw_write (const void *ptr, size_t len)
 			      get_obj_handle_count (select_sem), NULL);
 	  /* 0 bytes returned?  EAGAIN.  See above. */
 	  if (NT_SUCCESS (status) && nbytes == 0)
-	    set_errno (EAGAIN);
+	    {
+	      if (reader_closed ())
+		{
+		  set_errno (EPIPE);
+		  raise (SIGPIPE);
+		}
+	      else if (is_nonblocking ())
+		set_errno (EAGAIN);
+	      else
+		{
+		  cygwait (select_sem, 10);
+		  continue;
+		}
+	    }
 	}
       else if (STATUS_PIPE_IS_CLOSED (status))
 	{
@@ -522,6 +521,10 @@ fhandler_pipe::fixup_after_fork (HANDLE parent)
     fork_fixup (parent, read_mtx, "read_mtx");
   if (select_sem)
     fork_fixup (parent, select_sem, "select_sem");
+  /* Do not duplicate query_hdl if it has been already inrherited. */
+  if (query_hdl && !get_obj_handle_count (query_hdl))
+    fork_fixup (parent, query_hdl, "query_hdl");
+
   fhandler_base::fixup_after_fork (parent);
 }
 
@@ -552,6 +555,15 @@ fhandler_pipe::dup (fhandler_base *child, int flags)
       ftp->close ();
       res = -1;
     }
+  else if (query_hdl &&
+	   !DuplicateHandle (GetCurrentProcess (), query_hdl,
+			    GetCurrentProcess (), &ftp->query_hdl,
+			    0, !(flags & O_CLOEXEC), DUPLICATE_SAME_ACCESS))
+    {
+      __seterrno ();
+      ftp->close ();
+      res = -1;
+    }
 
   debug_printf ("res %d", res);
   return res;
@@ -567,6 +579,8 @@ fhandler_pipe::close ()
       ReleaseSemaphore (select_sem, get_obj_handle_count (select_sem), NULL);
       CloseHandle (select_sem);
     }
+  if (query_hdl)
+    CloseHandle (query_hdl);
   return fhandler_base::close ();
 }
 
@@ -791,6 +805,23 @@ fhandler_pipe::create (fhandler_pipe *fhs[2], unsigned psize, int mode)
 	DuplicateHandle (GetCurrentProcess (), fhs[0]->select_sem,
 			 GetCurrentProcess (), &fhs[1]->select_sem,
 			 0, 1, DUPLICATE_SAME_ACCESS);
+      if (!DuplicateHandle (GetCurrentProcess (), r,
+			    GetCurrentProcess (), &fhs[1]->query_hdl,
+			    GENERIC_READ, !(mode & O_CLOEXEC), 0))
+	{
+	  CloseHandle (fhs[0]->select_sem);
+	  delete fhs[0];
+	  CloseHandle (r);
+	  CloseHandle (fhs[1]->select_sem);
+	  delete fhs[1];
+	  CloseHandle (w);
+	}
+      else
+	{
+	  /* Call set_pipe_non_blocking() again after creating query_hdl. */
+	  fhs[1]->set_pipe_non_blocking (fhs[1]->is_nonblocking ());
+	  res = 0;
+	}
     }
 
   debug_printf ("%R = pipe([%p, %p], %d, %y)", res, fhs[0], fhs[1], psize, mode);
diff --git a/winsup/cygwin/select.cc b/winsup/cygwin/select.cc
index 5e583434c..c569a059c 100644
--- a/winsup/cygwin/select.cc
+++ b/winsup/cygwin/select.cc
@@ -608,12 +608,43 @@ pipe_data_available (int fd, fhandler_base *fh, HANDLE h, bool writing)
     }
   if (writing)
     {
-      /* WriteQuotaAvailable is decremented by the number of bytes requested
-	 by a blocking reader on the other side of the pipe.  Cygwin readers
-	 are serialized and never request a number of bytes equivalent to the
-	 full buffer size.  So WriteQuotaAvailable is 0 only if either the
-	 read buffer on the other side is really full, or if we have non-Cygwin
-	 readers. */
+      /* If there is anything available in the pipe buffer then signal
+        that.  This means that a pipe could still block since you could
+        be trying to write more to the pipe than is available in the
+        buffer but that is the hazard of select().
+
+        Note that WriteQuotaAvailable is unreliable.
+
+        Usually WriteQuotaAvailable on the write side reflects the space
+        available in the inbound buffer on the read side.  However, if a
+        pipe read is currently pending, WriteQuotaAvailable on the write side
+        is decremented by the number of bytes the read side is requesting.
+        So it's possible (even likely) that WriteQuotaAvailable is 0, even
+        if the inbound buffer on the read side is not full.  This can lead to
+        a deadlock situation: The reader is waiting for data, but select
+        on the writer side assumes that no space is available in the read
+        side inbound buffer.
+
+        Consequentially, the only reliable information is available on the
+        read side, so fetch info from the read side via the pipe-specific
+        query handle.  Use fpli.WriteQuotaAvailable as storage for the actual
+        interesting value, which is the OutboundQuote on the write side,
+        decremented by the number of bytes of data in that buffer. */
+      /* Note: Do not use NtQueryInformationFile() for query_hdl because
+	 NtQueryInformationFile() seems to interfere with reading pipes
+	 in non-cygwin apps. Instead, use PeekNamedPipe() here. */
+      if (fh->get_device () == FH_PIPEW)
+	{
+	  HANDLE query_hdl = ((fhandler_pipe *) fh)->get_query_handle ();
+	  if (query_hdl)
+	    {
+	      DWORD nbytes_in_pipe;
+	      PeekNamedPipe (query_hdl, NULL, 0, NULL, &nbytes_in_pipe, NULL);
+	      fpli.WriteQuotaAvailable = fpli.OutboundQuota - nbytes_in_pipe;
+	    }
+	  else
+	    return 1;
+	}
       if (fpli.WriteQuotaAvailable > 0)
 	{
 	  paranoid_printf ("fd %d, %s, write: size %u, avail %u", fd,
@@ -712,6 +743,13 @@ out:
   h = fh->get_output_handle ();
   if (s->write_selected && dev != FH_PIPER)
     {
+      if (dev == FH_PIPEW && ((fhandler_pipe *) fh)->reader_closed ())
+	{
+	  gotone += s->write_ready = true;
+	  if (s->except_selected)
+	    gotone += s->except_ready = true;
+	  return gotone;
+	}
       gotone += s->write_ready =  pipe_data_available (s->fd, fh, h, true);
       select_printf ("write: %s, gotone %d", fh->get_name (), gotone);
     }
diff --git a/winsup/cygwin/spawn.cc b/winsup/cygwin/spawn.cc
index 0bde0b04d..e902b5080 100644
--- a/winsup/cygwin/spawn.cc
+++ b/winsup/cygwin/spawn.cc
@@ -657,6 +657,8 @@ child_info_spawn::worker (const char *prog_arg, const char *const *argv,
 		ptys->create_invisible_console ();
 		ptys->setup_locale ();
 	      }
+	    else if (cfd->get_dev () == FH_PIPEW)
+	      ((fhandler_pipe *)(fhandler_base *) cfd)->close_query_handle ();
 	}
 
       bool enable_pcon = false;
-- 
2.33.0


^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-12  8:48                                   ` Takashi Yano
@ 2021-09-12 11:04                                     ` Takashi Yano
  2021-09-12 15:10                                       ` Ken Brown
                                                         ` (2 more replies)
  0 siblings, 3 replies; 250+ messages in thread
From: Takashi Yano @ 2021-09-12 11:04 UTC (permalink / raw)
  To: cygwin-developers

[-- Attachment #1: Type: text/plain, Size: 4233 bytes --]

On Sun, 12 Sep 2021 17:48:49 +0900
Takashi Yano wrote:
> On Sat, 11 Sep 2021 09:12:02 -0400
> Ken Brown wrote:
> > On 9/10/2021 10:35 PM, Takashi Yano wrote:
> > > On Fri, 10 Sep 2021 22:17:21 -0400
> > > Ken Brown wrote:
> > >> On 9/10/2021 6:57 PM, Takashi Yano wrote:
> > >>> On Fri, 10 Sep 2021 11:17:58 -0400
> > >>> Ken Brown wrote:
> > >>>> I've rerun your test with the latest version, and the test results are similar.
> > >>>>     I've also run a suite of fifo tests that I've accumulated, and they all pass
> > >>>> also, so I pushed your patch.
> > >>>>
> > >>>> I think we're in pretty good shape now.  The only detail remaining, AFAIK, is
> > >>>> how to best avoid a deadlock if the pipe has been created by a non-Cygwin
> > >>>> process.  I've proposed a timeout, but maybe there's a better idea.
> > >>>
> > >>> I am not pretty sure what is the problem, but is not the following
> > >>> patch enough?
> > >>>
> > >>> diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
> > >>> index d309be2f7..13fba9a14 100644
> > >>> --- a/winsup/cygwin/fhandler.h
> > >>> +++ b/winsup/cygwin/fhandler.h
> > >>> @@ -1205,6 +1205,7 @@ public:
> > >>>      select_record *select_except (select_stuff *);
> > >>>      char *get_proc_fd_name (char *buf);
> > >>>      int open (int flags, mode_t mode = 0);
> > >>> +  void open_setup (int flags);
> > >>>      void fixup_after_fork (HANDLE);
> > >>>      int dup (fhandler_base *child, int);
> > >>>      int close ();
> > >>> diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
> > >>> index 6994a5dce..d84e6ad84 100644
> > >>> --- a/winsup/cygwin/fhandler_pipe.cc
> > >>> +++ b/winsup/cygwin/fhandler_pipe.cc
> > >>> @@ -191,6 +191,17 @@ out:
> > >>>      return 0;
> > >>>    }
> > >>>
> > >>> +void
> > >>> +fhandler_pipe::open_setup (int flags)
> > >>> +{
> > >>> +  fhandler_base::open_setup (flags);
> > >>> +  if (get_dev () == FH_PIPER && !read_mtx)
> > >>> +    {
> > >>> +      SECURITY_ATTRIBUTES *sa = sec_none_cloexec (flags);
> > >>> +      read_mtx = CreateMutexW (sa, FALSE, NULL);
> > >>> +    }
> > >>> +}
> > >>> +
> > >>>    off_t
> > >>>    fhandler_pipe::lseek (off_t offset, int whence)
> > >>>    {
> > >>>
> > >>>
> > >>> AFAIK, another problem remaining is:
> > >>>
> > >>> On Mon, 6 Sep 2021 14:49:55 +0200
> > >>> Corinna Vinschen wrote:
> > >>>> - What about calling select for writing on pipes read by non-Cygwin
> > >>>>     processes?  In that case, we still can't rely on WriteQuotaAvailable,
> > >>>>     just as before.
> > >>
> > >> This is the problem I was talking about.  In this case the non-Cygwin process
> > >> might have a large pending read, so that the Cygwin process calling select on
> > >> the write side will see WriteQuotaAvailable == 0.  This could lead to a deadlock
> > >> with the Cygwin process waiting for write ready while the non-Cygwin process is
> > >> blocked trying to read.
> > > 
> > > Then, the above patch is for another issue.
> > > The problem happes when:
> > > 1) Start command prompt.
> > > 2) Run 'echo AAAAAAAAAAAA | \cygwin64\bin\cat
> > > This causes hang up in cat. In this case, pipe is created by cmd.exe.
> > > Therefore, read_mtx is not created.
> > 
> > Confirmed, and your patch fixes it.  Maybe you should check for error in the 
> > call to CreateMutexW and print a debug message in that case.
> > 
> > >> My suggestion is that we impose a timeout in this situation, after which select
> > >> reports write ready.
> > > 
> > > Keeping read handle in write pipe (Corinna's query_hdl) causes problem
> > > that write side cannot detect close on read side.
> > > Is it possible to open read handle temporally when pipe_data_available()
> > > is called?
> > 
> > That would be nice, but I have no idea how you could do that.
> 
> Hmm. Then, what about PoC code attached? This returns to Corinna's
> query_hdl, and counts read/write handles to detect closing reader side.
> 
> If the number of read handles is equal to number of write handles,
> only the pairs of write handle and query_hdl are alive. So, read pipe
> supposed to be closed.
> 
> This patch depends another patch I posted a few hours ago.

Revised a bit.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

[-- Attachment #2: 0001-Cygwin-pipe-Return-to-query_hdl-strategy-with-counti.patch --]
[-- Type: application/octet-stream, Size: 9924 bytes --]

From fc611be411e1d380f1258bb0b225d99757b4ef59 Mon Sep 17 00:00:00 2001
From: Takashi Yano <takashi.yano@nifty.ne.jp>
Date: Sun, 12 Sep 2021 19:59:53 +0900
Subject: [PATCH] Cygwin: pipe: Return to query_hdl strategy with counting r/w
 handles.

---
 winsup/cygwin/fhandler.h       |  5 ++
 winsup/cygwin/fhandler_pipe.cc | 98 ++++++++++++++++++++++++----------
 winsup/cygwin/select.cc        | 50 ++++++++++++++---
 winsup/cygwin/spawn.cc         |  2 +
 4 files changed, 121 insertions(+), 34 deletions(-)

diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
index 13fba9a14..f09af2c37 100644
--- a/winsup/cygwin/fhandler.h
+++ b/winsup/cygwin/fhandler.h
@@ -1176,10 +1176,15 @@ class fhandler_pipe_fifo: public fhandler_base
 {
  protected:
   size_t pipe_buf_size;
+  HANDLE query_hdl;
 
  public:
   fhandler_pipe_fifo ();
 
+  HANDLE get_query_handle () const { return query_hdl; }
+  void close_query_handle () { CloseHandle (query_hdl); query_hdl = NULL; }
+  bool reader_closed ();
+
   ssize_t __reg3 raw_write (const void *ptr, size_t len);
 
 };
diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index 9b4255cfd..b051f5c03 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -56,6 +56,8 @@ fhandler_pipe::set_pipe_non_blocking (bool nonblocking)
   fpi.ReadMode = FILE_PIPE_BYTE_STREAM_MODE;
   fpi.CompletionMode = nonblocking ? FILE_PIPE_COMPLETE_OPERATION
     : FILE_PIPE_QUEUE_OPERATION;
+  if (query_hdl)
+    fpi.CompletionMode = FILE_PIPE_COMPLETE_OPERATION;
   status = NtSetInformationFile (get_handle (), &io, &fpi, sizeof fpi,
 				 FilePipeInformation);
   if (!NT_SUCCESS (status))
@@ -202,6 +204,8 @@ fhandler_pipe::open_setup (int flags)
       if (!read_mtx)
 	debug_printf ("CreateMutex failed: %E");
     }
+  if (get_dev () == FH_PIPEW && !query_hdl)
+    set_pipe_non_blocking (is_nonblocking ());
 }
 
 off_t
@@ -268,39 +272,22 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
   while (nbytes < len)
     {
       ULONG_PTR nbytes_now = 0;
-      size_t left = len - nbytes;
-      ULONG len1 = (ULONG) left;
+      ULONG len1 = (ULONG) (len - nbytes);
       waitret = WAIT_OBJECT_0;
 
       if (evt)
 	ResetEvent (evt);
-      if (!is_nonblocking ())
+      FILE_PIPE_LOCAL_INFORMATION fpli;
+      status = NtQueryInformationFile (get_handle (), &io,
+				       &fpli, sizeof (fpli),
+				       FilePipeLocalInformation);
+      if (NT_SUCCESS (status))
 	{
-	  FILE_PIPE_LOCAL_INFORMATION fpli;
-
-	  /* If the pipe is empty, don't request more bytes than pipe
-	     buffer size - 1. Pending read lowers WriteQuotaAvailable on
-	     the write side and thus affects select's ability to return
-	     more or less reliable info whether a write succeeds or not. */
-	  ULONG chunk = pipe_buf_size - 1;
-	  status = NtQueryInformationFile (get_handle (), &io,
-					   &fpli, sizeof (fpli),
-					   FilePipeLocalInformation);
-	  if (NT_SUCCESS (status))
-	    {
-	      if (fpli.ReadDataAvailable > 0)
-		chunk = left;
-	      else if (nbytes != 0)
-		break;
-	      else
-		chunk = fpli.InboundQuota - 1;
-	    }
-	  else if (nbytes != 0)
-	    break;
-
-	  if (len1 > chunk)
-	    len1 = chunk;
+	if (fpli.ReadDataAvailable == 0 && nbytes != 0)
+	  break;
 	}
+      else if (nbytes != 0)
+	break;
       status = NtReadFile (get_handle (), evt, NULL, NULL, &io, ptr,
 			   len1, NULL, NULL);
       if (evt && status == STATUS_PENDING)
@@ -385,6 +372,16 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
   len = nbytes;
 }
 
+bool
+fhandler_pipe_fifo::reader_closed ()
+{
+  if (!query_hdl)
+    return false;
+  int n_reader = get_obj_handle_count (query_hdl);
+  int n_writer = get_obj_handle_count (get_handle ());
+  return n_reader == n_writer;
+}
+
 ssize_t __reg3
 fhandler_pipe_fifo::raw_write (const void *ptr, size_t len)
 {
@@ -493,7 +490,20 @@ fhandler_pipe_fifo::raw_write (const void *ptr, size_t len)
 			      get_obj_handle_count (select_sem), NULL);
 	  /* 0 bytes returned?  EAGAIN.  See above. */
 	  if (NT_SUCCESS (status) && nbytes == 0)
-	    set_errno (EAGAIN);
+	    {
+	      if (reader_closed ())
+		{
+		  set_errno (EPIPE);
+		  raise (SIGPIPE);
+		}
+	      else if (is_nonblocking ())
+		set_errno (EAGAIN);
+	      else
+		{
+		  cygwait (select_sem, 10);
+		  continue;
+		}
+	    }
 	}
       else if (STATUS_PIPE_IS_CLOSED (status))
 	{
@@ -522,6 +532,10 @@ fhandler_pipe::fixup_after_fork (HANDLE parent)
     fork_fixup (parent, read_mtx, "read_mtx");
   if (select_sem)
     fork_fixup (parent, select_sem, "select_sem");
+  /* Do not duplicate query_hdl if it has been already inherited. */
+  if (query_hdl && !get_obj_handle_count (query_hdl))
+    fork_fixup (parent, query_hdl, "query_hdl");
+
   fhandler_base::fixup_after_fork (parent);
 }
 
@@ -552,6 +566,15 @@ fhandler_pipe::dup (fhandler_base *child, int flags)
       ftp->close ();
       res = -1;
     }
+  else if (query_hdl &&
+	   !DuplicateHandle (GetCurrentProcess (), query_hdl,
+			    GetCurrentProcess (), &ftp->query_hdl,
+			    0, !(flags & O_CLOEXEC), DUPLICATE_SAME_ACCESS))
+    {
+      __seterrno ();
+      ftp->close ();
+      res = -1;
+    }
 
   debug_printf ("res %d", res);
   return res;
@@ -567,6 +590,8 @@ fhandler_pipe::close ()
       ReleaseSemaphore (select_sem, get_obj_handle_count (select_sem), NULL);
       CloseHandle (select_sem);
     }
+  if (query_hdl)
+    CloseHandle (query_hdl);
   return fhandler_base::close ();
 }
 
@@ -791,6 +816,23 @@ fhandler_pipe::create (fhandler_pipe *fhs[2], unsigned psize, int mode)
 	DuplicateHandle (GetCurrentProcess (), fhs[0]->select_sem,
 			 GetCurrentProcess (), &fhs[1]->select_sem,
 			 0, 1, DUPLICATE_SAME_ACCESS);
+      if (!DuplicateHandle (GetCurrentProcess (), r,
+			    GetCurrentProcess (), &fhs[1]->query_hdl,
+			    GENERIC_READ, !(mode & O_CLOEXEC), 0))
+	{
+	  CloseHandle (fhs[0]->select_sem);
+	  delete fhs[0];
+	  CloseHandle (r);
+	  CloseHandle (fhs[1]->select_sem);
+	  delete fhs[1];
+	  CloseHandle (w);
+	}
+      else
+	{
+	  /* Call set_pipe_non_blocking() again after creating query_hdl. */
+	  fhs[1]->set_pipe_non_blocking (fhs[1]->is_nonblocking ());
+	  res = 0;
+	}
     }
 
   debug_printf ("%R = pipe([%p, %p], %d, %y)", res, fhs[0], fhs[1], psize, mode);
diff --git a/winsup/cygwin/select.cc b/winsup/cygwin/select.cc
index 5e583434c..c569a059c 100644
--- a/winsup/cygwin/select.cc
+++ b/winsup/cygwin/select.cc
@@ -608,12 +608,43 @@ pipe_data_available (int fd, fhandler_base *fh, HANDLE h, bool writing)
     }
   if (writing)
     {
-      /* WriteQuotaAvailable is decremented by the number of bytes requested
-	 by a blocking reader on the other side of the pipe.  Cygwin readers
-	 are serialized and never request a number of bytes equivalent to the
-	 full buffer size.  So WriteQuotaAvailable is 0 only if either the
-	 read buffer on the other side is really full, or if we have non-Cygwin
-	 readers. */
+      /* If there is anything available in the pipe buffer then signal
+        that.  This means that a pipe could still block since you could
+        be trying to write more to the pipe than is available in the
+        buffer but that is the hazard of select().
+
+        Note that WriteQuotaAvailable is unreliable.
+
+        Usually WriteQuotaAvailable on the write side reflects the space
+        available in the inbound buffer on the read side.  However, if a
+        pipe read is currently pending, WriteQuotaAvailable on the write side
+        is decremented by the number of bytes the read side is requesting.
+        So it's possible (even likely) that WriteQuotaAvailable is 0, even
+        if the inbound buffer on the read side is not full.  This can lead to
+        a deadlock situation: The reader is waiting for data, but select
+        on the writer side assumes that no space is available in the read
+        side inbound buffer.
+
+        Consequentially, the only reliable information is available on the
+        read side, so fetch info from the read side via the pipe-specific
+        query handle.  Use fpli.WriteQuotaAvailable as storage for the actual
+        interesting value, which is the OutboundQuote on the write side,
+        decremented by the number of bytes of data in that buffer. */
+      /* Note: Do not use NtQueryInformationFile() for query_hdl because
+	 NtQueryInformationFile() seems to interfere with reading pipes
+	 in non-cygwin apps. Instead, use PeekNamedPipe() here. */
+      if (fh->get_device () == FH_PIPEW)
+	{
+	  HANDLE query_hdl = ((fhandler_pipe *) fh)->get_query_handle ();
+	  if (query_hdl)
+	    {
+	      DWORD nbytes_in_pipe;
+	      PeekNamedPipe (query_hdl, NULL, 0, NULL, &nbytes_in_pipe, NULL);
+	      fpli.WriteQuotaAvailable = fpli.OutboundQuota - nbytes_in_pipe;
+	    }
+	  else
+	    return 1;
+	}
       if (fpli.WriteQuotaAvailable > 0)
 	{
 	  paranoid_printf ("fd %d, %s, write: size %u, avail %u", fd,
@@ -712,6 +743,13 @@ out:
   h = fh->get_output_handle ();
   if (s->write_selected && dev != FH_PIPER)
     {
+      if (dev == FH_PIPEW && ((fhandler_pipe *) fh)->reader_closed ())
+	{
+	  gotone += s->write_ready = true;
+	  if (s->except_selected)
+	    gotone += s->except_ready = true;
+	  return gotone;
+	}
       gotone += s->write_ready =  pipe_data_available (s->fd, fh, h, true);
       select_printf ("write: %s, gotone %d", fh->get_name (), gotone);
     }
diff --git a/winsup/cygwin/spawn.cc b/winsup/cygwin/spawn.cc
index 0bde0b04d..e902b5080 100644
--- a/winsup/cygwin/spawn.cc
+++ b/winsup/cygwin/spawn.cc
@@ -657,6 +657,8 @@ child_info_spawn::worker (const char *prog_arg, const char *const *argv,
 		ptys->create_invisible_console ();
 		ptys->setup_locale ();
 	      }
+	    else if (cfd->get_dev () == FH_PIPEW)
+	      ((fhandler_pipe *)(fhandler_base *) cfd)->close_query_handle ();
 	}
 
       bool enable_pcon = false;
-- 
2.33.0


^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-12  6:23                                   ` Takashi Yano
@ 2021-09-12 14:39                                     ` Ken Brown
  2021-09-13  9:11                                       ` Corinna Vinschen
  0 siblings, 1 reply; 250+ messages in thread
From: Ken Brown @ 2021-09-12 14:39 UTC (permalink / raw)
  To: cygwin-developers

On 9/12/2021 2:23 AM, Takashi Yano wrote:
> On Sat, 11 Sep 2021 09:12:02 -0400
> Ken Brown wrote:
>> On 9/10/2021 10:35 PM, Takashi Yano wrote:
>>> On Fri, 10 Sep 2021 22:17:21 -0400
>>> Ken Brown wrote:
>>>> On 9/10/2021 6:57 PM, Takashi Yano wrote:
>>>>> On Fri, 10 Sep 2021 11:17:58 -0400
>>>>> Ken Brown wrote:
>>>>>> I've rerun your test with the latest version, and the test results are similar.
>>>>>>      I've also run a suite of fifo tests that I've accumulated, and they all pass
>>>>>> also, so I pushed your patch.
>>>>>>
>>>>>> I think we're in pretty good shape now.  The only detail remaining, AFAIK, is
>>>>>> how to best avoid a deadlock if the pipe has been created by a non-Cygwin
>>>>>> process.  I've proposed a timeout, but maybe there's a better idea.
>>>>>
>>>>> I am not pretty sure what is the problem, but is not the following
>>>>> patch enough?
>>>>>
>>>>> diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
>>>>> index d309be2f7..13fba9a14 100644
>>>>> --- a/winsup/cygwin/fhandler.h
>>>>> +++ b/winsup/cygwin/fhandler.h
>>>>> @@ -1205,6 +1205,7 @@ public:
>>>>>       select_record *select_except (select_stuff *);
>>>>>       char *get_proc_fd_name (char *buf);
>>>>>       int open (int flags, mode_t mode = 0);
>>>>> +  void open_setup (int flags);
>>>>>       void fixup_after_fork (HANDLE);
>>>>>       int dup (fhandler_base *child, int);
>>>>>       int close ();
>>>>> diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
>>>>> index 6994a5dce..d84e6ad84 100644
>>>>> --- a/winsup/cygwin/fhandler_pipe.cc
>>>>> +++ b/winsup/cygwin/fhandler_pipe.cc
>>>>> @@ -191,6 +191,17 @@ out:
>>>>>       return 0;
>>>>>     }
>>>>>
>>>>> +void
>>>>> +fhandler_pipe::open_setup (int flags)
>>>>> +{
>>>>> +  fhandler_base::open_setup (flags);
>>>>> +  if (get_dev () == FH_PIPER && !read_mtx)
>>>>> +    {
>>>>> +      SECURITY_ATTRIBUTES *sa = sec_none_cloexec (flags);
>>>>> +      read_mtx = CreateMutexW (sa, FALSE, NULL);
>>>>> +    }
>>>>> +}
>>>>> +
>>>>>     off_t
>>>>>     fhandler_pipe::lseek (off_t offset, int whence)
>>>>>     {
>>>>>
>>>>>
>>>>> AFAIK, another problem remaining is:
>>>>>
>>>>> On Mon, 6 Sep 2021 14:49:55 +0200
>>>>> Corinna Vinschen wrote:
>>>>>> - What about calling select for writing on pipes read by non-Cygwin
>>>>>>      processes?  In that case, we still can't rely on WriteQuotaAvailable,
>>>>>>      just as before.
>>>>
>>>> This is the problem I was talking about.  In this case the non-Cygwin process
>>>> might have a large pending read, so that the Cygwin process calling select on
>>>> the write side will see WriteQuotaAvailable == 0.  This could lead to a deadlock
>>>> with the Cygwin process waiting for write ready while the non-Cygwin process is
>>>> blocked trying to read.
>>>
>>> Then, the above patch is for another issue.
>>> The problem happes when:
>>> 1) Start command prompt.
>>> 2) Run 'echo AAAAAAAAAAAA | \cygwin64\bin\cat
>>> This causes hang up in cat. In this case, pipe is created by cmd.exe.
>>> Therefore, read_mtx is not created.
>>
>> Confirmed, and your patch fixes it.  Maybe you should check for error in the
>> call to CreateMutexW and print a debug message in that case.
> 
> I added the debug message.

LGTM.

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-12 11:04                                     ` Takashi Yano
@ 2021-09-12 15:10                                       ` Ken Brown
  2021-09-12 21:46                                         ` Ken Brown
  2021-09-12 23:41                                         ` Takashi Yano
  2021-09-13 17:42                                       ` Ken Brown
  2021-09-13 18:32                                       ` Corinna Vinschen
  2 siblings, 2 replies; 250+ messages in thread
From: Ken Brown @ 2021-09-12 15:10 UTC (permalink / raw)
  To: cygwin-developers

On 9/12/2021 7:04 AM, Takashi Yano wrote:
> On Sun, 12 Sep 2021 17:48:49 +0900
> Takashi Yano wrote:
>> On Sat, 11 Sep 2021 09:12:02 -0400
>> Ken Brown wrote:
>>> On 9/10/2021 10:35 PM, Takashi Yano wrote:
>>>> On Fri, 10 Sep 2021 22:17:21 -0400
>>>> Ken Brown wrote:
>>>>> On 9/10/2021 6:57 PM, Takashi Yano wrote:
>>>>>> On Fri, 10 Sep 2021 11:17:58 -0400
>>>>>> Ken Brown wrote:
>>>>>>> I've rerun your test with the latest version, and the test results are similar.
>>>>>>>      I've also run a suite of fifo tests that I've accumulated, and they all pass
>>>>>>> also, so I pushed your patch.
>>>>>>>
>>>>>>> I think we're in pretty good shape now.  The only detail remaining, AFAIK, is
>>>>>>> how to best avoid a deadlock if the pipe has been created by a non-Cygwin
>>>>>>> process.  I've proposed a timeout, but maybe there's a better idea.
>>>>>>
>>>>>> I am not pretty sure what is the problem, but is not the following
>>>>>> patch enough?
>>>>>>
>>>>>> diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
>>>>>> index d309be2f7..13fba9a14 100644
>>>>>> --- a/winsup/cygwin/fhandler.h
>>>>>> +++ b/winsup/cygwin/fhandler.h
>>>>>> @@ -1205,6 +1205,7 @@ public:
>>>>>>       select_record *select_except (select_stuff *);
>>>>>>       char *get_proc_fd_name (char *buf);
>>>>>>       int open (int flags, mode_t mode = 0);
>>>>>> +  void open_setup (int flags);
>>>>>>       void fixup_after_fork (HANDLE);
>>>>>>       int dup (fhandler_base *child, int);
>>>>>>       int close ();
>>>>>> diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
>>>>>> index 6994a5dce..d84e6ad84 100644
>>>>>> --- a/winsup/cygwin/fhandler_pipe.cc
>>>>>> +++ b/winsup/cygwin/fhandler_pipe.cc
>>>>>> @@ -191,6 +191,17 @@ out:
>>>>>>       return 0;
>>>>>>     }
>>>>>>
>>>>>> +void
>>>>>> +fhandler_pipe::open_setup (int flags)
>>>>>> +{
>>>>>> +  fhandler_base::open_setup (flags);
>>>>>> +  if (get_dev () == FH_PIPER && !read_mtx)
>>>>>> +    {
>>>>>> +      SECURITY_ATTRIBUTES *sa = sec_none_cloexec (flags);
>>>>>> +      read_mtx = CreateMutexW (sa, FALSE, NULL);
>>>>>> +    }
>>>>>> +}
>>>>>> +
>>>>>>     off_t
>>>>>>     fhandler_pipe::lseek (off_t offset, int whence)
>>>>>>     {
>>>>>>
>>>>>>
>>>>>> AFAIK, another problem remaining is:
>>>>>>
>>>>>> On Mon, 6 Sep 2021 14:49:55 +0200
>>>>>> Corinna Vinschen wrote:
>>>>>>> - What about calling select for writing on pipes read by non-Cygwin
>>>>>>>      processes?  In that case, we still can't rely on WriteQuotaAvailable,
>>>>>>>      just as before.
>>>>>
>>>>> This is the problem I was talking about.  In this case the non-Cygwin process
>>>>> might have a large pending read, so that the Cygwin process calling select on
>>>>> the write side will see WriteQuotaAvailable == 0.  This could lead to a deadlock
>>>>> with the Cygwin process waiting for write ready while the non-Cygwin process is
>>>>> blocked trying to read.
>>>>
>>>> Then, the above patch is for another issue.
>>>> The problem happes when:
>>>> 1) Start command prompt.
>>>> 2) Run 'echo AAAAAAAAAAAA | \cygwin64\bin\cat
>>>> This causes hang up in cat. In this case, pipe is created by cmd.exe.
>>>> Therefore, read_mtx is not created.
>>>
>>> Confirmed, and your patch fixes it.  Maybe you should check for error in the
>>> call to CreateMutexW and print a debug message in that case.
>>>
>>>>> My suggestion is that we impose a timeout in this situation, after which select
>>>>> reports write ready.
>>>>
>>>> Keeping read handle in write pipe (Corinna's query_hdl) causes problem
>>>> that write side cannot detect close on read side.
>>>> Is it possible to open read handle temporally when pipe_data_available()
>>>> is called?
>>>
>>> That would be nice, but I have no idea how you could do that.
>>
>> Hmm. Then, what about PoC code attached? This returns to Corinna's
>> query_hdl, and counts read/write handles to detect closing reader side.
>>
>> If the number of read handles is equal to number of write handles,
>> only the pairs of write handle and query_hdl are alive. So, read pipe
>> supposed to be closed.
>>
>> This patch depends another patch I posted a few hours ago.
> 
> Revised a bit.

I don't see how this solves the problem.  In the case we were worried about 
where we have a non-Cygwin reader, the writer has no query_hdl, and you're just 
always reporting write ready, aren't you?  Or am I missing something?

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-12 15:10                                       ` Ken Brown
@ 2021-09-12 21:46                                         ` Ken Brown
  2021-09-12 23:54                                           ` Takashi Yano
  2021-09-13  9:42                                           ` Corinna Vinschen
  2021-09-12 23:41                                         ` Takashi Yano
  1 sibling, 2 replies; 250+ messages in thread
From: Ken Brown @ 2021-09-12 21:46 UTC (permalink / raw)
  To: cygwin-developers

On 9/12/2021 11:10 AM, Ken Brown wrote:
> On 9/12/2021 7:04 AM, Takashi Yano wrote:
>> On Sun, 12 Sep 2021 17:48:49 +0900
>> Takashi Yano wrote:
>>> On Sat, 11 Sep 2021 09:12:02 -0400
>>> Ken Brown wrote:
>>>> On 9/10/2021 10:35 PM, Takashi Yano wrote:
>>>>> On Fri, 10 Sep 2021 22:17:21 -0400
>>>>> Ken Brown wrote:
>>>>>> On 9/10/2021 6:57 PM, Takashi Yano wrote:
>>>>>>> On Fri, 10 Sep 2021 11:17:58 -0400
>>>>>>> Ken Brown wrote:
>>>>>>>> I've rerun your test with the latest version, and the test results are 
>>>>>>>> similar.
>>>>>>>>      I've also run a suite of fifo tests that I've accumulated, and they 
>>>>>>>> all pass
>>>>>>>> also, so I pushed your patch.
>>>>>>>>
>>>>>>>> I think we're in pretty good shape now.  The only detail remaining, 
>>>>>>>> AFAIK, is
>>>>>>>> how to best avoid a deadlock if the pipe has been created by a non-Cygwin
>>>>>>>> process.  I've proposed a timeout, but maybe there's a better idea.
>>>>>>>
>>>>>>> I am not pretty sure what is the problem, but is not the following
>>>>>>> patch enough?
>>>>>>>
>>>>>>> diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
>>>>>>> index d309be2f7..13fba9a14 100644
>>>>>>> --- a/winsup/cygwin/fhandler.h
>>>>>>> +++ b/winsup/cygwin/fhandler.h
>>>>>>> @@ -1205,6 +1205,7 @@ public:
>>>>>>>       select_record *select_except (select_stuff *);
>>>>>>>       char *get_proc_fd_name (char *buf);
>>>>>>>       int open (int flags, mode_t mode = 0);
>>>>>>> +  void open_setup (int flags);
>>>>>>>       void fixup_after_fork (HANDLE);
>>>>>>>       int dup (fhandler_base *child, int);
>>>>>>>       int close ();
>>>>>>> diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
>>>>>>> index 6994a5dce..d84e6ad84 100644
>>>>>>> --- a/winsup/cygwin/fhandler_pipe.cc
>>>>>>> +++ b/winsup/cygwin/fhandler_pipe.cc
>>>>>>> @@ -191,6 +191,17 @@ out:
>>>>>>>       return 0;
>>>>>>>     }
>>>>>>>
>>>>>>> +void
>>>>>>> +fhandler_pipe::open_setup (int flags)
>>>>>>> +{
>>>>>>> +  fhandler_base::open_setup (flags);
>>>>>>> +  if (get_dev () == FH_PIPER && !read_mtx)
>>>>>>> +    {
>>>>>>> +      SECURITY_ATTRIBUTES *sa = sec_none_cloexec (flags);
>>>>>>> +      read_mtx = CreateMutexW (sa, FALSE, NULL);
>>>>>>> +    }
>>>>>>> +}
>>>>>>> +
>>>>>>>     off_t
>>>>>>>     fhandler_pipe::lseek (off_t offset, int whence)
>>>>>>>     {
>>>>>>>
>>>>>>>
>>>>>>> AFAIK, another problem remaining is:
>>>>>>>
>>>>>>> On Mon, 6 Sep 2021 14:49:55 +0200
>>>>>>> Corinna Vinschen wrote:
>>>>>>>> - What about calling select for writing on pipes read by non-Cygwin
>>>>>>>>      processes?  In that case, we still can't rely on WriteQuotaAvailable,
>>>>>>>>      just as before.
>>>>>>
>>>>>> This is the problem I was talking about.  In this case the non-Cygwin process
>>>>>> might have a large pending read, so that the Cygwin process calling select on
>>>>>> the write side will see WriteQuotaAvailable == 0.  This could lead to a 
>>>>>> deadlock
>>>>>> with the Cygwin process waiting for write ready while the non-Cygwin 
>>>>>> process is
>>>>>> blocked trying to read.
>>>>>
>>>>> Then, the above patch is for another issue.
>>>>> The problem happes when:
>>>>> 1) Start command prompt.
>>>>> 2) Run 'echo AAAAAAAAAAAA | \cygwin64\bin\cat
>>>>> This causes hang up in cat. In this case, pipe is created by cmd.exe.
>>>>> Therefore, read_mtx is not created.
>>>>
>>>> Confirmed, and your patch fixes it.  Maybe you should check for error in the
>>>> call to CreateMutexW and print a debug message in that case.
>>>>
>>>>>> My suggestion is that we impose a timeout in this situation, after which 
>>>>>> select
>>>>>> reports write ready.
>>>>>
>>>>> Keeping read handle in write pipe (Corinna's query_hdl) causes problem
>>>>> that write side cannot detect close on read side.
>>>>> Is it possible to open read handle temporally when pipe_data_available()
>>>>> is called?
>>>>
>>>> That would be nice, but I have no idea how you could do that.
>>>
>>> Hmm. Then, what about PoC code attached? This returns to Corinna's
>>> query_hdl, and counts read/write handles to detect closing reader side.
>>>
>>> If the number of read handles is equal to number of write handles,
>>> only the pairs of write handle and query_hdl are alive. So, read pipe
>>> supposed to be closed.
>>>
>>> This patch depends another patch I posted a few hours ago.
>>
>> Revised a bit.
> 
> I don't see how this solves the problem.  In the case we were worried about 
> where we have a non-Cygwin reader, the writer has no query_hdl, and you're just 
> always reporting write ready, aren't you?  Or am I missing something?

BTW, we could just decide that always reporting write ready in this corner case 
is acceptable.  But then we could just do that without going back to query_hdl.

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-12 15:10                                       ` Ken Brown
  2021-09-12 21:46                                         ` Ken Brown
@ 2021-09-12 23:41                                         ` Takashi Yano
  1 sibling, 0 replies; 250+ messages in thread
From: Takashi Yano @ 2021-09-12 23:41 UTC (permalink / raw)
  To: cygwin-developers

On Sun, 12 Sep 2021 11:10:54 -0400
Ken Brown wrote:
> On 9/12/2021 7:04 AM, Takashi Yano wrote:
> > On Sun, 12 Sep 2021 17:48:49 +0900
> > Takashi Yano wrote:
> >> On Sat, 11 Sep 2021 09:12:02 -0400
> >> Ken Brown wrote:
> >>> On 9/10/2021 10:35 PM, Takashi Yano wrote:
> >>>> On Fri, 10 Sep 2021 22:17:21 -0400
> >>>> Ken Brown wrote:
> >>>>> On 9/10/2021 6:57 PM, Takashi Yano wrote:
> >>>>>> On Fri, 10 Sep 2021 11:17:58 -0400
> >>>>>> Ken Brown wrote:
> >>>>>>> I've rerun your test with the latest version, and the test results are similar.
> >>>>>>>      I've also run a suite of fifo tests that I've accumulated, and they all pass
> >>>>>>> also, so I pushed your patch.
> >>>>>>>
> >>>>>>> I think we're in pretty good shape now.  The only detail remaining, AFAIK, is
> >>>>>>> how to best avoid a deadlock if the pipe has been created by a non-Cygwin
> >>>>>>> process.  I've proposed a timeout, but maybe there's a better idea.
> >>>>>>
> >>>>>> I am not pretty sure what is the problem, but is not the following
> >>>>>> patch enough?
> >>>>>>
> >>>>>> diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
> >>>>>> index d309be2f7..13fba9a14 100644
> >>>>>> --- a/winsup/cygwin/fhandler.h
> >>>>>> +++ b/winsup/cygwin/fhandler.h
> >>>>>> @@ -1205,6 +1205,7 @@ public:
> >>>>>>       select_record *select_except (select_stuff *);
> >>>>>>       char *get_proc_fd_name (char *buf);
> >>>>>>       int open (int flags, mode_t mode = 0);
> >>>>>> +  void open_setup (int flags);
> >>>>>>       void fixup_after_fork (HANDLE);
> >>>>>>       int dup (fhandler_base *child, int);
> >>>>>>       int close ();
> >>>>>> diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
> >>>>>> index 6994a5dce..d84e6ad84 100644
> >>>>>> --- a/winsup/cygwin/fhandler_pipe.cc
> >>>>>> +++ b/winsup/cygwin/fhandler_pipe.cc
> >>>>>> @@ -191,6 +191,17 @@ out:
> >>>>>>       return 0;
> >>>>>>     }
> >>>>>>
> >>>>>> +void
> >>>>>> +fhandler_pipe::open_setup (int flags)
> >>>>>> +{
> >>>>>> +  fhandler_base::open_setup (flags);
> >>>>>> +  if (get_dev () == FH_PIPER && !read_mtx)
> >>>>>> +    {
> >>>>>> +      SECURITY_ATTRIBUTES *sa = sec_none_cloexec (flags);
> >>>>>> +      read_mtx = CreateMutexW (sa, FALSE, NULL);
> >>>>>> +    }
> >>>>>> +}
> >>>>>> +
> >>>>>>     off_t
> >>>>>>     fhandler_pipe::lseek (off_t offset, int whence)
> >>>>>>     {
> >>>>>>
> >>>>>>
> >>>>>> AFAIK, another problem remaining is:
> >>>>>>
> >>>>>> On Mon, 6 Sep 2021 14:49:55 +0200
> >>>>>> Corinna Vinschen wrote:
> >>>>>>> - What about calling select for writing on pipes read by non-Cygwin
> >>>>>>>      processes?  In that case, we still can't rely on WriteQuotaAvailable,
> >>>>>>>      just as before.
> >>>>>
> >>>>> This is the problem I was talking about.  In this case the non-Cygwin process
> >>>>> might have a large pending read, so that the Cygwin process calling select on
> >>>>> the write side will see WriteQuotaAvailable == 0.  This could lead to a deadlock
> >>>>> with the Cygwin process waiting for write ready while the non-Cygwin process is
> >>>>> blocked trying to read.
> >>>>
> >>>> Then, the above patch is for another issue.
> >>>> The problem happes when:
> >>>> 1) Start command prompt.
> >>>> 2) Run 'echo AAAAAAAAAAAA | \cygwin64\bin\cat
> >>>> This causes hang up in cat. In this case, pipe is created by cmd.exe.
> >>>> Therefore, read_mtx is not created.
> >>>
> >>> Confirmed, and your patch fixes it.  Maybe you should check for error in the
> >>> call to CreateMutexW and print a debug message in that case.
> >>>
> >>>>> My suggestion is that we impose a timeout in this situation, after which select
> >>>>> reports write ready.
> >>>>
> >>>> Keeping read handle in write pipe (Corinna's query_hdl) causes problem
> >>>> that write side cannot detect close on read side.
> >>>> Is it possible to open read handle temporally when pipe_data_available()
> >>>> is called?
> >>>
> >>> That would be nice, but I have no idea how you could do that.
> >>
> >> Hmm. Then, what about PoC code attached? This returns to Corinna's
> >> query_hdl, and counts read/write handles to detect closing reader side.
> >>
> >> If the number of read handles is equal to number of write handles,
> >> only the pairs of write handle and query_hdl are alive. So, read pipe
> >> supposed to be closed.
> >>
> >> This patch depends another patch I posted a few hours ago.
> > 
> > Revised a bit.
> 
> I don't see how this solves the problem.  In the case we were worried about 
> where we have a non-Cygwin reader, the writer has no query_hdl, and you're just 
> always reporting write ready, aren't you?  Or am I missing something?

Do you assume pipe is created by non-cygwin app?

My assumption is:
1) Pipe is created by cygwin.
2) Writer is cygwin app using select().
3) Reader is non-cygwin app reading large block.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-12 21:46                                         ` Ken Brown
@ 2021-09-12 23:54                                           ` Takashi Yano
  2021-09-13  2:19                                             ` Ken Brown
  2021-09-13  8:40                                             ` Takashi Yano
  2021-09-13  9:42                                           ` Corinna Vinschen
  1 sibling, 2 replies; 250+ messages in thread
From: Takashi Yano @ 2021-09-12 23:54 UTC (permalink / raw)
  To: cygwin-developers

On Sun, 12 Sep 2021 17:46:47 -0400
Ken Brown wrote:
> On 9/12/2021 11:10 AM, Ken Brown wrote:
> > On 9/12/2021 7:04 AM, Takashi Yano wrote:
> >> On Sun, 12 Sep 2021 17:48:49 +0900
> >> Takashi Yano wrote:
> >>> On Sat, 11 Sep 2021 09:12:02 -0400
> >>> Ken Brown wrote:
> >>>> On 9/10/2021 10:35 PM, Takashi Yano wrote:
> >>>>> On Fri, 10 Sep 2021 22:17:21 -0400
> >>>>> Ken Brown wrote:
> >>>>>> On 9/10/2021 6:57 PM, Takashi Yano wrote:
> >>>>>>> On Fri, 10 Sep 2021 11:17:58 -0400
> >>>>>>> Ken Brown wrote:
> >>>>>>>> I've rerun your test with the latest version, and the test results are 
> >>>>>>>> similar.
> >>>>>>>>      I've also run a suite of fifo tests that I've accumulated, and they 
> >>>>>>>> all pass
> >>>>>>>> also, so I pushed your patch.
> >>>>>>>>
> >>>>>>>> I think we're in pretty good shape now.  The only detail remaining, 
> >>>>>>>> AFAIK, is
> >>>>>>>> how to best avoid a deadlock if the pipe has been created by a non-Cygwin
> >>>>>>>> process.  I've proposed a timeout, but maybe there's a better idea.
> >>>>>>>
> >>>>>>> I am not pretty sure what is the problem, but is not the following
> >>>>>>> patch enough?
> >>>>>>>
> >>>>>>> diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
> >>>>>>> index d309be2f7..13fba9a14 100644
> >>>>>>> --- a/winsup/cygwin/fhandler.h
> >>>>>>> +++ b/winsup/cygwin/fhandler.h
> >>>>>>> @@ -1205,6 +1205,7 @@ public:
> >>>>>>>       select_record *select_except (select_stuff *);
> >>>>>>>       char *get_proc_fd_name (char *buf);
> >>>>>>>       int open (int flags, mode_t mode = 0);
> >>>>>>> +  void open_setup (int flags);
> >>>>>>>       void fixup_after_fork (HANDLE);
> >>>>>>>       int dup (fhandler_base *child, int);
> >>>>>>>       int close ();
> >>>>>>> diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
> >>>>>>> index 6994a5dce..d84e6ad84 100644
> >>>>>>> --- a/winsup/cygwin/fhandler_pipe.cc
> >>>>>>> +++ b/winsup/cygwin/fhandler_pipe.cc
> >>>>>>> @@ -191,6 +191,17 @@ out:
> >>>>>>>       return 0;
> >>>>>>>     }
> >>>>>>>
> >>>>>>> +void
> >>>>>>> +fhandler_pipe::open_setup (int flags)
> >>>>>>> +{
> >>>>>>> +  fhandler_base::open_setup (flags);
> >>>>>>> +  if (get_dev () == FH_PIPER && !read_mtx)
> >>>>>>> +    {
> >>>>>>> +      SECURITY_ATTRIBUTES *sa = sec_none_cloexec (flags);
> >>>>>>> +      read_mtx = CreateMutexW (sa, FALSE, NULL);
> >>>>>>> +    }
> >>>>>>> +}
> >>>>>>> +
> >>>>>>>     off_t
> >>>>>>>     fhandler_pipe::lseek (off_t offset, int whence)
> >>>>>>>     {
> >>>>>>>
> >>>>>>>
> >>>>>>> AFAIK, another problem remaining is:
> >>>>>>>
> >>>>>>> On Mon, 6 Sep 2021 14:49:55 +0200
> >>>>>>> Corinna Vinschen wrote:
> >>>>>>>> - What about calling select for writing on pipes read by non-Cygwin
> >>>>>>>>      processes?  In that case, we still can't rely on WriteQuotaAvailable,
> >>>>>>>>      just as before.
> >>>>>>
> >>>>>> This is the problem I was talking about.  In this case the non-Cygwin process
> >>>>>> might have a large pending read, so that the Cygwin process calling select on
> >>>>>> the write side will see WriteQuotaAvailable == 0.  This could lead to a 
> >>>>>> deadlock
> >>>>>> with the Cygwin process waiting for write ready while the non-Cygwin 
> >>>>>> process is
> >>>>>> blocked trying to read.
> >>>>>
> >>>>> Then, the above patch is for another issue.
> >>>>> The problem happes when:
> >>>>> 1) Start command prompt.
> >>>>> 2) Run 'echo AAAAAAAAAAAA | \cygwin64\bin\cat
> >>>>> This causes hang up in cat. In this case, pipe is created by cmd.exe.
> >>>>> Therefore, read_mtx is not created.
> >>>>
> >>>> Confirmed, and your patch fixes it.  Maybe you should check for error in the
> >>>> call to CreateMutexW and print a debug message in that case.
> >>>>
> >>>>>> My suggestion is that we impose a timeout in this situation, after which 
> >>>>>> select
> >>>>>> reports write ready.
> >>>>>
> >>>>> Keeping read handle in write pipe (Corinna's query_hdl) causes problem
> >>>>> that write side cannot detect close on read side.
> >>>>> Is it possible to open read handle temporally when pipe_data_available()
> >>>>> is called?
> >>>>
> >>>> That would be nice, but I have no idea how you could do that.
> >>>
> >>> Hmm. Then, what about PoC code attached? This returns to Corinna's
> >>> query_hdl, and counts read/write handles to detect closing reader side.
> >>>
> >>> If the number of read handles is equal to number of write handles,
> >>> only the pairs of write handle and query_hdl are alive. So, read pipe
> >>> supposed to be closed.
> >>>
> >>> This patch depends another patch I posted a few hours ago.
> >>
> >> Revised a bit.
> > 
> > I don't see how this solves the problem.  In the case we were worried about 
> > where we have a non-Cygwin reader, the writer has no query_hdl, and you're just 
> > always reporting write ready, aren't you?  Or am I missing something?
> 
> BTW, we could just decide that always reporting write ready in this corner case 
> is acceptable.  But then we could just do that without going back to query_hdl.

The various combination of cygwin and non-cygwin cases resuts in:

W P R: current, query_hdl
c c c: OK     , OK
c c n: NG     , OK
c n c: OK     , OK
c n n: NG     , select() always report write ready

where

W: Writer
P: Pipe
R: Reder
c: cygwin
n: non-cygwin

* Reder requests larger block than pipe size.
* Writer cannot be non-cygwin because we assume the case
  that writer uses select().

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-12 23:54                                           ` Takashi Yano
@ 2021-09-13  2:19                                             ` Ken Brown
  2021-09-13  8:40                                             ` Takashi Yano
  1 sibling, 0 replies; 250+ messages in thread
From: Ken Brown @ 2021-09-13  2:19 UTC (permalink / raw)
  To: cygwin-developers

On 9/12/2021 7:54 PM, Takashi Yano wrote:
> On Sun, 12 Sep 2021 17:46:47 -0400
> Ken Brown wrote:
>> On 9/12/2021 11:10 AM, Ken Brown wrote:
>>> On 9/12/2021 7:04 AM, Takashi Yano wrote:
>>>> On Sun, 12 Sep 2021 17:48:49 +0900
>>>> Takashi Yano wrote:
>>>>> On Sat, 11 Sep 2021 09:12:02 -0400
>>>>> Ken Brown wrote:
>>>>>> On 9/10/2021 10:35 PM, Takashi Yano wrote:
>>>>>>> On Fri, 10 Sep 2021 22:17:21 -0400
>>>>>>> Ken Brown wrote:
>>>>>>>> On 9/10/2021 6:57 PM, Takashi Yano wrote:
>>>>>>>>> On Fri, 10 Sep 2021 11:17:58 -0400
>>>>>>>>> Ken Brown wrote:
>>>>>>>>>> I've rerun your test with the latest version, and the test results are
>>>>>>>>>> similar.
>>>>>>>>>>       I've also run a suite of fifo tests that I've accumulated, and they
>>>>>>>>>> all pass
>>>>>>>>>> also, so I pushed your patch.
>>>>>>>>>>
>>>>>>>>>> I think we're in pretty good shape now.  The only detail remaining,
>>>>>>>>>> AFAIK, is
>>>>>>>>>> how to best avoid a deadlock if the pipe has been created by a non-Cygwin
>>>>>>>>>> process.  I've proposed a timeout, but maybe there's a better idea.
>>>>>>>>>
>>>>>>>>> I am not pretty sure what is the problem, but is not the following
>>>>>>>>> patch enough?
>>>>>>>>>
>>>>>>>>> diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
>>>>>>>>> index d309be2f7..13fba9a14 100644
>>>>>>>>> --- a/winsup/cygwin/fhandler.h
>>>>>>>>> +++ b/winsup/cygwin/fhandler.h
>>>>>>>>> @@ -1205,6 +1205,7 @@ public:
>>>>>>>>>        select_record *select_except (select_stuff *);
>>>>>>>>>        char *get_proc_fd_name (char *buf);
>>>>>>>>>        int open (int flags, mode_t mode = 0);
>>>>>>>>> +  void open_setup (int flags);
>>>>>>>>>        void fixup_after_fork (HANDLE);
>>>>>>>>>        int dup (fhandler_base *child, int);
>>>>>>>>>        int close ();
>>>>>>>>> diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
>>>>>>>>> index 6994a5dce..d84e6ad84 100644
>>>>>>>>> --- a/winsup/cygwin/fhandler_pipe.cc
>>>>>>>>> +++ b/winsup/cygwin/fhandler_pipe.cc
>>>>>>>>> @@ -191,6 +191,17 @@ out:
>>>>>>>>>        return 0;
>>>>>>>>>      }
>>>>>>>>>
>>>>>>>>> +void
>>>>>>>>> +fhandler_pipe::open_setup (int flags)
>>>>>>>>> +{
>>>>>>>>> +  fhandler_base::open_setup (flags);
>>>>>>>>> +  if (get_dev () == FH_PIPER && !read_mtx)
>>>>>>>>> +    {
>>>>>>>>> +      SECURITY_ATTRIBUTES *sa = sec_none_cloexec (flags);
>>>>>>>>> +      read_mtx = CreateMutexW (sa, FALSE, NULL);
>>>>>>>>> +    }
>>>>>>>>> +}
>>>>>>>>> +
>>>>>>>>>      off_t
>>>>>>>>>      fhandler_pipe::lseek (off_t offset, int whence)
>>>>>>>>>      {
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> AFAIK, another problem remaining is:
>>>>>>>>>
>>>>>>>>> On Mon, 6 Sep 2021 14:49:55 +0200
>>>>>>>>> Corinna Vinschen wrote:
>>>>>>>>>> - What about calling select for writing on pipes read by non-Cygwin
>>>>>>>>>>       processes?  In that case, we still can't rely on WriteQuotaAvailable,
>>>>>>>>>>       just as before.
>>>>>>>>
>>>>>>>> This is the problem I was talking about.  In this case the non-Cygwin process
>>>>>>>> might have a large pending read, so that the Cygwin process calling select on
>>>>>>>> the write side will see WriteQuotaAvailable == 0.  This could lead to a
>>>>>>>> deadlock
>>>>>>>> with the Cygwin process waiting for write ready while the non-Cygwin
>>>>>>>> process is
>>>>>>>> blocked trying to read.
>>>>>>>
>>>>>>> Then, the above patch is for another issue.
>>>>>>> The problem happes when:
>>>>>>> 1) Start command prompt.
>>>>>>> 2) Run 'echo AAAAAAAAAAAA | \cygwin64\bin\cat
>>>>>>> This causes hang up in cat. In this case, pipe is created by cmd.exe.
>>>>>>> Therefore, read_mtx is not created.
>>>>>>
>>>>>> Confirmed, and your patch fixes it.  Maybe you should check for error in the
>>>>>> call to CreateMutexW and print a debug message in that case.
>>>>>>
>>>>>>>> My suggestion is that we impose a timeout in this situation, after which
>>>>>>>> select
>>>>>>>> reports write ready.
>>>>>>>
>>>>>>> Keeping read handle in write pipe (Corinna's query_hdl) causes problem
>>>>>>> that write side cannot detect close on read side.
>>>>>>> Is it possible to open read handle temporally when pipe_data_available()
>>>>>>> is called?
>>>>>>
>>>>>> That would be nice, but I have no idea how you could do that.
>>>>>
>>>>> Hmm. Then, what about PoC code attached? This returns to Corinna's
>>>>> query_hdl, and counts read/write handles to detect closing reader side.
>>>>>
>>>>> If the number of read handles is equal to number of write handles,
>>>>> only the pairs of write handle and query_hdl are alive. So, read pipe
>>>>> supposed to be closed.
>>>>>
>>>>> This patch depends another patch I posted a few hours ago.
>>>>
>>>> Revised a bit.
>>>
>>> I don't see how this solves the problem.  In the case we were worried about
>>> where we have a non-Cygwin reader, the writer has no query_hdl, and you're just
>>> always reporting write ready, aren't you?  Or am I missing something?
>>
>> BTW, we could just decide that always reporting write ready in this corner case
>> is acceptable.  But then we could just do that without going back to query_hdl.
> 
> The various combination of cygwin and non-cygwin cases resuts in:
> 
> W P R: current, query_hdl
> c c c: OK     , OK
> c c n: NG     , OK
> c n c: OK     , OK
> c n n: NG     , select() always report write ready

This is the case I've been talking about.  I'm sorry for not being more clear.

> where
> 
> W: Writer
> P: Pipe
> R: Reder
> c: cygwin
> n: non-cygwin
> 
> * Reder requests larger block than pipe size.
> * Writer cannot be non-cygwin because we assume the case
>    that writer uses select().

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-12 23:54                                           ` Takashi Yano
  2021-09-13  2:19                                             ` Ken Brown
@ 2021-09-13  8:40                                             ` Takashi Yano
  2021-09-13 12:51                                               ` Ken Brown
  1 sibling, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-13  8:40 UTC (permalink / raw)
  To: cygwin-developers

On Mon, 13 Sep 2021 08:54:31 +0900
Takashi Yano wrote:
> On Sun, 12 Sep 2021 17:46:47 -0400
> Ken Brown wrote:
> > On 9/12/2021 11:10 AM, Ken Brown wrote:
> > > On 9/12/2021 7:04 AM, Takashi Yano wrote:
> > >> On Sun, 12 Sep 2021 17:48:49 +0900
> > >> Takashi Yano wrote:
> > >>> On Sat, 11 Sep 2021 09:12:02 -0400
> > >>> Ken Brown wrote:
> > >>>> On 9/10/2021 10:35 PM, Takashi Yano wrote:
> > >>>>> On Fri, 10 Sep 2021 22:17:21 -0400
> > >>>>> Ken Brown wrote:
> > >>>>>> On 9/10/2021 6:57 PM, Takashi Yano wrote:
> > >>>>>>> On Fri, 10 Sep 2021 11:17:58 -0400
> > >>>>>>> Ken Brown wrote:
> > >>>>>>>> I've rerun your test with the latest version, and the test results are 
> > >>>>>>>> similar.
> > >>>>>>>>      I've also run a suite of fifo tests that I've accumulated, and they 
> > >>>>>>>> all pass
> > >>>>>>>> also, so I pushed your patch.
> > >>>>>>>>
> > >>>>>>>> I think we're in pretty good shape now.  The only detail remaining, 
> > >>>>>>>> AFAIK, is
> > >>>>>>>> how to best avoid a deadlock if the pipe has been created by a non-Cygwin
> > >>>>>>>> process.  I've proposed a timeout, but maybe there's a better idea.
> > >>>>>>>
> > >>>>>>> I am not pretty sure what is the problem, but is not the following
> > >>>>>>> patch enough?
> > >>>>>>>
> > >>>>>>> diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
> > >>>>>>> index d309be2f7..13fba9a14 100644
> > >>>>>>> --- a/winsup/cygwin/fhandler.h
> > >>>>>>> +++ b/winsup/cygwin/fhandler.h
> > >>>>>>> @@ -1205,6 +1205,7 @@ public:
> > >>>>>>>       select_record *select_except (select_stuff *);
> > >>>>>>>       char *get_proc_fd_name (char *buf);
> > >>>>>>>       int open (int flags, mode_t mode = 0);
> > >>>>>>> +  void open_setup (int flags);
> > >>>>>>>       void fixup_after_fork (HANDLE);
> > >>>>>>>       int dup (fhandler_base *child, int);
> > >>>>>>>       int close ();
> > >>>>>>> diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
> > >>>>>>> index 6994a5dce..d84e6ad84 100644
> > >>>>>>> --- a/winsup/cygwin/fhandler_pipe.cc
> > >>>>>>> +++ b/winsup/cygwin/fhandler_pipe.cc
> > >>>>>>> @@ -191,6 +191,17 @@ out:
> > >>>>>>>       return 0;
> > >>>>>>>     }
> > >>>>>>>
> > >>>>>>> +void
> > >>>>>>> +fhandler_pipe::open_setup (int flags)
> > >>>>>>> +{
> > >>>>>>> +  fhandler_base::open_setup (flags);
> > >>>>>>> +  if (get_dev () == FH_PIPER && !read_mtx)
> > >>>>>>> +    {
> > >>>>>>> +      SECURITY_ATTRIBUTES *sa = sec_none_cloexec (flags);
> > >>>>>>> +      read_mtx = CreateMutexW (sa, FALSE, NULL);
> > >>>>>>> +    }
> > >>>>>>> +}
> > >>>>>>> +
> > >>>>>>>     off_t
> > >>>>>>>     fhandler_pipe::lseek (off_t offset, int whence)
> > >>>>>>>     {
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> AFAIK, another problem remaining is:
> > >>>>>>>
> > >>>>>>> On Mon, 6 Sep 2021 14:49:55 +0200
> > >>>>>>> Corinna Vinschen wrote:
> > >>>>>>>> - What about calling select for writing on pipes read by non-Cygwin
> > >>>>>>>>      processes?  In that case, we still can't rely on WriteQuotaAvailable,
> > >>>>>>>>      just as before.
> > >>>>>>
> > >>>>>> This is the problem I was talking about.  In this case the non-Cygwin process
> > >>>>>> might have a large pending read, so that the Cygwin process calling select on
> > >>>>>> the write side will see WriteQuotaAvailable == 0.  This could lead to a 
> > >>>>>> deadlock
> > >>>>>> with the Cygwin process waiting for write ready while the non-Cygwin 
> > >>>>>> process is
> > >>>>>> blocked trying to read.
> > >>>>>
> > >>>>> Then, the above patch is for another issue.
> > >>>>> The problem happes when:
> > >>>>> 1) Start command prompt.
> > >>>>> 2) Run 'echo AAAAAAAAAAAA | \cygwin64\bin\cat
> > >>>>> This causes hang up in cat. In this case, pipe is created by cmd.exe.
> > >>>>> Therefore, read_mtx is not created.
> > >>>>
> > >>>> Confirmed, and your patch fixes it.  Maybe you should check for error in the
> > >>>> call to CreateMutexW and print a debug message in that case.
> > >>>>
> > >>>>>> My suggestion is that we impose a timeout in this situation, after which 
> > >>>>>> select
> > >>>>>> reports write ready.
> > >>>>>
> > >>>>> Keeping read handle in write pipe (Corinna's query_hdl) causes problem
> > >>>>> that write side cannot detect close on read side.
> > >>>>> Is it possible to open read handle temporally when pipe_data_available()
> > >>>>> is called?
> > >>>>
> > >>>> That would be nice, but I have no idea how you could do that.
> > >>>
> > >>> Hmm. Then, what about PoC code attached? This returns to Corinna's
> > >>> query_hdl, and counts read/write handles to detect closing reader side.
> > >>>
> > >>> If the number of read handles is equal to number of write handles,
> > >>> only the pairs of write handle and query_hdl are alive. So, read pipe
> > >>> supposed to be closed.
> > >>>
> > >>> This patch depends another patch I posted a few hours ago.
> > >>
> > >> Revised a bit.
> > > 
> > > I don't see how this solves the problem.  In the case we were worried about 
> > > where we have a non-Cygwin reader, the writer has no query_hdl, and you're just 
> > > always reporting write ready, aren't you?  Or am I missing something?
> > 
> > BTW, we could just decide that always reporting write ready in this corner case 
> > is acceptable.  But then we could just do that without going back to query_hdl.
> 
> The various combination of cygwin and non-cygwin cases resuts in:
> 
> W P R: current, query_hdl
> c c c: OK     , OK
> c c n: NG     , OK
> c n c: OK     , OK
> c n n: NG     , select() always report write ready

Sorry, this was not correct. In fact,
W P R: current, query_hdl
c c c: OK     , OK
c c n: NG     , OK
c n c: OK     , select() always report write ready
c n n: NG     , select() always report write ready

:-(

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-11  2:35                               ` Takashi Yano
  2021-09-11 13:12                                 ` Ken Brown
@ 2021-09-13  9:07                                 ` Corinna Vinschen
  2021-09-20 12:52                                   ` Takashi Yano
  1 sibling, 1 reply; 250+ messages in thread
From: Corinna Vinschen @ 2021-09-13  9:07 UTC (permalink / raw)
  To: cygwin-developers

On Sep 11 11:35, Takashi Yano wrote:
> On Fri, 10 Sep 2021 22:17:21 -0400
> Ken Brown wrote:
> > My suggestion is that we impose a timeout in this situation, after which select 
> > reports write ready.
> 
> Keeping read handle in write pipe (Corinna's query_hdl) causes problem
> that write side cannot detect close on read side.
> Is it possible to open read handle temporally when pipe_data_available()
> is called?

1. You would have to know which process keeps the other side of the pipe.
2. You would have to have the permission to open the other process to
   duplicate the pipe into your own process
3. You would have to know the HANDLE value of the read side of your pipe
   in that other process.

Point 1 is (kind of) doable using GetNamedPipeClientProcessId or
GetNamedPipeServerProcessId.  ZIt's not clear how reliable these
functions are, given that both pipe sides are created by the same
process and then usually inherited by two child processes communicating
over that pipe.

Point 2 is most of the time the case, especially when talking with
native processes.

Point 3 requires some sort of IPC.

Having said that, I think this is too complicated.


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-12 14:39                                     ` Ken Brown
@ 2021-09-13  9:11                                       ` Corinna Vinschen
  2021-09-13 12:30                                         ` Ken Brown
  0 siblings, 1 reply; 250+ messages in thread
From: Corinna Vinschen @ 2021-09-13  9:11 UTC (permalink / raw)
  To: cygwin-developers

On Sep 12 10:39, Ken Brown wrote:
> On 9/12/2021 2:23 AM, Takashi Yano wrote:
> > > Confirmed, and your patch fixes it.  Maybe you should check for error in the
> > > call to CreateMutexW and print a debug message in that case.
> > 
> > I added the debug message.
> 
> LGTM.

Are you going to push this, Ken?


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-12 21:46                                         ` Ken Brown
  2021-09-12 23:54                                           ` Takashi Yano
@ 2021-09-13  9:42                                           ` Corinna Vinschen
  2021-09-13 13:03                                             ` Ken Brown
  1 sibling, 1 reply; 250+ messages in thread
From: Corinna Vinschen @ 2021-09-13  9:42 UTC (permalink / raw)
  To: cygwin-developers

[Guys, is it asked too much to trim your mails to the required context,
 rather than always performing a Full Quote?  Pretty please...]

On Sep 12 17:46, Ken Brown wrote:
> On 9/12/2021 11:10 AM, Ken Brown wrote:
> > On 9/12/2021 7:04 AM, Takashi Yano wrote:
> > > On Sun, 12 Sep 2021 17:48:49 +0900
> > > Takashi Yano wrote:
> > > > Hmm. Then, what about PoC code attached? This returns to Corinna's
> > > > query_hdl, and counts read/write handles to detect closing reader side.
> > > > 
> > > > If the number of read handles is equal to number of write handles,
> > > > only the pairs of write handle and query_hdl are alive. So, read pipe
> > > > supposed to be closed.
> > > > 
> > > > This patch depends another patch I posted a few hours ago.
> > > [...]
> > I don't see how this solves the problem.  In the case we were worried
> > about where we have a non-Cygwin reader, the writer has no query_hdl,
> > and you're just always reporting write ready, aren't you?  Or am I
> > missing something?
> 
> BTW, we could just decide that always reporting write ready in this corner
> case is acceptable.  But then we could just do that without going back to
> query_hdl.

The problem with the corner case is, how to find out?  You could have
arbitrarily complex process trees with the pipe inherited by grand
children, one of which is a non-Cygwin process.  How does a Cygwin
process tree member learn about that fact, if it didn't start the
non-Cygwin process by itself?

Looks like we have three choices:

- Reintroducing query_hdl as in Takashi's patch.

- select timeouts

- always return "pipe writable"

I think we might try Takashi's idea for as start, no?

Didn't we also have a problem with C# in terms of non-blocking pipes?
I wonder if we could just do the following: As soon as we spawn a
non-Cygwin process, just call set_pipe_non_blocking(false) for all
pipes.  Blocking vs. nonblocking mode is a per-handle thingy anyway.

What do you think?


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-13  9:11                                       ` Corinna Vinschen
@ 2021-09-13 12:30                                         ` Ken Brown
  0 siblings, 0 replies; 250+ messages in thread
From: Ken Brown @ 2021-09-13 12:30 UTC (permalink / raw)
  To: cygwin-developers

On 9/13/2021 5:11 AM, Corinna Vinschen wrote:
> On Sep 12 10:39, Ken Brown wrote:
>> On 9/12/2021 2:23 AM, Takashi Yano wrote:
>>>> Confirmed, and your patch fixes it.  Maybe you should check for error in the
>>>> call to CreateMutexW and print a debug message in that case.
>>>
>>> I added the debug message.
>>
>> LGTM.
> 
> Are you going to push this, Ken?

Done.

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-13  8:40                                             ` Takashi Yano
@ 2021-09-13 12:51                                               ` Ken Brown
  2021-09-13 17:05                                                 ` Ken Brown
  0 siblings, 1 reply; 250+ messages in thread
From: Ken Brown @ 2021-09-13 12:51 UTC (permalink / raw)
  To: cygwin-developers

On 9/13/2021 4:40 AM, Takashi Yano wrote:
> On Mon, 13 Sep 2021 08:54:31 +0900
> Takashi Yano wrote:
>> On Sun, 12 Sep 2021 17:46:47 -0400
>> Ken Brown wrote:
>>> On 9/12/2021 11:10 AM, Ken Brown wrote:
>>>> I don't see how this solves the problem.  In the case we were worried about
>>>> where we have a non-Cygwin reader, the writer has no query_hdl, and you're just
>>>> always reporting write ready, aren't you?  Or am I missing something?
>>>
>>> BTW, we could just decide that always reporting write ready in this corner case
>>> is acceptable.  But then we could just do that without going back to query_hdl.
>>
>> The various combination of cygwin and non-cygwin cases resuts in:
>>
>> W P R: current, query_hdl
>> c c c: OK     , OK
>> c c n: NG     , OK
>> c n c: OK     , OK
>> c n n: NG     , select() always report write ready
> 
> Sorry, this was not correct. In fact,
> W P R: current, query_hdl
> c c c: OK     , OK
> c c n: NG     , OK
> c n c: OK     , select() always report write ready
> c n n: NG     , select() always report write ready

What if you use query_hdl to fix the ccn case, but also keep the current code in 
raw_read that serializes the reads and avoids a large blocking read?  Would that 
avoid breaking the cnc case?

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-13  9:42                                           ` Corinna Vinschen
@ 2021-09-13 13:03                                             ` Ken Brown
  2021-09-13 18:39                                               ` Takashi Yano
  0 siblings, 1 reply; 250+ messages in thread
From: Ken Brown @ 2021-09-13 13:03 UTC (permalink / raw)
  To: cygwin-developers

On 9/13/2021 5:42 AM, Corinna Vinschen wrote:
> On Sep 12 17:46, Ken Brown wrote:
>> BTW, we could just decide that always reporting write ready in this corner
>> case is acceptable.  But then we could just do that without going back to
>> query_hdl.
> 
> The problem with the corner case is, how to find out?  You could have
> arbitrarily complex process trees with the pipe inherited by grand
> children, one of which is a non-Cygwin process.  How does a Cygwin
> process tree member learn about that fact, if it didn't start the
> non-Cygwin process by itself?
> 
> Looks like we have three choices:
> 
> - Reintroducing query_hdl as in Takashi's patch.
> 
> - select timeouts
> 
> - always return "pipe writable"
> 
> I think we might try Takashi's idea for as start, no?

That sounds good to me, provided he can fix the "cnc" case I just asked him 
about -- Cygwin reader and writer on a pipe created by a non-Cygwin process.

> Didn't we also have a problem with C# in terms of non-blocking pipes?
> I wonder if we could just do the following: As soon as we spawn a
> non-Cygwin process, just call set_pipe_non_blocking(false) for all
> pipes.  Blocking vs. nonblocking mode is a per-handle thingy anyway.
> 
> What do you think?

I don't remember exactly what the issue was with C# programs, so I'll defer to 
Takashi on this.

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-13 12:51                                               ` Ken Brown
@ 2021-09-13 17:05                                                 ` Ken Brown
  0 siblings, 0 replies; 250+ messages in thread
From: Ken Brown @ 2021-09-13 17:05 UTC (permalink / raw)
  To: cygwin-developers

On 9/13/2021 8:51 AM, Ken Brown wrote:
> On 9/13/2021 4:40 AM, Takashi Yano wrote:
>> On Mon, 13 Sep 2021 08:54:31 +0900
>> Takashi Yano wrote:
>>> On Sun, 12 Sep 2021 17:46:47 -0400
>>> Ken Brown wrote:
>>>> On 9/12/2021 11:10 AM, Ken Brown wrote:
>>>>> I don't see how this solves the problem.  In the case we were worried about
>>>>> where we have a non-Cygwin reader, the writer has no query_hdl, and you're 
>>>>> just
>>>>> always reporting write ready, aren't you?  Or am I missing something?
>>>>
>>>> BTW, we could just decide that always reporting write ready in this corner case
>>>> is acceptable.  But then we could just do that without going back to query_hdl.
>>>
>>> The various combination of cygwin and non-cygwin cases resuts in:
>>>
>>> W P R: current, query_hdl
>>> c c c: OK     , OK
>>> c c n: NG     , OK
>>> c n c: OK     , OK
>>> c n n: NG     , select() always report write ready
>>
>> Sorry, this was not correct. In fact,
>> W P R: current, query_hdl
>> c c c: OK     , OK
>> c c n: NG     , OK
>> c n c: OK     , select() always report write ready
>> c n n: NG     , select() always report write ready
> 
> What if you use query_hdl to fix the ccn case, but also keep the current code in 
> raw_read that serializes the reads and avoids a large blocking read?  Would that 
> avoid breaking the cnc case?

Never mind.  I don't think that makes sense.  The way I see it now, if we have a 
query_hdl, we should use it.  If we don't have a query_hdl, then we know there 
were non-Cygwin processes in the process tree, and then if WriteQuotaAvailable 
== 0, the pipe buffer might actually be empty.  So I don't see any option other 
than reporting write ready, as in your patch.

I do have a couple of small comments about your patch, which I'll send later.

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-12 11:04                                     ` Takashi Yano
  2021-09-12 15:10                                       ` Ken Brown
@ 2021-09-13 17:42                                       ` Ken Brown
  2021-09-13 18:54                                         ` Takashi Yano
  2021-09-13 18:32                                       ` Corinna Vinschen
  2 siblings, 1 reply; 250+ messages in thread
From: Ken Brown @ 2021-09-13 17:42 UTC (permalink / raw)
  To: cygwin-developers

On 9/12/2021 7:04 AM, Takashi Yano wrote:
> On Sun, 12 Sep 2021 17:48:49 +0900
> Takashi Yano wrote:
>> Hmm. Then, what about PoC code attached? This returns to Corinna's
>> query_hdl, and counts read/write handles to detect closing reader side.
>>
>> If the number of read handles is equal to number of write handles,
>> only the pairs of write handle and query_hdl are alive. So, read pipe
>> supposed to be closed.
>>
>> This patch depends another patch I posted a few hours ago.
> 
> Revised a bit.

A few small comments/questions:

> diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
> index 13fba9a14..f09af2c37 100644
> --- a/winsup/cygwin/fhandler.h
> +++ b/winsup/cygwin/fhandler.h
> @@ -1176,10 +1176,15 @@ class fhandler_pipe_fifo: public fhandler_base
>  {
>   protected:
>    size_t pipe_buf_size;
> +  HANDLE query_hdl;
>  
>   public:
>    fhandler_pipe_fifo ();
>  
> +  HANDLE get_query_handle () const { return query_hdl; }
> +  void close_query_handle () { CloseHandle (query_hdl); query_hdl = NULL; }

Should you use if(query_hdl) here?  Or is it up to the caller to check that?

> @@ -522,6 +532,10 @@ fhandler_pipe::fixup_after_fork (HANDLE parent)
>      fork_fixup (parent, read_mtx, "read_mtx");
>    if (select_sem)
>      fork_fixup (parent, select_sem, "select_sem");
> +  /* Do not duplicate query_hdl if it has been already inherited. */
> +  if (query_hdl && !get_obj_handle_count (query_hdl))
> +    fork_fixup (parent, query_hdl, "query_hdl");


Why do you need to call get_obj_handle_count here?  Shouldn't fork_fixup take 
care of the case where the handle has been inherited?

> @@ -608,12 +608,43 @@ pipe_data_available (int fd, fhandler_base *fh, HANDLE h, bool writing)
>      }
>    if (writing)
>      {
> -      /* WriteQuotaAvailable is decremented by the number of bytes requested
> -	 by a blocking reader on the other side of the pipe.  Cygwin readers
> -	 are serialized and never request a number of bytes equivalent to the
> -	 full buffer size.  So WriteQuotaAvailable is 0 only if either the
> -	 read buffer on the other side is really full, or if we have non-Cygwin
> -	 readers. */
> +      /* If there is anything available in the pipe buffer then signal
> +        that.  This means that a pipe could still block since you could
> +        be trying to write more to the pipe than is available in the
> +        buffer but that is the hazard of select().
> +
> +        Note that WriteQuotaAvailable is unreliable.
> +
> +        Usually WriteQuotaAvailable on the write side reflects the space
> +        available in the inbound buffer on the read side.  However, if a
> +        pipe read is currently pending, WriteQuotaAvailable on the write side
> +        is decremented by the number of bytes the read side is requesting.
> +        So it's possible (even likely) that WriteQuotaAvailable is 0, even
> +        if the inbound buffer on the read side is not full.  This can lead to
> +        a deadlock situation: The reader is waiting for data, but select
> +        on the writer side assumes that no space is available in the read
> +        side inbound buffer.
> +
> +        Consequentially, the only reliable information is available on the
> +        read side, so fetch info from the read side via the pipe-specific
> +        query handle.  Use fpli.WriteQuotaAvailable as storage for the actual
> +        interesting value, which is the OutboundQuote on the write side,

I thought Corinna's experiments showed that InboundQuota and OutboundQuota are 
the same on the read and write sides, and that InboundQuota is the one we should 
be using.

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-12 11:04                                     ` Takashi Yano
  2021-09-12 15:10                                       ` Ken Brown
  2021-09-13 17:42                                       ` Ken Brown
@ 2021-09-13 18:32                                       ` Corinna Vinschen
  2021-09-13 19:37                                         ` Takashi Yano
  2 siblings, 1 reply; 250+ messages in thread
From: Corinna Vinschen @ 2021-09-13 18:32 UTC (permalink / raw)
  To: cygwin-developers

On Sep 12 20:04, Takashi Yano wrote:
> On Sun, 12 Sep 2021 17:48:49 +0900
> Takashi Yano wrote:
> > Hmm. Then, what about PoC code attached? This returns to Corinna's
> > query_hdl, and counts read/write handles to detect closing reader side.
> > 
> > If the number of read handles is equal to number of write handles,
> > only the pairs of write handle and query_hdl are alive. So, read pipe
> > supposed to be closed.
> > 
> > This patch depends another patch I posted a few hours ago.
> 
> Revised a bit.
> [...]

What I miss is a bit more detailed commit message...

> diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
> index 13fba9a14..f09af2c37 100644
> --- a/winsup/cygwin/fhandler.h
> +++ b/winsup/cygwin/fhandler.h
> @@ -1176,10 +1176,15 @@ class fhandler_pipe_fifo: public fhandler_base
>  {
>   protected:
>    size_t pipe_buf_size;
> +  HANDLE query_hdl;
>  
>   public:
>    fhandler_pipe_fifo ();
>  
> +  HANDLE get_query_handle () const { return query_hdl; }
> +  void close_query_handle () { CloseHandle (query_hdl); query_hdl = NULL; }
> +  bool reader_closed ();
> +
>    ssize_t __reg3 raw_write (const void *ptr, size_t len);
>  
>  };
> diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
> index 9b4255cfd..b051f5c03 100644
> --- a/winsup/cygwin/fhandler_pipe.cc
> +++ b/winsup/cygwin/fhandler_pipe.cc
> @@ -56,6 +56,8 @@ fhandler_pipe::set_pipe_non_blocking (bool nonblocking)
>    fpi.ReadMode = FILE_PIPE_BYTE_STREAM_MODE;
>    fpi.CompletionMode = nonblocking ? FILE_PIPE_COMPLETE_OPERATION
>      : FILE_PIPE_QUEUE_OPERATION;
> +  if (query_hdl)
> +    fpi.CompletionMode = FILE_PIPE_COMPLETE_OPERATION;

This should be a single expression, i.e.

   fpi.CompletionMode = nonblocking || query_hdl
                        ? FILE_PIPE_COMPLETE_OPERATION
                        : FILE_PIPE_QUEUE_OPERATION;

ideally combined with a comment.

But then again... you're basically switching the write side of
a pipe to nonblocking mode unconditionally.  The downside is a
busy wait:

>  fhandler_pipe_fifo::raw_write (const void *ptr, size_t len)
>  {
> @@ -493,7 +490,20 @@ fhandler_pipe_fifo::raw_write (const void *ptr, size_t len)
>  			      get_obj_handle_count (select_sem), NULL);
>  	  /* 0 bytes returned?  EAGAIN.  See above. */
>  	  if (NT_SUCCESS (status) && nbytes == 0)
> -	    set_errno (EAGAIN);
> +	    {
> +	      if (reader_closed ())
> +		{
> +		  set_errno (EPIPE);
> +		  raise (SIGPIPE);
> +		}
> +	      else if (is_nonblocking ())
> +		set_errno (EAGAIN);
> +	      else
> +		{
> +		  cygwait (select_sem, 10);
> +		  continue;

I'm a bit puzzled.  The cygwait branch neglects to check if select_sem
is NULL (the preceeding ReleaseSemaphore expression does!)
And then it doesn't matter if the caller got blocked or not, it will
always perform a continue.  So why do it at all?  Worse, if this
expression loops, it will eat up the semaphore, because each call will
decrement the semaphore count until it blocks.  That sounds wrong to me.

Btw., while looking into the current pipe code, I wonder what select_sem
is doing in the pipe code at all so far.  It gets released, but it never
gets waited on?!?  Am I missing something?

>  	{
> @@ -522,6 +532,10 @@ fhandler_pipe::fixup_after_fork (HANDLE parent)
>      fork_fixup (parent, read_mtx, "read_mtx");
>    if (select_sem)
>      fork_fixup (parent, select_sem, "select_sem");
> +  /* Do not duplicate query_hdl if it has been already inherited. */
> +  if (query_hdl && !get_obj_handle_count (query_hdl))
> +    fork_fixup (parent, query_hdl, "query_hdl");

I don't understand why calling fork_fixup on query_hdl should depend
on the handle count.  If you duplicate a writer, you always have to
duplicate query_hdl as well to keep the count, no?  Inheritence is
handled by the O_CLOEXEC flag and fork_fixup will do the right thing.

> +      if (!DuplicateHandle (GetCurrentProcess (), r,
> +			    GetCurrentProcess (), &fhs[1]->query_hdl,
> +			    GENERIC_READ, !(mode & O_CLOEXEC), 0))

This is a bug I introduced accidentally during testing.  This
GENERIC_READ is actually supposed to be a FILE_READ_ATTRIBUTES.
Sorry about that.


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-13 13:03                                             ` Ken Brown
@ 2021-09-13 18:39                                               ` Takashi Yano
  0 siblings, 0 replies; 250+ messages in thread
From: Takashi Yano @ 2021-09-13 18:39 UTC (permalink / raw)
  To: cygwin-developers

On Mon, 13 Sep 2021 09:03:04 -0400
Ken Brown wrote:
> On 9/13/2021 5:42 AM, Corinna Vinschen wrote:
> > Didn't we also have a problem with C# in terms of non-blocking pipes?
> > I wonder if we could just do the following: As soon as we spawn a
> > non-Cygwin process, just call set_pipe_non_blocking(false) for all
> > pipes.  Blocking vs. nonblocking mode is a per-handle thingy anyway.
> > 
> > What do you think?
> 
> I don't remember exactly what the issue was with C# programs, so I'll defer to 
> Takashi on this.

Actually, piping C# program has a problem even if setting the pipe
blocking. It needs also pipe_byte and FILE_SYNCHRONOUS_IO_NONALERT.
However, I agree it is safer to set the pipe blocking for non-cygwin
apps.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-13 17:42                                       ` Ken Brown
@ 2021-09-13 18:54                                         ` Takashi Yano
  0 siblings, 0 replies; 250+ messages in thread
From: Takashi Yano @ 2021-09-13 18:54 UTC (permalink / raw)
  To: cygwin-developers

On Mon, 13 Sep 2021 13:42:46 -0400
Ken Brown wrote:
> > diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
> > index 13fba9a14..f09af2c37 100644
> > --- a/winsup/cygwin/fhandler.h
> > +++ b/winsup/cygwin/fhandler.h
> > @@ -1176,10 +1176,15 @@ class fhandler_pipe_fifo: public fhandler_base
> >  {
> >   protected:
> >    size_t pipe_buf_size;
> > +  HANDLE query_hdl;
> >  
> >   public:
> >    fhandler_pipe_fifo ();
> >  
> > +  HANDLE get_query_handle () const { return query_hdl; }
> > +  void close_query_handle () { CloseHandle (query_hdl); query_hdl = NULL; }
> 
> Should you use if(query_hdl) here?  Or is it up to the caller to check that?

Right. I will fix it.

> > @@ -522,6 +532,10 @@ fhandler_pipe::fixup_after_fork (HANDLE parent)
> >      fork_fixup (parent, read_mtx, "read_mtx");
> >    if (select_sem)
> >      fork_fixup (parent, select_sem, "select_sem");
> > +  /* Do not duplicate query_hdl if it has been already inherited. */
> > +  if (query_hdl && !get_obj_handle_count (query_hdl))
> > +    fork_fixup (parent, query_hdl, "query_hdl");
> 
> 
> Why do you need to call get_obj_handle_count here?  Shouldn't fork_fixup take 
> care of the case where the handle has been inherited?

I also thought so, however, counting query_hdl did not work
as expected without this check. I am not sure why...

There seems to be the case that handle is already inherited
here without fork_fixup.

> > @@ -608,12 +608,43 @@ pipe_data_available (int fd, fhandler_base *fh, HANDLE h, bool writing)
> >      }
> >    if (writing)
> >      {
> > -      /* WriteQuotaAvailable is decremented by the number of bytes requested
> > -	 by a blocking reader on the other side of the pipe.  Cygwin readers
> > -	 are serialized and never request a number of bytes equivalent to the
> > -	 full buffer size.  So WriteQuotaAvailable is 0 only if either the
> > -	 read buffer on the other side is really full, or if we have non-Cygwin
> > -	 readers. */
> > +      /* If there is anything available in the pipe buffer then signal
> > +        that.  This means that a pipe could still block since you could
> > +        be trying to write more to the pipe than is available in the
> > +        buffer but that is the hazard of select().
> > +
> > +        Note that WriteQuotaAvailable is unreliable.
> > +
> > +        Usually WriteQuotaAvailable on the write side reflects the space
> > +        available in the inbound buffer on the read side.  However, if a
> > +        pipe read is currently pending, WriteQuotaAvailable on the write side
> > +        is decremented by the number of bytes the read side is requesting.
> > +        So it's possible (even likely) that WriteQuotaAvailable is 0, even
> > +        if the inbound buffer on the read side is not full.  This can lead to
> > +        a deadlock situation: The reader is waiting for data, but select
> > +        on the writer side assumes that no space is available in the read
> > +        side inbound buffer.
> > +
> > +        Consequentially, the only reliable information is available on the
> > +        read side, so fetch info from the read side via the pipe-specific
> > +        query handle.  Use fpli.WriteQuotaAvailable as storage for the actual
> > +        interesting value, which is the OutboundQuote on the write side,
> 
> I thought Corinna's experiments showed that InboundQuota and OutboundQuota are 
> the same on the read and write sides, and that InboundQuota is the one we should 
> be using.

I have confirmed that behaviour. You are right. Using InboundQuota
is right thing. I'll fix it. Thanks.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-13 18:32                                       ` Corinna Vinschen
@ 2021-09-13 19:37                                         ` Takashi Yano
  2021-09-13 20:15                                           ` Corinna Vinschen
  2021-09-14  8:08                                           ` Takashi Yano
  0 siblings, 2 replies; 250+ messages in thread
From: Takashi Yano @ 2021-09-13 19:37 UTC (permalink / raw)
  To: cygwin-developers

On Mon, 13 Sep 2021 20:32:33 +0200
Corinna Vinschen wrote:

> On Sep 12 20:04, Takashi Yano wrote:
> > On Sun, 12 Sep 2021 17:48:49 +0900
> > Takashi Yano wrote:
> > > Hmm. Then, what about PoC code attached? This returns to Corinna's
> > > query_hdl, and counts read/write handles to detect closing reader side.
> > > 
> > > If the number of read handles is equal to number of write handles,
> > > only the pairs of write handle and query_hdl are alive. So, read pipe
> > > supposed to be closed.
> > > 
> > > This patch depends another patch I posted a few hours ago.
> > 
> > Revised a bit.
> > [...]
> 
> What I miss is a bit more detailed commit message...

I am sorry, I will add more detail commit message.

> > diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
> > index 9b4255cfd..b051f5c03 100644
> > --- a/winsup/cygwin/fhandler_pipe.cc
> > +++ b/winsup/cygwin/fhandler_pipe.cc
> > @@ -56,6 +56,8 @@ fhandler_pipe::set_pipe_non_blocking (bool nonblocking)
> >    fpi.ReadMode = FILE_PIPE_BYTE_STREAM_MODE;
> >    fpi.CompletionMode = nonblocking ? FILE_PIPE_COMPLETE_OPERATION
> >      : FILE_PIPE_QUEUE_OPERATION;
> > +  if (query_hdl)
> > +    fpi.CompletionMode = FILE_PIPE_COMPLETE_OPERATION;
> 
> This should be a single expression, i.e.
> 
>    fpi.CompletionMode = nonblocking || query_hdl
>                         ? FILE_PIPE_COMPLETE_OPERATION
>                         : FILE_PIPE_QUEUE_OPERATION;
> 
> ideally combined with a comment.

Thanks. I'll do that.

> But then again... you're basically switching the write side of
> a pipe to nonblocking mode unconditionally.  The downside is a
> busy wait:
> 
> >  fhandler_pipe_fifo::raw_write (const void *ptr, size_t len)
> >  {
> > @@ -493,7 +490,20 @@ fhandler_pipe_fifo::raw_write (const void *ptr, size_t len)
> >  			      get_obj_handle_count (select_sem), NULL);
> >  	  /* 0 bytes returned?  EAGAIN.  See above. */
> >  	  if (NT_SUCCESS (status) && nbytes == 0)
> > -	    set_errno (EAGAIN);
> > +	    {
> > +	      if (reader_closed ())
> > +		{
> > +		  set_errno (EPIPE);
> > +		  raise (SIGPIPE);
> > +		}
> > +	      else if (is_nonblocking ())
> > +		set_errno (EAGAIN);
> > +	      else
> > +		{
> > +		  cygwait (select_sem, 10);
> > +		  continue;
> 
> I'm a bit puzzled.  The cygwait branch neglects to check if select_sem
> is NULL (the preceeding ReleaseSemaphore expression does!)
> And then it doesn't matter if the caller got blocked or not, it will
> always perform a continue.  So why do it at all?  Worse, if this
> expression loops, it will eat up the semaphore, because each call will
> decrement the semaphore count until it blocks.  That sounds wrong to me.

It is by design. ReleaseSemaphore() releases maximum number of semaphore
which the waiter can exists. If only one writer and one reader exist,
ReleaseSemaphore releases 2 semaphores. Then cygwait here consume semaphore
two times and return to wait state.
This wait state is released by raw_read() or close().
 
> Btw., while looking into the current pipe code, I wonder what select_sem
> is doing in the pipe code at all so far.  It gets released, but it never
> gets waited on?!?  Am I missing something?

The semaphore is waited in select.cc.
But, wait. Wat happens if select() is not called? Released semaphore
can be accumulated up to INT32_MAX!!?

Let me consider.

> >  	{
> > @@ -522,6 +532,10 @@ fhandler_pipe::fixup_after_fork (HANDLE parent)
> >      fork_fixup (parent, read_mtx, "read_mtx");
> >    if (select_sem)
> >      fork_fixup (parent, select_sem, "select_sem");
> > +  /* Do not duplicate query_hdl if it has been already inherited. */
> > +  if (query_hdl && !get_obj_handle_count (query_hdl))
> > +    fork_fixup (parent, query_hdl, "query_hdl");
> 
> I don't understand why calling fork_fixup on query_hdl should depend
> on the handle count.  If you duplicate a writer, you always have to
> duplicate query_hdl as well to keep the count, no?  Inheritence is
> handled by the O_CLOEXEC flag and fork_fixup will do the right thing.

I thought so, however, counting query_hdl cannot work as expected
without this check. The number of query_hdl opend seems to exceed
the number of writer.

There seems to be the case that handle is already inherited here
without fork_fixup. Any idea?

> 
> > +      if (!DuplicateHandle (GetCurrentProcess (), r,
> > +			    GetCurrentProcess (), &fhs[1]->query_hdl,
> > +			    GENERIC_READ, !(mode & O_CLOEXEC), 0))
> 
> This is a bug I introduced accidentally during testing.  This
> GENERIC_READ is actually supposed to be a FILE_READ_ATTRIBUTES.
> Sorry about that.

The PoC code uses PeekNamedPipe for query_hdl, so GENERIC_READ is
necessary I think.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-13 19:37                                         ` Takashi Yano
@ 2021-09-13 20:15                                           ` Corinna Vinschen
  2021-09-14  8:07                                             ` Takashi Yano
  2021-09-14  8:08                                           ` Takashi Yano
  1 sibling, 1 reply; 250+ messages in thread
From: Corinna Vinschen @ 2021-09-13 20:15 UTC (permalink / raw)
  To: cygwin-developers

On Sep 14 04:37, Takashi Yano wrote:
> On Mon, 13 Sep 2021 20:32:33 +0200
> Corinna Vinschen wrote:
> > Btw., while looking into the current pipe code, I wonder what select_sem
> > is doing in the pipe code at all so far.  It gets released, but it never
> > gets waited on?!?  Am I missing something?
> 
> The semaphore is waited in select.cc.

Ouch, I missed the get_select_sem call, sorry.

> > I don't understand why calling fork_fixup on query_hdl should depend
> > on the handle count.  If you duplicate a writer, you always have to
> > duplicate query_hdl as well to keep the count, no?  Inheritence is
> > handled by the O_CLOEXEC flag and fork_fixup will do the right thing.
> 
> I thought so, however, counting query_hdl cannot work as expected
> without this check. The number of query_hdl opend seems to exceed
> the number of writer.

If the write handle as well as the query handle are both opened,
duplicated and closed in the same manner, they should never have a
different count, unless the write side is inherited by a non-Cygwin
client.

> There seems to be the case that handle is already inherited here
> without fork_fixup. Any idea?

That should depend on the O_CLOEXEC setting, but identically for
all handles in the fhandler.

I pushed two more patches to topic/pipe in terms of inheritence,
maybe that gives a clue?

> > 
> > > +      if (!DuplicateHandle (GetCurrentProcess (), r,
> > > +			    GetCurrentProcess (), &fhs[1]->query_hdl,
> > > +			    GENERIC_READ, !(mode & O_CLOEXEC), 0))
> > 
> > This is a bug I introduced accidentally during testing.  This
> > GENERIC_READ is actually supposed to be a FILE_READ_ATTRIBUTES.
> > Sorry about that.
> 
> The PoC code uses PeekNamedPipe for query_hdl, so GENERIC_READ is
> necessary I think.

Oh, right, that's how it's documented.  Funny enough, the descriptions
of FSCTL_PIPE_PEEK does not mention any permissions at all.  I tried the
permissions and it's FILE_READ_DATA which is required.


Thanks,
Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-13 20:15                                           ` Corinna Vinschen
@ 2021-09-14  8:07                                             ` Takashi Yano
  2021-09-14  8:47                                               ` Corinna Vinschen
  0 siblings, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-14  8:07 UTC (permalink / raw)
  To: cygwin-developers

[-- Attachment #1: Type: text/plain, Size: 2099 bytes --]

On Mon, 13 Sep 2021 22:15:25 +0200
Corinna Vinschen wrote:
> On Sep 14 04:37, Takashi Yano wrote:
> > On Mon, 13 Sep 2021 20:32:33 +0200
> > Corinna Vinschen wrote:
> > > I don't understand why calling fork_fixup on query_hdl should depend
> > > on the handle count.  If you duplicate a writer, you always have to
> > > duplicate query_hdl as well to keep the count, no?  Inheritence is
> > > handled by the O_CLOEXEC flag and fork_fixup will do the right thing.
> > 
> > I thought so, however, counting query_hdl cannot work as expected
> > without this check. The number of query_hdl opend seems to exceed
> > the number of writer.
> 
> If the write handle as well as the query handle are both opened,
> duplicated and closed in the same manner, they should never have a
> different count, unless the write side is inherited by a non-Cygwin
> client.
> 
> > There seems to be the case that handle is already inherited here
> > without fork_fixup. Any idea?
> 
> That should depend on the O_CLOEXEC setting, but identically for
> all handles in the fhandler.

I found the cause. set_close_on_exec() in fhandler_pipe is missing.
set_no_inheritance() calls for all adjunct handles are necessary.

> I pushed two more patches to topic/pipe in terms of inheritence,
> maybe that gives a clue?

I attached two additional patch for this issue.

> > > > +      if (!DuplicateHandle (GetCurrentProcess (), r,
> > > > +			    GetCurrentProcess (), &fhs[1]->query_hdl,
> > > > +			    GENERIC_READ, !(mode & O_CLOEXEC), 0))
> > > 
> > > This is a bug I introduced accidentally during testing.  This
> > > GENERIC_READ is actually supposed to be a FILE_READ_ATTRIBUTES.
> > > Sorry about that.
> > 
> > The PoC code uses PeekNamedPipe for query_hdl, so GENERIC_READ is
> > necessary I think.
> 
> Oh, right, that's how it's documented.  Funny enough, the descriptions
> of FSCTL_PIPE_PEEK does not mention any permissions at all.  I tried the
> permissions and it's FILE_READ_DATA which is required.

Thanks. I revised the patch and attach it the subsequent mail.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

[-- Attachment #2: 0001-Cygwin-fhandler_base-dup-Reflect-O_CLOEXEC-to-inheri.patch --]
[-- Type: application/octet-stream, Size: 1257 bytes --]

From 8dd2bdff579762497898044e9e5304b4b6ab5e93 Mon Sep 17 00:00:00 2001
From: Takashi Yano <takashi.yano@nifty.ne.jp>
Date: Tue, 14 Sep 2021 12:48:03 +0900
Subject: [PATCH 1/2] Cygwin: fhandler_base::dup Reflect O_CLOEXEC to
 inheritance flag.

- Currently fhandler_base::dup duplicates handles with bInheritHandle
  TRUE unconditionally. This patch reflects O_CLOEXEC flag to that
  parameter.
---
 winsup/cygwin/fhandler.cc | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/winsup/cygwin/fhandler.cc b/winsup/cygwin/fhandler.cc
index 39fe2640a..9dfe70be3 100644
--- a/winsup/cygwin/fhandler.cc
+++ b/winsup/cygwin/fhandler.cc
@@ -1308,7 +1308,7 @@ fhandler_base::init (HANDLE f, DWORD a, mode_t bin)
 }
 
 int
-fhandler_base::dup (fhandler_base *child, int)
+fhandler_base::dup (fhandler_base *child, int flags)
 {
   debug_printf ("in fhandler_base dup");
 
@@ -1317,7 +1317,7 @@ fhandler_base::dup (fhandler_base *child, int)
     {
       if (!DuplicateHandle (GetCurrentProcess (), get_handle (),
 			    GetCurrentProcess (), &nh,
-			    0, TRUE, DUPLICATE_SAME_ACCESS))
+			    0, !(flags & O_CLOEXEC), DUPLICATE_SAME_ACCESS))
 	{
 	  debug_printf ("dup(%s) failed, handle %p, %E",
 			get_name (), get_handle ());
-- 
2.33.0


[-- Attachment #3: 0002-Cygwin-pipe-fifo-Call-set_no_inheritance-for-adjunct.patch --]
[-- Type: application/octet-stream, Size: 1962 bytes --]

From a3c223f72f98baf14df3b58941761c72690f5eee Mon Sep 17 00:00:00 2001
From: Takashi Yano <takashi.yano@nifty.ne.jp>
Date: Tue, 14 Sep 2021 12:49:35 +0900
Subject: [PATCH 2/2] Cygwin: pipe, fifo: Call set_no_inheritance() for adjunct
 handles.

- Currntly, set_no_inheritance() is not called for the adjunct handles
  such as select_sem. This patch fixes the issue.
---
 winsup/cygwin/fhandler.h       |  1 +
 winsup/cygwin/fhandler_fifo.cc |  2 ++
 winsup/cygwin/fhandler_pipe.cc | 10 ++++++++++
 3 files changed, 13 insertions(+)

diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
index 13fba9a14..46381c397 100644
--- a/winsup/cygwin/fhandler.h
+++ b/winsup/cygwin/fhandler.h
@@ -1208,6 +1208,7 @@ public:
   void open_setup (int flags);
   void fixup_after_fork (HANDLE);
   int dup (fhandler_base *child, int);
+  void set_close_on_exec (bool val);
   int close ();
   void __reg3 raw_read (void *ptr, size_t& len);
   int ioctl (unsigned int cmd, void *);
diff --git a/winsup/cygwin/fhandler_fifo.cc b/winsup/cygwin/fhandler_fifo.cc
index aa89fa7ae..37498f547 100644
--- a/winsup/cygwin/fhandler_fifo.cc
+++ b/winsup/cygwin/fhandler_fifo.cc
@@ -1817,4 +1817,6 @@ fhandler_fifo::set_close_on_exec (bool val)
 	set_no_inheritance (fc_handler[i].h, val);
       fifo_client_unlock ();
     }
+  if (select_sem)
+    set_no_inheritance (select_sem, val);
 }
diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index 70cfa3784..da473a1dc 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -515,6 +515,16 @@ fhandler_pipe_fifo::raw_write (const void *ptr, size_t len)
   return nbytes ?: -1;
 }
 
+void
+fhandler_pipe::set_close_on_exec (bool val)
+{
+  fhandler_base::set_close_on_exec (val);
+  if (read_mtx)
+    set_no_inheritance (read_mtx, val);
+  if (select_sem)
+    set_no_inheritance (select_sem, val);
+}
+
 void
 fhandler_pipe::fixup_after_fork (HANDLE parent)
 {
-- 
2.33.0


^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-13 19:37                                         ` Takashi Yano
  2021-09-13 20:15                                           ` Corinna Vinschen
@ 2021-09-14  8:08                                           ` Takashi Yano
  2021-09-14  9:03                                             ` Corinna Vinschen
  1 sibling, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-14  8:08 UTC (permalink / raw)
  To: cygwin-developers

[-- Attachment #1: Type: text/plain, Size: 4129 bytes --]

On Tue, 14 Sep 2021 04:37:18 +0900
Takashi Yano wrote:
> On Mon, 13 Sep 2021 20:32:33 +0200
> Corinna Vinschen wrote:
> > On Sep 12 20:04, Takashi Yano wrote:
> > > On Sun, 12 Sep 2021 17:48:49 +0900
> > > Takashi Yano wrote:
> > > > Hmm. Then, what about PoC code attached? This returns to Corinna's
> > > > query_hdl, and counts read/write handles to detect closing reader side.
> > > > 
> > > > If the number of read handles is equal to number of write handles,
> > > > only the pairs of write handle and query_hdl are alive. So, read pipe
> > > > supposed to be closed.
> > > > 
> > > > This patch depends another patch I posted a few hours ago.
> > > 
> > > Revised a bit.
> > > [...]
> > 
> > What I miss is a bit more detailed commit message...
> 
> I am sorry, I will add more detail commit message.

Done.

> > > diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
> > > index 9b4255cfd..b051f5c03 100644
> > > --- a/winsup/cygwin/fhandler_pipe.cc
> > > +++ b/winsup/cygwin/fhandler_pipe.cc
> > > @@ -56,6 +56,8 @@ fhandler_pipe::set_pipe_non_blocking (bool nonblocking)
> > >    fpi.ReadMode = FILE_PIPE_BYTE_STREAM_MODE;
> > >    fpi.CompletionMode = nonblocking ? FILE_PIPE_COMPLETE_OPERATION
> > >      : FILE_PIPE_QUEUE_OPERATION;
> > > +  if (query_hdl)
> > > +    fpi.CompletionMode = FILE_PIPE_COMPLETE_OPERATION;
> > 
> > This should be a single expression, i.e.
> > 
> >    fpi.CompletionMode = nonblocking || query_hdl
> >                         ? FILE_PIPE_COMPLETE_OPERATION
> >                         : FILE_PIPE_QUEUE_OPERATION;
> > 
> > ideally combined with a comment.
> 
> Thanks. I'll do that.

Done.


> > But then again... you're basically switching the write side of
> > a pipe to nonblocking mode unconditionally.  The downside is a
> > busy wait:
> > 
> > >  fhandler_pipe_fifo::raw_write (const void *ptr, size_t len)
> > >  {
> > > @@ -493,7 +490,20 @@ fhandler_pipe_fifo::raw_write (const void *ptr, size_t len)
> > >  			      get_obj_handle_count (select_sem), NULL);
> > >  	  /* 0 bytes returned?  EAGAIN.  See above. */
> > >  	  if (NT_SUCCESS (status) && nbytes == 0)
> > > -	    set_errno (EAGAIN);
> > > +	    {
> > > +	      if (reader_closed ())
> > > +		{
> > > +		  set_errno (EPIPE);
> > > +		  raise (SIGPIPE);
> > > +		}
> > > +	      else if (is_nonblocking ())
> > > +		set_errno (EAGAIN);
> > > +	      else
> > > +		{
> > > +		  cygwait (select_sem, 10);
> > > +		  continue;
> > 
> > I'm a bit puzzled.  The cygwait branch neglects to check if select_sem
> > is NULL (the preceeding ReleaseSemaphore expression does!)
> > And then it doesn't matter if the caller got blocked or not, it will
> > always perform a continue.  So why do it at all?  Worse, if this
> > expression loops, it will eat up the semaphore, because each call will
> > decrement the semaphore count until it blocks.  That sounds wrong to me.
> 
> It is by design. ReleaseSemaphore() releases maximum number of semaphore
> which the waiter can exists. If only one writer and one reader exist,
> ReleaseSemaphore releases 2 semaphores. Then cygwait here consume semaphore
> two times and return to wait state.
> This wait state is released by raw_read() or close().
>  
> > Btw., while looking into the current pipe code, I wonder what select_sem
> > is doing in the pipe code at all so far.  It gets released, but it never
> > gets waited on?!?  Am I missing something?
> 
> The semaphore is waited in select.cc.
> But, wait. Wat happens if select() is not called? Released semaphore
> can be accumulated up to INT32_MAX!!?
> 
> Let me consider.

See the second patch attached. With this patch, only minimum number
of semaphores needed are released.


Please apply following two patches I attached to previous mail first:
0001-Cygwin-fhandler_base-dup-Reflect-O_CLOEXEC-to-inheri.patch
0002-Cygwin-pipe-fifo-Call-set_no_inheritance-for-adjunct.patch

Then, apply the patches attached this mail.
0001-Cygwin-pipe-Use-read-pipe-handle-for-select-on-write.patch
0002-Cygwin-pipe-fifo-Release-select_sem-semaphore-as-muc.patch

Thanks

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

[-- Attachment #2: 0001-Cygwin-pipe-Use-read-pipe-handle-for-select-on-write.patch --]
[-- Type: application/octet-stream, Size: 12170 bytes --]

From 611ac5f87df0b5156e3ec82e98af27892a9c8882 Mon Sep 17 00:00:00 2001
From: Takashi Yano <takashi.yano@nifty.ne.jp>
Date: Tue, 14 Sep 2021 12:27:33 +0900
Subject: [PATCH 1/2] Cygwin: pipe: Use read pipe handle for select() on write
 pipe.

- Usually WriteQuotaAvailable retrieved by NtQueryInformationFile()
  on the write side reflects the space available in the inbound buffer
  on the read side. However, if a pipe read is currently pending,
  WriteQuotaAvailable on the write side is decremented by the number
  of bytes the read side is requesting. So it's possible (even likely)
  that WriteQuotaAvailable is 0, even if the inbound buffer on the
  read side is not full. This can lead to a deadlock situation:
  The reader is waiting for data, but select on the writer side
  assumes that no space is available in the read side inbound buffer.

  Currently, to avoid this stuation, read() does not request larger
  block than pipe size - 1. However, this mechanism does not take
  effect if the reader side is non-cygwin app.

  The only reliable information is available on the read side, so
  fetch info from the read side via the pipe-specific query handle
  (query_hdl) introduced.
---
 winsup/cygwin/fhandler.h       |  14 ++++-
 winsup/cygwin/fhandler_pipe.cc | 105 +++++++++++++++++++++++----------
 winsup/cygwin/select.cc        |  52 +++++++++++++---
 winsup/cygwin/spawn.cc         |  11 ++++
 4 files changed, 144 insertions(+), 38 deletions(-)

diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
index 46381c397..db2325144 100644
--- a/winsup/cygwin/fhandler.h
+++ b/winsup/cygwin/fhandler.h
@@ -1176,10 +1176,22 @@ class fhandler_pipe_fifo: public fhandler_base
 {
  protected:
   size_t pipe_buf_size;
+  HANDLE query_hdl;
 
  public:
   fhandler_pipe_fifo ();
 
+  HANDLE get_query_handle () const { return query_hdl; }
+  void close_query_handle ()
+  {
+    if (query_hdl)
+      {
+	CloseHandle (query_hdl);
+	query_hdl = NULL;
+      }
+  }
+  bool reader_closed ();
+
   ssize_t __reg3 raw_write (const void *ptr, size_t len);
 
 };
@@ -1189,7 +1201,6 @@ class fhandler_pipe: public fhandler_pipe_fifo
 private:
   HANDLE read_mtx;
   pid_t popen_pid;
-  void set_pipe_non_blocking (bool nonblocking);
 public:
   fhandler_pipe ();
 
@@ -1237,6 +1248,7 @@ public:
     fh->copy_from (this);
     return fh;
   }
+  void set_pipe_non_blocking (bool nonblocking);
 };
 
 #define CYGWIN_FIFO_PIPE_NAME_LEN     47
diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index da473a1dc..4dab3015d 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -54,8 +54,12 @@ fhandler_pipe::set_pipe_non_blocking (bool nonblocking)
   FILE_PIPE_INFORMATION fpi;
 
   fpi.ReadMode = FILE_PIPE_BYTE_STREAM_MODE;
-  fpi.CompletionMode = nonblocking ? FILE_PIPE_COMPLETE_OPERATION
-    : FILE_PIPE_QUEUE_OPERATION;
+  /* If query_hdl is set, write pipe should check reader_closed()
+     while raw_read(). If the pipe is blocking, raw_write() stops
+     at NtWriteFile() and loses the chance to check it. Therefore,
+     always set write pipe to non-blocking. */
+  fpi.CompletionMode = (nonblocking || query_hdl)
+    ? FILE_PIPE_COMPLETE_OPERATION : FILE_PIPE_QUEUE_OPERATION;
   status = NtSetInformationFile (get_handle (), &io, &fpi, sizeof fpi,
 				 FilePipeInformation);
   if (!NT_SUCCESS (status))
@@ -202,6 +206,8 @@ fhandler_pipe::open_setup (int flags)
       if (!read_mtx)
 	debug_printf ("CreateMutex failed: %E");
     }
+  if (get_dev () == FH_PIPEW && !query_hdl)
+    set_pipe_non_blocking (is_nonblocking ());
 }
 
 off_t
@@ -268,39 +274,22 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
   while (nbytes < len)
     {
       ULONG_PTR nbytes_now = 0;
-      size_t left = len - nbytes;
-      ULONG len1 = (ULONG) left;
+      ULONG len1 = (ULONG) (len - nbytes);
       waitret = WAIT_OBJECT_0;
 
       if (evt)
 	ResetEvent (evt);
-      if (!is_nonblocking ())
+      FILE_PIPE_LOCAL_INFORMATION fpli;
+      status = NtQueryInformationFile (get_handle (), &io,
+				       &fpli, sizeof (fpli),
+				       FilePipeLocalInformation);
+      if (NT_SUCCESS (status))
 	{
-	  FILE_PIPE_LOCAL_INFORMATION fpli;
-
-	  /* If the pipe is empty, don't request more bytes than pipe
-	     buffer size - 1. Pending read lowers WriteQuotaAvailable on
-	     the write side and thus affects select's ability to return
-	     more or less reliable info whether a write succeeds or not. */
-	  ULONG chunk = pipe_buf_size - 1;
-	  status = NtQueryInformationFile (get_handle (), &io,
-					   &fpli, sizeof (fpli),
-					   FilePipeLocalInformation);
-	  if (NT_SUCCESS (status))
-	    {
-	      if (fpli.ReadDataAvailable > 0)
-		chunk = left;
-	      else if (nbytes != 0)
-		break;
-	      else
-		chunk = fpli.InboundQuota - 1;
-	    }
-	  else if (nbytes != 0)
-	    break;
-
-	  if (len1 > chunk)
-	    len1 = chunk;
+	if (fpli.ReadDataAvailable == 0 && nbytes != 0)
+	  break;
 	}
+      else if (nbytes != 0)
+	break;
       status = NtReadFile (get_handle (), evt, NULL, NULL, &io, ptr,
 			   len1, NULL, NULL);
       if (evt && status == STATUS_PENDING)
@@ -385,6 +374,16 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
   len = nbytes;
 }
 
+bool
+fhandler_pipe_fifo::reader_closed ()
+{
+  if (!query_hdl)
+    return false;
+  int n_reader = get_obj_handle_count (query_hdl);
+  int n_writer = get_obj_handle_count (get_handle ());
+  return n_reader == n_writer;
+}
+
 ssize_t __reg3
 fhandler_pipe_fifo::raw_write (const void *ptr, size_t len)
 {
@@ -493,7 +492,20 @@ fhandler_pipe_fifo::raw_write (const void *ptr, size_t len)
 			      get_obj_handle_count (select_sem), NULL);
 	  /* 0 bytes returned?  EAGAIN.  See above. */
 	  if (NT_SUCCESS (status) && nbytes == 0)
-	    set_errno (EAGAIN);
+	    {
+	      if (reader_closed ())
+		{
+		  set_errno (EPIPE);
+		  raise (SIGPIPE);
+		}
+	      else if (is_nonblocking ())
+		set_errno (EAGAIN);
+	      else
+		{
+		  cygwait (select_sem, 10);
+		  continue;
+		}
+	    }
 	}
       else if (STATUS_PIPE_IS_CLOSED (status))
 	{
@@ -523,6 +535,8 @@ fhandler_pipe::set_close_on_exec (bool val)
     set_no_inheritance (read_mtx, val);
   if (select_sem)
     set_no_inheritance (select_sem, val);
+  if (query_hdl)
+    set_no_inheritance (query_hdl, val);
 }
 
 void
@@ -532,6 +546,9 @@ fhandler_pipe::fixup_after_fork (HANDLE parent)
     fork_fixup (parent, read_mtx, "read_mtx");
   if (select_sem)
     fork_fixup (parent, select_sem, "select_sem");
+  if (query_hdl)
+    fork_fixup (parent, query_hdl, "query_hdl");
+
   fhandler_base::fixup_after_fork (parent);
 }
 
@@ -562,6 +579,15 @@ fhandler_pipe::dup (fhandler_base *child, int flags)
       ftp->close ();
       res = -1;
     }
+  else if (query_hdl &&
+	   !DuplicateHandle (GetCurrentProcess (), query_hdl,
+			    GetCurrentProcess (), &ftp->query_hdl,
+			    0, !(flags & O_CLOEXEC), DUPLICATE_SAME_ACCESS))
+    {
+      __seterrno ();
+      ftp->close ();
+      res = -1;
+    }
 
   debug_printf ("res %d", res);
   return res;
@@ -577,6 +603,8 @@ fhandler_pipe::close ()
       ReleaseSemaphore (select_sem, get_obj_handle_count (select_sem), NULL);
       CloseHandle (select_sem);
     }
+  if (query_hdl)
+    CloseHandle (query_hdl);
   return fhandler_base::close ();
 }
 
@@ -797,6 +825,23 @@ fhandler_pipe::create (fhandler_pipe *fhs[2], unsigned psize, int mode)
 	DuplicateHandle (GetCurrentProcess (), fhs[0]->select_sem,
 			 GetCurrentProcess (), &fhs[1]->select_sem,
 			 0, sa->bInheritHandle, DUPLICATE_SAME_ACCESS);
+      if (!DuplicateHandle (GetCurrentProcess (), r,
+			    GetCurrentProcess (), &fhs[1]->query_hdl,
+			    FILE_READ_DATA, sa->bInheritHandle, 0))
+	{
+	  CloseHandle (fhs[0]->select_sem);
+	  delete fhs[0];
+	  CloseHandle (r);
+	  CloseHandle (fhs[1]->select_sem);
+	  delete fhs[1];
+	  CloseHandle (w);
+	}
+      else
+	{
+	  /* Call set_pipe_non_blocking() again after creating query_hdl. */
+	  fhs[1]->set_pipe_non_blocking (fhs[1]->is_nonblocking ());
+	  res = 0;
+	}
     }
 
   debug_printf ("%R = pipe([%p, %p], %d, %y)", res, fhs[0], fhs[1], psize, mode);
diff --git a/winsup/cygwin/select.cc b/winsup/cygwin/select.cc
index 5e583434c..ac2f3a9e0 100644
--- a/winsup/cygwin/select.cc
+++ b/winsup/cygwin/select.cc
@@ -608,16 +608,47 @@ pipe_data_available (int fd, fhandler_base *fh, HANDLE h, bool writing)
     }
   if (writing)
     {
-      /* WriteQuotaAvailable is decremented by the number of bytes requested
-	 by a blocking reader on the other side of the pipe.  Cygwin readers
-	 are serialized and never request a number of bytes equivalent to the
-	 full buffer size.  So WriteQuotaAvailable is 0 only if either the
-	 read buffer on the other side is really full, or if we have non-Cygwin
-	 readers. */
+      /* If there is anything available in the pipe buffer then signal
+        that.  This means that a pipe could still block since you could
+        be trying to write more to the pipe than is available in the
+        buffer but that is the hazard of select().
+
+        Note that WriteQuotaAvailable is unreliable.
+
+        Usually WriteQuotaAvailable on the write side reflects the space
+        available in the inbound buffer on the read side.  However, if a
+        pipe read is currently pending, WriteQuotaAvailable on the write side
+        is decremented by the number of bytes the read side is requesting.
+        So it's possible (even likely) that WriteQuotaAvailable is 0, even
+        if the inbound buffer on the read side is not full.  This can lead to
+        a deadlock situation: The reader is waiting for data, but select
+        on the writer side assumes that no space is available in the read
+        side inbound buffer.
+
+        Consequentially, the only reliable information is available on the
+        read side, so fetch info from the read side via the pipe-specific
+        query handle.  Use fpli.WriteQuotaAvailable as storage for the actual
+        interesting value, which is the InboundQuote on the write side,
+        decremented by the number of bytes of data in that buffer. */
+      /* Note: Do not use NtQueryInformationFile() for query_hdl because
+	 NtQueryInformationFile() seems to interfere with reading pipes
+	 in non-cygwin apps. Instead, use PeekNamedPipe() here. */
+      if (fh->get_device () == FH_PIPEW)
+	{
+	  HANDLE query_hdl = ((fhandler_pipe *) fh)->get_query_handle ();
+	  if (query_hdl)
+	    {
+	      DWORD nbytes_in_pipe;
+	      PeekNamedPipe (query_hdl, NULL, 0, NULL, &nbytes_in_pipe, NULL);
+	      fpli.WriteQuotaAvailable = fpli.InboundQuota - nbytes_in_pipe;
+	    }
+	  else
+	    return 1;
+	}
       if (fpli.WriteQuotaAvailable > 0)
 	{
 	  paranoid_printf ("fd %d, %s, write: size %u, avail %u", fd,
-			   fh->get_name (), fpli.OutboundQuota,
+			   fh->get_name (), fpli.InboundQuota,
 			   fpli.WriteQuotaAvailable);
 	  return 1;
 	}
@@ -712,6 +743,13 @@ out:
   h = fh->get_output_handle ();
   if (s->write_selected && dev != FH_PIPER)
     {
+      if (dev == FH_PIPEW && ((fhandler_pipe *) fh)->reader_closed ())
+	{
+	  gotone += s->write_ready = true;
+	  if (s->except_selected)
+	    gotone += s->except_ready = true;
+	  return gotone;
+	}
       gotone += s->write_ready =  pipe_data_available (s->fd, fh, h, true);
       select_printf ("write: %s, gotone %d", fh->get_name (), gotone);
     }
diff --git a/winsup/cygwin/spawn.cc b/winsup/cygwin/spawn.cc
index 0bde0b04d..6b2026776 100644
--- a/winsup/cygwin/spawn.cc
+++ b/winsup/cygwin/spawn.cc
@@ -657,6 +657,17 @@ child_info_spawn::worker (const char *prog_arg, const char *const *argv,
 		ptys->create_invisible_console ();
 		ptys->setup_locale ();
 	      }
+	    else if (cfd->get_dev () == FH_PIPEW)
+	      {
+		fhandler_pipe *pipe = (fhandler_pipe *)(fhandler_base *) cfd;
+		pipe->close_query_handle ();
+		pipe->set_pipe_non_blocking (false);
+	      }
+	    else if (cfd->get_dev () == FH_PIPER)
+	      {
+		fhandler_pipe *pipe = (fhandler_pipe *)(fhandler_base *) cfd;
+		pipe->set_pipe_non_blocking (false);
+	      }
 	}
 
       bool enable_pcon = false;
-- 
2.33.0


[-- Attachment #3: 0002-Cygwin-pipe-fifo-Release-select_sem-semaphore-as-muc.patch --]
[-- Type: application/octet-stream, Size: 5277 bytes --]

From e5c64960fddd43f08dae7afbe3ae1c75bd41c81d Mon Sep 17 00:00:00 2001
From: Takashi Yano <takashi.yano@nifty.ne.jp>
Date: Tue, 14 Sep 2021 13:10:54 +0900
Subject: [PATCH 2/2] Cygwin: pipe, fifo: Release select_sem semaphore as much
 as needed.

- Currently, raw_read(), raw_write() and close() release select_sem
  unconditionally even if no waiter for select_sem exists. With this
  patch, only the minimum number of semaphores required is released.
---
 winsup/cygwin/fhandler.h       |  4 ++++
 winsup/cygwin/fhandler_fifo.cc | 28 ++++++++++++++++++++++------
 winsup/cygwin/fhandler_pipe.cc | 28 +++++++++++++++++++++-------
 3 files changed, 47 insertions(+), 13 deletions(-)

diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
index db2325144..9580a698c 100644
--- a/winsup/cygwin/fhandler.h
+++ b/winsup/cygwin/fhandler.h
@@ -1177,6 +1177,7 @@ class fhandler_pipe_fifo: public fhandler_base
  protected:
   size_t pipe_buf_size;
   HANDLE query_hdl;
+  virtual void release_select_sem (const char *) {};
 
  public:
   fhandler_pipe_fifo ();
@@ -1201,6 +1202,7 @@ class fhandler_pipe: public fhandler_pipe_fifo
 private:
   HANDLE read_mtx;
   pid_t popen_pid;
+  void release_select_sem (const char *);
 public:
   fhandler_pipe ();
 
@@ -1444,6 +1446,8 @@ class fhandler_fifo: public fhandler_pipe_fifo
   void shared_fc_handler_updated (bool val)
   { shmem->shared_fc_handler_updated (val); }
 
+  void release_select_sem (const char *);
+
 public:
   fhandler_fifo ();
   ~fhandler_fifo ()
diff --git a/winsup/cygwin/fhandler_fifo.cc b/winsup/cygwin/fhandler_fifo.cc
index 37498f547..489ba528c 100644
--- a/winsup/cygwin/fhandler_fifo.cc
+++ b/winsup/cygwin/fhandler_fifo.cc
@@ -1185,6 +1185,22 @@ fhandler_fifo::take_ownership (DWORD timeout)
   return ret;
 }
 
+void
+fhandler_fifo::release_select_sem (const char *from)
+{
+  LONG n_release;
+  if (reader) /* Number of select() call. */
+    n_release = get_obj_handle_count (select_sem)
+      - get_obj_handle_count (read_ready);
+  else /* Number of select() and reader */
+    n_release = get_obj_handle_count (select_sem)
+      - get_obj_handle_count (get_handle ());
+  debug_printf("%s(%s) release %d", from,
+	       reader ? "reader" : "writer", n_release);
+  if (n_release)
+    ReleaseSemaphore (select_sem, n_release, NULL);
+}
+
 void __reg3
 fhandler_fifo::raw_read (void *in_ptr, size_t& len)
 {
@@ -1372,7 +1388,7 @@ out:
   fifo_client_unlock ();
   reading_unlock ();
   if (select_sem)
-    ReleaseSemaphore (select_sem, get_obj_handle_count (select_sem), NULL);
+    release_select_sem ("raw_read");
 }
 
 int __reg2
@@ -1483,6 +1499,11 @@ fhandler_fifo::cancel_reader_thread ()
 int
 fhandler_fifo::close ()
 {
+  if (select_sem)
+    {
+      release_select_sem ("close");
+      NtClose (select_sem);
+    }
   if (writer)
     {
       nwriters_lock ();
@@ -1574,11 +1595,6 @@ fhandler_fifo::close ()
     NtClose (write_ready);
   if (writer_opening)
     NtClose (writer_opening);
-  if (select_sem)
-    {
-      ReleaseSemaphore (select_sem, get_obj_handle_count (select_sem), NULL);
-      NtClose (select_sem);
-    }
   if (nohandle ())
     return 0;
   else
diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index 4dab3015d..c0379e81d 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -237,6 +237,22 @@ fhandler_pipe::get_proc_fd_name (char *buf)
   return buf;
 }
 
+void
+fhandler_pipe::release_select_sem (const char *from)
+{
+  LONG n_release;
+  if (get_dev () == FH_PIPER) /* Number of select() and writer */
+    n_release = get_obj_handle_count (select_sem)
+      - get_obj_handle_count (read_mtx);
+  else /* Number of select() call */
+    n_release = get_obj_handle_count (select_sem)
+      - get_obj_handle_count (query_hdl);
+  debug_printf("%s(%s) release %d", from,
+	       get_dev () == FH_PIPER ? "PIPER" : "PIPEW", n_release);
+  if (n_release)
+    ReleaseSemaphore (select_sem, n_release, NULL);
+}
+
 void __reg3
 fhandler_pipe::raw_read (void *ptr, size_t& len)
 {
@@ -328,8 +344,7 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
 	  ptr = ((char *) ptr) + nbytes_now;
 	  nbytes += nbytes_now;
 	  if (select_sem && nbytes_now > 0)
-	    ReleaseSemaphore (select_sem,
-			      get_obj_handle_count (select_sem), NULL);
+	    release_select_sem ("raw_read");
 	}
       else
 	{
@@ -488,8 +503,7 @@ fhandler_pipe_fifo::raw_write (const void *ptr, size_t len)
 	  ptr = ((char *) ptr) + nbytes_now;
 	  nbytes += nbytes_now;
 	  if (select_sem && nbytes_now > 0)
-	    ReleaseSemaphore (select_sem,
-			      get_obj_handle_count (select_sem), NULL);
+	    release_select_sem ("raw_write");
 	  /* 0 bytes returned?  EAGAIN.  See above. */
 	  if (NT_SUCCESS (status) && nbytes == 0)
 	    {
@@ -596,13 +610,13 @@ fhandler_pipe::dup (fhandler_base *child, int flags)
 int
 fhandler_pipe::close ()
 {
-  if (read_mtx)
-    CloseHandle (read_mtx);
   if (select_sem)
     {
-      ReleaseSemaphore (select_sem, get_obj_handle_count (select_sem), NULL);
+      release_select_sem ("close");
       CloseHandle (select_sem);
     }
+  if (read_mtx)
+    CloseHandle (read_mtx);
   if (query_hdl)
     CloseHandle (query_hdl);
   return fhandler_base::close ();
-- 
2.33.0


^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-14  8:07                                             ` Takashi Yano
@ 2021-09-14  8:47                                               ` Corinna Vinschen
  2021-09-14 12:38                                                 ` Ken Brown
  0 siblings, 1 reply; 250+ messages in thread
From: Corinna Vinschen @ 2021-09-14  8:47 UTC (permalink / raw)
  To: cygwin-developers

On Sep 14 17:07, Takashi Yano wrote:
> On Mon, 13 Sep 2021 22:15:25 +0200
> Corinna Vinschen wrote:
> > That should depend on the O_CLOEXEC setting, but identically for
> > all handles in the fhandler.
> 
> I found the cause. set_close_on_exec() in fhandler_pipe is missing.
> set_no_inheritance() calls for all adjunct handles are necessary.
> 
> > I pushed two more patches to topic/pipe in terms of inheritence,
> > maybe that gives a clue?
> 
> I attached two additional patch for this issue.

Uh oh!  This patch to fhandler_base::dup made me check other fhandlers
and, yeah, we have more unconditional inheritence ignoring O_CLOEXEC
(fhandler_tape for instance).  We should fix that at one point, but that
requires your patch to go to master first.  Let's just keep that in mind
for now.

I'll push both patches in a bit.


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-14  8:08                                           ` Takashi Yano
@ 2021-09-14  9:03                                             ` Corinna Vinschen
  2021-09-14  9:56                                               ` Takashi Yano
  0 siblings, 1 reply; 250+ messages in thread
From: Corinna Vinschen @ 2021-09-14  9:03 UTC (permalink / raw)
  To: cygwin-developers

On Sep 14 17:08, Takashi Yano wrote:
> @@ -54,8 +54,12 @@ fhandler_pipe::set_pipe_non_blocking (bool nonblocking)
>    FILE_PIPE_INFORMATION fpi;
>  
>    fpi.ReadMode = FILE_PIPE_BYTE_STREAM_MODE;
> -  fpi.CompletionMode = nonblocking ? FILE_PIPE_COMPLETE_OPERATION
> -    : FILE_PIPE_QUEUE_OPERATION;
> +  /* If query_hdl is set, write pipe should check reader_closed()
> +     while raw_read(). If the pipe is blocking, raw_write() stops
> +     at NtWriteFile() and loses the chance to check it. Therefore,
> +     always set write pipe to non-blocking. */
> +  fpi.CompletionMode = (nonblocking || query_hdl)
> +    ? FILE_PIPE_COMPLETE_OPERATION : FILE_PIPE_QUEUE_OPERATION;
>    status = NtSetInformationFile (get_handle (), &io, &fpi, sizeof fpi,
>  				 FilePipeInformation);
>    if (!NT_SUCCESS (status))

I don't quite follow the argument.  Blocking pipes are using
asynchronous IO, so they are in fact not blocking calls on the
OS level.  After calling NtWriteFile, the blocking variation
will go into the subsequent

  waitret = cygwait (evt, INFINITE, cw_cancel | cw_sig);

So, wouldn't you get the same effect by keeping the pipe in
FILE_PIPE_QUEUE_OPERATION mode and just add a timeout to the above
cygwait call and handle select_sem in a not yet existing WAIT_TIMEOUT
conditional?


Corinna


P.S.: Maybe the cygwait call is just too simple.  It would be nice if it
      had been defined to take an array of handles, rather than just a
      single handle.  Another change we should keep in mind.

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-14  9:03                                             ` Corinna Vinschen
@ 2021-09-14  9:56                                               ` Takashi Yano
  2021-09-14 10:19                                                 ` Takashi Yano
  0 siblings, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-14  9:56 UTC (permalink / raw)
  To: cygwin-developers

[-- Attachment #1: Type: text/plain, Size: 1545 bytes --]

On Tue, 14 Sep 2021 11:03:39 +0200
Corinna Vinschen wrote:
> On Sep 14 17:08, Takashi Yano wrote:
> > @@ -54,8 +54,12 @@ fhandler_pipe::set_pipe_non_blocking (bool nonblocking)
> >    FILE_PIPE_INFORMATION fpi;
> >  
> >    fpi.ReadMode = FILE_PIPE_BYTE_STREAM_MODE;
> > -  fpi.CompletionMode = nonblocking ? FILE_PIPE_COMPLETE_OPERATION
> > -    : FILE_PIPE_QUEUE_OPERATION;
> > +  /* If query_hdl is set, write pipe should check reader_closed()
> > +     while raw_read(). If the pipe is blocking, raw_write() stops
> > +     at NtWriteFile() and loses the chance to check it. Therefore,
> > +     always set write pipe to non-blocking. */
> > +  fpi.CompletionMode = (nonblocking || query_hdl)
> > +    ? FILE_PIPE_COMPLETE_OPERATION : FILE_PIPE_QUEUE_OPERATION;
> >    status = NtSetInformationFile (get_handle (), &io, &fpi, sizeof fpi,
> >  				 FilePipeInformation);
> >    if (!NT_SUCCESS (status))
> 
> I don't quite follow the argument.  Blocking pipes are using
> asynchronous IO, so they are in fact not blocking calls on the
> OS level.  After calling NtWriteFile, the blocking variation
> will go into the subsequent
> 
>   waitret = cygwait (evt, INFINITE, cw_cancel | cw_sig);
> 
> So, wouldn't you get the same effect by keeping the pipe in
> FILE_PIPE_QUEUE_OPERATION mode and just add a timeout to the above
> cygwait call and handle select_sem in a not yet existing WAIT_TIMEOUT
> conditional?

Sounds reasonable. I revised the patches. Do you mean something like
patch attached?

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

[-- Attachment #2: 0001-Cygwin-pipe-Use-read-pipe-handle-for-select-on-write.patch --]
[-- Type: application/octet-stream, Size: 11763 bytes --]

From cf6808184a2c2f47f035dca930adbc2e4bf7f53a Mon Sep 17 00:00:00 2001
From: Takashi Yano <takashi.yano@nifty.ne.jp>
Date: Tue, 14 Sep 2021 18:40:27 +0900
Subject: [PATCH 1/2] Cygwin: pipe: Use read pipe handle for select() on write
 pipe.

- Usually WriteQuotaAvailable retrieved by NtQueryInformationFile()
  on the write side reflects the space available in the inbound buffer
  on the read side. However, if a pipe read is currently pending,
  WriteQuotaAvailable on the write side is decremented by the number
  of bytes the read side is requesting. So it's possible (even likely)
  that WriteQuotaAvailable is 0, even if the inbound buffer on the
  read side is not full. This can lead to a deadlock situation:
  The reader is waiting for data, but select on the writer side
  assumes that no space is available in the read side inbound buffer.

  Currently, to avoid this stuation, read() does not request larger
  block than pipe size - 1. However, this mechanism does not take
  effect if the reader side is non-cygwin app.

  The only reliable information is available on the read side, so
  fetch info from the read side via the pipe-specific query handle
  (query_hdl) introduced.

  If the query_hdl (read handle) is kept in write side, writer can
  not detect closure of read pipe. Therefore, raw_write() counts
  write handle and query_hdl. If they are equal, only the pairs of
  write handle and query_hdl are alive. In this case, raw_write()
  returns EPIPE and raises SIGPIPE.
---
 winsup/cygwin/fhandler.h       | 14 ++++-
 winsup/cygwin/fhandler_pipe.cc | 96 ++++++++++++++++++++++++----------
 winsup/cygwin/select.cc        | 52 +++++++++++++++---
 winsup/cygwin/spawn.cc         | 11 ++++
 4 files changed, 137 insertions(+), 36 deletions(-)

diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
index 46381c397..db2325144 100644
--- a/winsup/cygwin/fhandler.h
+++ b/winsup/cygwin/fhandler.h
@@ -1176,10 +1176,22 @@ class fhandler_pipe_fifo: public fhandler_base
 {
  protected:
   size_t pipe_buf_size;
+  HANDLE query_hdl;
 
  public:
   fhandler_pipe_fifo ();
 
+  HANDLE get_query_handle () const { return query_hdl; }
+  void close_query_handle ()
+  {
+    if (query_hdl)
+      {
+	CloseHandle (query_hdl);
+	query_hdl = NULL;
+      }
+  }
+  bool reader_closed ();
+
   ssize_t __reg3 raw_write (const void *ptr, size_t len);
 
 };
@@ -1189,7 +1201,6 @@ class fhandler_pipe: public fhandler_pipe_fifo
 private:
   HANDLE read_mtx;
   pid_t popen_pid;
-  void set_pipe_non_blocking (bool nonblocking);
 public:
   fhandler_pipe ();
 
@@ -1237,6 +1248,7 @@ public:
     fh->copy_from (this);
     return fh;
   }
+  void set_pipe_non_blocking (bool nonblocking);
 };
 
 #define CYGWIN_FIFO_PIPE_NAME_LEN     47
diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index da473a1dc..b2c95b56e 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -202,6 +202,8 @@ fhandler_pipe::open_setup (int flags)
       if (!read_mtx)
 	debug_printf ("CreateMutex failed: %E");
     }
+  if (get_dev () == FH_PIPEW && !query_hdl)
+    set_pipe_non_blocking (is_nonblocking ());
 }
 
 off_t
@@ -268,39 +270,22 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
   while (nbytes < len)
     {
       ULONG_PTR nbytes_now = 0;
-      size_t left = len - nbytes;
-      ULONG len1 = (ULONG) left;
+      ULONG len1 = (ULONG) (len - nbytes);
       waitret = WAIT_OBJECT_0;
 
       if (evt)
 	ResetEvent (evt);
-      if (!is_nonblocking ())
+      FILE_PIPE_LOCAL_INFORMATION fpli;
+      status = NtQueryInformationFile (get_handle (), &io,
+				       &fpli, sizeof (fpli),
+				       FilePipeLocalInformation);
+      if (NT_SUCCESS (status))
 	{
-	  FILE_PIPE_LOCAL_INFORMATION fpli;
-
-	  /* If the pipe is empty, don't request more bytes than pipe
-	     buffer size - 1. Pending read lowers WriteQuotaAvailable on
-	     the write side and thus affects select's ability to return
-	     more or less reliable info whether a write succeeds or not. */
-	  ULONG chunk = pipe_buf_size - 1;
-	  status = NtQueryInformationFile (get_handle (), &io,
-					   &fpli, sizeof (fpli),
-					   FilePipeLocalInformation);
-	  if (NT_SUCCESS (status))
-	    {
-	      if (fpli.ReadDataAvailable > 0)
-		chunk = left;
-	      else if (nbytes != 0)
-		break;
-	      else
-		chunk = fpli.InboundQuota - 1;
-	    }
-	  else if (nbytes != 0)
-	    break;
-
-	  if (len1 > chunk)
-	    len1 = chunk;
+	if (fpli.ReadDataAvailable == 0 && nbytes != 0)
+	  break;
 	}
+      else if (nbytes != 0)
+	break;
       status = NtReadFile (get_handle (), evt, NULL, NULL, &io, ptr,
 			   len1, NULL, NULL);
       if (evt && status == STATUS_PENDING)
@@ -385,6 +370,16 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
   len = nbytes;
 }
 
+bool
+fhandler_pipe_fifo::reader_closed ()
+{
+  if (!query_hdl)
+    return false;
+  int n_reader = get_obj_handle_count (query_hdl);
+  int n_writer = get_obj_handle_count (get_handle ());
+  return n_reader == n_writer;
+}
+
 ssize_t __reg3
 fhandler_pipe_fifo::raw_write (const void *ptr, size_t len)
 {
@@ -457,7 +452,23 @@ fhandler_pipe_fifo::raw_write (const void *ptr, size_t len)
 	}
       if (evt && status == STATUS_PENDING)
 	{
-	  waitret = cygwait (evt, INFINITE, cw_cancel | cw_sig);
+	  do
+	    {
+	      waitret = cygwait (evt, (DWORD) 0, cw_cancel | cw_sig);
+	      if (waitret == WAIT_TIMEOUT)
+		{
+		  if (reader_closed ())
+		    {
+		      CancelIo (get_handle ());
+		      set_errno (EPIPE);
+		      raise (SIGPIPE);
+		      break;
+		    }
+		  else
+		    cygwait (select_sem, 10);
+		}
+	    }
+	  while (waitret == WAIT_TIMEOUT);
 	  /* If io.Status is STATUS_CANCELLED after CancelIo, IO has actually
 	     been cancelled and io.Information contains the number of bytes
 	     processed so far.
@@ -523,6 +534,8 @@ fhandler_pipe::set_close_on_exec (bool val)
     set_no_inheritance (read_mtx, val);
   if (select_sem)
     set_no_inheritance (select_sem, val);
+  if (query_hdl)
+    set_no_inheritance (query_hdl, val);
 }
 
 void
@@ -532,6 +545,9 @@ fhandler_pipe::fixup_after_fork (HANDLE parent)
     fork_fixup (parent, read_mtx, "read_mtx");
   if (select_sem)
     fork_fixup (parent, select_sem, "select_sem");
+  if (query_hdl)
+    fork_fixup (parent, query_hdl, "query_hdl");
+
   fhandler_base::fixup_after_fork (parent);
 }
 
@@ -562,6 +578,15 @@ fhandler_pipe::dup (fhandler_base *child, int flags)
       ftp->close ();
       res = -1;
     }
+  else if (query_hdl &&
+	   !DuplicateHandle (GetCurrentProcess (), query_hdl,
+			    GetCurrentProcess (), &ftp->query_hdl,
+			    0, !(flags & O_CLOEXEC), DUPLICATE_SAME_ACCESS))
+    {
+      __seterrno ();
+      ftp->close ();
+      res = -1;
+    }
 
   debug_printf ("res %d", res);
   return res;
@@ -577,6 +602,8 @@ fhandler_pipe::close ()
       ReleaseSemaphore (select_sem, get_obj_handle_count (select_sem), NULL);
       CloseHandle (select_sem);
     }
+  if (query_hdl)
+    CloseHandle (query_hdl);
   return fhandler_base::close ();
 }
 
@@ -797,6 +824,19 @@ fhandler_pipe::create (fhandler_pipe *fhs[2], unsigned psize, int mode)
 	DuplicateHandle (GetCurrentProcess (), fhs[0]->select_sem,
 			 GetCurrentProcess (), &fhs[1]->select_sem,
 			 0, sa->bInheritHandle, DUPLICATE_SAME_ACCESS);
+      if (!DuplicateHandle (GetCurrentProcess (), r,
+			    GetCurrentProcess (), &fhs[1]->query_hdl,
+			    FILE_READ_DATA, sa->bInheritHandle, 0))
+	{
+	  CloseHandle (fhs[0]->select_sem);
+	  delete fhs[0];
+	  CloseHandle (r);
+	  CloseHandle (fhs[1]->select_sem);
+	  delete fhs[1];
+	  CloseHandle (w);
+	}
+      else
+	res = 0;
     }
 
   debug_printf ("%R = pipe([%p, %p], %d, %y)", res, fhs[0], fhs[1], psize, mode);
diff --git a/winsup/cygwin/select.cc b/winsup/cygwin/select.cc
index 5e583434c..ac2f3a9e0 100644
--- a/winsup/cygwin/select.cc
+++ b/winsup/cygwin/select.cc
@@ -608,16 +608,47 @@ pipe_data_available (int fd, fhandler_base *fh, HANDLE h, bool writing)
     }
   if (writing)
     {
-      /* WriteQuotaAvailable is decremented by the number of bytes requested
-	 by a blocking reader on the other side of the pipe.  Cygwin readers
-	 are serialized and never request a number of bytes equivalent to the
-	 full buffer size.  So WriteQuotaAvailable is 0 only if either the
-	 read buffer on the other side is really full, or if we have non-Cygwin
-	 readers. */
+      /* If there is anything available in the pipe buffer then signal
+        that.  This means that a pipe could still block since you could
+        be trying to write more to the pipe than is available in the
+        buffer but that is the hazard of select().
+
+        Note that WriteQuotaAvailable is unreliable.
+
+        Usually WriteQuotaAvailable on the write side reflects the space
+        available in the inbound buffer on the read side.  However, if a
+        pipe read is currently pending, WriteQuotaAvailable on the write side
+        is decremented by the number of bytes the read side is requesting.
+        So it's possible (even likely) that WriteQuotaAvailable is 0, even
+        if the inbound buffer on the read side is not full.  This can lead to
+        a deadlock situation: The reader is waiting for data, but select
+        on the writer side assumes that no space is available in the read
+        side inbound buffer.
+
+        Consequentially, the only reliable information is available on the
+        read side, so fetch info from the read side via the pipe-specific
+        query handle.  Use fpli.WriteQuotaAvailable as storage for the actual
+        interesting value, which is the InboundQuote on the write side,
+        decremented by the number of bytes of data in that buffer. */
+      /* Note: Do not use NtQueryInformationFile() for query_hdl because
+	 NtQueryInformationFile() seems to interfere with reading pipes
+	 in non-cygwin apps. Instead, use PeekNamedPipe() here. */
+      if (fh->get_device () == FH_PIPEW)
+	{
+	  HANDLE query_hdl = ((fhandler_pipe *) fh)->get_query_handle ();
+	  if (query_hdl)
+	    {
+	      DWORD nbytes_in_pipe;
+	      PeekNamedPipe (query_hdl, NULL, 0, NULL, &nbytes_in_pipe, NULL);
+	      fpli.WriteQuotaAvailable = fpli.InboundQuota - nbytes_in_pipe;
+	    }
+	  else
+	    return 1;
+	}
       if (fpli.WriteQuotaAvailable > 0)
 	{
 	  paranoid_printf ("fd %d, %s, write: size %u, avail %u", fd,
-			   fh->get_name (), fpli.OutboundQuota,
+			   fh->get_name (), fpli.InboundQuota,
 			   fpli.WriteQuotaAvailable);
 	  return 1;
 	}
@@ -712,6 +743,13 @@ out:
   h = fh->get_output_handle ();
   if (s->write_selected && dev != FH_PIPER)
     {
+      if (dev == FH_PIPEW && ((fhandler_pipe *) fh)->reader_closed ())
+	{
+	  gotone += s->write_ready = true;
+	  if (s->except_selected)
+	    gotone += s->except_ready = true;
+	  return gotone;
+	}
       gotone += s->write_ready =  pipe_data_available (s->fd, fh, h, true);
       select_printf ("write: %s, gotone %d", fh->get_name (), gotone);
     }
diff --git a/winsup/cygwin/spawn.cc b/winsup/cygwin/spawn.cc
index 0bde0b04d..6b2026776 100644
--- a/winsup/cygwin/spawn.cc
+++ b/winsup/cygwin/spawn.cc
@@ -657,6 +657,17 @@ child_info_spawn::worker (const char *prog_arg, const char *const *argv,
 		ptys->create_invisible_console ();
 		ptys->setup_locale ();
 	      }
+	    else if (cfd->get_dev () == FH_PIPEW)
+	      {
+		fhandler_pipe *pipe = (fhandler_pipe *)(fhandler_base *) cfd;
+		pipe->close_query_handle ();
+		pipe->set_pipe_non_blocking (false);
+	      }
+	    else if (cfd->get_dev () == FH_PIPER)
+	      {
+		fhandler_pipe *pipe = (fhandler_pipe *)(fhandler_base *) cfd;
+		pipe->set_pipe_non_blocking (false);
+	      }
 	}
 
       bool enable_pcon = false;
-- 
2.33.0


[-- Attachment #3: 0002-Cygwin-pipe-fifo-Release-select_sem-semaphore-as-muc.patch --]
[-- Type: application/octet-stream, Size: 5295 bytes --]

From 0b41e75d5e8bfc62f97af85e977c7672393c4194 Mon Sep 17 00:00:00 2001
From: Takashi Yano <takashi.yano@nifty.ne.jp>
Date: Tue, 14 Sep 2021 18:50:49 +0900
Subject: [PATCH 2/2] Cygwin: pipe, fifo: Release select_sem semaphore as much
 as needed.

- Currently, raw_read(), raw_write() and close() release select_sem
  unconditionally even if no waiter for select_sem exists. With this
  patch, only the minimum number of semaphores required is released.
---
 winsup/cygwin/fhandler.h       |  4 ++++
 winsup/cygwin/fhandler_fifo.cc | 28 ++++++++++++++++++++++------
 winsup/cygwin/fhandler_pipe.cc | 28 +++++++++++++++++++++-------
 3 files changed, 47 insertions(+), 13 deletions(-)

diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
index db2325144..9580a698c 100644
--- a/winsup/cygwin/fhandler.h
+++ b/winsup/cygwin/fhandler.h
@@ -1177,6 +1177,7 @@ class fhandler_pipe_fifo: public fhandler_base
  protected:
   size_t pipe_buf_size;
   HANDLE query_hdl;
+  virtual void release_select_sem (const char *) {};
 
  public:
   fhandler_pipe_fifo ();
@@ -1201,6 +1202,7 @@ class fhandler_pipe: public fhandler_pipe_fifo
 private:
   HANDLE read_mtx;
   pid_t popen_pid;
+  void release_select_sem (const char *);
 public:
   fhandler_pipe ();
 
@@ -1444,6 +1446,8 @@ class fhandler_fifo: public fhandler_pipe_fifo
   void shared_fc_handler_updated (bool val)
   { shmem->shared_fc_handler_updated (val); }
 
+  void release_select_sem (const char *);
+
 public:
   fhandler_fifo ();
   ~fhandler_fifo ()
diff --git a/winsup/cygwin/fhandler_fifo.cc b/winsup/cygwin/fhandler_fifo.cc
index 37498f547..489ba528c 100644
--- a/winsup/cygwin/fhandler_fifo.cc
+++ b/winsup/cygwin/fhandler_fifo.cc
@@ -1185,6 +1185,22 @@ fhandler_fifo::take_ownership (DWORD timeout)
   return ret;
 }
 
+void
+fhandler_fifo::release_select_sem (const char *from)
+{
+  LONG n_release;
+  if (reader) /* Number of select() call. */
+    n_release = get_obj_handle_count (select_sem)
+      - get_obj_handle_count (read_ready);
+  else /* Number of select() and reader */
+    n_release = get_obj_handle_count (select_sem)
+      - get_obj_handle_count (get_handle ());
+  debug_printf("%s(%s) release %d", from,
+	       reader ? "reader" : "writer", n_release);
+  if (n_release)
+    ReleaseSemaphore (select_sem, n_release, NULL);
+}
+
 void __reg3
 fhandler_fifo::raw_read (void *in_ptr, size_t& len)
 {
@@ -1372,7 +1388,7 @@ out:
   fifo_client_unlock ();
   reading_unlock ();
   if (select_sem)
-    ReleaseSemaphore (select_sem, get_obj_handle_count (select_sem), NULL);
+    release_select_sem ("raw_read");
 }
 
 int __reg2
@@ -1483,6 +1499,11 @@ fhandler_fifo::cancel_reader_thread ()
 int
 fhandler_fifo::close ()
 {
+  if (select_sem)
+    {
+      release_select_sem ("close");
+      NtClose (select_sem);
+    }
   if (writer)
     {
       nwriters_lock ();
@@ -1574,11 +1595,6 @@ fhandler_fifo::close ()
     NtClose (write_ready);
   if (writer_opening)
     NtClose (writer_opening);
-  if (select_sem)
-    {
-      ReleaseSemaphore (select_sem, get_obj_handle_count (select_sem), NULL);
-      NtClose (select_sem);
-    }
   if (nohandle ())
     return 0;
   else
diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index b2c95b56e..0310b2b68 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -233,6 +233,22 @@ fhandler_pipe::get_proc_fd_name (char *buf)
   return buf;
 }
 
+void
+fhandler_pipe::release_select_sem (const char *from)
+{
+  LONG n_release;
+  if (get_dev () == FH_PIPER) /* Number of select() and writer */
+    n_release = get_obj_handle_count (select_sem)
+      - get_obj_handle_count (read_mtx);
+  else /* Number of select() call */
+    n_release = get_obj_handle_count (select_sem)
+      - get_obj_handle_count (query_hdl);
+  debug_printf("%s(%s) release %d", from,
+	       get_dev () == FH_PIPER ? "PIPER" : "PIPEW", n_release);
+  if (n_release)
+    ReleaseSemaphore (select_sem, n_release, NULL);
+}
+
 void __reg3
 fhandler_pipe::raw_read (void *ptr, size_t& len)
 {
@@ -324,8 +340,7 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
 	  ptr = ((char *) ptr) + nbytes_now;
 	  nbytes += nbytes_now;
 	  if (select_sem && nbytes_now > 0)
-	    ReleaseSemaphore (select_sem,
-			      get_obj_handle_count (select_sem), NULL);
+	    release_select_sem ("raw_read");
 	}
       else
 	{
@@ -500,8 +515,7 @@ fhandler_pipe_fifo::raw_write (const void *ptr, size_t len)
 	  ptr = ((char *) ptr) + nbytes_now;
 	  nbytes += nbytes_now;
 	  if (select_sem && nbytes_now > 0)
-	    ReleaseSemaphore (select_sem,
-			      get_obj_handle_count (select_sem), NULL);
+	    release_select_sem ("raw_write");
 	  /* 0 bytes returned?  EAGAIN.  See above. */
 	  if (NT_SUCCESS (status) && nbytes == 0)
 	    set_errno (EAGAIN);
@@ -595,13 +609,13 @@ fhandler_pipe::dup (fhandler_base *child, int flags)
 int
 fhandler_pipe::close ()
 {
-  if (read_mtx)
-    CloseHandle (read_mtx);
   if (select_sem)
     {
-      ReleaseSemaphore (select_sem, get_obj_handle_count (select_sem), NULL);
+      release_select_sem ("close");
       CloseHandle (select_sem);
     }
+  if (read_mtx)
+    CloseHandle (read_mtx);
   if (query_hdl)
     CloseHandle (query_hdl);
   return fhandler_base::close ();
-- 
2.33.0


^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-14  9:56                                               ` Takashi Yano
@ 2021-09-14 10:19                                                 ` Takashi Yano
  2021-09-14 11:03                                                   ` Corinna Vinschen
  0 siblings, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-14 10:19 UTC (permalink / raw)
  To: cygwin-developers

[-- Attachment #1: Type: text/plain, Size: 797 bytes --]

On Tue, 14 Sep 2021 18:56:23 +0900
Takashi Yano wrote:
> On Tue, 14 Sep 2021 11:03:39 +0200
> Corinna Vinschen wrote:
> > I don't quite follow the argument.  Blocking pipes are using
> > asynchronous IO, so they are in fact not blocking calls on the
> > OS level.  After calling NtWriteFile, the blocking variation
> > will go into the subsequent
> > 
> >   waitret = cygwait (evt, INFINITE, cw_cancel | cw_sig);
> > 
> > So, wouldn't you get the same effect by keeping the pipe in
> > FILE_PIPE_QUEUE_OPERATION mode and just add a timeout to the above
> > cygwait call and handle select_sem in a not yet existing WAIT_TIMEOUT
> > conditional?
> 
> Sounds reasonable. I revised the patches. Do you mean something like
> patch attached?

Revised a bit.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

[-- Attachment #2: 0001-Cygwin-pipe-Use-read-pipe-handle-for-select-on-write.patch --]
[-- Type: application/octet-stream, Size: 11677 bytes --]

From 6fc36b5ec7a265d78e80d449c8864ddee46eda10 Mon Sep 17 00:00:00 2001
From: Takashi Yano <takashi.yano@nifty.ne.jp>
Date: Tue, 14 Sep 2021 19:10:48 +0900
Subject: [PATCH 1/2] Cygwin: pipe: Use read pipe handle for select() on write
 pipe.

- Usually WriteQuotaAvailable retrieved by NtQueryInformationFile()
  on the write side reflects the space available in the inbound buffer
  on the read side. However, if a pipe read is currently pending,
  WriteQuotaAvailable on the write side is decremented by the number
  of bytes the read side is requesting. So it's possible (even likely)
  that WriteQuotaAvailable is 0, even if the inbound buffer on the
  read side is not full. This can lead to a deadlock situation:
  The reader is waiting for data, but select on the writer side
  assumes that no space is available in the read side inbound buffer.

  Currently, to avoid this stuation, read() does not request larger
  block than pipe size - 1. However, this mechanism does not take
  effect if the reader side is non-cygwin app.

  The only reliable information is available on the read side, so
  fetch info from the read side via the pipe-specific query handle
  (query_hdl) introduced.

  If the query_hdl (read handle) is kept in write side, writer can
  not detect closure of read pipe. Therefore, raw_write() counts
  write handle and query_hdl. If they are equal, only the pairs of
  write handle and query_hdl are alive. In this case, raw_write()
  returns EPIPE and raises SIGPIPE.
---
 winsup/cygwin/fhandler.h       | 14 +++++-
 winsup/cygwin/fhandler_pipe.cc | 92 +++++++++++++++++++++++-----------
 winsup/cygwin/select.cc        | 52 ++++++++++++++++---
 winsup/cygwin/spawn.cc         | 11 ++++
 4 files changed, 133 insertions(+), 36 deletions(-)

diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
index 46381c397..db2325144 100644
--- a/winsup/cygwin/fhandler.h
+++ b/winsup/cygwin/fhandler.h
@@ -1176,10 +1176,22 @@ class fhandler_pipe_fifo: public fhandler_base
 {
  protected:
   size_t pipe_buf_size;
+  HANDLE query_hdl;
 
  public:
   fhandler_pipe_fifo ();
 
+  HANDLE get_query_handle () const { return query_hdl; }
+  void close_query_handle ()
+  {
+    if (query_hdl)
+      {
+	CloseHandle (query_hdl);
+	query_hdl = NULL;
+      }
+  }
+  bool reader_closed ();
+
   ssize_t __reg3 raw_write (const void *ptr, size_t len);
 
 };
@@ -1189,7 +1201,6 @@ class fhandler_pipe: public fhandler_pipe_fifo
 private:
   HANDLE read_mtx;
   pid_t popen_pid;
-  void set_pipe_non_blocking (bool nonblocking);
 public:
   fhandler_pipe ();
 
@@ -1237,6 +1248,7 @@ public:
     fh->copy_from (this);
     return fh;
   }
+  void set_pipe_non_blocking (bool nonblocking);
 };
 
 #define CYGWIN_FIFO_PIPE_NAME_LEN     47
diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index da473a1dc..b7d4cb5cc 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -202,6 +202,8 @@ fhandler_pipe::open_setup (int flags)
       if (!read_mtx)
 	debug_printf ("CreateMutex failed: %E");
     }
+  if (get_dev () == FH_PIPEW && !query_hdl)
+    set_pipe_non_blocking (is_nonblocking ());
 }
 
 off_t
@@ -268,39 +270,22 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
   while (nbytes < len)
     {
       ULONG_PTR nbytes_now = 0;
-      size_t left = len - nbytes;
-      ULONG len1 = (ULONG) left;
+      ULONG len1 = (ULONG) (len - nbytes);
       waitret = WAIT_OBJECT_0;
 
       if (evt)
 	ResetEvent (evt);
-      if (!is_nonblocking ())
+      FILE_PIPE_LOCAL_INFORMATION fpli;
+      status = NtQueryInformationFile (get_handle (), &io,
+				       &fpli, sizeof (fpli),
+				       FilePipeLocalInformation);
+      if (NT_SUCCESS (status))
 	{
-	  FILE_PIPE_LOCAL_INFORMATION fpli;
-
-	  /* If the pipe is empty, don't request more bytes than pipe
-	     buffer size - 1. Pending read lowers WriteQuotaAvailable on
-	     the write side and thus affects select's ability to return
-	     more or less reliable info whether a write succeeds or not. */
-	  ULONG chunk = pipe_buf_size - 1;
-	  status = NtQueryInformationFile (get_handle (), &io,
-					   &fpli, sizeof (fpli),
-					   FilePipeLocalInformation);
-	  if (NT_SUCCESS (status))
-	    {
-	      if (fpli.ReadDataAvailable > 0)
-		chunk = left;
-	      else if (nbytes != 0)
-		break;
-	      else
-		chunk = fpli.InboundQuota - 1;
-	    }
-	  else if (nbytes != 0)
-	    break;
-
-	  if (len1 > chunk)
-	    len1 = chunk;
+	if (fpli.ReadDataAvailable == 0 && nbytes != 0)
+	  break;
 	}
+      else if (nbytes != 0)
+	break;
       status = NtReadFile (get_handle (), evt, NULL, NULL, &io, ptr,
 			   len1, NULL, NULL);
       if (evt && status == STATUS_PENDING)
@@ -385,6 +370,16 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
   len = nbytes;
 }
 
+bool
+fhandler_pipe_fifo::reader_closed ()
+{
+  if (!query_hdl)
+    return false;
+  int n_reader = get_obj_handle_count (query_hdl);
+  int n_writer = get_obj_handle_count (get_handle ());
+  return n_reader == n_writer;
+}
+
 ssize_t __reg3
 fhandler_pipe_fifo::raw_write (const void *ptr, size_t len)
 {
@@ -457,7 +452,19 @@ fhandler_pipe_fifo::raw_write (const void *ptr, size_t len)
 	}
       if (evt && status == STATUS_PENDING)
 	{
-	  waitret = cygwait (evt, INFINITE, cw_cancel | cw_sig);
+	  while (WAIT_TIMEOUT ==
+		 (waitret = cygwait (evt, (DWORD) 0, cw_cancel | cw_sig)))
+	    {
+	      if (reader_closed ())
+		{
+		  CancelIo (get_handle ());
+		  set_errno (EPIPE);
+		  raise (SIGPIPE);
+		  break;
+		}
+	      else
+		cygwait (select_sem, 10);
+	    }
 	  /* If io.Status is STATUS_CANCELLED after CancelIo, IO has actually
 	     been cancelled and io.Information contains the number of bytes
 	     processed so far.
@@ -523,6 +530,8 @@ fhandler_pipe::set_close_on_exec (bool val)
     set_no_inheritance (read_mtx, val);
   if (select_sem)
     set_no_inheritance (select_sem, val);
+  if (query_hdl)
+    set_no_inheritance (query_hdl, val);
 }
 
 void
@@ -532,6 +541,9 @@ fhandler_pipe::fixup_after_fork (HANDLE parent)
     fork_fixup (parent, read_mtx, "read_mtx");
   if (select_sem)
     fork_fixup (parent, select_sem, "select_sem");
+  if (query_hdl)
+    fork_fixup (parent, query_hdl, "query_hdl");
+
   fhandler_base::fixup_after_fork (parent);
 }
 
@@ -562,6 +574,15 @@ fhandler_pipe::dup (fhandler_base *child, int flags)
       ftp->close ();
       res = -1;
     }
+  else if (query_hdl &&
+	   !DuplicateHandle (GetCurrentProcess (), query_hdl,
+			    GetCurrentProcess (), &ftp->query_hdl,
+			    0, !(flags & O_CLOEXEC), DUPLICATE_SAME_ACCESS))
+    {
+      __seterrno ();
+      ftp->close ();
+      res = -1;
+    }
 
   debug_printf ("res %d", res);
   return res;
@@ -577,6 +598,8 @@ fhandler_pipe::close ()
       ReleaseSemaphore (select_sem, get_obj_handle_count (select_sem), NULL);
       CloseHandle (select_sem);
     }
+  if (query_hdl)
+    CloseHandle (query_hdl);
   return fhandler_base::close ();
 }
 
@@ -797,6 +820,19 @@ fhandler_pipe::create (fhandler_pipe *fhs[2], unsigned psize, int mode)
 	DuplicateHandle (GetCurrentProcess (), fhs[0]->select_sem,
 			 GetCurrentProcess (), &fhs[1]->select_sem,
 			 0, sa->bInheritHandle, DUPLICATE_SAME_ACCESS);
+      if (!DuplicateHandle (GetCurrentProcess (), r,
+			    GetCurrentProcess (), &fhs[1]->query_hdl,
+			    FILE_READ_DATA, sa->bInheritHandle, 0))
+	{
+	  CloseHandle (fhs[0]->select_sem);
+	  delete fhs[0];
+	  CloseHandle (r);
+	  CloseHandle (fhs[1]->select_sem);
+	  delete fhs[1];
+	  CloseHandle (w);
+	}
+      else
+	res = 0;
     }
 
   debug_printf ("%R = pipe([%p, %p], %d, %y)", res, fhs[0], fhs[1], psize, mode);
diff --git a/winsup/cygwin/select.cc b/winsup/cygwin/select.cc
index 5e583434c..ac2f3a9e0 100644
--- a/winsup/cygwin/select.cc
+++ b/winsup/cygwin/select.cc
@@ -608,16 +608,47 @@ pipe_data_available (int fd, fhandler_base *fh, HANDLE h, bool writing)
     }
   if (writing)
     {
-      /* WriteQuotaAvailable is decremented by the number of bytes requested
-	 by a blocking reader on the other side of the pipe.  Cygwin readers
-	 are serialized and never request a number of bytes equivalent to the
-	 full buffer size.  So WriteQuotaAvailable is 0 only if either the
-	 read buffer on the other side is really full, or if we have non-Cygwin
-	 readers. */
+      /* If there is anything available in the pipe buffer then signal
+        that.  This means that a pipe could still block since you could
+        be trying to write more to the pipe than is available in the
+        buffer but that is the hazard of select().
+
+        Note that WriteQuotaAvailable is unreliable.
+
+        Usually WriteQuotaAvailable on the write side reflects the space
+        available in the inbound buffer on the read side.  However, if a
+        pipe read is currently pending, WriteQuotaAvailable on the write side
+        is decremented by the number of bytes the read side is requesting.
+        So it's possible (even likely) that WriteQuotaAvailable is 0, even
+        if the inbound buffer on the read side is not full.  This can lead to
+        a deadlock situation: The reader is waiting for data, but select
+        on the writer side assumes that no space is available in the read
+        side inbound buffer.
+
+        Consequentially, the only reliable information is available on the
+        read side, so fetch info from the read side via the pipe-specific
+        query handle.  Use fpli.WriteQuotaAvailable as storage for the actual
+        interesting value, which is the InboundQuote on the write side,
+        decremented by the number of bytes of data in that buffer. */
+      /* Note: Do not use NtQueryInformationFile() for query_hdl because
+	 NtQueryInformationFile() seems to interfere with reading pipes
+	 in non-cygwin apps. Instead, use PeekNamedPipe() here. */
+      if (fh->get_device () == FH_PIPEW)
+	{
+	  HANDLE query_hdl = ((fhandler_pipe *) fh)->get_query_handle ();
+	  if (query_hdl)
+	    {
+	      DWORD nbytes_in_pipe;
+	      PeekNamedPipe (query_hdl, NULL, 0, NULL, &nbytes_in_pipe, NULL);
+	      fpli.WriteQuotaAvailable = fpli.InboundQuota - nbytes_in_pipe;
+	    }
+	  else
+	    return 1;
+	}
       if (fpli.WriteQuotaAvailable > 0)
 	{
 	  paranoid_printf ("fd %d, %s, write: size %u, avail %u", fd,
-			   fh->get_name (), fpli.OutboundQuota,
+			   fh->get_name (), fpli.InboundQuota,
 			   fpli.WriteQuotaAvailable);
 	  return 1;
 	}
@@ -712,6 +743,13 @@ out:
   h = fh->get_output_handle ();
   if (s->write_selected && dev != FH_PIPER)
     {
+      if (dev == FH_PIPEW && ((fhandler_pipe *) fh)->reader_closed ())
+	{
+	  gotone += s->write_ready = true;
+	  if (s->except_selected)
+	    gotone += s->except_ready = true;
+	  return gotone;
+	}
       gotone += s->write_ready =  pipe_data_available (s->fd, fh, h, true);
       select_printf ("write: %s, gotone %d", fh->get_name (), gotone);
     }
diff --git a/winsup/cygwin/spawn.cc b/winsup/cygwin/spawn.cc
index 0bde0b04d..6b2026776 100644
--- a/winsup/cygwin/spawn.cc
+++ b/winsup/cygwin/spawn.cc
@@ -657,6 +657,17 @@ child_info_spawn::worker (const char *prog_arg, const char *const *argv,
 		ptys->create_invisible_console ();
 		ptys->setup_locale ();
 	      }
+	    else if (cfd->get_dev () == FH_PIPEW)
+	      {
+		fhandler_pipe *pipe = (fhandler_pipe *)(fhandler_base *) cfd;
+		pipe->close_query_handle ();
+		pipe->set_pipe_non_blocking (false);
+	      }
+	    else if (cfd->get_dev () == FH_PIPER)
+	      {
+		fhandler_pipe *pipe = (fhandler_pipe *)(fhandler_base *) cfd;
+		pipe->set_pipe_non_blocking (false);
+	      }
 	}
 
       bool enable_pcon = false;
-- 
2.33.0


[-- Attachment #3: 0002-Cygwin-pipe-fifo-Release-select_sem-semaphore-as-muc.patch --]
[-- Type: application/octet-stream, Size: 5295 bytes --]

From b8efd2b36cb7992b1fb5ad2fdc43370a2df1cd53 Mon Sep 17 00:00:00 2001
From: Takashi Yano <takashi.yano@nifty.ne.jp>
Date: Tue, 14 Sep 2021 18:50:49 +0900
Subject: [PATCH 2/2] Cygwin: pipe, fifo: Release select_sem semaphore as much
 as needed.

- Currently, raw_read(), raw_write() and close() release select_sem
  unconditionally even if no waiter for select_sem exists. With this
  patch, only the minimum number of semaphores required is released.
---
 winsup/cygwin/fhandler.h       |  4 ++++
 winsup/cygwin/fhandler_fifo.cc | 28 ++++++++++++++++++++++------
 winsup/cygwin/fhandler_pipe.cc | 28 +++++++++++++++++++++-------
 3 files changed, 47 insertions(+), 13 deletions(-)

diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
index db2325144..9580a698c 100644
--- a/winsup/cygwin/fhandler.h
+++ b/winsup/cygwin/fhandler.h
@@ -1177,6 +1177,7 @@ class fhandler_pipe_fifo: public fhandler_base
  protected:
   size_t pipe_buf_size;
   HANDLE query_hdl;
+  virtual void release_select_sem (const char *) {};
 
  public:
   fhandler_pipe_fifo ();
@@ -1201,6 +1202,7 @@ class fhandler_pipe: public fhandler_pipe_fifo
 private:
   HANDLE read_mtx;
   pid_t popen_pid;
+  void release_select_sem (const char *);
 public:
   fhandler_pipe ();
 
@@ -1444,6 +1446,8 @@ class fhandler_fifo: public fhandler_pipe_fifo
   void shared_fc_handler_updated (bool val)
   { shmem->shared_fc_handler_updated (val); }
 
+  void release_select_sem (const char *);
+
 public:
   fhandler_fifo ();
   ~fhandler_fifo ()
diff --git a/winsup/cygwin/fhandler_fifo.cc b/winsup/cygwin/fhandler_fifo.cc
index 37498f547..489ba528c 100644
--- a/winsup/cygwin/fhandler_fifo.cc
+++ b/winsup/cygwin/fhandler_fifo.cc
@@ -1185,6 +1185,22 @@ fhandler_fifo::take_ownership (DWORD timeout)
   return ret;
 }
 
+void
+fhandler_fifo::release_select_sem (const char *from)
+{
+  LONG n_release;
+  if (reader) /* Number of select() call. */
+    n_release = get_obj_handle_count (select_sem)
+      - get_obj_handle_count (read_ready);
+  else /* Number of select() and reader */
+    n_release = get_obj_handle_count (select_sem)
+      - get_obj_handle_count (get_handle ());
+  debug_printf("%s(%s) release %d", from,
+	       reader ? "reader" : "writer", n_release);
+  if (n_release)
+    ReleaseSemaphore (select_sem, n_release, NULL);
+}
+
 void __reg3
 fhandler_fifo::raw_read (void *in_ptr, size_t& len)
 {
@@ -1372,7 +1388,7 @@ out:
   fifo_client_unlock ();
   reading_unlock ();
   if (select_sem)
-    ReleaseSemaphore (select_sem, get_obj_handle_count (select_sem), NULL);
+    release_select_sem ("raw_read");
 }
 
 int __reg2
@@ -1483,6 +1499,11 @@ fhandler_fifo::cancel_reader_thread ()
 int
 fhandler_fifo::close ()
 {
+  if (select_sem)
+    {
+      release_select_sem ("close");
+      NtClose (select_sem);
+    }
   if (writer)
     {
       nwriters_lock ();
@@ -1574,11 +1595,6 @@ fhandler_fifo::close ()
     NtClose (write_ready);
   if (writer_opening)
     NtClose (writer_opening);
-  if (select_sem)
-    {
-      ReleaseSemaphore (select_sem, get_obj_handle_count (select_sem), NULL);
-      NtClose (select_sem);
-    }
   if (nohandle ())
     return 0;
   else
diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index b7d4cb5cc..7e5ab328c 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -233,6 +233,22 @@ fhandler_pipe::get_proc_fd_name (char *buf)
   return buf;
 }
 
+void
+fhandler_pipe::release_select_sem (const char *from)
+{
+  LONG n_release;
+  if (get_dev () == FH_PIPER) /* Number of select() and writer */
+    n_release = get_obj_handle_count (select_sem)
+      - get_obj_handle_count (read_mtx);
+  else /* Number of select() call */
+    n_release = get_obj_handle_count (select_sem)
+      - get_obj_handle_count (query_hdl);
+  debug_printf("%s(%s) release %d", from,
+	       get_dev () == FH_PIPER ? "PIPER" : "PIPEW", n_release);
+  if (n_release)
+    ReleaseSemaphore (select_sem, n_release, NULL);
+}
+
 void __reg3
 fhandler_pipe::raw_read (void *ptr, size_t& len)
 {
@@ -324,8 +340,7 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
 	  ptr = ((char *) ptr) + nbytes_now;
 	  nbytes += nbytes_now;
 	  if (select_sem && nbytes_now > 0)
-	    ReleaseSemaphore (select_sem,
-			      get_obj_handle_count (select_sem), NULL);
+	    release_select_sem ("raw_read");
 	}
       else
 	{
@@ -496,8 +511,7 @@ fhandler_pipe_fifo::raw_write (const void *ptr, size_t len)
 	  ptr = ((char *) ptr) + nbytes_now;
 	  nbytes += nbytes_now;
 	  if (select_sem && nbytes_now > 0)
-	    ReleaseSemaphore (select_sem,
-			      get_obj_handle_count (select_sem), NULL);
+	    release_select_sem ("raw_write");
 	  /* 0 bytes returned?  EAGAIN.  See above. */
 	  if (NT_SUCCESS (status) && nbytes == 0)
 	    set_errno (EAGAIN);
@@ -591,13 +605,13 @@ fhandler_pipe::dup (fhandler_base *child, int flags)
 int
 fhandler_pipe::close ()
 {
-  if (read_mtx)
-    CloseHandle (read_mtx);
   if (select_sem)
     {
-      ReleaseSemaphore (select_sem, get_obj_handle_count (select_sem), NULL);
+      release_select_sem ("close");
       CloseHandle (select_sem);
     }
+  if (read_mtx)
+    CloseHandle (read_mtx);
   if (query_hdl)
     CloseHandle (query_hdl);
   return fhandler_base::close ();
-- 
2.33.0


^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-14 10:19                                                 ` Takashi Yano
@ 2021-09-14 11:03                                                   ` Corinna Vinschen
  2021-09-14 12:05                                                     ` Takashi Yano
  0 siblings, 1 reply; 250+ messages in thread
From: Corinna Vinschen @ 2021-09-14 11:03 UTC (permalink / raw)
  To: cygwin-developers

On Sep 14 19:19, Takashi Yano wrote:
> On Tue, 14 Sep 2021 18:56:23 +0900
> Takashi Yano wrote:
> > On Tue, 14 Sep 2021 11:03:39 +0200
> > Corinna Vinschen wrote:
> > > I don't quite follow the argument.  Blocking pipes are using
> > > asynchronous IO, so they are in fact not blocking calls on the
> > > OS level.  After calling NtWriteFile, the blocking variation
> > > will go into the subsequent
> > > 
> > >   waitret = cygwait (evt, INFINITE, cw_cancel | cw_sig);
> > > 
> > > So, wouldn't you get the same effect by keeping the pipe in
> > > FILE_PIPE_QUEUE_OPERATION mode and just add a timeout to the above
> > > cygwait call and handle select_sem in a not yet existing WAIT_TIMEOUT
> > > conditional?
> > 
> > Sounds reasonable. I revised the patches. Do you mean something like
> > patch attached?
> 
> Revised a bit.

Pushed.  I added a bit of text to the commit message in terms of
setting pipes to blocking when exec'ing a non-Cygwin tool.


Thanks,
Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-14 11:03                                                   ` Corinna Vinschen
@ 2021-09-14 12:05                                                     ` Takashi Yano
  2021-09-14 14:17                                                       ` Corinna Vinschen
  2021-09-14 22:14                                                       ` Ken Brown
  0 siblings, 2 replies; 250+ messages in thread
From: Takashi Yano @ 2021-09-14 12:05 UTC (permalink / raw)
  To: cygwin-developers

[-- Attachment #1: Type: text/plain, Size: 1205 bytes --]

On Tue, 14 Sep 2021 13:03:55 +0200
Corinna Vinschen wrote:
> On Sep 14 19:19, Takashi Yano wrote:
> > On Tue, 14 Sep 2021 18:56:23 +0900
> > Takashi Yano wrote:
> > > On Tue, 14 Sep 2021 11:03:39 +0200
> > > Corinna Vinschen wrote:
> > > > I don't quite follow the argument.  Blocking pipes are using
> > > > asynchronous IO, so they are in fact not blocking calls on the
> > > > OS level.  After calling NtWriteFile, the blocking variation
> > > > will go into the subsequent
> > > > 
> > > >   waitret = cygwait (evt, INFINITE, cw_cancel | cw_sig);
> > > > 
> > > > So, wouldn't you get the same effect by keeping the pipe in
> > > > FILE_PIPE_QUEUE_OPERATION mode and just add a timeout to the above
> > > > cygwait call and handle select_sem in a not yet existing WAIT_TIMEOUT
> > > > conditional?
> > > 
> > > Sounds reasonable. I revised the patches. Do you mean something like
> > > patch attached?
> > 
> > Revised a bit.
> 
> Pushed.  I added a bit of text to the commit message in terms of
> setting pipes to blocking when exec'ing a non-Cygwin tool.

Thanks. I found the handling of EPIPE and SIGPIPE in raw_write()
was not correct. Patch attached.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

[-- Attachment #2: 0001-Cygwin-pipe-Fix-handling-of-EPIPE-and-SIGPIPE-in-raw.patch --]
[-- Type: application/octet-stream, Size: 1287 bytes --]

From c70d7e9713dfe55f549abb6c0750812676ba4699 Mon Sep 17 00:00:00 2001
From: Takashi Yano <takashi.yano@nifty.ne.jp>
Date: Tue, 14 Sep 2021 20:59:08 +0900
Subject: [PATCH] Cygwin: pipe: Fix handling of EPIPE and SIGPIPE in
 raw_write().

---
 winsup/cygwin/fhandler_pipe.cc | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index 7e5ab328c..c2f4353f6 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -407,6 +407,13 @@ fhandler_pipe_fifo::raw_write (const void *ptr, size_t len)
   if (!len)
     return 0;
 
+  if (reader_closed ())
+    {
+      set_errno (EPIPE);
+      raise (SIGPIPE);
+      return -1;
+    }
+
   if (len <= pipe_buf_size)
     chunk = len;
   else if (is_nonblocking ())
@@ -475,7 +482,7 @@ fhandler_pipe_fifo::raw_write (const void *ptr, size_t len)
 		  CancelIo (get_handle ());
 		  set_errno (EPIPE);
 		  raise (SIGPIPE);
-		  break;
+		  goto out;
 		}
 	      else
 		cygwait (select_sem, 10);
@@ -527,6 +534,7 @@ fhandler_pipe_fifo::raw_write (const void *ptr, size_t len)
       if (nbytes_now == 0)
 	break;
     }
+out:
   if (evt)
     CloseHandle (evt);
   if (status == STATUS_THREAD_SIGNALED && nbytes == 0)
-- 
2.33.0


^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-14  8:47                                               ` Corinna Vinschen
@ 2021-09-14 12:38                                                 ` Ken Brown
  2021-09-14 14:15                                                   ` Corinna Vinschen
  0 siblings, 1 reply; 250+ messages in thread
From: Ken Brown @ 2021-09-14 12:38 UTC (permalink / raw)
  To: cygwin-developers

On 9/14/2021 4:47 AM, Corinna Vinschen wrote:
> On Sep 14 17:07, Takashi Yano wrote:
>> On Mon, 13 Sep 2021 22:15:25 +0200
>> Corinna Vinschen wrote:
>>> That should depend on the O_CLOEXEC setting, but identically for
>>> all handles in the fhandler.
>>
>> I found the cause. set_close_on_exec() in fhandler_pipe is missing.
>> set_no_inheritance() calls for all adjunct handles are necessary.
>>
>>> I pushed two more patches to topic/pipe in terms of inheritence,
>>> maybe that gives a clue?
>>
>> I attached two additional patch for this issue.
> 
> Uh oh!  This patch to fhandler_base::dup made me check other fhandlers
> and, yeah, we have more unconditional inheritence ignoring O_CLOEXEC
> (fhandler_tape for instance).  We should fix that at one point, but that
> requires your patch to go to master first.  Let's just keep that in mind
> for now.

I think the change to fhandler_base::dup was unnecessary (technically), because 
dtable::dup_worker takes care of that with

   newfh->set_close_on_exec (!!(flags & O_CLOEXEC));

But the code is certainly more clear after the change, and I agree that it 
should be changed in the other fhandlers also.

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-14 12:38                                                 ` Ken Brown
@ 2021-09-14 14:15                                                   ` Corinna Vinschen
  0 siblings, 0 replies; 250+ messages in thread
From: Corinna Vinschen @ 2021-09-14 14:15 UTC (permalink / raw)
  To: cygwin-developers

On Sep 14 08:38, Ken Brown wrote:
> On 9/14/2021 4:47 AM, Corinna Vinschen wrote:
> > On Sep 14 17:07, Takashi Yano wrote:
> > > On Mon, 13 Sep 2021 22:15:25 +0200
> > > Corinna Vinschen wrote:
> > > > That should depend on the O_CLOEXEC setting, but identically for
> > > > all handles in the fhandler.
> > > 
> > > I found the cause. set_close_on_exec() in fhandler_pipe is missing.
> > > set_no_inheritance() calls for all adjunct handles are necessary.
> > > 
> > > > I pushed two more patches to topic/pipe in terms of inheritence,
> > > > maybe that gives a clue?
> > > 
> > > I attached two additional patch for this issue.
> > 
> > Uh oh!  This patch to fhandler_base::dup made me check other fhandlers
> > and, yeah, we have more unconditional inheritence ignoring O_CLOEXEC
> > (fhandler_tape for instance).  We should fix that at one point, but that
> > requires your patch to go to master first.  Let's just keep that in mind
> > for now.
> 
> I think the change to fhandler_base::dup was unnecessary (technically),
> because dtable::dup_worker takes care of that with
> 
>   newfh->set_close_on_exec (!!(flags & O_CLOEXEC));

Right...  and that reminds me we have to check all fhandlers for
set_close_on_exec being correct, too :}

> But the code is certainly more clear after the change, and I agree that it
> should be changed in the other fhandlers also.

:+1:


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-14 12:05                                                     ` Takashi Yano
@ 2021-09-14 14:17                                                       ` Corinna Vinschen
  2021-09-14 22:14                                                       ` Ken Brown
  1 sibling, 0 replies; 250+ messages in thread
From: Corinna Vinschen @ 2021-09-14 14:17 UTC (permalink / raw)
  To: cygwin-developers

On Sep 14 21:05, Takashi Yano wrote:
> Thanks. I found the handling of EPIPE and SIGPIPE in raw_write()
> was not correct. Patch attached.

Pushed.


Thanks,
Corinna



^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-14 12:05                                                     ` Takashi Yano
  2021-09-14 14:17                                                       ` Corinna Vinschen
@ 2021-09-14 22:14                                                       ` Ken Brown
  2021-09-15  0:21                                                         ` Takashi Yano
  1 sibling, 1 reply; 250+ messages in thread
From: Ken Brown @ 2021-09-14 22:14 UTC (permalink / raw)
  To: cygwin-developers

Hi Takashi,

On 9/14/2021 8:05 AM, Takashi Yano wrote:
> Thanks. I found the handling of EPIPE and SIGPIPE in raw_write()
> was not correct. Patch attached.

After pushing this, Corinna merged the topic/pipe branch into master.  We're 
planning to make a test release of Cygwin 3.3.0 shortly.  Do you have any more 
patches before we do this?

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-14 22:14                                                       ` Ken Brown
@ 2021-09-15  0:21                                                         ` Takashi Yano
  2021-09-15  0:44                                                           ` Takashi Yano
  0 siblings, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-15  0:21 UTC (permalink / raw)
  To: cygwin-developers

[-- Attachment #1: Type: text/plain, Size: 518 bytes --]

Hi Ken,

On Tue, 14 Sep 2021 18:14:29 -0400
Ken Brown wrote:
> On 9/14/2021 8:05 AM, Takashi Yano wrote:
> > Thanks. I found the handling of EPIPE and SIGPIPE in raw_write()
> > was not correct. Patch attached.
> 
> After pushing this, Corinna merged the topic/pipe branch into master.  We're 
> planning to make a test release of Cygwin 3.3.0 shortly.  Do you have any more 
> patches before we do this?

I have one patch attched. This make select() on pipe faster a bit.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

[-- Attachment #2: 0001-Cygwin-pipe-Do-not-call-PeekNamedPipe-if-it-is-not-n.patch --]
[-- Type: application/octet-stream, Size: 1422 bytes --]

From 1825f26399c3f44b012bfcb09ff2f1b5dd610259 Mon Sep 17 00:00:00 2001
From: Takashi Yano <takashi.yano@nifty.ne.jp>
Date: Wed, 15 Sep 2021 08:50:52 +0900
Subject: [PATCH] Cygwin: pipe: Do not call PeekNamedPipe() if it is not
 necessary.

- In pipe_data_available() in select.cc, PeekNamedPipe() call is
  not needed if WriteQuotaAvailable is non-zero because we already
  know the write pipe has a space. Therefore, with this patch,
  PeekNamedPipe() is called only when WriteQuotaAvailable is zero.
  This makes select() on pipe faster a bit.
---
 winsup/cygwin/select.cc | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/winsup/cygwin/select.cc b/winsup/cygwin/select.cc
index ac2f3a9e0..8633e88ac 100644
--- a/winsup/cygwin/select.cc
+++ b/winsup/cygwin/select.cc
@@ -636,14 +636,14 @@ pipe_data_available (int fd, fhandler_base *fh, HANDLE h, bool writing)
       if (fh->get_device () == FH_PIPEW)
 	{
 	  HANDLE query_hdl = ((fhandler_pipe *) fh)->get_query_handle ();
-	  if (query_hdl)
+	  if (!query_hdl)
+	    return 1; /* We cannot know actual write pipe space. */
+	  else if (fpli.WriteQuotaAvailable == 0)
 	    {
 	      DWORD nbytes_in_pipe;
 	      PeekNamedPipe (query_hdl, NULL, 0, NULL, &nbytes_in_pipe, NULL);
 	      fpli.WriteQuotaAvailable = fpli.InboundQuota - nbytes_in_pipe;
 	    }
-	  else
-	    return 1;
 	}
       if (fpli.WriteQuotaAvailable > 0)
 	{
-- 
2.33.0


^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-15  0:21                                                         ` Takashi Yano
@ 2021-09-15  0:44                                                           ` Takashi Yano
  2021-09-15  0:59                                                             ` Takashi Yano
  0 siblings, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-15  0:44 UTC (permalink / raw)
  To: cygwin-developers

[-- Attachment #1: Type: text/plain, Size: 212 bytes --]

On Wed, 15 Sep 2021 09:21:00 +0900
Takashi Yano <takashi.yano@nifty.ne.jp> wrote:
> I have one patch attched. This make select() on pipe faster a bit.

Revised a bit.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

[-- Attachment #2: v2-0001-Cygwin-pipe-Do-not-call-PeekNamedPipe-if-it-is-no.patch --]
[-- Type: application/octet-stream, Size: 1829 bytes --]

From 1b9e36cf8a42bc4f646b3cfe8326de2be26185a2 Mon Sep 17 00:00:00 2001
From: Takashi Yano <takashi.yano@nifty.ne.jp>
Date: Wed, 15 Sep 2021 09:42:46 +0900
Subject: [PATCH v2] Cygwin: pipe: Do not call PeekNamedPipe() if it is not
 necessary.

- In pipe_data_available() in select.cc, PeekNamedPipe() call is
  not needed if WriteQuotaAvailable is non-zero because we already
  know the write pipe has a space. Therefore, with this patch,
  PeekNamedPipe() is called only when WriteQuotaAvailable is zero.
  This makes select() on pipe faster a bit.
---
 winsup/cygwin/select.cc | 15 ++++++---------
 1 file changed, 6 insertions(+), 9 deletions(-)

diff --git a/winsup/cygwin/select.cc b/winsup/cygwin/select.cc
index ac2f3a9e0..584a6fb47 100644
--- a/winsup/cygwin/select.cc
+++ b/winsup/cygwin/select.cc
@@ -633,17 +633,14 @@ pipe_data_available (int fd, fhandler_base *fh, HANDLE h, bool writing)
       /* Note: Do not use NtQueryInformationFile() for query_hdl because
 	 NtQueryInformationFile() seems to interfere with reading pipes
 	 in non-cygwin apps. Instead, use PeekNamedPipe() here. */
-      if (fh->get_device () == FH_PIPEW)
+      if (fh->get_device () == FH_PIPEW && fpli.WriteQuotaAvailable == 0)
 	{
 	  HANDLE query_hdl = ((fhandler_pipe *) fh)->get_query_handle ();
-	  if (query_hdl)
-	    {
-	      DWORD nbytes_in_pipe;
-	      PeekNamedPipe (query_hdl, NULL, 0, NULL, &nbytes_in_pipe, NULL);
-	      fpli.WriteQuotaAvailable = fpli.InboundQuota - nbytes_in_pipe;
-	    }
-	  else
-	    return 1;
+	  if (!query_hdl)
+	    return 1; /* We cannot know actual write pipe space. */
+	  DWORD nbytes_in_pipe;
+	  PeekNamedPipe (query_hdl, NULL, 0, NULL, &nbytes_in_pipe, NULL);
+	  fpli.WriteQuotaAvailable = fpli.InboundQuota - nbytes_in_pipe;
 	}
       if (fpli.WriteQuotaAvailable > 0)
 	{
-- 
2.33.0


^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-15  0:44                                                           ` Takashi Yano
@ 2021-09-15  0:59                                                             ` Takashi Yano
  2021-09-15  9:57                                                               ` Corinna Vinschen
  0 siblings, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-15  0:59 UTC (permalink / raw)
  To: cygwin-developers

[-- Attachment #1: Type: text/plain, Size: 306 bytes --]

On Wed, 15 Sep 2021 09:44:54 +0900
Takashi Yano wrote:
> On Wed, 15 Sep 2021 09:21:00 +0900
> Takashi Yano <takashi.yano@nifty.ne.jp> wrote:
> > I have one patch attched. This make select() on pipe faster a bit.
> 
> Revised a bit.

Add missing error handling.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

[-- Attachment #2: v3-0001-Cygwin-pipe-Do-not-call-PeekNamedPipe-if-it-is-no.patch --]
[-- Type: application/octet-stream, Size: 1833 bytes --]

From ce5c88696013894ea6a0b78cd42859c14532a10f Mon Sep 17 00:00:00 2001
From: Takashi Yano <takashi.yano@nifty.ne.jp>
Date: Wed, 15 Sep 2021 09:55:57 +0900
Subject: [PATCH v3] Cygwin: pipe: Do not call PeekNamedPipe() if it is not
 necessary.

- In pipe_data_available() in select.cc, PeekNamedPipe() call is
  not needed if WriteQuotaAvailable is non-zero because we already
  know the write pipe has a space. Therefore, with this patch,
  PeekNamedPipe() is called only when WriteQuotaAvailable is zero.
  This makes select() on pipe faster a bit.
---
 winsup/cygwin/select.cc | 14 ++++++--------
 1 file changed, 6 insertions(+), 8 deletions(-)

diff --git a/winsup/cygwin/select.cc b/winsup/cygwin/select.cc
index ac2f3a9e0..fd2312298 100644
--- a/winsup/cygwin/select.cc
+++ b/winsup/cygwin/select.cc
@@ -633,17 +633,15 @@ pipe_data_available (int fd, fhandler_base *fh, HANDLE h, bool writing)
       /* Note: Do not use NtQueryInformationFile() for query_hdl because
 	 NtQueryInformationFile() seems to interfere with reading pipes
 	 in non-cygwin apps. Instead, use PeekNamedPipe() here. */
-      if (fh->get_device () == FH_PIPEW)
+      if (fh->get_device () == FH_PIPEW && fpli.WriteQuotaAvailable == 0)
 	{
 	  HANDLE query_hdl = ((fhandler_pipe *) fh)->get_query_handle ();
-	  if (query_hdl)
-	    {
-	      DWORD nbytes_in_pipe;
-	      PeekNamedPipe (query_hdl, NULL, 0, NULL, &nbytes_in_pipe, NULL);
-	      fpli.WriteQuotaAvailable = fpli.InboundQuota - nbytes_in_pipe;
-	    }
-	  else
+	  if (!query_hdl)
+	    return 1; /* We cannot know actual write pipe space. */
+	  DWORD nbytes_in_pipe;
+	  if (!PeekNamedPipe (query_hdl, NULL, 0, NULL, &nbytes_in_pipe, NULL))
 	    return 1;
+	  fpli.WriteQuotaAvailable = fpli.InboundQuota - nbytes_in_pipe;
 	}
       if (fpli.WriteQuotaAvailable > 0)
 	{
-- 
2.33.0


^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-15  0:59                                                             ` Takashi Yano
@ 2021-09-15  9:57                                                               ` Corinna Vinschen
  2021-09-15 10:48                                                                 ` Takashi Yano
  0 siblings, 1 reply; 250+ messages in thread
From: Corinna Vinschen @ 2021-09-15  9:57 UTC (permalink / raw)
  To: cygwin-developers

On Sep 15 09:59, Takashi Yano wrote:
> On Wed, 15 Sep 2021 09:44:54 +0900
> Takashi Yano wrote:
> > On Wed, 15 Sep 2021 09:21:00 +0900
> > Takashi Yano <takashi.yano@nifty.ne.jp> wrote:
> > > I have one patch attched. This make select() on pipe faster a bit.
> > 
> > Revised a bit.
> 
> Add missing error handling.

Patch pushed, but I wonder...

> @@ -633,17 +633,15 @@ pipe_data_available (int fd, fhandler_base *fh, HANDLE h, bool writing)
>        /* Note: Do not use NtQueryInformationFile() for query_hdl because
>  	 NtQueryInformationFile() seems to interfere with reading pipes
>  	 in non-cygwin apps. Instead, use PeekNamedPipe() here. */

...so we're not using NtQueryInformationFile on the query_hdl...

> -      if (fh->get_device () == FH_PIPEW)
> +      if (fh->get_device () == FH_PIPEW && fpli.WriteQuotaAvailable == 0)
>  	{
>  	  HANDLE query_hdl = ((fhandler_pipe *) fh)->get_query_handle ();
> -	  if (query_hdl)
> -	    {
> -	      DWORD nbytes_in_pipe;
> -	      PeekNamedPipe (query_hdl, NULL, 0, NULL, &nbytes_in_pipe, NULL);
> -	      fpli.WriteQuotaAvailable = fpli.InboundQuota - nbytes_in_pipe;
> -	    }
> -	  else
> +	  if (!query_hdl)
> +	    return 1; /* We cannot know actual write pipe space. */
> +	  DWORD nbytes_in_pipe;
> +	  if (!PeekNamedPipe (query_hdl, NULL, 0, NULL, &nbytes_in_pipe, NULL))
>  	    return 1;
> +	  fpli.WriteQuotaAvailable = fpli.InboundQuota - nbytes_in_pipe;
>  	}
>        if (fpli.WriteQuotaAvailable > 0)

...but we're unconditionally using NtQueryInformationFile at the start
of pipe_data_available.  If this is the read side of the pipe, isn't
there a chance that another (non-Cygwin) process is performing a blocking
read and this call will hang?

Btw., this doesn't work as expected:

$ yes | /cygdrive/c/Windows/System32/more

It just hangs until pressing Ctrl-C.  Is that expected currently?


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-15  9:57                                                               ` Corinna Vinschen
@ 2021-09-15 10:48                                                                 ` Takashi Yano
  2021-09-15 10:58                                                                   ` Takashi Yano
  2021-09-15 11:13                                                                   ` Corinna Vinschen
  0 siblings, 2 replies; 250+ messages in thread
From: Takashi Yano @ 2021-09-15 10:48 UTC (permalink / raw)
  To: cygwin-developers

On Wed, 15 Sep 2021 11:57:44 +0200
Corinna Vinschen wrote:
> On Sep 15 09:59, Takashi Yano wrote:
> > @@ -633,17 +633,15 @@ pipe_data_available (int fd, fhandler_base *fh, HANDLE h, bool writing)
> >        /* Note: Do not use NtQueryInformationFile() for query_hdl because
> >  	 NtQueryInformationFile() seems to interfere with reading pipes
> >  	 in non-cygwin apps. Instead, use PeekNamedPipe() here. */
> 
> ...so we're not using NtQueryInformationFile on the query_hdl...
> 
> > -      if (fh->get_device () == FH_PIPEW)
> > +      if (fh->get_device () == FH_PIPEW && fpli.WriteQuotaAvailable == 0)
> >  	{
> >  	  HANDLE query_hdl = ((fhandler_pipe *) fh)->get_query_handle ();
> > -	  if (query_hdl)
> > -	    {
> > -	      DWORD nbytes_in_pipe;
> > -	      PeekNamedPipe (query_hdl, NULL, 0, NULL, &nbytes_in_pipe, NULL);
> > -	      fpli.WriteQuotaAvailable = fpli.InboundQuota - nbytes_in_pipe;
> > -	    }
> > -	  else
> > +	  if (!query_hdl)
> > +	    return 1; /* We cannot know actual write pipe space. */
> > +	  DWORD nbytes_in_pipe;
> > +	  if (!PeekNamedPipe (query_hdl, NULL, 0, NULL, &nbytes_in_pipe, NULL))
> >  	    return 1;
> > +	  fpli.WriteQuotaAvailable = fpli.InboundQuota - nbytes_in_pipe;
> >  	}
> >        if (fpli.WriteQuotaAvailable > 0)
> 
> ...but we're unconditionally using NtQueryInformationFile at the start
> of pipe_data_available.  If this is the read side of the pipe, isn't
> there a chance that another (non-Cygwin) process is performing a blocking
> read and this call will hang?

I will check that.

> Btw., this doesn't work as expected:
> 
> $ yes | /cygdrive/c/Windows/System32/more
> 
> It just hangs until pressing Ctrl-C.  Is that expected currently?

Indeed. However, this also does not work in cygwin 3.2.0,

while
yes AAAAAA | /cygdrive/c/Windows/System32/more
works...

I am not sure why so far.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-15 10:48                                                                 ` Takashi Yano
@ 2021-09-15 10:58                                                                   ` Takashi Yano
  2021-09-15 11:34                                                                     ` Corinna Vinschen
  2021-09-15 11:13                                                                   ` Corinna Vinschen
  1 sibling, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-15 10:58 UTC (permalink / raw)
  To: cygwin-developers

On Wed, 15 Sep 2021 19:48:58 +0900
Takashi Yano wrote:
> On Wed, 15 Sep 2021 11:57:44 +0200
> Corinna Vinschen wrote:
> > Btw., this doesn't work as expected:
> > 
> > $ yes | /cygdrive/c/Windows/System32/more
> > 
> > It just hangs until pressing Ctrl-C.  Is that expected currently?
> 
> Indeed. However, this also does not work in cygwin 3.2.0,
> 
> while
> yes AAAAAA | /cygdrive/c/Windows/System32/more
> works...
> 
> I am not sure why so far.

\cygwin64\bin\yes | more
in command prompt also does not work.

Maybe newline code problem?

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-15 10:48                                                                 ` Takashi Yano
  2021-09-15 10:58                                                                   ` Takashi Yano
@ 2021-09-15 11:13                                                                   ` Corinna Vinschen
  2021-09-15 11:41                                                                     ` Ken Brown
  2021-09-15 11:54                                                                     ` Takashi Yano
  1 sibling, 2 replies; 250+ messages in thread
From: Corinna Vinschen @ 2021-09-15 11:13 UTC (permalink / raw)
  To: cygwin-developers

On Sep 15 19:48, Takashi Yano wrote:
> On Wed, 15 Sep 2021 11:57:44 +0200
> Corinna Vinschen wrote:
> > ...but we're unconditionally using NtQueryInformationFile at the start
> > of pipe_data_available.  If this is the read side of the pipe, isn't
> > there a chance that another (non-Cygwin) process is performing a blocking
> > read and this call will hang?
> 
> I will check that.

Something like that comes to mind:

diff --git a/winsup/cygwin/select.cc b/winsup/cygwin/select.cc
index fd23122988c3..cf9ad87538f2 100644
--- a/winsup/cygwin/select.cc
+++ b/winsup/cygwin/select.cc
@@ -594,18 +594,6 @@ pipe_data_available (int fd, fhandler_base *fh, HANDLE h, bool writing)
   if (fh->has_ongoing_io ())
     return 0;
 
-  status = NtQueryInformationFile (h, &iosb, &fpli, sizeof (fpli),
-				   FilePipeLocalInformation);
-  if (!NT_SUCCESS (status))
-    {
-      /* If NtQueryInformationFile fails, optimistically assume the
-	 pipe is writable.  This could happen if we somehow
-	 inherit a pipe that doesn't permit FILE_READ_ATTRIBUTES
-	 access on the write end.  */
-      select_printf ("fd %d, %s, NtQueryInformationFile failed, status %y",
-		     fd, fh->get_name (), status);
-      return writing ? 1 : -1;
-    }
   if (writing)
     {
       /* If there is anything available in the pipe buffer then signal
@@ -633,6 +621,18 @@ pipe_data_available (int fd, fhandler_base *fh, HANDLE h, bool writing)
       /* Note: Do not use NtQueryInformationFile() for query_hdl because
 	 NtQueryInformationFile() seems to interfere with reading pipes
 	 in non-cygwin apps. Instead, use PeekNamedPipe() here. */
+      status = NtQueryInformationFile (h, &iosb, &fpli, sizeof (fpli),
+				       FilePipeLocalInformation);
+      if (!NT_SUCCESS (status))
+	{
+	  /* If NtQueryInformationFile fails, optimistically assume the
+	     pipe is writable.  This could happen if we somehow
+	     inherit a pipe that doesn't permit FILE_READ_ATTRIBUTES
+	     access on the write end.  */
+	  select_printf ("fd %d, %s, NtQueryInformationFile failed, status %y",
+			 fd, fh->get_name (), status);
+	  return 1;
+	}
       if (fh->get_device () == FH_PIPEW && fpli.WriteQuotaAvailable == 0)
 	{
 	  HANDLE query_hdl = ((fhandler_pipe *) fh)->get_query_handle ();
@@ -652,11 +652,15 @@ pipe_data_available (int fd, fhandler_base *fh, HANDLE h, bool writing)
 	}
       /* TODO: Buffer really full or non-Cygwin reader? */
     }
-  else if (fpli.ReadDataAvailable)
+  else
     {
-      paranoid_printf ("fd %d, %s, read avail %u", fd, fh->get_name (),
-		       fpli.ReadDataAvailable);
-      return 1;
+      DWORD nbytes_in_pipe;
+      if (PeekNamedPipe (h, NULL, 0, NULL, &nbytes_in_pipe, NULL))
+	{
+	  paranoid_printf ("fd %d, %s, read avail %u", fd, fh->get_name (),
+			   nbytes_in_pipe);
+	  return nbytes_in_pipe > 0;
+	}
     }
   if (fpli.NamedPipeState & FILE_PIPE_CLOSING_STATE)
     return -1;


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-15 10:58                                                                   ` Takashi Yano
@ 2021-09-15 11:34                                                                     ` Corinna Vinschen
  2021-09-15 11:40                                                                       ` Corinna Vinschen
  0 siblings, 1 reply; 250+ messages in thread
From: Corinna Vinschen @ 2021-09-15 11:34 UTC (permalink / raw)
  To: cygwin-developers

On Sep 15 19:58, Takashi Yano wrote:
> On Wed, 15 Sep 2021 19:48:58 +0900
> Takashi Yano wrote:
> > On Wed, 15 Sep 2021 11:57:44 +0200
> > Corinna Vinschen wrote:
> > > Btw., this doesn't work as expected:
> > > 
> > > $ yes | /cygdrive/c/Windows/System32/more
> > > 
> > > It just hangs until pressing Ctrl-C.  Is that expected currently?
> > 
> > Indeed. However, this also does not work in cygwin 3.2.0,
> > 
> > while
> > yes AAAAAA | /cygdrive/c/Windows/System32/more
> > works...
> > 
> > I am not sure why so far.
> 
> \cygwin64\bin\yes | more
> in command prompt also does not work.
> 
> Maybe newline code problem?

It's weird.   `yes AA | ...' works but `yes A | ...' doesn't.
Maybe you're right and the missing CR confuses more...


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-15 11:34                                                                     ` Corinna Vinschen
@ 2021-09-15 11:40                                                                       ` Corinna Vinschen
  0 siblings, 0 replies; 250+ messages in thread
From: Corinna Vinschen @ 2021-09-15 11:40 UTC (permalink / raw)
  To: cygwin-developers

On Sep 15 13:34, Corinna Vinschen wrote:
> On Sep 15 19:58, Takashi Yano wrote:
> > On Wed, 15 Sep 2021 19:48:58 +0900
> > Takashi Yano wrote:
> > > On Wed, 15 Sep 2021 11:57:44 +0200
> > > Corinna Vinschen wrote:
> > > > Btw., this doesn't work as expected:
> > > > 
> > > > $ yes | /cygdrive/c/Windows/System32/more
> > > > 
> > > > It just hangs until pressing Ctrl-C.  Is that expected currently?
> > > 
> > > Indeed. However, this also does not work in cygwin 3.2.0,
> > > 
> > > while
> > > yes AAAAAA | /cygdrive/c/Windows/System32/more
> > > works...
> > > 
> > > I am not sure why so far.
> > 
> > \cygwin64\bin\yes | more
> > in command prompt also does not work.
> > 
> > Maybe newline code problem?
> 
> It's weird.   `yes AA | ...' works but `yes A | ...' doesn't.
> Maybe you're right and the missing CR confuses more...

Yes, that's the problem, apparently.  I created a sophisticated
testcase:

$ cat > yes.c <<EOF
include <stdio.h>

int
main ()
{
  while (1)
    printf ("y\010\n");
}
EOF
$ gcc -g -o yes yes.c
$ ./yes | /cygdrive/c/Windows/System32/more
y
y
y
y
--  MORE  --


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-15 11:13                                                                   ` Corinna Vinschen
@ 2021-09-15 11:41                                                                     ` Ken Brown
  2021-09-15 11:49                                                                       ` Corinna Vinschen
  2021-09-15 11:54                                                                     ` Takashi Yano
  1 sibling, 1 reply; 250+ messages in thread
From: Ken Brown @ 2021-09-15 11:41 UTC (permalink / raw)
  To: cygwin-developers

On 9/15/2021 7:13 AM, Corinna Vinschen wrote:
> On Sep 15 19:48, Takashi Yano wrote:
>> On Wed, 15 Sep 2021 11:57:44 +0200
>> Corinna Vinschen wrote:
>>> ...but we're unconditionally using NtQueryInformationFile at the start
>>> of pipe_data_available.  If this is the read side of the pipe, isn't
>>> there a chance that another (non-Cygwin) process is performing a blocking
>>> read and this call will hang?
>>
>> I will check that.
> 
> Something like that comes to mind:
> 
> diff --git a/winsup/cygwin/select.cc b/winsup/cygwin/select.cc
> index fd23122988c3..cf9ad87538f2 100644
> --- a/winsup/cygwin/select.cc
> +++ b/winsup/cygwin/select.cc
> @@ -594,18 +594,6 @@ pipe_data_available (int fd, fhandler_base *fh, HANDLE h, bool writing)
>     if (fh->has_ongoing_io ())
>       return 0;
>   
> -  status = NtQueryInformationFile (h, &iosb, &fpli, sizeof (fpli),
> -				   FilePipeLocalInformation);
> -  if (!NT_SUCCESS (status))
> -    {
> -      /* If NtQueryInformationFile fails, optimistically assume the
> -	 pipe is writable.  This could happen if we somehow
> -	 inherit a pipe that doesn't permit FILE_READ_ATTRIBUTES
> -	 access on the write end.  */
> -      select_printf ("fd %d, %s, NtQueryInformationFile failed, status %y",
> -		     fd, fh->get_name (), status);
> -      return writing ? 1 : -1;
> -    }
>     if (writing)
>       {
>         /* If there is anything available in the pipe buffer then signal
> @@ -633,6 +621,18 @@ pipe_data_available (int fd, fhandler_base *fh, HANDLE h, bool writing)
>         /* Note: Do not use NtQueryInformationFile() for query_hdl because
>   	 NtQueryInformationFile() seems to interfere with reading pipes
>   	 in non-cygwin apps. Instead, use PeekNamedPipe() here. */
> +      status = NtQueryInformationFile (h, &iosb, &fpli, sizeof (fpli),
> +				       FilePipeLocalInformation);
> +      if (!NT_SUCCESS (status))
> +	{
> +	  /* If NtQueryInformationFile fails, optimistically assume the
> +	     pipe is writable.  This could happen if we somehow
> +	     inherit a pipe that doesn't permit FILE_READ_ATTRIBUTES
> +	     access on the write end.  */
> +	  select_printf ("fd %d, %s, NtQueryInformationFile failed, status %y",
> +			 fd, fh->get_name (), status);
> +	  return 1;
> +	}
>         if (fh->get_device () == FH_PIPEW && fpli.WriteQuotaAvailable == 0)
>   	{
>   	  HANDLE query_hdl = ((fhandler_pipe *) fh)->get_query_handle ();
> @@ -652,11 +652,15 @@ pipe_data_available (int fd, fhandler_base *fh, HANDLE h, bool writing)
>   	}
>         /* TODO: Buffer really full or non-Cygwin reader? */
>       }
> -  else if (fpli.ReadDataAvailable)
> +  else
>       {
> -      paranoid_printf ("fd %d, %s, read avail %u", fd, fh->get_name (),
> -		       fpli.ReadDataAvailable);
> -      return 1;
> +      DWORD nbytes_in_pipe;
> +      if (PeekNamedPipe (h, NULL, 0, NULL, &nbytes_in_pipe, NULL))
> +	{
> +	  paranoid_printf ("fd %d, %s, read avail %u", fd, fh->get_name (),
> +			   nbytes_in_pipe);
> +	  return nbytes_in_pipe > 0;
> +	}
>       }
>     if (fpli.NamedPipeState & FILE_PIPE_CLOSING_STATE)
>       return -1;

Maybe we want to call NtQueryInformationFile in all cases except FH_PIPER, 
keeping in mind that there are callers of pipe_data_available other than pipes. 
  That way we can still test for FILE_PIPE_CLOSING_STATE at the end in many cases.

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-15 11:41                                                                     ` Ken Brown
@ 2021-09-15 11:49                                                                       ` Corinna Vinschen
  0 siblings, 0 replies; 250+ messages in thread
From: Corinna Vinschen @ 2021-09-15 11:49 UTC (permalink / raw)
  To: cygwin-developers

On Sep 15 07:41, Ken Brown wrote:
> On 9/15/2021 7:13 AM, Corinna Vinschen wrote:
> > On Sep 15 19:48, Takashi Yano wrote:
> > > On Wed, 15 Sep 2021 11:57:44 +0200
> > > Corinna Vinschen wrote:
> > > > ...but we're unconditionally using NtQueryInformationFile at the start
> > > > of pipe_data_available.  If this is the read side of the pipe, isn't
> > > > there a chance that another (non-Cygwin) process is performing a blocking
> > > > read and this call will hang?
> > > 
> > > I will check that.
> > 
> > Something like that comes to mind:
> > [...]
> >     if (fpli.NamedPipeState & FILE_PIPE_CLOSING_STATE)
> >       return -1;
> 
> Maybe we want to call NtQueryInformationFile in all cases except FH_PIPER,
> keeping in mind that there are callers of pipe_data_available other than
> pipes.  That way we can still test for FILE_PIPE_CLOSING_STATE at the end in
> many cases.

Ouch, right.  I totally missed this.


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-15 11:13                                                                   ` Corinna Vinschen
  2021-09-15 11:41                                                                     ` Ken Brown
@ 2021-09-15 11:54                                                                     ` Takashi Yano
  2021-09-15 12:20                                                                       ` Corinna Vinschen
  1 sibling, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-15 11:54 UTC (permalink / raw)
  To: cygwin-developers

On Wed, 15 Sep 2021 13:13:25 +0200
Corinna Vinschen wrote:
> @@ -652,11 +652,15 @@ pipe_data_available (int fd, fhandler_base *fh, HANDLE h, bool writing)
>  	}
>        /* TODO: Buffer really full or non-Cygwin reader? */
>      }
> -  else if (fpli.ReadDataAvailable)
> +  else
>      {
> -      paranoid_printf ("fd %d, %s, read avail %u", fd, fh->get_name (),
> -		       fpli.ReadDataAvailable);
> -      return 1;
> +      DWORD nbytes_in_pipe;
> +      if (PeekNamedPipe (h, NULL, 0, NULL, &nbytes_in_pipe, NULL))
> +	{
> +	  paranoid_printf ("fd %d, %s, read avail %u", fd, fh->get_name (),
> +			   nbytes_in_pipe);
> +	  return nbytes_in_pipe > 0;
> +	}
>      }
>    if (fpli.NamedPipeState & FILE_PIPE_CLOSING_STATE)
>      return -1;

This shoudl be:

@@ -652,11 +652,17 @@ pipe_data_available (int fd, fhandler_base *fh, HANDLE h, bool writing)
 	}
       /* TODO: Buffer really full or non-Cygwin reader? */
     }
-  else if (fpli.ReadDataAvailable)
+  else
     {
-      paranoid_printf ("fd %d, %s, read avail %u", fd, fh->get_name (),
-		       fpli.ReadDataAvailable);
-      return 1;
+      DWORD nbytes_in_pipe;
+      if (PeekNamedPipe (h, NULL, 0, NULL, &nbytes_in_pipe, NULL))
+	{
+	  paranoid_printf ("fd %d, %s, read avail %u", fd, fh->get_name (),
+			   nbytes_in_pipe);
+	  return nbytes_in_pipe > 0;
+	}
+      else if (GetLastError () == ERROR_BROKEN_PIPE)
+	return -1;
     }
   if (fpli.NamedPipeState & FILE_PIPE_CLOSING_STATE)
     return -1;

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-15 11:54                                                                     ` Takashi Yano
@ 2021-09-15 12:20                                                                       ` Corinna Vinschen
  2021-09-15 13:04                                                                         ` Takashi Yano
  0 siblings, 1 reply; 250+ messages in thread
From: Corinna Vinschen @ 2021-09-15 12:20 UTC (permalink / raw)
  To: cygwin-developers

On Sep 15 20:54, Takashi Yano wrote:
> This shoudl be:
> 
> @@ -652,11 +652,17 @@ pipe_data_available (int fd, fhandler_base *fh, HANDLE h, bool writing)
>  	}
>        /* TODO: Buffer really full or non-Cygwin reader? */
>      }
> -  else if (fpli.ReadDataAvailable)
> +  else
>      {
> -      paranoid_printf ("fd %d, %s, read avail %u", fd, fh->get_name (),
> -		       fpli.ReadDataAvailable);
> -      return 1;
> +      DWORD nbytes_in_pipe;
> +      if (PeekNamedPipe (h, NULL, 0, NULL, &nbytes_in_pipe, NULL))
> +	{
> +	  paranoid_printf ("fd %d, %s, read avail %u", fd, fh->get_name (),
> +			   nbytes_in_pipe);
> +	  return nbytes_in_pipe > 0;
> +	}
> +      else if (GetLastError () == ERROR_BROKEN_PIPE)
> +	return -1;
>      }
>    if (fpli.NamedPipeState & FILE_PIPE_CLOSING_STATE)
>      return -1;

After Ken's comment, I was going to suggest this one:

From 34b14470406cb9551f98707bf63175811a506523 Mon Sep 17 00:00:00 2001
From: Corinna Vinschen
Date: Wed, 15 Sep 2021 14:17:59 +0200
Subject: [PATCH] Cygwin: pipes: don't call NtQueryInformationFile on read side
 of pipes

NtQueryInformationFile hangs if it's called on the read side handle of
a pipe while another thread or process is performing a blocking read.

Avoid select potentially hanging by calling NtQueryInformationFile
only on the write side of the pipe and using PeekNamedPipe otherwise.

---
 winsup/cygwin/select.cc | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/winsup/cygwin/select.cc b/winsup/cygwin/select.cc
index a09d8a34da2e..566cf66d653b 100644
--- a/winsup/cygwin/select.cc
+++ b/winsup/cygwin/select.cc
@@ -587,6 +587,14 @@ no_verify (select_record *, fd_set *, fd_set *, fd_set *)
 static int
 pipe_data_available (int fd, fhandler_base *fh, HANDLE h, bool writing)
 {
+  if (fh->get_device () == FH_PIPER)
+    {
+      DWORD nbytes_in_pipe;
+      if (!writing && PeekNamedPipe (h, NULL, 0, NULL, &nbytes_in_pipe, NULL))
+	return nbytes_in_pipe > 0;
+      return -1;
+    }
+
   IO_STATUS_BLOCK iosb = {{0}, 0};
   FILE_PIPE_LOCAL_INFORMATION fpli = {0};
   NTSTATUS status;
-- 
2.31.1


^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-15 12:20                                                                       ` Corinna Vinschen
@ 2021-09-15 13:04                                                                         ` Takashi Yano
  2021-09-15 13:42                                                                           ` Corinna Vinschen
  0 siblings, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-15 13:04 UTC (permalink / raw)
  To: cygwin-developers

On Wed, 15 Sep 2021 14:20:48 +0200
Corinna Vinschen wrote:
> After Ken's comment, I was going to suggest this one:
> 
> From 34b14470406cb9551f98707bf63175811a506523 Mon Sep 17 00:00:00 2001
> From: Corinna Vinschen
> Date: Wed, 15 Sep 2021 14:17:59 +0200
> Subject: [PATCH] Cygwin: pipes: don't call NtQueryInformationFile on read side
>  of pipes
> 
> NtQueryInformationFile hangs if it's called on the read side handle of
> a pipe while another thread or process is performing a blocking read.
> 
> Avoid select potentially hanging by calling NtQueryInformationFile
> only on the write side of the pipe and using PeekNamedPipe otherwise.
> 
> ---
>  winsup/cygwin/select.cc | 8 ++++++++
>  1 file changed, 8 insertions(+)
> 
> diff --git a/winsup/cygwin/select.cc b/winsup/cygwin/select.cc
> index a09d8a34da2e..566cf66d653b 100644
> --- a/winsup/cygwin/select.cc
> +++ b/winsup/cygwin/select.cc
> @@ -587,6 +587,14 @@ no_verify (select_record *, fd_set *, fd_set *, fd_set *)
>  static int
>  pipe_data_available (int fd, fhandler_base *fh, HANDLE h, bool writing)
>  {
> +  if (fh->get_device () == FH_PIPER)
> +    {
> +      DWORD nbytes_in_pipe;
> +      if (!writing && PeekNamedPipe (h, NULL, 0, NULL, &nbytes_in_pipe, NULL))
> +	return nbytes_in_pipe > 0;
> +      return -1;
> +    }
> +
>    IO_STATUS_BLOCK iosb = {{0}, 0};
>    FILE_PIPE_LOCAL_INFORMATION fpli = {0};
>    NTSTATUS status;
> -- 
> 2.31.1
> 

LGTM.
I also confirmed a several test cases are all passed with this patch.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-15 13:04                                                                         ` Takashi Yano
@ 2021-09-15 13:42                                                                           ` Corinna Vinschen
  2021-09-15 16:22                                                                             ` Ken Brown
  0 siblings, 1 reply; 250+ messages in thread
From: Corinna Vinschen @ 2021-09-15 13:42 UTC (permalink / raw)
  To: cygwin-developers

On Sep 15 22:04, Takashi Yano wrote:
> On Wed, 15 Sep 2021 14:20:48 +0200
> Corinna Vinschen wrote:
> > After Ken's comment, I was going to suggest this one:
> > 
> > From 34b14470406cb9551f98707bf63175811a506523 Mon Sep 17 00:00:00 2001
> > From: Corinna Vinschen
> > Date: Wed, 15 Sep 2021 14:17:59 +0200
> > Subject: [PATCH] Cygwin: pipes: don't call NtQueryInformationFile on read side
> >  of pipes
> > 
> > NtQueryInformationFile hangs if it's called on the read side handle of
> > a pipe while another thread or process is performing a blocking read.
> > 
> > Avoid select potentially hanging by calling NtQueryInformationFile
> > only on the write side of the pipe and using PeekNamedPipe otherwise.
> > 
> > ---
> >  winsup/cygwin/select.cc | 8 ++++++++
> >  1 file changed, 8 insertions(+)
> > [...]
> 
> LGTM.
> I also confirmed a several test cases are all passed with this patch.

Thanks for reviewing.  Pushed.


Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-15 13:42                                                                           ` Corinna Vinschen
@ 2021-09-15 16:22                                                                             ` Ken Brown
  2021-09-15 17:09                                                                               ` Ken Brown
  2021-09-16  0:13                                                                               ` Takashi Yano
  0 siblings, 2 replies; 250+ messages in thread
From: Ken Brown @ 2021-09-15 16:22 UTC (permalink / raw)
  To: cygwin-developers

One other small detail:

diff --git a/winsup/cygwin/select.cc b/winsup/cygwin/select.cc
index 566cf66d6..cdb213a42 100644
--- a/winsup/cygwin/select.cc
+++ b/winsup/cygwin/select.cc
@@ -645,7 +645,7 @@ pipe_data_available (int fd, fhandler_base *fh, HANDLE h, 
bool writing)
             return 1; /* We cannot know actual write pipe space. */
           DWORD nbytes_in_pipe;
           if (!PeekNamedPipe (query_hdl, NULL, 0, NULL, &nbytes_in_pipe, NULL))
-           return 1;
+           return -1;
           fpli.WriteQuotaAvailable = fpli.InboundQuota - nbytes_in_pipe;
         }
        if (fpli.WriteQuotaAvailable > 0)

Having noticed that, I checked the callers of pipe_data_available with writing 
== true, and they don't check for a return of -1.  So I think I need to fix that 
too.

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-15 16:22                                                                             ` Ken Brown
@ 2021-09-15 17:09                                                                               ` Ken Brown
  2021-09-16  0:22                                                                                 ` Takashi Yano
  2021-09-16  9:09                                                                                 ` Takashi Yano
  2021-09-16  0:13                                                                               ` Takashi Yano
  1 sibling, 2 replies; 250+ messages in thread
From: Ken Brown @ 2021-09-15 17:09 UTC (permalink / raw)
  To: cygwin-developers

[-- Attachment #1: Type: text/plain, Size: 1079 bytes --]

On 9/15/2021 12:22 PM, Ken Brown wrote:
> One other small detail:
> 
> diff --git a/winsup/cygwin/select.cc b/winsup/cygwin/select.cc
> index 566cf66d6..cdb213a42 100644
> --- a/winsup/cygwin/select.cc
> +++ b/winsup/cygwin/select.cc
> @@ -645,7 +645,7 @@ pipe_data_available (int fd, fhandler_base *fh, HANDLE h, 
> bool writing)
>              return 1; /* We cannot know actual write pipe space. */
>            DWORD nbytes_in_pipe;
>            if (!PeekNamedPipe (query_hdl, NULL, 0, NULL, &nbytes_in_pipe, NULL))
> -           return 1;
> +           return -1;
>            fpli.WriteQuotaAvailable = fpli.InboundQuota - nbytes_in_pipe;
>          }
>         if (fpli.WriteQuotaAvailable > 0)
> 
> Having noticed that, I checked the callers of pipe_data_available with writing 
> == true, and they don't check for a return of -1.  So I think I need to fix that 
> too.

Could one of you (Corinna or Takashi) please check that I haven't done anything 
stupid in the attached patches?  If they look OK, I'll push them, and then maybe 
we're ready for a test release.

Ken

[-- Attachment #2: 0001-Cygwin-pipes-fix-a-return-value.patch --]
[-- Type: text/plain, Size: 936 bytes --]

From fc8d2b2ccaf3b519e6b606fd95217f94b4a6fcbc Mon Sep 17 00:00:00 2001
From: Ken Brown <kbrown@cornell.edu>
Date: Wed, 15 Sep 2021 11:24:25 -0400
Subject: [PATCH 1/2] Cygwin: pipes: fix a return value

If PeekNamedPipe fails in select.cc:pipe_data_available, return -1
rather than 1.
---
 winsup/cygwin/select.cc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/winsup/cygwin/select.cc b/winsup/cygwin/select.cc
index 566cf66d6..cdb213a42 100644
--- a/winsup/cygwin/select.cc
+++ b/winsup/cygwin/select.cc
@@ -645,7 +645,7 @@ pipe_data_available (int fd, fhandler_base *fh, HANDLE h, bool writing)
 	    return 1; /* We cannot know actual write pipe space. */
 	  DWORD nbytes_in_pipe;
 	  if (!PeekNamedPipe (query_hdl, NULL, 0, NULL, &nbytes_in_pipe, NULL))
-	    return 1;
+	    return -1;
 	  fpli.WriteQuotaAvailable = fpli.InboundQuota - nbytes_in_pipe;
 	}
       if (fpli.WriteQuotaAvailable > 0)
-- 
2.33.0


[-- Attachment #3: 0002-Cygwin-select-check-for-negative-return-from-pipe_da.patch --]
[-- Type: text/plain, Size: 2170 bytes --]

From 9d00b583c3d622c5ab8fcbdc3ba089eef8dd9ce5 Mon Sep 17 00:00:00 2001
From: Ken Brown <kbrown@cornell.edu>
Date: Wed, 15 Sep 2021 12:56:34 -0400
Subject: [PATCH 2/2] Cygwin: select: check for negative return from
 pipe_data_available

Make sure except_ready is set (if except_selected) on a negative
return from pipe_data_available.
---
 winsup/cygwin/select.cc | 22 +++++++++++++++-------
 1 file changed, 15 insertions(+), 7 deletions(-)

diff --git a/winsup/cygwin/select.cc b/winsup/cygwin/select.cc
index cdb213a42..cf9fbbbd3 100644
--- a/winsup/cygwin/select.cc
+++ b/winsup/cygwin/select.cc
@@ -753,8 +753,11 @@ out:
 	    gotone += s->except_ready = true;
 	  return gotone;
 	}
-      gotone += s->write_ready =  pipe_data_available (s->fd, fh, h, true);
-      select_printf ("write: %s, gotone %d", fh->get_name (), gotone);
+      int n = pipe_data_available (s->fd, fh, h, true);
+      select_printf ("write: %s, n %d", fh->get_name (), n);
+      gotone += s->write_ready = n;
+      if (n < 0 && s->except_selected)
+	gotone += s->except_ready = true;
     }
   return gotone;
 }
@@ -953,9 +956,11 @@ peek_fifo (select_record *s, bool from_select)
 out:
   if (s->write_selected)
     {
-      gotone += s->write_ready
-	= pipe_data_available (s->fd, fh, fh->get_handle (), true);
-      select_printf ("write: %s, gotone %d", fh->get_name (), gotone);
+      int n = pipe_data_available (s->fd, fh, fh->get_handle (), true);
+      select_printf ("write: %s, n %d", fh->get_name (), n);
+      gotone += s->write_ready = n;
+      if (n < 0 && s->except_selected)
+	gotone += s->except_ready = true;
     }
   return gotone;
 }
@@ -1394,8 +1399,11 @@ out:
   HANDLE h = ptys->get_output_handle ();
   if (s->write_selected)
     {
-      gotone += s->write_ready =  pipe_data_available (s->fd, fh, h, true);
-      select_printf ("write: %s, gotone %d", fh->get_name (), gotone);
+      int n = pipe_data_available (s->fd, fh, h, true);
+      select_printf ("write: %s, n %d", fh->get_name (), n);
+      gotone += s->write_ready = n;
+      if (n < 0 && s->except_selected)
+	gotone += s->except_ready = true;
     }
   return gotone;
 }
-- 
2.33.0


^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-15 16:22                                                                             ` Ken Brown
  2021-09-15 17:09                                                                               ` Ken Brown
@ 2021-09-16  0:13                                                                               ` Takashi Yano
  2021-09-16  2:26                                                                                 ` Ken Brown
  1 sibling, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-16  0:13 UTC (permalink / raw)
  To: cygwin-developers

On Wed, 15 Sep 2021 12:22:58 -0400
Ken Brown wrote:
> diff --git a/winsup/cygwin/select.cc b/winsup/cygwin/select.cc
> index 566cf66d6..cdb213a42 100644
> --- a/winsup/cygwin/select.cc
> +++ b/winsup/cygwin/select.cc
> @@ -645,7 +645,7 @@ pipe_data_available (int fd, fhandler_base *fh, HANDLE h, 
> bool writing)
>              return 1; /* We cannot know actual write pipe space. */
>            DWORD nbytes_in_pipe;
>            if (!PeekNamedPipe (query_hdl, NULL, 0, NULL, &nbytes_in_pipe, NULL))
> -           return 1;
> +           return -1;
>            fpli.WriteQuotaAvailable = fpli.InboundQuota - nbytes_in_pipe;
>          }
>         if (fpli.WriteQuotaAvailable > 0)

I think this is not correct. IIUC, return value -1 means that the
other side of pipe is closed. However, in this case, NtQueryInformationFile
is succeeded previously. So the other side of pipe is still alive. I cannot
imagine when PeekNamedPipe fails, however, if query_hdl is lost accidentally,
returning 1 is the correct thing just as the case query_hdl == NULL.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-15 17:09                                                                               ` Ken Brown
@ 2021-09-16  0:22                                                                                 ` Takashi Yano
  2021-09-16  2:28                                                                                   ` Ken Brown
  2021-09-16  9:09                                                                                 ` Takashi Yano
  1 sibling, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-16  0:22 UTC (permalink / raw)
  To: cygwin-developers

Hi Ken,

On Wed, 15 Sep 2021 13:09:30 -0400
Ken Brown wrote:
> Could one of you (Corinna or Takashi) please check that I haven't done anything 
> stupid in the attached patches?  If they look OK, I'll push them, and then maybe 
> we're ready for a test release.

The second patch
0002-Cygwin-select-check-for-negative-return-from-pipe_da-2.patch
LGTM. Thank you for noticing that!

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-16  0:13                                                                               ` Takashi Yano
@ 2021-09-16  2:26                                                                                 ` Ken Brown
  0 siblings, 0 replies; 250+ messages in thread
From: Ken Brown @ 2021-09-16  2:26 UTC (permalink / raw)
  To: cygwin-developers

On 9/15/2021 8:13 PM, Takashi Yano wrote:
> On Wed, 15 Sep 2021 12:22:58 -0400
> Ken Brown wrote:
>> diff --git a/winsup/cygwin/select.cc b/winsup/cygwin/select.cc
>> index 566cf66d6..cdb213a42 100644
>> --- a/winsup/cygwin/select.cc
>> +++ b/winsup/cygwin/select.cc
>> @@ -645,7 +645,7 @@ pipe_data_available (int fd, fhandler_base *fh, HANDLE h,
>> bool writing)
>>               return 1; /* We cannot know actual write pipe space. */
>>             DWORD nbytes_in_pipe;
>>             if (!PeekNamedPipe (query_hdl, NULL, 0, NULL, &nbytes_in_pipe, NULL))
>> -           return 1;
>> +           return -1;
>>             fpli.WriteQuotaAvailable = fpli.InboundQuota - nbytes_in_pipe;
>>           }
>>          if (fpli.WriteQuotaAvailable > 0)
> 
> I think this is not correct. IIUC, return value -1 means that the
> other side of pipe is closed. However, in this case, NtQueryInformationFile
> is succeeded previously. So the other side of pipe is still alive. I cannot
> imagine when PeekNamedPipe fails, however, if query_hdl is lost accidentally,
> returning 1 is the correct thing just as the case query_hdl == NULL.

Ah, you're right of course.  A failure of a call on query_hdl doesn't indicate 
an exceptional condition on the write side of the pipe.

Thanks.

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-16  0:22                                                                                 ` Takashi Yano
@ 2021-09-16  2:28                                                                                   ` Ken Brown
  0 siblings, 0 replies; 250+ messages in thread
From: Ken Brown @ 2021-09-16  2:28 UTC (permalink / raw)
  To: cygwin-developers

On 9/15/2021 8:22 PM, Takashi Yano wrote:
> Hi Ken,
> 
> On Wed, 15 Sep 2021 13:09:30 -0400
> Ken Brown wrote:
>> Could one of you (Corinna or Takashi) please check that I haven't done anything
>> stupid in the attached patches?  If they look OK, I'll push them, and then maybe
>> we're ready for a test release.
> 
> The second patch
> 0002-Cygwin-select-check-for-negative-return-from-pipe_da-2.patch
> LGTM. Thank you for noticing that!

Thanks for the review.

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-15 17:09                                                                               ` Ken Brown
  2021-09-16  0:22                                                                                 ` Takashi Yano
@ 2021-09-16  9:09                                                                                 ` Takashi Yano
  2021-09-16 13:02                                                                                   ` Takashi Yano
  1 sibling, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-16  9:09 UTC (permalink / raw)
  To: cygwin-developers

On Wed, 15 Sep 2021 13:09:30 -0400
Ken Brown wrote:
> Could one of you (Corinna or Takashi) please check that I haven't done anything 
> stupid in the attached patches?  If they look OK, I'll push them, and then maybe 
> we're ready for a test release.

I encountered a problem with current git head.

To reproduce the problem:
1) Make the CPU hight load state (100% load for all cores).
2) Run '/bin/echo AAAAAAAAAAA 2>&1 |cat' several times.
Sometimes, no output is shown.

This seems to be a race issue between multiple pipe writers.
Now I am investigating the problem.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-16  9:09                                                                                 ` Takashi Yano
@ 2021-09-16 13:02                                                                                   ` Takashi Yano
  2021-09-16 13:25                                                                                     ` Corinna Vinschen
  0 siblings, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-16 13:02 UTC (permalink / raw)
  To: cygwin-developers

[-- Attachment #1: Type: text/plain, Size: 625 bytes --]

On Thu, 16 Sep 2021 18:09:05 +0900
Takashi Yano wrote:
> I encountered a problem with current git head.
> 
> To reproduce the problem:
> 1) Make the CPU hight load state (100% load for all cores).
> 2) Run '/bin/echo AAAAAAAAAAA 2>&1 |cat' several times.
> Sometimes, no output is shown.
> 
> This seems to be a race issue between multiple pipe writers.
> Now I am investigating the problem.

I would like to propose three additional patches, two are for
the issue above (0001 and 0003), and the other one (0002) fixes
the error handling.

Please have a look at patches attached.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

[-- Attachment #2: 0001-Cygwin-close_all_files-Do-not-duplicate-stderr-for-w.patch --]
[-- Type: application/octet-stream, Size: 1088 bytes --]

From a57ad3e77a5b4760fc161406c411cd9d5084c39a Mon Sep 17 00:00:00 2001
From: Takashi Yano <takashi.yano@nifty.ne.jp>
Date: Thu, 16 Sep 2021 20:50:54 +0900
Subject: [PATCH 1/3] Cygwin: close_all_files: Do not duplicate stderr for
 write pipe.

- Currently, the stderr handle is duplicated in close_all_files().
  This interferes the handle counting for detecting closure of read
  pipe, which is introduced by commit f79a4611. This patch stops
  duplicating stderr handle if it is write pipe.
---
 winsup/cygwin/syscalls.cc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/winsup/cygwin/syscalls.cc b/winsup/cygwin/syscalls.cc
index 11af079cb..a3ffe3532 100644
--- a/winsup/cygwin/syscalls.cc
+++ b/winsup/cygwin/syscalls.cc
@@ -95,7 +95,7 @@ close_all_files (bool norelease)
       if (cfd >= 0)
 	{
 	  debug_only_printf ("closing fd %d", i);
-	  if (i == 2)
+	  if (i == 2 && cfd->get_dev () != FH_PIPEW)
 	    DuplicateHandle (GetCurrentProcess (), cfd->get_output_handle (),
 			     GetCurrentProcess (), &h,
 			     0, false, DUPLICATE_SAME_ACCESS);
-- 
2.33.0


[-- Attachment #3: 0002-Cygwin-pipe-Fix-error-handling-in-fhandler_pip-creat.patch --]
[-- Type: application/octet-stream, Size: 2315 bytes --]

From c967493799fe261c6bc0d3d36a5664763ec29f31 Mon Sep 17 00:00:00 2001
From: Takashi Yano <takashi.yano@nifty.ne.jp>
Date: Thu, 16 Sep 2021 21:27:24 +0900
Subject: [PATCH 2/3] Cygwin: pipe: Fix error handling in
 fhandler_pip::create().

- Currently, error handling in fhandler_pipe::create() is broken.
  This patch fixes that.
---
 winsup/cygwin/fhandler_pipe.cc | 43 ++++++++++++++++++++++++----------
 1 file changed, 31 insertions(+), 12 deletions(-)

diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index c2f4353f6..fd2ab87af 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -831,32 +831,51 @@ fhandler_pipe::create (fhandler_pipe *fhs[2], unsigned psize, int mode)
 	  NtClose (r);
 	  delete fhs[1];
 	  NtClose (w);
+	  goto out;
 	}
-      else
+      fhs[0]->set_read_mutex (mtx);
+
+      fhs[0]->select_sem = CreateSemaphore (sa, 0, INT32_MAX, NULL);
+      if (!fhs[0]->select_sem)
 	{
-	  fhs[0]->set_read_mutex (mtx);
-	  res = 0;
+	  CloseHandle (fhs[0]->read_mtx);
+	  delete fhs[0];
+	  NtClose (r);
+	  delete fhs[1];
+	  NtClose (w);
+	  goto out;
 	}
-      fhs[0]->select_sem = CreateSemaphore (sa, 0, INT32_MAX, NULL);
-      if (fhs[0]->select_sem)
-	DuplicateHandle (GetCurrentProcess (), fhs[0]->select_sem,
-			 GetCurrentProcess (), &fhs[1]->select_sem,
-			 0, sa->bInheritHandle, DUPLICATE_SAME_ACCESS);
+      if (!DuplicateHandle (GetCurrentProcess (), fhs[0]->select_sem,
+			    GetCurrentProcess (), &fhs[1]->select_sem,
+			    0, sa->bInheritHandle, DUPLICATE_SAME_ACCESS))
+	{
+	  CloseHandle (fhs[0]->read_mtx);
+	  CloseHandle (fhs[0]->select_sem);
+	  delete fhs[0];
+	  NtClose (r);
+	  delete fhs[1];
+	  NtClose (w);
+	  goto out;
+	}
+
       if (!DuplicateHandle (GetCurrentProcess (), r,
 			    GetCurrentProcess (), &fhs[1]->query_hdl,
 			    FILE_READ_DATA, sa->bInheritHandle, 0))
 	{
+	  CloseHandle (fhs[0]->read_mtx);
 	  CloseHandle (fhs[0]->select_sem);
 	  delete fhs[0];
-	  CloseHandle (r);
+	  NtClose (r);
 	  CloseHandle (fhs[1]->select_sem);
 	  delete fhs[1];
-	  CloseHandle (w);
+	  NtClose (w);
+	  goto out;
 	}
-      else
-	res = 0;
+
+      res = 0;
     }
 
+out:
   debug_printf ("%R = pipe([%p, %p], %d, %y)", res, fhs[0], fhs[1], psize, mode);
   return res;
 }
-- 
2.33.0


[-- Attachment #4: 0003-Cygwin-pipe-Fix-race-issue-regarding-handle-count.patch --]
[-- Type: application/octet-stream, Size: 5549 bytes --]

From 8c21f43b181aa24dd4dde91d6d3f45f3f86f4d81 Mon Sep 17 00:00:00 2001
From: Takashi Yano <takashi.yano@nifty.ne.jp>
Date: Thu, 16 Sep 2021 21:44:09 +0900
Subject: [PATCH 3/3] Cygwin: pipe: Fix race issue regarding handle count.

- This patch fixes the race issue in the handle counting to detect
  closure of read pipe, which is introduced by commit f79a4611.
  A mutex hdl_cnt_mtx is introduced for this issue.
---
 winsup/cygwin/fhandler.h       |  1 +
 winsup/cygwin/fhandler_pipe.cc | 74 +++++++++++++++++++++++++++++++---
 2 files changed, 69 insertions(+), 6 deletions(-)

diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
index 919655012..e0a2dee09 100644
--- a/winsup/cygwin/fhandler.h
+++ b/winsup/cygwin/fhandler.h
@@ -1176,6 +1176,7 @@ class fhandler_pipe_fifo: public fhandler_base
  protected:
   size_t pipe_buf_size;
   HANDLE query_hdl;
+  HANDLE hdl_cnt_mtx;
   virtual void release_select_sem (const char *) {};
 
  public:
diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index fd2ab87af..7eedc010c 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -200,7 +200,14 @@ fhandler_pipe::open_setup (int flags)
       SECURITY_ATTRIBUTES *sa = sec_none_cloexec (flags);
       read_mtx = CreateMutex (sa, FALSE, NULL);
       if (!read_mtx)
-	debug_printf ("CreateMutex failed: %E");
+	debug_printf ("CreateMutex read_mtx failed: %E");
+    }
+  if (!hdl_cnt_mtx)
+    {
+      SECURITY_ATTRIBUTES *sa = sec_none_cloexec (flags);
+      hdl_cnt_mtx = CreateMutex (sa, FALSE, NULL);
+      if (!hdl_cnt_mtx)
+	debug_printf ("CreateMutex hdl_cnt_mtx failed: %E");
     }
   if (get_dev () == FH_PIPEW && !query_hdl)
     set_pipe_non_blocking (is_nonblocking ());
@@ -390,8 +397,10 @@ fhandler_pipe_fifo::reader_closed ()
 {
   if (!query_hdl)
     return false;
+  cygwait (hdl_cnt_mtx);
   int n_reader = get_obj_handle_count (query_hdl);
   int n_writer = get_obj_handle_count (get_handle ());
+  ReleaseMutex (hdl_cnt_mtx);
   return n_reader == n_writer;
 }
 
@@ -554,11 +563,18 @@ fhandler_pipe::set_close_on_exec (bool val)
     set_no_inheritance (select_sem, val);
   if (query_hdl)
     set_no_inheritance (query_hdl, val);
+  if (hdl_cnt_mtx)
+    set_no_inheritance (hdl_cnt_mtx, val);
 }
 
 void
 fhandler_pipe::fixup_after_fork (HANDLE parent)
 {
+  if (hdl_cnt_mtx)
+    {
+      fork_fixup (parent, hdl_cnt_mtx, "hdl_cnt_mtx");
+      cygwait (hdl_cnt_mtx);
+    }
   if (read_mtx)
     fork_fixup (parent, read_mtx, "read_mtx");
   if (select_sem)
@@ -567,6 +583,8 @@ fhandler_pipe::fixup_after_fork (HANDLE parent)
     fork_fixup (parent, query_hdl, "query_hdl");
 
   fhandler_base::fixup_after_fork (parent);
+  if (hdl_cnt_mtx)
+    ReleaseMutex (hdl_cnt_mtx);
 }
 
 int
@@ -576,6 +594,7 @@ fhandler_pipe::dup (fhandler_base *child, int flags)
   ftp->set_popen_pid (0);
 
   int res = 0;
+  cygwait (hdl_cnt_mtx);
   if (fhandler_base::dup (child, flags))
     res = -1;
   else if (read_mtx &&
@@ -589,8 +608,8 @@ fhandler_pipe::dup (fhandler_base *child, int flags)
     }
   else if (select_sem &&
 	   !DuplicateHandle (GetCurrentProcess (), select_sem,
-			    GetCurrentProcess (), &ftp->select_sem,
-			    0, !(flags & O_CLOEXEC), DUPLICATE_SAME_ACCESS))
+			     GetCurrentProcess (), &ftp->select_sem,
+			     0, !(flags & O_CLOEXEC), DUPLICATE_SAME_ACCESS))
     {
       __seterrno ();
       ftp->close ();
@@ -598,13 +617,23 @@ fhandler_pipe::dup (fhandler_base *child, int flags)
     }
   else if (query_hdl &&
 	   !DuplicateHandle (GetCurrentProcess (), query_hdl,
-			    GetCurrentProcess (), &ftp->query_hdl,
-			    0, !(flags & O_CLOEXEC), DUPLICATE_SAME_ACCESS))
+			     GetCurrentProcess (), &ftp->query_hdl,
+			     0, !(flags & O_CLOEXEC), DUPLICATE_SAME_ACCESS))
+    {
+      __seterrno ();
+      ftp->close ();
+      res = -1;
+    }
+  else if (hdl_cnt_mtx &&
+	   !DuplicateHandle (GetCurrentProcess (), hdl_cnt_mtx,
+			     GetCurrentProcess (), &ftp->hdl_cnt_mtx,
+			     0, !(flags & O_CLOEXEC), DUPLICATE_SAME_ACCESS))
     {
       __seterrno ();
       ftp->close ();
       res = -1;
     }
+  ReleaseMutex (hdl_cnt_mtx);
 
   debug_printf ("res %d", res);
   return res;
@@ -620,9 +649,13 @@ fhandler_pipe::close ()
     }
   if (read_mtx)
     CloseHandle (read_mtx);
+  cygwait (hdl_cnt_mtx);
   if (query_hdl)
     CloseHandle (query_hdl);
-  return fhandler_base::close ();
+  int ret = fhandler_base::close ();
+  ReleaseMutex (hdl_cnt_mtx);
+  CloseHandle (hdl_cnt_mtx);
+  return ret;
 }
 
 #define PIPE_INTRO "\\\\.\\pipe\\cygwin-"
@@ -872,6 +905,35 @@ fhandler_pipe::create (fhandler_pipe *fhs[2], unsigned psize, int mode)
 	  goto out;
 	}
 
+      fhs[0]->hdl_cnt_mtx = CreateMutexW (sa, FALSE, NULL);
+      if (!fhs[0]->hdl_cnt_mtx)
+	{
+	  CloseHandle (fhs[0]->read_mtx);
+	  CloseHandle (fhs[0]->select_sem);
+	  delete fhs[0];
+	  NtClose (r);
+	  CloseHandle (fhs[1]->select_sem);
+	  CloseHandle (fhs[1]->query_hdl);
+	  delete fhs[1];
+	  NtClose (w);
+	  goto out;
+	}
+      if (!DuplicateHandle (GetCurrentProcess (), fhs[0]->hdl_cnt_mtx,
+			    GetCurrentProcess (), &fhs[1]->hdl_cnt_mtx,
+			    0, sa->bInheritHandle, DUPLICATE_SAME_ACCESS))
+	{
+	  CloseHandle (fhs[0]->read_mtx);
+	  CloseHandle (fhs[0]->select_sem);
+	  CloseHandle (fhs[0]->hdl_cnt_mtx);
+	  delete fhs[0];
+	  NtClose (r);
+	  CloseHandle (fhs[1]->select_sem);
+	  CloseHandle (fhs[1]->query_hdl);
+	  delete fhs[1];
+	  NtClose (w);
+	  goto out;
+	}
+
       res = 0;
     }
 
-- 
2.33.0


^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-16 13:02                                                                                   ` Takashi Yano
@ 2021-09-16 13:25                                                                                     ` Corinna Vinschen
  2021-09-16 14:27                                                                                       ` Takashi Yano
  0 siblings, 1 reply; 250+ messages in thread
From: Corinna Vinschen @ 2021-09-16 13:25 UTC (permalink / raw)
  To: cygwin-developers

On Sep 16 22:02, Takashi Yano wrote:
> On Thu, 16 Sep 2021 18:09:05 +0900
> Takashi Yano wrote:
> > I encountered a problem with current git head.
> > 
> > To reproduce the problem:
> > 1) Make the CPU hight load state (100% load for all cores).
> > 2) Run '/bin/echo AAAAAAAAAAA 2>&1 |cat' several times.
> > Sometimes, no output is shown.
> > 
> > This seems to be a race issue between multiple pipe writers.
> > Now I am investigating the problem.
> 
> I would like to propose three additional patches, two are for
> the issue above (0001 and 0003), and the other one (0002) fixes
> the error handling.
> 
> Please have a look at patches attached.

> diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
> index fd2ab87af..7eedc010c 100644
> --- a/winsup/cygwin/fhandler_pipe.cc
> +++ b/winsup/cygwin/fhandler_pipe.cc
> @@ -200,7 +200,14 @@ fhandler_pipe::open_setup (int flags)
>        SECURITY_ATTRIBUTES *sa = sec_none_cloexec (flags);
>        read_mtx = CreateMutex (sa, FALSE, NULL);
>        if (!read_mtx)
> -	debug_printf ("CreateMutex failed: %E");
> +	debug_printf ("CreateMutex read_mtx failed: %E");
> +    }
> +  if (!hdl_cnt_mtx)
> +    {
> +      SECURITY_ATTRIBUTES *sa = sec_none_cloexec (flags);
> +      hdl_cnt_mtx = CreateMutex (sa, FALSE, NULL);
> +      if (!hdl_cnt_mtx)
> +	debug_printf ("CreateMutex hdl_cnt_mtx failed: %E");
>      }
>    if (get_dev () == FH_PIPEW && !query_hdl)
>      set_pipe_non_blocking (is_nonblocking ());
> @@ -390,8 +397,10 @@ fhandler_pipe_fifo::reader_closed ()
>  {
>    if (!query_hdl)
>      return false;
> +  cygwait (hdl_cnt_mtx);

This is missing an `if (hdl_cnt_mtx)' check.

Also, given you're using hdl_cnt_mtx only for short-lived blocking,
you may want to reduce the overhead and just call WFSO with INFINITE
timeout.

>    int n_reader = get_obj_handle_count (query_hdl);
>    int n_writer = get_obj_handle_count (get_handle ());
> +  ReleaseMutex (hdl_cnt_mtx);
>    return n_reader == n_writer;
>  }
>  
>  int
> @@ -576,6 +594,7 @@ fhandler_pipe::dup (fhandler_base *child, int flags)
>    ftp->set_popen_pid (0);
>  
>    int res = 0;
> +  cygwait (hdl_cnt_mtx);

Same here and all the other cygwait(hdl_cnt_mtx) calls.

It would be great if open_setup() would be converted to a method which
is allowed to fail, rather than ignoring errors in sync object creation
and having to test the handle throughout the code.  Given there's only a
single caller of that function (dtable::init_std_file_from_handle), that
shouldn't be much work.  But it's certainly better than ignoring creation
failures in the long run.

> +      fhs[0]->hdl_cnt_mtx = CreateMutexW (sa, FALSE, NULL);
> +      if (!fhs[0]->hdl_cnt_mtx)
> +	{
> +	  CloseHandle (fhs[0]->read_mtx);
> +	  CloseHandle (fhs[0]->select_sem);
> +	  delete fhs[0];
> +	  NtClose (r);
> +	  CloseHandle (fhs[1]->select_sem);
> +	  CloseHandle (fhs[1]->query_hdl);
> +	  delete fhs[1];
> +	  NtClose (w);
> +	  goto out;
> +	}
> +      if (!DuplicateHandle (GetCurrentProcess (), fhs[0]->hdl_cnt_mtx,
> +			    GetCurrentProcess (), &fhs[1]->hdl_cnt_mtx,
> +			    0, sa->bInheritHandle, DUPLICATE_SAME_ACCESS))
> +	{
> +	  CloseHandle (fhs[0]->read_mtx);
> +	  CloseHandle (fhs[0]->select_sem);
> +	  CloseHandle (fhs[0]->hdl_cnt_mtx);
> +	  delete fhs[0];
> +	  NtClose (r);
> +	  CloseHandle (fhs[1]->select_sem);
> +	  CloseHandle (fhs[1]->query_hdl);
> +	  delete fhs[1];
> +	  NtClose (w);
> +	  goto out;
> +	}
> +
>        res = 0;
>      }

What about converting this to a goto error chain as in
fhandler_pty_slave::setup_pseudoconsole?  This makes error handling
much cleaner, IMHO.


Corinna


^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-16 13:25                                                                                     ` Corinna Vinschen
@ 2021-09-16 14:27                                                                                       ` Takashi Yano
  2021-09-16 15:01                                                                                         ` Corinna Vinschen
  0 siblings, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-16 14:27 UTC (permalink / raw)
  To: cygwin-developers

[-- Attachment #1: Type: text/plain, Size: 3487 bytes --]

Hi Corinna,

Thanks for reviewing the patches.

On Thu, 16 Sep 2021 15:25:46 +0200
Corinna Vinschen wrote:
> > diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
> > index fd2ab87af..7eedc010c 100644
> > --- a/winsup/cygwin/fhandler_pipe.cc
> > +++ b/winsup/cygwin/fhandler_pipe.cc
> > @@ -200,7 +200,14 @@ fhandler_pipe::open_setup (int flags)
> >        SECURITY_ATTRIBUTES *sa = sec_none_cloexec (flags);
> >        read_mtx = CreateMutex (sa, FALSE, NULL);
> >        if (!read_mtx)
> > -	debug_printf ("CreateMutex failed: %E");
> > +	debug_printf ("CreateMutex read_mtx failed: %E");
> > +    }
> > +  if (!hdl_cnt_mtx)
> > +    {
> > +      SECURITY_ATTRIBUTES *sa = sec_none_cloexec (flags);
> > +      hdl_cnt_mtx = CreateMutex (sa, FALSE, NULL);
> > +      if (!hdl_cnt_mtx)
> > +	debug_printf ("CreateMutex hdl_cnt_mtx failed: %E");
> >      }
> >    if (get_dev () == FH_PIPEW && !query_hdl)
> >      set_pipe_non_blocking (is_nonblocking ());
> > @@ -390,8 +397,10 @@ fhandler_pipe_fifo::reader_closed ()
> >  {
> >    if (!query_hdl)
> >      return false;
> > +  cygwait (hdl_cnt_mtx);
> 
> This is missing an `if (hdl_cnt_mtx)' check.

Done.

> Also, given you're using hdl_cnt_mtx only for short-lived blocking,
> you may want to reduce the overhead and just call WFSO with INFINITE
> timeout.
> 
> >    int n_reader = get_obj_handle_count (query_hdl);
> >    int n_writer = get_obj_handle_count (get_handle ());
> > +  ReleaseMutex (hdl_cnt_mtx);
> >    return n_reader == n_writer;
> >  }
> >  
> >  int
> > @@ -576,6 +594,7 @@ fhandler_pipe::dup (fhandler_base *child, int flags)
> >    ftp->set_popen_pid (0);
> >  
> >    int res = 0;
> > +  cygwait (hdl_cnt_mtx);
> 
> Same here and all the other cygwait(hdl_cnt_mtx) calls.

Done.

> It would be great if open_setup() would be converted to a method which
> is allowed to fail, rather than ignoring errors in sync object creation
> and having to test the handle throughout the code.  Given there's only a
> single caller of that function (dtable::init_std_file_from_handle), that
> shouldn't be much work.  But it's certainly better than ignoring creation
> failures in the long run.

I do not have confidence to do this correctly in a short time.
Leave it as a homework. Or may I leave it to you?

> > +      fhs[0]->hdl_cnt_mtx = CreateMutexW (sa, FALSE, NULL);
> > +      if (!fhs[0]->hdl_cnt_mtx)
> > +	{
> > +	  CloseHandle (fhs[0]->read_mtx);
> > +	  CloseHandle (fhs[0]->select_sem);
> > +	  delete fhs[0];
> > +	  NtClose (r);
> > +	  CloseHandle (fhs[1]->select_sem);
> > +	  CloseHandle (fhs[1]->query_hdl);
> > +	  delete fhs[1];
> > +	  NtClose (w);
> > +	  goto out;
> > +	}
> > +      if (!DuplicateHandle (GetCurrentProcess (), fhs[0]->hdl_cnt_mtx,
> > +			    GetCurrentProcess (), &fhs[1]->hdl_cnt_mtx,
> > +			    0, sa->bInheritHandle, DUPLICATE_SAME_ACCESS))
> > +	{
> > +	  CloseHandle (fhs[0]->read_mtx);
> > +	  CloseHandle (fhs[0]->select_sem);
> > +	  CloseHandle (fhs[0]->hdl_cnt_mtx);
> > +	  delete fhs[0];
> > +	  NtClose (r);
> > +	  CloseHandle (fhs[1]->select_sem);
> > +	  CloseHandle (fhs[1]->query_hdl);
> > +	  delete fhs[1];
> > +	  NtClose (w);
> > +	  goto out;
> > +	}
> > +
> >        res = 0;
> >      }
> 
> What about converting this to a goto error chain as in
> fhandler_pty_slave::setup_pseudoconsole?  This makes error handling
> much cleaner, IMHO.

Done.

I have attached the patches revised.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

[-- Attachment #2: v2-0001-Cygwin-close_all_files-Do-not-duplicate-stderr-fo.patch --]
[-- Type: application/octet-stream, Size: 1091 bytes --]

From 4bbd03dc7d482966ca5ace2e130424a8ba2cbf9d Mon Sep 17 00:00:00 2001
From: Takashi Yano <takashi.yano@nifty.ne.jp>
Date: Thu, 16 Sep 2021 20:50:54 +0900
Subject: [PATCH v2 1/3] Cygwin: close_all_files: Do not duplicate stderr for
 write pipe.

- Currently, the stderr handle is duplicated in close_all_files().
  This interferes the handle counting for detecting closure of read
  pipe, which is introduced by commit f79a4611. This patch stops
  duplicating stderr handle if it is write pipe.
---
 winsup/cygwin/syscalls.cc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/winsup/cygwin/syscalls.cc b/winsup/cygwin/syscalls.cc
index 11af079cb..a3ffe3532 100644
--- a/winsup/cygwin/syscalls.cc
+++ b/winsup/cygwin/syscalls.cc
@@ -95,7 +95,7 @@ close_all_files (bool norelease)
       if (cfd >= 0)
 	{
 	  debug_only_printf ("closing fd %d", i);
-	  if (i == 2)
+	  if (i == 2 && cfd->get_dev () != FH_PIPEW)
 	    DuplicateHandle (GetCurrentProcess (), cfd->get_output_handle (),
 			     GetCurrentProcess (), &h,
 			     0, false, DUPLICATE_SAME_ACCESS);
-- 
2.33.0


[-- Attachment #3: v2-0002-Cygwin-pipe-Fix-error-handling-in-fhandler_pip-cr.patch --]
[-- Type: application/octet-stream, Size: 4410 bytes --]

From 4b784d72ecffe5293f10baee314ff892d5ddc59b Mon Sep 17 00:00:00 2001
From: Takashi Yano <takashi.yano@nifty.ne.jp>
Date: Thu, 16 Sep 2021 23:15:17 +0900
Subject: [PATCH v2 2/3] Cygwin: pipe: Fix error handling in
 fhandler_pip::create().

- Currently, error handling in fhandler_pipe::create() is broken.
  This patch fixes that.
---
 winsup/cygwin/fhandler.h       |   1 -
 winsup/cygwin/fhandler_pipe.cc | 101 ++++++++++++++++-----------------
 2 files changed, 48 insertions(+), 54 deletions(-)

diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
index 919655012..eb312ccb6 100644
--- a/winsup/cygwin/fhandler.h
+++ b/winsup/cygwin/fhandler.h
@@ -1206,7 +1206,6 @@ public:
   fhandler_pipe ();
 
   bool ispipe() const { return true; }
-  void set_read_mutex (HANDLE mtx) { read_mtx = mtx; }
   void set_pipe_buf_size ();
 
   void set_popen_pid (pid_t pid) {popen_pid = pid;}
diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index c2f4353f6..582b4d564 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -803,61 +803,56 @@ fhandler_pipe::create (fhandler_pipe *fhs[2], unsigned psize, int mode)
 
   int ret = nt_create (sa, r, w, psize, &unique_id);
   if (ret)
-    __seterrno_from_win_error (ret);
-  else if ((fhs[0] = (fhandler_pipe *) build_fh_dev (*piper_dev)) == NULL)
     {
-      NtClose (r);
-      NtClose (w);
-    }
-  else if ((fhs[1] = (fhandler_pipe *) build_fh_dev (*pipew_dev)) == NULL)
-    {
-      delete fhs[0];
-      NtClose (r);
-      NtClose (w);
-    }
-  else
-    {
-      mode |= mode & O_TEXT ?: O_BINARY;
-      fhs[0]->init (r, FILE_CREATE_PIPE_INSTANCE | GENERIC_READ, mode,
-		    unique_id);
-      fhs[1]->init (w, FILE_CREATE_PIPE_INSTANCE | GENERIC_WRITE, mode,
-		    unique_id);
-      /* For the read side of the pipe, add a mutex.  See raw_read for the
-	 usage. */
-      HANDLE mtx = CreateMutexW (sa, FALSE, NULL);
-      if (!mtx)
-	{
-	  delete fhs[0];
-	  NtClose (r);
-	  delete fhs[1];
-	  NtClose (w);
-	}
-      else
-	{
-	  fhs[0]->set_read_mutex (mtx);
-	  res = 0;
-	}
-      fhs[0]->select_sem = CreateSemaphore (sa, 0, INT32_MAX, NULL);
-      if (fhs[0]->select_sem)
-	DuplicateHandle (GetCurrentProcess (), fhs[0]->select_sem,
-			 GetCurrentProcess (), &fhs[1]->select_sem,
-			 0, sa->bInheritHandle, DUPLICATE_SAME_ACCESS);
-      if (!DuplicateHandle (GetCurrentProcess (), r,
-			    GetCurrentProcess (), &fhs[1]->query_hdl,
-			    FILE_READ_DATA, sa->bInheritHandle, 0))
-	{
-	  CloseHandle (fhs[0]->select_sem);
-	  delete fhs[0];
-	  CloseHandle (r);
-	  CloseHandle (fhs[1]->select_sem);
-	  delete fhs[1];
-	  CloseHandle (w);
-	}
-      else
-	res = 0;
+      __seterrno_from_win_error (ret);
+      goto out;
     }
-
-  debug_printf ("%R = pipe([%p, %p], %d, %y)", res, fhs[0], fhs[1], psize, mode);
+  if ((fhs[0] = (fhandler_pipe *) build_fh_dev (*piper_dev)) == NULL)
+    goto err_close_rw_handle;
+  if ((fhs[1] = (fhandler_pipe *) build_fh_dev (*pipew_dev)) == NULL)
+    goto err_delete_fhs0;
+  mode |= mode & O_TEXT ?: O_BINARY;
+  fhs[0]->init (r, FILE_CREATE_PIPE_INSTANCE | GENERIC_READ, mode, unique_id);
+  fhs[1]->init (w, FILE_CREATE_PIPE_INSTANCE | GENERIC_WRITE, mode, unique_id);
+
+  /* For the read side of the pipe, add a mutex.  See raw_read for the
+     usage. */
+  fhs[0]->read_mtx = CreateMutexW (sa, FALSE, NULL);
+  if (!fhs[0]->read_mtx)
+    goto err_delete_fhs1;
+
+  fhs[0]->select_sem = CreateSemaphore (sa, 0, INT32_MAX, NULL);
+  if (!fhs[0]->select_sem)
+    goto err_close_read_mtx;
+  if (!DuplicateHandle (GetCurrentProcess (), fhs[0]->select_sem,
+			GetCurrentProcess (), &fhs[1]->select_sem,
+			0, sa->bInheritHandle, DUPLICATE_SAME_ACCESS))
+    goto err_close_select_sem0;
+
+  if (!DuplicateHandle (GetCurrentProcess (), r,
+			GetCurrentProcess (), &fhs[1]->query_hdl,
+			FILE_READ_DATA, sa->bInheritHandle, 0))
+    goto err_close_select_sem1;
+
+  res = 0;
+  goto out;
+
+err_close_select_sem1:
+  CloseHandle (fhs[1]->select_sem);
+err_close_select_sem0:
+  CloseHandle (fhs[0]->select_sem);
+err_close_read_mtx:
+  CloseHandle (fhs[0]->read_mtx);
+err_delete_fhs1:
+  delete fhs[1];
+err_delete_fhs0:
+  delete fhs[0];
+err_close_rw_handle:
+  NtClose (r);
+  NtClose (w);
+out:
+  debug_printf ("%R = pipe([%p, %p], %d, %y)",
+		res, fhs[0], fhs[1], psize, mode);
   return res;
 }
 
-- 
2.33.0


[-- Attachment #4: v2-0003-Cygwin-pipe-Fix-race-issue-regarding-handle-count.patch --]
[-- Type: application/octet-stream, Size: 5579 bytes --]

From bc1a2d65cb36b839c5e9246f65865abd62cb2587 Mon Sep 17 00:00:00 2001
From: Takashi Yano <takashi.yano@nifty.ne.jp>
Date: Thu, 16 Sep 2021 23:21:57 +0900
Subject: [PATCH v2 3/3] Cygwin: pipe: Fix race issue regarding handle count.

- This patch fixes the race issue in the handle counting to detect
  closure of read pipe, which is introduced by commit f79a4611.
  A mutex hdl_cnt_mtx is introduced for this issue.
---
 winsup/cygwin/fhandler.h       |  1 +
 winsup/cygwin/fhandler_pipe.cc | 65 ++++++++++++++++++++++++++++++----
 2 files changed, 60 insertions(+), 6 deletions(-)

diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
index eb312ccb6..31edb1822 100644
--- a/winsup/cygwin/fhandler.h
+++ b/winsup/cygwin/fhandler.h
@@ -1176,6 +1176,7 @@ class fhandler_pipe_fifo: public fhandler_base
  protected:
   size_t pipe_buf_size;
   HANDLE query_hdl;
+  HANDLE hdl_cnt_mtx;
   virtual void release_select_sem (const char *) {};
 
  public:
diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index 582b4d564..2068a943e 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -200,7 +200,14 @@ fhandler_pipe::open_setup (int flags)
       SECURITY_ATTRIBUTES *sa = sec_none_cloexec (flags);
       read_mtx = CreateMutex (sa, FALSE, NULL);
       if (!read_mtx)
-	debug_printf ("CreateMutex failed: %E");
+	debug_printf ("CreateMutex read_mtx failed: %E");
+    }
+  if (!hdl_cnt_mtx)
+    {
+      SECURITY_ATTRIBUTES *sa = sec_none_cloexec (flags);
+      hdl_cnt_mtx = CreateMutex (sa, FALSE, NULL);
+      if (!hdl_cnt_mtx)
+	debug_printf ("CreateMutex hdl_cnt_mtx failed: %E");
     }
   if (get_dev () == FH_PIPEW && !query_hdl)
     set_pipe_non_blocking (is_nonblocking ());
@@ -390,8 +397,12 @@ fhandler_pipe_fifo::reader_closed ()
 {
   if (!query_hdl)
     return false;
+  if (hdl_cnt_mtx)
+    WaitForSingleObject (hdl_cnt_mtx, INFINITE);
   int n_reader = get_obj_handle_count (query_hdl);
   int n_writer = get_obj_handle_count (get_handle ());
+  if (hdl_cnt_mtx)
+    ReleaseMutex (hdl_cnt_mtx);
   return n_reader == n_writer;
 }
 
@@ -554,11 +565,18 @@ fhandler_pipe::set_close_on_exec (bool val)
     set_no_inheritance (select_sem, val);
   if (query_hdl)
     set_no_inheritance (query_hdl, val);
+  if (hdl_cnt_mtx)
+    set_no_inheritance (hdl_cnt_mtx, val);
 }
 
 void
 fhandler_pipe::fixup_after_fork (HANDLE parent)
 {
+  if (hdl_cnt_mtx)
+    {
+      fork_fixup (parent, hdl_cnt_mtx, "hdl_cnt_mtx");
+      WaitForSingleObject (hdl_cnt_mtx, INFINITE);
+    }
   if (read_mtx)
     fork_fixup (parent, read_mtx, "read_mtx");
   if (select_sem)
@@ -567,6 +585,8 @@ fhandler_pipe::fixup_after_fork (HANDLE parent)
     fork_fixup (parent, query_hdl, "query_hdl");
 
   fhandler_base::fixup_after_fork (parent);
+  if (hdl_cnt_mtx)
+    ReleaseMutex (hdl_cnt_mtx);
 }
 
 int
@@ -576,6 +596,8 @@ fhandler_pipe::dup (fhandler_base *child, int flags)
   ftp->set_popen_pid (0);
 
   int res = 0;
+  if (hdl_cnt_mtx)
+    WaitForSingleObject (hdl_cnt_mtx, INFINITE);
   if (fhandler_base::dup (child, flags))
     res = -1;
   else if (read_mtx &&
@@ -589,8 +611,8 @@ fhandler_pipe::dup (fhandler_base *child, int flags)
     }
   else if (select_sem &&
 	   !DuplicateHandle (GetCurrentProcess (), select_sem,
-			    GetCurrentProcess (), &ftp->select_sem,
-			    0, !(flags & O_CLOEXEC), DUPLICATE_SAME_ACCESS))
+			     GetCurrentProcess (), &ftp->select_sem,
+			     0, !(flags & O_CLOEXEC), DUPLICATE_SAME_ACCESS))
     {
       __seterrno ();
       ftp->close ();
@@ -598,13 +620,24 @@ fhandler_pipe::dup (fhandler_base *child, int flags)
     }
   else if (query_hdl &&
 	   !DuplicateHandle (GetCurrentProcess (), query_hdl,
-			    GetCurrentProcess (), &ftp->query_hdl,
-			    0, !(flags & O_CLOEXEC), DUPLICATE_SAME_ACCESS))
+			     GetCurrentProcess (), &ftp->query_hdl,
+			     0, !(flags & O_CLOEXEC), DUPLICATE_SAME_ACCESS))
+    {
+      __seterrno ();
+      ftp->close ();
+      res = -1;
+    }
+  else if (hdl_cnt_mtx &&
+	   !DuplicateHandle (GetCurrentProcess (), hdl_cnt_mtx,
+			     GetCurrentProcess (), &ftp->hdl_cnt_mtx,
+			     0, !(flags & O_CLOEXEC), DUPLICATE_SAME_ACCESS))
     {
       __seterrno ();
       ftp->close ();
       res = -1;
     }
+  if (hdl_cnt_mtx)
+    ReleaseMutex (hdl_cnt_mtx);
 
   debug_printf ("res %d", res);
   return res;
@@ -620,9 +653,17 @@ fhandler_pipe::close ()
     }
   if (read_mtx)
     CloseHandle (read_mtx);
+  if (hdl_cnt_mtx)
+    WaitForSingleObject (hdl_cnt_mtx, INFINITE);
   if (query_hdl)
     CloseHandle (query_hdl);
-  return fhandler_base::close ();
+  int ret = fhandler_base::close ();
+  if (hdl_cnt_mtx)
+    {
+      ReleaseMutex (hdl_cnt_mtx);
+      CloseHandle (hdl_cnt_mtx);
+    }
+  return ret;
 }
 
 #define PIPE_INTRO "\\\\.\\pipe\\cygwin-"
@@ -834,9 +875,21 @@ fhandler_pipe::create (fhandler_pipe *fhs[2], unsigned psize, int mode)
 			FILE_READ_DATA, sa->bInheritHandle, 0))
     goto err_close_select_sem1;
 
+  fhs[0]->hdl_cnt_mtx = CreateMutexW (sa, FALSE, NULL);
+  if (!fhs[0]->hdl_cnt_mtx)
+    goto err_close_query_hdl;
+  if (!DuplicateHandle (GetCurrentProcess (), fhs[0]->hdl_cnt_mtx,
+			GetCurrentProcess (), &fhs[1]->hdl_cnt_mtx,
+			0, sa->bInheritHandle, DUPLICATE_SAME_ACCESS))
+    goto err_close_hdl_cnt_mtx0;
+
   res = 0;
   goto out;
 
+err_close_hdl_cnt_mtx0:
+  CloseHandle (fhs[0]->hdl_cnt_mtx);
+err_close_query_hdl:
+  CloseHandle (fhs[1]->query_hdl);
 err_close_select_sem1:
   CloseHandle (fhs[1]->select_sem);
 err_close_select_sem0:
-- 
2.33.0


^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-16 14:27                                                                                       ` Takashi Yano
@ 2021-09-16 15:01                                                                                         ` Corinna Vinschen
  2021-09-16 15:46                                                                                           ` Ken Brown
  2021-09-18 13:44                                                                                           ` Ken Brown
  0 siblings, 2 replies; 250+ messages in thread
From: Corinna Vinschen @ 2021-09-16 15:01 UTC (permalink / raw)
  To: cygwin-developers

On Sep 16 23:27, Takashi Yano wrote:
> Hi Corinna,
> 
> Thanks for reviewing the patches.
> 
> On Thu, 16 Sep 2021 15:25:46 +0200
> Corinna Vinschen wrote:
> > [...]
> > It would be great if open_setup() would be converted to a method which
> > is allowed to fail, rather than ignoring errors in sync object creation
> > and having to test the handle throughout the code.  Given there's only a
> > single caller of that function (dtable::init_std_file_from_handle), that
> > shouldn't be much work.  But it's certainly better than ignoring creation
> > failures in the long run.
> 
> I do not have confidence to do this correctly in a short time.
> Leave it as a homework. Or may I leave it to you?

That's not a short-term problem, but I think it would be good to do this
with a bit more paranoia at creation time and then rely on the handles
being available in the subsequent code.  Maybe I can get to it later but
it certainly wouldn't hurt my feelings if one of you guys look into it
while I'm abroad. :)

> I have attached the patches revised.

LGTM.  I leave it to Ken to take another look and push it.


Thanks,
Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-16 15:01                                                                                         ` Corinna Vinschen
@ 2021-09-16 15:46                                                                                           ` Ken Brown
  2021-09-16 16:02                                                                                             ` Ken Brown
  2021-09-18 13:44                                                                                           ` Ken Brown
  1 sibling, 1 reply; 250+ messages in thread
From: Ken Brown @ 2021-09-16 15:46 UTC (permalink / raw)
  To: cygwin-developers

On 9/16/2021 11:01 AM, Corinna Vinschen wrote:
> On Sep 16 23:27, Takashi Yano wrote:
>> On Thu, 16 Sep 2021 15:25:46 +0200
>> Corinna Vinschen wrote:
>>> [...]
>>> It would be great if open_setup() would be converted to a method which
>>> is allowed to fail, rather than ignoring errors in sync object creation
>>> and having to test the handle throughout the code.  Given there's only a
>>> single caller of that function (dtable::init_std_file_from_handle), that
>>> shouldn't be much work.  But it's certainly better than ignoring creation
>>> failures in the long run.
>>
>> I do not have confidence to do this correctly in a short time.
>> Leave it as a homework. Or may I leave it to you?
> 
> That's not a short-term problem, but I think it would be good to do this
> with a bit more paranoia at creation time and then rely on the handles
> being available in the subsequent code.  Maybe I can get to it later but
> it certainly wouldn't hurt my feelings if one of you guys look into it
> while I'm abroad. :)
> 
>> I have attached the patches revised.
> 
> LGTM.  I leave it to Ken to take another look and push it.

Done.

One minor thing I just noticed: If I'm not mistaken, query_hdl, hdl_cnt_mtx, and 
reader_closed() are used only for pipes, but they're defined in 
fhandler_pipe_fifo rather than fhandler_pipe.  Is there a reason for that?

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-16 15:46                                                                                           ` Ken Brown
@ 2021-09-16 16:02                                                                                             ` Ken Brown
  2021-09-16 19:42                                                                                               ` Takashi Yano
  2021-09-16 19:48                                                                                               ` Ken Brown
  0 siblings, 2 replies; 250+ messages in thread
From: Ken Brown @ 2021-09-16 16:02 UTC (permalink / raw)
  To: cygwin-developers

On 9/16/2021 11:46 AM, Ken Brown wrote:
> One minor thing I just noticed: If I'm not mistaken, query_hdl, hdl_cnt_mtx, and 
> reader_closed() are used only for pipes, but they're defined in 
> fhandler_pipe_fifo rather than fhandler_pipe.  Is there a reason for that?

Never mind.  I see the reason: reader_closed() is called by raw_write.

I'll wait about 24 hours to see if anything else turns up, and then I'll make a 
test release of Cygwin 3.3.0.

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-16 16:02                                                                                             ` Ken Brown
@ 2021-09-16 19:42                                                                                               ` Takashi Yano
  2021-09-16 20:28                                                                                                 ` Ken Brown
  2021-09-16 19:48                                                                                               ` Ken Brown
  1 sibling, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-16 19:42 UTC (permalink / raw)
  To: cygwin-developers

[-- Attachment #1: Type: text/plain, Size: 677 bytes --]

On Thu, 16 Sep 2021 12:02:04 -0400
Ken Brown wrote:
> On 9/16/2021 11:46 AM, Ken Brown wrote:
> > One minor thing I just noticed: If I'm not mistaken, query_hdl, hdl_cnt_mtx, and 
> > reader_closed() are used only for pipes, but they're defined in 
> > fhandler_pipe_fifo rather than fhandler_pipe.  Is there a reason for that?
> 
> Never mind.  I see the reason: reader_closed() is called by raw_write.

You are right. However, I think it is beter that only reader_closed()
is declared in fhandler_pipe_fifo as virtual, and query_hdl and
hdl_cnt_mtx are declared in fhandler_pipe.

Thank you for making me aware.

Patch attached.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

[-- Attachment #2: 0001-Cygwin-pipe-fifo-Move-query_hdl-and-hdl_cnt_mtx-to-f.patch --]
[-- Type: application/octet-stream, Size: 2338 bytes --]

From 6525876ff00197a3e5bbfa59773b19e54c3b7e57 Mon Sep 17 00:00:00 2001
From: Takashi Yano <takashi.yano@nifty.ne.jp>
Date: Fri, 17 Sep 2021 04:23:12 +0900
Subject: [PATCH] Cygwin: pipe, fifo: Move query_hdl and hdl_cnt_mtx to
 fhandler_pipe.

- query_hdl and hdl_cnt_mtx are moved from fhandler_pipe_fifo to
  fhandler_pipe. Then reader_closed() is changed to virtual and
  overridden in fhandler_pipe.
---
 winsup/cygwin/fhandler.h       | 27 +++++++++++++--------------
 winsup/cygwin/fhandler_pipe.cc |  2 +-
 2 files changed, 14 insertions(+), 15 deletions(-)

diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
index 31edb1822..61113e698 100644
--- a/winsup/cygwin/fhandler.h
+++ b/winsup/cygwin/fhandler.h
@@ -1175,26 +1175,13 @@ class fhandler_pipe_fifo: public fhandler_base
 {
  protected:
   size_t pipe_buf_size;
-  HANDLE query_hdl;
-  HANDLE hdl_cnt_mtx;
   virtual void release_select_sem (const char *) {};
 
  public:
   fhandler_pipe_fifo ();
 
-  HANDLE get_query_handle () const { return query_hdl; }
-  void close_query_handle ()
-  {
-    if (query_hdl)
-      {
-	CloseHandle (query_hdl);
-	query_hdl = NULL;
-      }
-  }
-  bool reader_closed ();
-
+  virtual bool reader_closed () { return false; };
   ssize_t __reg3 raw_write (const void *ptr, size_t len);
-
 };
 
 class fhandler_pipe: public fhandler_pipe_fifo
@@ -1202,6 +1189,8 @@ class fhandler_pipe: public fhandler_pipe_fifo
 private:
   HANDLE read_mtx;
   pid_t popen_pid;
+  HANDLE query_hdl;
+  HANDLE hdl_cnt_mtx;
   void release_select_sem (const char *);
 public:
   fhandler_pipe ();
@@ -1250,6 +1239,16 @@ public:
     return fh;
   }
   void set_pipe_non_blocking (bool nonblocking);
+  HANDLE get_query_handle () const { return query_hdl; }
+  void close_query_handle ()
+  {
+    if (query_hdl)
+      {
+	CloseHandle (query_hdl);
+	query_hdl = NULL;
+      }
+  }
+  bool reader_closed ();
 };
 
 #define CYGWIN_FIFO_PIPE_NAME_LEN     47
diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index 2068a943e..73ace3ac5 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -393,7 +393,7 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
 }
 
 bool
-fhandler_pipe_fifo::reader_closed ()
+fhandler_pipe::reader_closed ()
 {
   if (!query_hdl)
     return false;
-- 
2.33.0


^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-16 16:02                                                                                             ` Ken Brown
  2021-09-16 19:42                                                                                               ` Takashi Yano
@ 2021-09-16 19:48                                                                                               ` Ken Brown
  2021-09-16 20:01                                                                                                 ` Takashi Yano
  1 sibling, 1 reply; 250+ messages in thread
From: Ken Brown @ 2021-09-16 19:48 UTC (permalink / raw)
  To: cygwin-developers

On 9/16/2021 12:02 PM, Ken Brown wrote:
> I'll wait about 24 hours to see if anything else turns up, and then I'll make a 
> test release of Cygwin 3.3.0.

I just noticed something very surprising.  Up to now, all my tests have been on 
64-bit.  But I just tested the current git head on 32-bit Cygwin, and found that 
mintty is very slow to start.  If I click on the Cygwin Terminal shortcut, 
there's a long delay (about 10 seconds on my system) before the window appears. 
  Moreover, the window doesn't have focus when it does appear.

Can anyone else reproduce this?

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-16 19:48                                                                                               ` Ken Brown
@ 2021-09-16 20:01                                                                                                 ` Takashi Yano
  2021-09-17  2:25                                                                                                   ` Ken Brown
  0 siblings, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-16 20:01 UTC (permalink / raw)
  To: cygwin-developers

On Thu, 16 Sep 2021 15:48:58 -0400
Ken Brown wrote:
> I just noticed something very surprising.  Up to now, all my tests have been on 
> 64-bit.  But I just tested the current git head on 32-bit Cygwin, and found that 
> mintty is very slow to start.  If I click on the Cygwin Terminal shortcut, 
> there's a long delay (about 10 seconds on my system) before the window appears. 
>   Moreover, the window doesn't have focus when it does appear.
> 
> Can anyone else reproduce this?

I cannot reproduce that.
Proberbly 'strace -o mintty.log mintty' may help to find
what is wrong. 

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-16 19:42                                                                                               ` Takashi Yano
@ 2021-09-16 20:28                                                                                                 ` Ken Brown
  0 siblings, 0 replies; 250+ messages in thread
From: Ken Brown @ 2021-09-16 20:28 UTC (permalink / raw)
  To: cygwin-developers

On 9/16/2021 3:42 PM, Takashi Yano wrote:
> On Thu, 16 Sep 2021 12:02:04 -0400
> Ken Brown wrote:
>> On 9/16/2021 11:46 AM, Ken Brown wrote:
>>> One minor thing I just noticed: If I'm not mistaken, query_hdl, hdl_cnt_mtx, and
>>> reader_closed() are used only for pipes, but they're defined in
>>> fhandler_pipe_fifo rather than fhandler_pipe.  Is there a reason for that?
>>
>> Never mind.  I see the reason: reader_closed() is called by raw_write.
> 
> You are right. However, I think it is beter that only reader_closed()
> is declared in fhandler_pipe_fifo as virtual, and query_hdl and
> hdl_cnt_mtx are declared in fhandler_pipe.
> 
> Thank you for making me aware.
> 
> Patch attached.

I agree that this is better.  Pushed.

Thanks.

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-16 20:01                                                                                                 ` Takashi Yano
@ 2021-09-17  2:25                                                                                                   ` Ken Brown
  2021-09-17  8:31                                                                                                     ` Takashi Yano
  0 siblings, 1 reply; 250+ messages in thread
From: Ken Brown @ 2021-09-17  2:25 UTC (permalink / raw)
  To: cygwin-developers

On 9/16/2021 4:01 PM, Takashi Yano wrote:
> On Thu, 16 Sep 2021 15:48:58 -0400
> Ken Brown wrote:
>> I just noticed something very surprising.  Up to now, all my tests have been on
>> 64-bit.  But I just tested the current git head on 32-bit Cygwin, and found that
>> mintty is very slow to start.  If I click on the Cygwin Terminal shortcut,
>> there's a long delay (about 10 seconds on my system) before the window appears.
>>    Moreover, the window doesn't have focus when it does appear.
>>
>> Can anyone else reproduce this?
> 
> I cannot reproduce that.
> Proberbly 'strace -o mintty.log mintty' may help to find
> what is wrong.

This is getting stranger and stranger.  When I try to run strace, I get a pop-up 
complaining that libgcc_s_sjlj-1.dll was not found.

I'll have to try to sort it out tomorrow.

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-17  2:25                                                                                                   ` Ken Brown
@ 2021-09-17  8:31                                                                                                     ` Takashi Yano
  2021-09-17 11:16                                                                                                       ` Ken Brown
  0 siblings, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-17  8:31 UTC (permalink / raw)
  To: cygwin-developers

On Thu, 16 Sep 2021 22:25:33 -0400
Ken Brown wrote:
> On 9/16/2021 4:01 PM, Takashi Yano wrote:
> > On Thu, 16 Sep 2021 15:48:58 -0400
> > Ken Brown wrote:
> >> I just noticed something very surprising.  Up to now, all my tests have been on
> >> 64-bit.  But I just tested the current git head on 32-bit Cygwin, and found that
> >> mintty is very slow to start.  If I click on the Cygwin Terminal shortcut,
> >> there's a long delay (about 10 seconds on my system) before the window appears.
> >>    Moreover, the window doesn't have focus when it does appear.
> >>
> >> Can anyone else reproduce this?
> > 
> > I cannot reproduce that.
> > Proberbly 'strace -o mintty.log mintty' may help to find
> > what is wrong.
> 
> This is getting stranger and stranger.  When I try to run strace, I get a pop-up 
> complaining that libgcc_s_sjlj-1.dll was not found.
> 
> I'll have to try to sort it out tomorrow.

32bit mintty seems to require only the following files.

bash.exe
cyggcc_s-1.dll
cygiconv-2.dll
cygintl-8.dll
cygncursesw-10.dll
cygreadline7.dll
cygwin1.dll
cygwin-console-helper.exe
mintty.exe

None of them depends on libgcc_s_sjlj-1.dll.

What happes if you put them into a new directory and run
mintty in it?

'strace mintty' also should work with these minimum files.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-17  8:31                                                                                                     ` Takashi Yano
@ 2021-09-17 11:16                                                                                                       ` Ken Brown
  2021-09-17 16:23                                                                                                         ` Takashi Yano
  0 siblings, 1 reply; 250+ messages in thread
From: Ken Brown @ 2021-09-17 11:16 UTC (permalink / raw)
  To: cygwin-developers

[-- Attachment #1: Type: text/plain, Size: 2142 bytes --]

On 9/17/2021 4:31 AM, Takashi Yano wrote:
> On Thu, 16 Sep 2021 22:25:33 -0400
> Ken Brown wrote:
>> On 9/16/2021 4:01 PM, Takashi Yano wrote:
>>> On Thu, 16 Sep 2021 15:48:58 -0400
>>> Ken Brown wrote:
>>>> I just noticed something very surprising.  Up to now, all my tests have been on
>>>> 64-bit.  But I just tested the current git head on 32-bit Cygwin, and found that
>>>> mintty is very slow to start.  If I click on the Cygwin Terminal shortcut,
>>>> there's a long delay (about 10 seconds on my system) before the window appears.
>>>>     Moreover, the window doesn't have focus when it does appear.
>>>>
>>>> Can anyone else reproduce this?
>>>
>>> I cannot reproduce that.
>>> Proberbly 'strace -o mintty.log mintty' may help to find
>>> what is wrong.
>>
>> This is getting stranger and stranger.  When I try to run strace, I get a pop-up
>> complaining that libgcc_s_sjlj-1.dll was not found.
>>
>> I'll have to try to sort it out tomorrow.
> 
> 32bit mintty seems to require only the following files.
> 
> bash.exe
> cyggcc_s-1.dll
> cygiconv-2.dll
> cygintl-8.dll
> cygncursesw-10.dll
> cygreadline7.dll
> cygwin1.dll
> cygwin-console-helper.exe
> mintty.exe
> 
> None of them depends on libgcc_s_sjlj-1.dll.
> 
> What happes if you put them into a new directory and run
> mintty in it?
> 
> 'strace mintty' also should work with these minimum files.

Sorry, I was unclear.  The pop-up wasn't saying that mintty needs 
libgcc_s_sjlj-1.dll; it was saying that strace needs it.  Even 'strace -h' 
causes that pop-up to appear.  In order to get strace to run, I have to copy 
libgcc_s_sjlj-1.dll and libwinpthread-1.dll to my CWD.  Then I run strace on 
mintty and it starts up instantly, and the strace output shows both of those 
libs being loaded.

One other thing I should mention is that for my testing, I did a complete 
install of the cygwin package (as produced by cygport), not just cygwin1.dll. 
My cygport file is attached.

I have a feeling that something is wrong with my 32-bit installation, perhaps as 
a result of prior testing.  I'm going to start fresh with a new 32-bit 
installation and see what happens.

Ken

[-- Attachment #2: cygwin.cygport --]
[-- Type: text/plain, Size: 3562 bytes --]

NAME="cygwin"
VERSION="3.3.0"
RELEASE="0.1.199482654b07"
HOMEPAGE="https://cygwin.com/"

GIT_URI="https://cygwin.com/git/newlib-cygwin.git"
# Release
#GIT_TAG="cygwin-${PV//\./_}-release"
# Test
GIT_REV="199482654b07"
inherit git

TOOLCHAIN_TARGET="native"
inherit toolchain

# From .appveyor.yml in Cygwin git repo
BUILD_REQUIRES="\
gcc-core \
gcc-g++ \
autoconf \
automake \
make \
perl \
patch \
cocom \
gettext-devel \
libiconv-devel \
zlib-devel \
mingw64-${ARCH}-gcc-core \
mingw64-${ARCH}-gcc-g++ \
mingw64-${ARCH}-zlib \
dejagnu \
dblatex \
docbook2X \
docbook-xml45 \
docbook-xsl \
texlive-collection-latexrecommended \
texlive-collection-fontsrecommended \
texlive-collection-pictures \
xmlto \
python38-lxml \
python38-ply"

PKG_NAMES="cygwin cygwin-devel cygwin-doc"

# cygwin base package

cygwin_CATEGORY="Base"
cygwin_SUMMARY="The UNIX emulation engine"
cygwin_DESCRIPTION="The UNIX emulation engine"
cygwin_CONTENTS="--exclude=cygwin1.dbg
		 etc/defaults/
		 usr/bin/
		 usr/sbin/
		 usr/share/cygwin/
		 usr/share/doc/Cygwin/"

# cygwin-devel package

cygwin_devel_CATEGORY="Devel"
cygwin_devel_SUMMARY="Core development files"
cygwin_devel_DESCRIPTION="Core development files required to build Cygwin packages"
cygwin_devel_CONTENTS="usr/include/ usr/lib/"

# cygwin doc package

cygwin_doc_CATEGORY="Doc"
cygwin_doc_SUMMARY="Cygwin-specific documentation, including man pages"
cygwin_doc_DESCRIPTION="Documentation for Cygwin and newlib, including man
pages and texinfo files."
cygwin_doc_CONTENTS="usr/share/man/ usr/share/info/ usr/share/doc/cygwin-doc/ usr/share/doc/newlib/ etc/postinstall/ etc/preremove/"

PKG_IGNORE="usr/bin/cygwin1.dbg"

DIFF_EXCLUDES="libc.info libm.info parser.out parsetab.py"

# This is needed since we otherwise end up with two dirs in usr/share/doc
# which only differ by case, "cygwin" and "Cygwin".  That's not feasible for
# the distro.  Therefore we suppress generation of the default docs and create
# our own cygwin-${PV} dir for the docs in src_install.
RESTRICT="postinst-doc"
CYGCONF_ARGS="--docdir=/usr/share/doc/cygwin-doc
	      --htmldir=\${docdir}/html"

src_compile() {
  cd ${S}
  winsup/autogen.sh
  cd ${B}
  # cygport compiles with _FORTIFY_SOURCE=2 -fstack-protector by default.
  # This doesn't work for the lib itself
  CFLAGS+=" -Wp,-U_FORTIFY_SOURCE -fno-stack-protector"
  CXXFLAGS+=" -Wp,-U_FORTIFY_SOURCE -fno-stack-protector"
  cygconf
  cygmake

  # make newlib info and manpage documentation
  cd ${B}/${CHOST}/newlib
  cygmake info man
}

src_install() {
  cd ${B}
  # tooldir and gcc_tooldir must be set to install correctly even when
  # building on a non-Cygwin build machine.
  cyginstall -j1 tooldir=/usr gcc_tooldir=/usr

  # Workaround the problem that the Cygwin DLL is already stripped and
  # a cygwin1.dbg file exists.
  insinto /usr/lib/debug/usr/bin
  doins ${B}/${ARCH}-pc-cygwin/winsup/cygwin/cygwin1.dbg
  dosym ../lib/debug/usr/bin/cygwin1.dbg /usr/bin/cygwin1.dbg

  # install newlib info and manpage documentation
  cd ${B}/${CHOST}/newlib
  cygmake DESTDIR=${D} install-info install-man

  # cygwin-doc postinstall
  cd ${I}
  cd ${B}/${CHOST}/winsup/doc
  cygmake DESTDIR=${D} install-etc

  cd ${S}
  docinto /cygwin-doc
  dodoc winsup/COPYING winsup/CYGWIN_LICENSE README

  docinto /newlib
  dodoc COPYING.NEWLIB newlib/HOWTO newlib/NEWS newlib/README

  cd ${D}
  rm -rf usr/include/iconv.h usr/include/unctrl.h usr/include/rpc \
	 usr/lib/libiberty.a usr/share/locale \
	 usr/share/info/standards.info usr/share/info/configure.info
}

SCALLYWAG="nobuild"

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-17 11:16                                                                                                       ` Ken Brown
@ 2021-09-17 16:23                                                                                                         ` Takashi Yano
  2021-09-17 17:08                                                                                                           ` Ken Brown
  0 siblings, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-17 16:23 UTC (permalink / raw)
  To: cygwin-developers

On Fri, 17 Sep 2021 07:16:03 -0400
Ken Brown wrote:
> Sorry, I was unclear.  The pop-up wasn't saying that mintty needs 
> libgcc_s_sjlj-1.dll; it was saying that strace needs it.  Even 'strace -h' 
> causes that pop-up to appear.  In order to get strace to run, I have to copy 
> libgcc_s_sjlj-1.dll and libwinpthread-1.dll to my CWD.  Then I run strace on 
> mintty and it starts up instantly, and the strace output shows both of those 
> libs being loaded.
> 
> One other thing I should mention is that for my testing, I did a complete 
> install of the cygwin package (as produced by cygport), not just cygwin1.dll. 
> My cygport file is attached.
> 
> I have a feeling that something is wrong with my 32-bit installation, perhaps as 
> a result of prior testing.  I'm going to start fresh with a new 32-bit 
> installation and see what happens.

I can reproduce your problem by replacing cygwin-console-helper.exe
to newly build one.

With cygwin-console-helper.exe of cygwin 3.2.0,
ldd /bin/cygwin-console-helper.exe shows
        ntdll.dll => /cygdrive/c/WINDOWS/SYSTEM32/ntdll.dll (0x77460000)
        KERNEL32.DLL => /cygdrive/c/WINDOWS/System32/KERNEL32.DLL (0x76ef0000)
        KERNELBASE.dll => /cygdrive/c/WINDOWS/System32/KERNELBASE.dll (0x75f10000)
        msvcrt.dll => /cygdrive/c/WINDOWS/System32/msvcrt.dll (0x75640000)

However, with newlib-built one,
ldd /bin/cygwin-console-helper.exe shows
        ntdll.dll => /cygdrive/c/WINDOWS/SYSTEM32/ntdll.dll (0x77460000)
        KERNEL32.DLL => /cygdrive/c/WINDOWS/System32/KERNEL32.DLL (0x76ef0000)
        KERNELBASE.dll => /cygdrive/c/WINDOWS/System32/KERNELBASE.dll (0x75f10000)
        msvcrt.dll => /cygdrive/c/WINDOWS/System32/msvcrt.dll (0x75640000)
        libgcc_s_sjlj-1.dll => not found

It seems to depend on libgcc_s_sjlj-1.dll and libwinpthread-1.dll.

It seems that the tools under winsup/utils/mingw get dynamically linked
while old ones are statically linked.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-17 16:23                                                                                                         ` Takashi Yano
@ 2021-09-17 17:08                                                                                                           ` Ken Brown
  2021-09-17 17:39                                                                                                             ` Jon Turney
  2021-09-17 17:43                                                                                                             ` Takashi Yano
  0 siblings, 2 replies; 250+ messages in thread
From: Ken Brown @ 2021-09-17 17:08 UTC (permalink / raw)
  To: cygwin-developers

On 9/17/2021 12:23 PM, Takashi Yano wrote:
> On Fri, 17 Sep 2021 07:16:03 -0400
> Ken Brown wrote:
>> Sorry, I was unclear.  The pop-up wasn't saying that mintty needs
>> libgcc_s_sjlj-1.dll; it was saying that strace needs it.  Even 'strace -h'
>> causes that pop-up to appear.  In order to get strace to run, I have to copy
>> libgcc_s_sjlj-1.dll and libwinpthread-1.dll to my CWD.  Then I run strace on
>> mintty and it starts up instantly, and the strace output shows both of those
>> libs being loaded.
>>
>> One other thing I should mention is that for my testing, I did a complete
>> install of the cygwin package (as produced by cygport), not just cygwin1.dll.
>> My cygport file is attached.
>>
>> I have a feeling that something is wrong with my 32-bit installation, perhaps as
>> a result of prior testing.  I'm going to start fresh with a new 32-bit
>> installation and see what happens.
> 
> I can reproduce your problem by replacing cygwin-console-helper.exe
> to newly build one.
> 
> With cygwin-console-helper.exe of cygwin 3.2.0,
> ldd /bin/cygwin-console-helper.exe shows
>          ntdll.dll => /cygdrive/c/WINDOWS/SYSTEM32/ntdll.dll (0x77460000)
>          KERNEL32.DLL => /cygdrive/c/WINDOWS/System32/KERNEL32.DLL (0x76ef0000)
>          KERNELBASE.dll => /cygdrive/c/WINDOWS/System32/KERNELBASE.dll (0x75f10000)
>          msvcrt.dll => /cygdrive/c/WINDOWS/System32/msvcrt.dll (0x75640000)
> 
> However, with newlib-built one,
> ldd /bin/cygwin-console-helper.exe shows
>          ntdll.dll => /cygdrive/c/WINDOWS/SYSTEM32/ntdll.dll (0x77460000)
>          KERNEL32.DLL => /cygdrive/c/WINDOWS/System32/KERNEL32.DLL (0x76ef0000)
>          KERNELBASE.dll => /cygdrive/c/WINDOWS/System32/KERNELBASE.dll (0x75f10000)
>          msvcrt.dll => /cygdrive/c/WINDOWS/System32/msvcrt.dll (0x75640000)
>          libgcc_s_sjlj-1.dll => not found
> 
> It seems to depend on libgcc_s_sjlj-1.dll and libwinpthread-1.dll.
> 
> It seems that the tools under winsup/utils/mingw get dynamically linked
> while old ones are statically linked.

This even happens on 64 bit for at least one tool...

$ ldd /bin/cygcheck.exe
         ntdll.dll => /c/WINDOWS/SYSTEM32/ntdll.dll (0x7ff8046b0000)
         KERNEL32.DLL => /c/WINDOWS/System32/KERNEL32.DLL (0x7ff802e60000)
         KERNELBASE.dll => /c/WINDOWS/System32/KERNELBASE.dll (0x7ff801f30000)
         ADVAPI32.dll => /c/WINDOWS/System32/ADVAPI32.dll (0x7ff802d80000)
         msvcrt.dll => /c/WINDOWS/System32/msvcrt.dll (0x7ff803a90000)
         sechost.dll => /c/WINDOWS/System32/sechost.dll (0x7ff803e20000)
         RPCRT4.dll => /c/WINDOWS/System32/RPCRT4.dll (0x7ff803b30000)
         PSAPI.DLL => /c/WINDOWS/System32/PSAPI.DLL (0x7ff803a70000)
         USER32.dll => /c/WINDOWS/System32/USER32.dll (0x7ff8030c0000)
         win32u.dll => /c/WINDOWS/System32/win32u.dll (0x7ff802540000)
         GDI32.dll => /c/WINDOWS/System32/GDI32.dll (0x7ff803030000)
         gdi32full.dll => /c/WINDOWS/System32/gdi32full.dll (0x7ff802400000)
         msvcp_win.dll => /c/WINDOWS/System32/msvcp_win.dll (0x7ff802200000)
         ucrtbase.dll => /c/WINDOWS/System32/ucrtbase.dll (0x7ff802600000)
         WININET.dll => /c/WINDOWS/SYSTEM32/WININET.dll (0x7fffea2e0000)
         zlib1.dll => not found

...but not for all...

$ ldd cygwin-console-helper.exe
         ntdll.dll => /c/WINDOWS/SYSTEM32/ntdll.dll (0x7ff8046b0000)
         KERNEL32.DLL => /c/WINDOWS/System32/KERNEL32.DLL (0x7ff802e60000)
         KERNELBASE.dll => /c/WINDOWS/System32/KERNELBASE.dll (0x7ff801f30000)
         msvcrt.dll => /c/WINDOWS/System32/msvcrt.dll (0x7ff803a90000)

Are we just missing AM_LDFLAGS containing "-static" in 
winsup/utils/mingw/Makefile.am?  But why is this just affecting some tools, and 
why the difference between 32 bit and 64 bit?

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-17 17:08                                                                                                           ` Ken Brown
@ 2021-09-17 17:39                                                                                                             ` Jon Turney
  2021-09-17 17:43                                                                                                             ` Takashi Yano
  1 sibling, 0 replies; 250+ messages in thread
From: Jon Turney @ 2021-09-17 17:39 UTC (permalink / raw)
  To: cygwin-developers

On 17/09/2021 18:08, Ken Brown wrote:
> On 9/17/2021 12:23 PM, Takashi Yano wrote:
>> On Fri, 17 Sep 2021 07:16:03 -0400
>> Ken Brown wrote:
>>> Sorry, I was unclear.  The pop-up wasn't saying that mintty needs
>>> libgcc_s_sjlj-1.dll; it was saying that strace needs it.  Even 
>>> 'strace -h'
>>> causes that pop-up to appear.  In order to get strace to run, I have 
>>> to copy
>>> libgcc_s_sjlj-1.dll and libwinpthread-1.dll to my CWD.  Then I run 
>>> strace on
>>> mintty and it starts up instantly, and the strace output shows both 
>>> of those
>>> libs being loaded.
>>>
>>> One other thing I should mention is that for my testing, I did a 
>>> complete
>>> install of the cygwin package (as produced by cygport), not just 
>>> cygwin1.dll.
>>> My cygport file is attached.
>>>
>>> I have a feeling that something is wrong with my 32-bit installation, 
>>> perhaps as
>>> a result of prior testing.  I'm going to start fresh with a new 32-bit
>>> installation and see what happens.
>>
>> I can reproduce your problem by replacing cygwin-console-helper.exe
>> to newly build one.
>>
>> With cygwin-console-helper.exe of cygwin 3.2.0,
>> ldd /bin/cygwin-console-helper.exe shows
>>          ntdll.dll => /cygdrive/c/WINDOWS/SYSTEM32/ntdll.dll (0x77460000)
>>          KERNEL32.DLL => /cygdrive/c/WINDOWS/System32/KERNEL32.DLL 
>> (0x76ef0000)
>>          KERNELBASE.dll => /cygdrive/c/WINDOWS/System32/KERNELBASE.dll 
>> (0x75f10000)
>>          msvcrt.dll => /cygdrive/c/WINDOWS/System32/msvcrt.dll 
>> (0x75640000)
>>
>> However, with newlib-built one,
>> ldd /bin/cygwin-console-helper.exe shows
>>          ntdll.dll => /cygdrive/c/WINDOWS/SYSTEM32/ntdll.dll (0x77460000)
>>          KERNEL32.DLL => /cygdrive/c/WINDOWS/System32/KERNEL32.DLL 
>> (0x76ef0000)
>>          KERNELBASE.dll => /cygdrive/c/WINDOWS/System32/KERNELBASE.dll 
>> (0x75f10000)
>>          msvcrt.dll => /cygdrive/c/WINDOWS/System32/msvcrt.dll 
>> (0x75640000)
>>          libgcc_s_sjlj-1.dll => not found
>>
>> It seems to depend on libgcc_s_sjlj-1.dll and libwinpthread-1.dll.
>>
>> It seems that the tools under winsup/utils/mingw get dynamically linked
>> while old ones are statically linked.
> 
> This even happens on 64 bit for at least one tool...
> 
> $ ldd /bin/cygcheck.exe
>          ntdll.dll => /c/WINDOWS/SYSTEM32/ntdll.dll (0x7ff8046b0000)
>          KERNEL32.DLL => /c/WINDOWS/System32/KERNEL32.DLL (0x7ff802e60000)
>          KERNELBASE.dll => /c/WINDOWS/System32/KERNELBASE.dll 
> (0x7ff801f30000)
>          ADVAPI32.dll => /c/WINDOWS/System32/ADVAPI32.dll (0x7ff802d80000)
>          msvcrt.dll => /c/WINDOWS/System32/msvcrt.dll (0x7ff803a90000)
>          sechost.dll => /c/WINDOWS/System32/sechost.dll (0x7ff803e20000)
>          RPCRT4.dll => /c/WINDOWS/System32/RPCRT4.dll (0x7ff803b30000)
>          PSAPI.DLL => /c/WINDOWS/System32/PSAPI.DLL (0x7ff803a70000)
>          USER32.dll => /c/WINDOWS/System32/USER32.dll (0x7ff8030c0000)
>          win32u.dll => /c/WINDOWS/System32/win32u.dll (0x7ff802540000)
>          GDI32.dll => /c/WINDOWS/System32/GDI32.dll (0x7ff803030000)
>          gdi32full.dll => /c/WINDOWS/System32/gdi32full.dll 
> (0x7ff802400000)
>          msvcp_win.dll => /c/WINDOWS/System32/msvcp_win.dll 
> (0x7ff802200000)
>          ucrtbase.dll => /c/WINDOWS/System32/ucrtbase.dll (0x7ff802600000)
>          WININET.dll => /c/WINDOWS/SYSTEM32/WININET.dll (0x7fffea2e0000)
>          zlib1.dll => not found
> 
> ...but not for all...
> 
> $ ldd cygwin-console-helper.exe
>          ntdll.dll => /c/WINDOWS/SYSTEM32/ntdll.dll (0x7ff8046b0000)
>          KERNEL32.DLL => /c/WINDOWS/System32/KERNEL32.DLL (0x7ff802e60000)
>          KERNELBASE.dll => /c/WINDOWS/System32/KERNELBASE.dll 
> (0x7ff801f30000)
>          msvcrt.dll => /c/WINDOWS/System32/msvcrt.dll (0x7ff803a90000)
> 
> Are we just missing AM_LDFLAGS containing "-static" in 
> winsup/utils/mingw/Makefile.am?  But why is this just affecting some 

Yes, it seems I missed that when doing the automake conversion.

Apologies for the inconvenience.

> tools, and why the difference between 32 bit and 64 bit?

I believe that on 64-bit, SEH exception handling is being used, which 
doesn't have a personality DLL.

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-17 17:08                                                                                                           ` Ken Brown
  2021-09-17 17:39                                                                                                             ` Jon Turney
@ 2021-09-17 17:43                                                                                                             ` Takashi Yano
  2021-09-17 19:53                                                                                                               ` Ken Brown
  1 sibling, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-17 17:43 UTC (permalink / raw)
  To: cygwin-developers

On Fri, 17 Sep 2021 13:08:35 -0400
Ken Brown wrote:
> On 9/17/2021 12:23 PM, Takashi Yano wrote:
> > With cygwin-console-helper.exe of cygwin 3.2.0,
> > ldd /bin/cygwin-console-helper.exe shows
> >          ntdll.dll => /cygdrive/c/WINDOWS/SYSTEM32/ntdll.dll (0x77460000)
> >          KERNEL32.DLL => /cygdrive/c/WINDOWS/System32/KERNEL32.DLL (0x76ef0000)
> >          KERNELBASE.dll => /cygdrive/c/WINDOWS/System32/KERNELBASE.dll (0x75f10000)
> >          msvcrt.dll => /cygdrive/c/WINDOWS/System32/msvcrt.dll (0x75640000)
> > 
> > However, with newlib-built one,
> > ldd /bin/cygwin-console-helper.exe shows
> >          ntdll.dll => /cygdrive/c/WINDOWS/SYSTEM32/ntdll.dll (0x77460000)
> >          KERNEL32.DLL => /cygdrive/c/WINDOWS/System32/KERNEL32.DLL (0x76ef0000)
> >          KERNELBASE.dll => /cygdrive/c/WINDOWS/System32/KERNELBASE.dll (0x75f10000)
> >          msvcrt.dll => /cygdrive/c/WINDOWS/System32/msvcrt.dll (0x75640000)
> >          libgcc_s_sjlj-1.dll => not found
> > 
> > It seems to depend on libgcc_s_sjlj-1.dll and libwinpthread-1.dll.
> > 
> > It seems that the tools under winsup/utils/mingw get dynamically linked
> > while old ones are statically linked.
> 
> Are we just missing AM_LDFLAGS containing "-static" in 
> winsup/utils/mingw/Makefile.am?  But why is this just affecting some tools, and 
> why the difference between 32 bit and 64 bit?

Adding 
AM_LDFLAGS = -static
in winsup/utils/mingw/Makefile.am
and rerun winsup/autogen.sh resolves the issue.

Just compiling the hello.cc:
#include <cstdio>
int main()
{
        printf("Hello\n");
        return 0;
}
causes same behaiviour.

With 32bit mingw compiler,
        ntdll.dll => /cygdrive/c/WINDOWS/SYSTEM32/ntdll.dll (0x77460000)
        KERNEL32.DLL => /cygdrive/c/WINDOWS/System32/KERNEL32.DLL (0x76ef0000)
        KERNELBASE.dll => /cygdrive/c/WINDOWS/System32/KERNELBASE.dll (0x75f10000)
        msvcrt.dll => /cygdrive/c/WINDOWS/System32/msvcrt.dll (0x75640000)
        libgcc_s_sjlj-1.dll => not found

With 64bit mingw compiler,
        ntdll.dll => /cygdrive/c/WINDOWS/SYSTEM32/ntdll.dll (0x7ffc56330000)
        KERNEL32.DLL => /cygdrive/c/WINDOWS/System32/KERNEL32.DLL (0x7ffc54460000)
        KERNELBASE.dll => /cygdrive/c/WINDOWS/System32/KERNELBASE.dll (0x7ffc53a50000)
        msvcrt.dll => /cygdrive/c/WINDOWS/System32/msvcrt.dll (0x7ffc54800000)

I also confirmed that mingw compiler (v10.3.0) from MSYS2
have such difference between 32bit and 64bit compiler.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-17 17:43                                                                                                             ` Takashi Yano
@ 2021-09-17 19:53                                                                                                               ` Ken Brown
  2021-09-18  1:30                                                                                                                 ` Takashi Yano
  0 siblings, 1 reply; 250+ messages in thread
From: Ken Brown @ 2021-09-17 19:53 UTC (permalink / raw)
  To: cygwin-developers

On 9/17/2021 1:43 PM, Takashi Yano wrote:
> Adding
> AM_LDFLAGS = -static
> in winsup/utils/mingw/Makefile.am
> and rerun winsup/autogen.sh resolves the issue.

I've made that change and uploaded a test release.  Interestingly, the upload 
process itself revealed a bug:

$ cygport *.cygport up
/usr/bin/cygport: line 426: echo: write error: Broken pipe
/usr/bin/cygport: line 426: echo: write error: Broken pipe
/usr/bin/cygport: line 426: echo: write error: Broken pipe
/usr/bin/cygport: line 426: echo: write error: Broken pipe
*** Info: Building native toolchain for x86_64-pc-cygwin host
/usr/bin/cygport: line 426: echo: write error: Broken pipe
/usr/bin/cygport: line 426: echo: write error: Broken pipe
 >>> Uploading cygwin-3.3.0-0.1.9814cfd8f693.x86_64
[...]

This happens in an xterm window, but not in mintty.

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-17 19:53                                                                                                               ` Ken Brown
@ 2021-09-18  1:30                                                                                                                 ` Takashi Yano
  2021-09-18  2:07                                                                                                                   ` Ken Brown
  0 siblings, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-18  1:30 UTC (permalink / raw)
  To: cygwin-developers

On Fri, 17 Sep 2021 15:53:18 -0400
Ken Brown wrote:
> On 9/17/2021 1:43 PM, Takashi Yano wrote:
> > Adding
> > AM_LDFLAGS = -static
> > in winsup/utils/mingw/Makefile.am
> > and rerun winsup/autogen.sh resolves the issue.
> 
> I've made that change and uploaded a test release.  Interestingly, the upload 
> process itself revealed a bug:
> 
> $ cygport *.cygport up
> /usr/bin/cygport: line 426: echo: write error: Broken pipe
> /usr/bin/cygport: line 426: echo: write error: Broken pipe
> /usr/bin/cygport: line 426: echo: write error: Broken pipe
> /usr/bin/cygport: line 426: echo: write error: Broken pipe
> *** Info: Building native toolchain for x86_64-pc-cygwin host
> /usr/bin/cygport: line 426: echo: write error: Broken pipe
> /usr/bin/cygport: line 426: echo: write error: Broken pipe
>  >>> Uploading cygwin-3.3.0-0.1.9814cfd8f693.x86_64
> [...]
> 
> This happens in an xterm window, but not in mintty.

I have installed cygwin 3.3.0 [TEST].
And I tryed cygpoort upload, but I cannot reproduce your problem
even in xterm window.

[yano@Express5800-S70 ~/inetutils-release/inetutils-2.2-1.src]$ cygport inetutils.cygport upload
>>> Uploading inetutils-2.2-1.i686
>>> Running lftp sftp://cygwin@cygwin.com
Password: 
cd ok, cwd=/x86/release
Transferring file `inetutils-2.2-1-src.hint'
Transferring file `inetutils-2.2-1-src.tar.xz'
Transferring file `inetutils-2.2-1.hint'
Transferring file `inetutils-2.2-1.tar.xz'
Transferring file `inetutils-debuginfo/inetutils-debuginfo-2.2-1.hint'
Transferring file `inetutils-debuginfo/inetutils-debuginfo-2.2-1.tar.xz'
Transferring file `inetutils-server/inetutils-server-2.2-1.hint'
Transferring file `inetutils-server/inetutils-server-2.2-1.tar.xz'
Transferring file `ping/ping-2.2-1.hint'
Transferring file `ping/ping-2.2-1.tar.xz'
Total: 3 directories, 10 files, 0 symlinks
New: 10 files, 0 symlinks
3426384 bytes transferred in 9 seconds (370.5 KiB/s)
Upload complete.
[yano@Express5800-S70 ~/inetutils-release/inetutils-2.2-1.src]$ 

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-18  1:30                                                                                                                 ` Takashi Yano
@ 2021-09-18  2:07                                                                                                                   ` Ken Brown
  2021-09-18  2:10                                                                                                                     ` Ken Brown
  0 siblings, 1 reply; 250+ messages in thread
From: Ken Brown @ 2021-09-18  2:07 UTC (permalink / raw)
  To: cygwin-developers

On 9/17/2021 9:30 PM, Takashi Yano wrote:
> On Fri, 17 Sep 2021 15:53:18 -0400
> Ken Brown wrote:
>> $ cygport *.cygport up
>> /usr/bin/cygport: line 426: echo: write error: Broken pipe
>> /usr/bin/cygport: line 426: echo: write error: Broken pipe
>> /usr/bin/cygport: line 426: echo: write error: Broken pipe
>> /usr/bin/cygport: line 426: echo: write error: Broken pipe
>> *** Info: Building native toolchain for x86_64-pc-cygwin host
>> /usr/bin/cygport: line 426: echo: write error: Broken pipe
>> /usr/bin/cygport: line 426: echo: write error: Broken pipe
>>   >>> Uploading cygwin-3.3.0-0.1.9814cfd8f693.x86_64
>> [...]
>>
>> This happens in an xterm window, but not in mintty.
> 
> I have installed cygwin 3.3.0 [TEST].
> And I tryed cygpoort upload, but I cannot reproduce your problem
> even in xterm window.

Hmm.  I wonder what's different about my environment that would explain this.  I 
start the X server with startxwin.  Is that what you do too?

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-18  2:07                                                                                                                   ` Ken Brown
@ 2021-09-18  2:10                                                                                                                     ` Ken Brown
  2021-09-18  8:03                                                                                                                       ` Takashi Yano
  0 siblings, 1 reply; 250+ messages in thread
From: Ken Brown @ 2021-09-18  2:10 UTC (permalink / raw)
  To: cygwin-developers

On 9/17/2021 10:07 PM, Ken Brown wrote:
> On 9/17/2021 9:30 PM, Takashi Yano wrote:
>> On Fri, 17 Sep 2021 15:53:18 -0400
>> Ken Brown wrote:
>>> $ cygport *.cygport up
>>> /usr/bin/cygport: line 426: echo: write error: Broken pipe
>>> /usr/bin/cygport: line 426: echo: write error: Broken pipe
>>> /usr/bin/cygport: line 426: echo: write error: Broken pipe
>>> /usr/bin/cygport: line 426: echo: write error: Broken pipe
>>> *** Info: Building native toolchain for x86_64-pc-cygwin host
>>> /usr/bin/cygport: line 426: echo: write error: Broken pipe
>>> /usr/bin/cygport: line 426: echo: write error: Broken pipe
>>>   >>> Uploading cygwin-3.3.0-0.1.9814cfd8f693.x86_64
>>> [...]
>>>
>>> This happens in an xterm window, but not in mintty.
>>
>> I have installed cygwin 3.3.0 [TEST].
>> And I tryed cygpoort upload, but I cannot reproduce your problem
>> even in xterm window.
> 
> Hmm.  I wonder what's different about my environment that would explain this.  I 
> start the X server with startxwin.

And then I start UXTerm from the xwin-xdg-menu.

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-18  2:10                                                                                                                     ` Ken Brown
@ 2021-09-18  8:03                                                                                                                       ` Takashi Yano
  2021-09-18 11:12                                                                                                                         ` Ken Brown
  0 siblings, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-18  8:03 UTC (permalink / raw)
  To: cygwin-developers; +Cc: Jon Turney

On Fri, 17 Sep 2021 22:10:39 -0400
Ken Brown wrote:
> On 9/17/2021 10:07 PM, Ken Brown wrote:
> > On 9/17/2021 9:30 PM, Takashi Yano wrote:
> >> On Fri, 17 Sep 2021 15:53:18 -0400
> >> Ken Brown wrote:
> >>> $ cygport *.cygport up
> >>> /usr/bin/cygport: line 426: echo: write error: Broken pipe
> >>> /usr/bin/cygport: line 426: echo: write error: Broken pipe
> >>> /usr/bin/cygport: line 426: echo: write error: Broken pipe
> >>> /usr/bin/cygport: line 426: echo: write error: Broken pipe
> >>> *** Info: Building native toolchain for x86_64-pc-cygwin host
> >>> /usr/bin/cygport: line 426: echo: write error: Broken pipe
> >>> /usr/bin/cygport: line 426: echo: write error: Broken pipe
> >>>   >>> Uploading cygwin-3.3.0-0.1.9814cfd8f693.x86_64
> >>> [...]
> >>>
> >>> This happens in an xterm window, but not in mintty.
> >>
> >> I have installed cygwin 3.3.0 [TEST].
> >> And I tryed cygpoort upload, but I cannot reproduce your problem
> >> even in xterm window.
> > 
> > Hmm.  I wonder what's different about my environment that would explain this.  I 
> > start the X server with startxwin.
> 
> And then I start UXTerm from the xwin-xdg-menu.

Ah, this seems to be the point.

If xterm is started from xwin-xdg-menu, minimized test case
(/bin/echo A; /bin/echo B) | head -1
also causes the error:
/bin/echo: write error: Broken pipe

Even in mintty, this error occurs if it is started from
xwin-xdg-menu.

I also confirmed that this also happen with cygwin 3.2.0.

However this does not happen if xterm is started from
mintty window by 'xterm -display :0'.

So, this is not a regression in cygwin 3.3.0 [TEST].
In other words, the change of pipe implementation is not
the culprit.

I have no idea why terminal behaves differently if it is
started from xwin-xdg-menu so far.

'yes |head -1' also behaves differently. In the termainl
not started from xwin-xdg-menu,
[yano@Express5800-S70 ~]$ yes |head -1
y
[yano@Express5800-S70 ~]$
however, if it is started from xwin-xdg-menu,
[yano@Express5800-S70 ~]$ yes |head -1
y
yes: standard output: Broken pipe
[yano@Express5800-S70 ~]$

It seems that SIGPIPE is handled differently. I guess
SIGPIPE is handled by SIG_IGN if the terminal started
from xwin-xdg-menu, otherwize, it is handled by SIG_DFL.


Ah, the following patch for xwin-xdg-menu may be the
right thing.

--- execute.c.orig	2021-06-23 23:59:37.000000000 +0900
+++ execute.c	2021-09-18 16:53:52.144248600 +0900
@@ -76,6 +76,7 @@
     {
         struct rlimit rl;
         unsigned int fd;
+        int sig;
 
         /* dup write end of pipes onto stderr and stdout */
         close(STDOUT_FILENO);
@@ -89,6 +90,10 @@
         for (fd = STDERR_FILENO + 1; fd < rl.rlim_cur; fd++)
             close(fd);
 
+        /* Set all signal handlers to SIG_DFL. */
+        for (sig = 1; sig < NSIG; sig++)
+            signal(sig, SIG_DFL);
+
         /* Disassociate any TTYs */
         setsid();
 

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-18  8:03                                                                                                                       ` Takashi Yano
@ 2021-09-18 11:12                                                                                                                         ` Ken Brown
  2021-09-18 11:35                                                                                                                           ` Takashi Yano
  0 siblings, 1 reply; 250+ messages in thread
From: Ken Brown @ 2021-09-18 11:12 UTC (permalink / raw)
  To: cygwin-developers; +Cc: Jon Turney

On 9/18/2021 4:03 AM, Takashi Yano wrote:
> If xterm is started from xwin-xdg-menu, minimized test case
> (/bin/echo A; /bin/echo B) | head -1
> also causes the error:
> /bin/echo: write error: Broken pipe
> 
> Even in mintty, this error occurs if it is started from
> xwin-xdg-menu.
> 
> I also confirmed that this also happen with cygwin 3.2.0.
> 
> However this does not happen if xterm is started from
> mintty window by 'xterm -display :0'.
> 
> So, this is not a regression in cygwin 3.3.0 [TEST].
> In other words, the change of pipe implementation is not
> the culprit.

> It seems that SIGPIPE is handled differently. I guess
> SIGPIPE is handled by SIG_IGN if the terminal started
> from xwin-xdg-menu, otherwize, it is handled by SIG_DFL.

Yes, I remember now that the same issue came up a couple years ago:

   https://cygwin.com/pipermail/cygwin/2019-August/242060.html

> Ah, the following patch for xwin-xdg-menu may be the
> right thing.
> 
> --- execute.c.orig	2021-06-23 23:59:37.000000000 +0900
> +++ execute.c	2021-09-18 16:53:52.144248600 +0900
> @@ -76,6 +76,7 @@
>       {
>           struct rlimit rl;
>           unsigned int fd;
> +        int sig;
>   
>           /* dup write end of pipes onto stderr and stdout */
>           close(STDOUT_FILENO);
> @@ -89,6 +90,10 @@
>           for (fd = STDERR_FILENO + 1; fd < rl.rlim_cur; fd++)
>               close(fd);
>   
> +        /* Set all signal handlers to SIG_DFL. */
> +        for (sig = 1; sig < NSIG; sig++)
> +            signal(sig, SIG_DFL);
> +
>           /* Disassociate any TTYs */
>           setsid();

This may be unnecessarily drastic, given the analysis in my message cited above.

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-18 11:12                                                                                                                         ` Ken Brown
@ 2021-09-18 11:35                                                                                                                           ` Takashi Yano
  2021-09-18 14:11                                                                                                                             ` Jon Turney
  0 siblings, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-18 11:35 UTC (permalink / raw)
  To: cygwin-developers; +Cc: Jon Turney

On Sat, 18 Sep 2021 07:12:56 -0400
Ken Brown wrote:
> > It seems that SIGPIPE is handled differently. I guess
> > SIGPIPE is handled by SIG_IGN if the terminal started
> > from xwin-xdg-menu, otherwize, it is handled by SIG_DFL.
> 
> Yes, I remember now that the same issue came up a couple years ago:
> 
>    https://cygwin.com/pipermail/cygwin/2019-August/242060.html
> 
> > Ah, the following patch for xwin-xdg-menu may be the
> > right thing.
> > 
> > --- execute.c.orig	2021-06-23 23:59:37.000000000 +0900
> > +++ execute.c	2021-09-18 16:53:52.144248600 +0900
> > @@ -76,6 +76,7 @@
> >       {
> >           struct rlimit rl;
> >           unsigned int fd;
> > +        int sig;
> >   
> >           /* dup write end of pipes onto stderr and stdout */
> >           close(STDOUT_FILENO);
> > @@ -89,6 +90,10 @@
> >           for (fd = STDERR_FILENO + 1; fd < rl.rlim_cur; fd++)
> >               close(fd);
> >   
> > +        /* Set all signal handlers to SIG_DFL. */
> > +        for (sig = 1; sig < NSIG; sig++)
> > +            signal(sig, SIG_DFL);
> > +
> >           /* Disassociate any TTYs */
> >           setsid();
> 
> This may be unnecessarily drastic, given the analysis in my message cited above.

It seems resetting handler only for SIGPIPE is enough according to
the message you posted two years ago. Thanks.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-16 15:01                                                                                         ` Corinna Vinschen
  2021-09-16 15:46                                                                                           ` Ken Brown
@ 2021-09-18 13:44                                                                                           ` Ken Brown
  2021-09-19  1:31                                                                                             ` Takashi Yano
  1 sibling, 1 reply; 250+ messages in thread
From: Ken Brown @ 2021-09-18 13:44 UTC (permalink / raw)
  To: cygwin-developers

On 9/16/2021 11:01 AM, Corinna Vinschen wrote:
> On Sep 16 23:27, Takashi Yano wrote:
>> On Thu, 16 Sep 2021 15:25:46 +0200
>> Corinna Vinschen wrote:
>>> It would be great if open_setup() would be converted to a method which
>>> is allowed to fail, rather than ignoring errors in sync object creation
>>> and having to test the handle throughout the code.  Given there's only a
>>> single caller of that function (dtable::init_std_file_from_handle), that
>>> shouldn't be much work.  But it's certainly better than ignoring creation
>>> failures in the long run.
>>
>> I do not have confidence to do this correctly in a short time.
>> Leave it as a homework. Or may I leave it to you?
> 
> That's not a short-term problem, but I think it would be good to do this
> with a bit more paranoia at creation time and then rely on the handles
> being available in the subsequent code.  Maybe I can get to it later but
> it certainly wouldn't hurt my feelings if one of you guys look into it
> while I'm abroad. :)

I took a stab at this on the topic/pipe branch.  I haven't done any testing, 
other than to make sure that the code compiles.

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-18 11:35                                                                                                                           ` Takashi Yano
@ 2021-09-18 14:11                                                                                                                             ` Jon Turney
  0 siblings, 0 replies; 250+ messages in thread
From: Jon Turney @ 2021-09-18 14:11 UTC (permalink / raw)
  To: cygwin-developers

On 18/09/2021 12:35, Takashi Yano wrote:
> On Sat, 18 Sep 2021 07:12:56 -0400
> Ken Brown wrote:
>>> It seems that SIGPIPE is handled differently. I guess
>>> SIGPIPE is handled by SIG_IGN if the terminal started
>>> from xwin-xdg-menu, otherwize, it is handled by SIG_DFL.
>>
>> Yes, I remember now that the same issue came up a couple years ago:
>>
>>     https://cygwin.com/pipermail/cygwin/2019-August/242060.html

Oops.  Somehow I missed the conclusion in that thread.

>>> Ah, the following patch for xwin-xdg-menu may be the
>>> right thing.
>>>
>>> --- execute.c.orig	2021-06-23 23:59:37.000000000 +0900
>>> +++ execute.c	2021-09-18 16:53:52.144248600 +0900
>>> @@ -76,6 +76,7 @@
>>>        {
>>>            struct rlimit rl;
>>>            unsigned int fd;
>>> +        int sig;
>>>    
>>>            /* dup write end of pipes onto stderr and stdout */
>>>            close(STDOUT_FILENO);
>>> @@ -89,6 +90,10 @@
>>>            for (fd = STDERR_FILENO + 1; fd < rl.rlim_cur; fd++)
>>>                close(fd);
>>>    
>>> +        /* Set all signal handlers to SIG_DFL. */
>>> +        for (sig = 1; sig < NSIG; sig++)
>>> +            signal(sig, SIG_DFL);
>>> +
>>>            /* Disassociate any TTYs */
>>>            setsid();
>>

Thanks for the patch!

>> This may be unnecessarily drastic, given the analysis in my message cited above.
> 
> It seems resetting handler only for SIGPIPE is enough according to
> the message you posted two years ago. Thanks.

I've applied Takashi-san's patch as is, and rebuilt the xwin-xdg-menu 
package.

I think NSIG is sufficiently small that it's worth doing this for all 
signals, just in case we ever ignore some other signal.

I probably need to audit that code more carefully for anything else it's 
leaking into the child that it shouldn't be.


^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-18 13:44                                                                                           ` Ken Brown
@ 2021-09-19  1:31                                                                                             ` Takashi Yano
  2021-09-19 14:35                                                                                               ` Ken Brown
  0 siblings, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-19  1:31 UTC (permalink / raw)
  To: cygwin-developers

On Sat, 18 Sep 2021 09:44:48 -0400
Ken Brown wrote:
> On 9/16/2021 11:01 AM, Corinna Vinschen wrote:
> > On Sep 16 23:27, Takashi Yano wrote:
> >> On Thu, 16 Sep 2021 15:25:46 +0200
> >> Corinna Vinschen wrote:
> >>> It would be great if open_setup() would be converted to a method which
> >>> is allowed to fail, rather than ignoring errors in sync object creation
> >>> and having to test the handle throughout the code.  Given there's only a
> >>> single caller of that function (dtable::init_std_file_from_handle), that
> >>> shouldn't be much work.  But it's certainly better than ignoring creation
> >>> failures in the long run.
> >>
> >> I do not have confidence to do this correctly in a short time.
> >> Leave it as a homework. Or may I leave it to you?
> > 
> > That's not a short-term problem, but I think it would be good to do this
> > with a bit more paranoia at creation time and then rely on the handles
> > being available in the subsequent code.  Maybe I can get to it later but
> > it certainly wouldn't hurt my feelings if one of you guys look into it
> > while I'm abroad. :)
> 
> I took a stab at this on the topic/pipe branch.  I haven't done any testing, 
> other than to make sure that the code compiles.

Thanks for working on this! I tested the head of topic/pipe branch,
and do not find any defect so far. However, I cannot test the case
open_setup() failure, so I am not sure this works as expected in
error case.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-19  1:31                                                                                             ` Takashi Yano
@ 2021-09-19 14:35                                                                                               ` Ken Brown
  2021-09-20  9:29                                                                                                 ` Takashi Yano
  0 siblings, 1 reply; 250+ messages in thread
From: Ken Brown @ 2021-09-19 14:35 UTC (permalink / raw)
  To: cygwin-developers

On 9/18/2021 9:31 PM, Takashi Yano wrote:
> On Sat, 18 Sep 2021 09:44:48 -0400
> Ken Brown wrote:
>> I took a stab at this on the topic/pipe branch.  I haven't done any testing,
>> other than to make sure that the code compiles.
> 
> Thanks for working on this! I tested the head of topic/pipe branch,
> and do not find any defect so far.

Thanks for testing.

> However, I cannot test the case
> open_setup() failure, so I am not sure this works as expected in
> error case.

Right, it's hard to test this.  I have no idea why it would ever fail.

I wonder if I should go ahead and push this to master, so that it will get some 
testing in the next test release.  Corinna can always make changes when she 
returns if she thinks there's a better way to do it.

WDYT?

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-19 14:35                                                                                               ` Ken Brown
@ 2021-09-20  9:29                                                                                                 ` Takashi Yano
  0 siblings, 0 replies; 250+ messages in thread
From: Takashi Yano @ 2021-09-20  9:29 UTC (permalink / raw)
  To: cygwin-developers

On Sun, 19 Sep 2021 10:35:10 -0400
Ken Brown wrote:
> On 9/18/2021 9:31 PM, Takashi Yano wrote:
> > On Sat, 18 Sep 2021 09:44:48 -0400
> > Ken Brown wrote:
> >> I took a stab at this on the topic/pipe branch.  I haven't done any testing,
> >> other than to make sure that the code compiles.
> > 
> > Thanks for working on this! I tested the head of topic/pipe branch,
> > and do not find any defect so far.
> 
> Thanks for testing.
> 
> > However, I cannot test the case
> > open_setup() failure, so I am not sure this works as expected in
> > error case.
> 
> Right, it's hard to test this.  I have no idea why it would ever fail.
> 
> I wonder if I should go ahead and push this to master, so that it will get some 
> testing in the next test release.  Corinna can always make changes when she 
> returns if she thinks there's a better way to do it.
> 
> WDYT?

I agree with you.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-13  9:07                                 ` Corinna Vinschen
@ 2021-09-20 12:52                                   ` Takashi Yano
  2021-09-20 19:14                                     ` Ken Brown
  0 siblings, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-20 12:52 UTC (permalink / raw)
  To: cygwin-developers

[-- Attachment #1: Type: text/plain, Size: 1882 bytes --]

On Mon, 13 Sep 2021 11:07:42 +0200
Corinna Vinschen wrote:
> On Sep 11 11:35, Takashi Yano wrote:
> > Keeping read handle in write pipe (Corinna's query_hdl) causes problem
> > that write side cannot detect close on read side.
> > Is it possible to open read handle temporally when pipe_data_available()
> > is called?
> 
> 1. You would have to know which process keeps the other side of the pipe.
> 2. You would have to have the permission to open the other process to
>    duplicate the pipe into your own process
> 3. You would have to know the HANDLE value of the read side of your pipe
>    in that other process.
> 
> Point 1 is (kind of) doable using GetNamedPipeClientProcessId or
> GetNamedPipeServerProcessId.  ZIt's not clear how reliable these
> functions are, given that both pipe sides are created by the same
> process and then usually inherited by two child processes communicating
> over that pipe.
> 
> Point 2 is most of the time the case, especially when talking with
> native processes.
> 
> Point 3 requires some sort of IPC.
> 
> Having said that, I think this is too complicated.

In the current implementation, there is a potential risk that
write side fails to detect the closure of read side if more than
one writers exist and one of them is non-cygwin process.

Therefore, I had tried to implement the above idea (tentative
query handle). Finally, I could convince myself to some extent,
so I attach a patch for that.

As for point 1 and 3, I used NtQuerySystemInformation() with
SystemHandleInformation. I also used NtQueryObject() with
ObjectNameInformation in order to know which handle is the
other side of the pipe. As for point 2, it is not possible to
open the process which is running as a service, so the current
query_hdl is used for such process as an exception.

Ken, WDYT of this implementation?

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

[-- Attachment #2: v2-0001-Cygwin-pipe-Introduce-tentative-query_hdl.patch --]
[-- Type: application/octet-stream, Size: 9164 bytes --]

From 98b0e3f0dba382751cbe85c1a8ae44327c592d4c Mon Sep 17 00:00:00 2001
From: Takashi Yano <takashi.yano@nifty.ne.jp>
Date: Mon, 20 Sep 2021 16:30:20 +0900
Subject: [PATCH v2] Cygwin: pipe: Introduce tentative query_hdl.

- The commit f79a4611 introduced query_hdl which is the read pipe
  handle kept in the write pipe instance in order to determine if
  the pipe is ready to write in select(). This implementation has
  a potential risk that write side fails to detect the closure of
  read side if more than one writers exist and one of them is non-
  cygwin process.

  With this patch, the strategy of commit f79a4611 is used only if
  the process is running as a service. For normal process, instead
  of keeping query_hdl in the write pipe instance, it is retrieved
  tentatively when select() is called. Actually, we want to use
  tentative query_hdl for all processes, however, it does not work
  for service processes due to OpenProcess() failure.
---
 winsup/cygwin/fhandler.h       |   3 +
 winsup/cygwin/fhandler_pipe.cc | 134 +++++++++++++++++++++++++++++++--
 winsup/cygwin/select.cc        |   8 +-
 3 files changed, 139 insertions(+), 6 deletions(-)

diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
index 3471e95b9..d1149d301 100644
--- a/winsup/cygwin/fhandler.h
+++ b/winsup/cygwin/fhandler.h
@@ -1191,6 +1191,8 @@ private:
   pid_t popen_pid;
   HANDLE query_hdl;
   HANDLE hdl_cnt_mtx;
+  HANDLE query_hdl_proc;
+  HANDLE query_hdl_value;
   void release_select_sem (const char *);
 public:
   fhandler_pipe ();
@@ -1249,6 +1251,7 @@ public:
       }
   }
   bool reader_closed ();
+  HANDLE tentative_query_hdl ();
 };
 
 #define CYGWIN_FIFO_PIPE_NAME_LEN     47
diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index 6b41a755f..fb2d06bb8 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -19,6 +19,7 @@ details. */
 #include "cygheap.h"
 #include "pinfo.h"
 #include "shared_info.h"
+#include "tls_pbuf.h"
 
 /* This is only to be used for writing.  When reading,
 STATUS_PIPE_EMPTY simply means there's no data to be read. */
@@ -220,8 +221,6 @@ fhandler_pipe::open_setup (int flags)
 	  goto err_close_read_mtx;
 	}
     }
-  if (get_dev () == FH_PIPEW && !query_hdl)
-    set_pipe_non_blocking (is_nonblocking ());
   return true;
 
 err_close_read_mtx:
@@ -267,7 +266,7 @@ fhandler_pipe::release_select_sem (const char *from)
       - get_obj_handle_count (read_mtx);
   else /* Number of select() call */
     n_release = get_obj_handle_count (select_sem)
-      - get_obj_handle_count (query_hdl);
+      - get_obj_handle_count (hdl_cnt_mtx);
   debug_printf("%s(%s) release %d", from,
 	       get_dev () == FH_PIPER ? "PIPER" : "PIPEW", n_release);
   if (n_release)
@@ -667,6 +666,8 @@ fhandler_pipe::close ()
   int ret = fhandler_base::close ();
   ReleaseMutex (hdl_cnt_mtx);
   CloseHandle (hdl_cnt_mtx);
+  if (query_hdl_proc)
+    CloseHandle (query_hdl_proc);
   return ret;
 }
 
@@ -820,6 +821,13 @@ fhandler_pipe::create (LPSECURITY_ATTRIBUTES sa_ptr, PHANDLE r, PHANDLE w,
   return 0;
 }
 
+inline static bool
+is_running_as_service (void)
+{
+  return check_token_membership (well_known_service_sid)
+    || cygheap->user.saved_sid () == well_known_system_sid;
+}
+
 /* The next version of fhandler_pipe::create used to call the previous
    version.  But the read handle created by the latter doesn't have
    FILE_WRITE_ATTRIBUTES access unless the pipe is created with
@@ -874,7 +882,8 @@ fhandler_pipe::create (fhandler_pipe *fhs[2], unsigned psize, int mode)
 			0, sa->bInheritHandle, DUPLICATE_SAME_ACCESS))
     goto err_close_select_sem0;
 
-  if (!DuplicateHandle (GetCurrentProcess (), r,
+  if (is_running_as_service () &&
+      !DuplicateHandle (GetCurrentProcess (), r,
 			GetCurrentProcess (), &fhs[1]->query_hdl,
 			FILE_READ_DATA, sa->bInheritHandle, 0))
     goto err_close_select_sem1;
@@ -893,7 +902,8 @@ fhandler_pipe::create (fhandler_pipe *fhs[2], unsigned psize, int mode)
 err_close_hdl_cnt_mtx0:
   CloseHandle (fhs[0]->hdl_cnt_mtx);
 err_close_query_hdl:
-  CloseHandle (fhs[1]->query_hdl);
+  if (fhs[1]->query_hdl)
+    CloseHandle (fhs[1]->query_hdl);
 err_close_select_sem1:
   CloseHandle (fhs[1]->select_sem);
 err_close_select_sem0:
@@ -946,6 +956,7 @@ nt_create (LPSECURITY_ATTRIBUTES sa_ptr, HANDLE &r, HANDLE &w,
 				 GetCurrentProcessId ());
 
   access = GENERIC_READ | FILE_WRITE_ATTRIBUTES | SYNCHRONIZE;
+  access |= FILE_WRITE_EA; /* Add this right as a marker of cygwin read pipe */
 
   ULONG pipe_type = pipe_byte ? FILE_PIPE_BYTE_STREAM_TYPE
     : FILE_PIPE_MESSAGE_TYPE;
@@ -1112,3 +1123,116 @@ fhandler_pipe::fstatvfs (struct statvfs *sfs)
   set_errno (EBADF);
   return -1;
 }
+
+HANDLE
+fhandler_pipe::tentative_query_hdl ()
+{
+  if (get_dev () != FH_PIPEW)
+    return NULL;
+  /* Try process handle opened and pipe handle value cached first
+     in order to reduce overhead. */
+  if (query_hdl_proc && query_hdl_value)
+    {
+      HANDLE h;
+      if (DuplicateHandle (query_hdl_proc, query_hdl_value,
+			   GetCurrentProcess (), &h, FILE_READ_DATA, 0, 0))
+	return h;
+      CloseHandle (query_hdl_proc);
+      query_hdl_proc = NULL;
+      query_hdl_value = NULL;
+    }
+
+  ULONG len;
+  NTSTATUS status;
+  tmp_pathbuf tp;
+  OBJECT_NAME_INFORMATION *ntfn =(OBJECT_NAME_INFORMATION *) tp.w_get ();
+  status = NtQueryObject (get_handle (), ObjectNameInformation, ntfn,
+			  65536, &len);
+  if (!NT_SUCCESS (status))
+    return NULL; /* Non cygwin pipe? */
+  WCHAR name[MAX_PATH];
+  int namelen = min (ntfn->Name.Length / sizeof (WCHAR), MAX_PATH-1);
+  memcpy (name, ntfn->Name.Buffer, namelen * sizeof (WCHAR));
+  name[namelen] = L'\0';
+  uint64_t key;
+  DWORD pid;
+  LONG id;
+  if (swscanf (name, L"\\Device\\NamedPipe\\%llx-%u-pipe-nt-0x%x",
+	       &key, &pid, &id) != 3)
+    return NULL; /* Non cygwin pipe? */
+
+  SIZE_T n_handle = 65536;
+  typedef struct
+    {
+      USHORT UniqueProcessId;
+      USHORT CreatorBackTraceIndex;
+      UCHAR ObjectTypeIndex;
+      UCHAR HandleAttributes;
+      USHORT HandleValue;
+      PVOID Object;
+      ULONG GrantedAccess;
+    } SYSTEM_HANDLE_TABLE_ENTRY_INFO;
+  typedef struct
+    {
+      ULONG NumberOfHandles;
+      SYSTEM_HANDLE_TABLE_ENTRY_INFO Handles[1];
+    } SYSTEM_HANDLE_INFORMATION;
+  SYSTEM_HANDLE_INFORMATION *shi;
+  do
+    {
+      SIZE_T nbytes =
+	sizeof (ULONG) + n_handle * sizeof (SYSTEM_HANDLE_TABLE_ENTRY_INFO);
+      shi = (SYSTEM_HANDLE_INFORMATION *) HeapAlloc (GetProcessHeap (),
+						     0, nbytes);
+      status = NtQuerySystemInformation (SystemHandleInformation,
+					 shi, nbytes, NULL);
+      if (NT_SUCCESS (status))
+	break;
+      HeapFree (GetProcessHeap (), 0, shi);
+      n_handle *= 2;
+    }
+  while (n_handle < (1L<<20));
+  if (!NT_SUCCESS (status))
+    return NULL;
+
+  HANDLE qh = NULL;
+  for (LONG i = (LONG) shi->NumberOfHandles - 1; i >= 0; i--)
+    {
+      /* Check for the peculiarity of cygwin read pipe */
+      DWORD access = FILE_READ_DATA | FILE_READ_EA | FILE_WRITE_EA /* marker */
+	| FILE_READ_ATTRIBUTES | FILE_WRITE_ATTRIBUTES
+	| READ_CONTROL | SYNCHRONIZE;
+      if (shi->Handles[i].GrantedAccess != access)
+	continue;
+
+      /* Retrieve handle */
+      HANDLE proc = OpenProcess (PROCESS_DUP_HANDLE, 0,
+				 shi->Handles[i].UniqueProcessId);
+      if (!proc)
+	continue;
+      HANDLE h = (HANDLE)(intptr_t) shi->Handles[i].HandleValue;
+      BOOL res  = DuplicateHandle (proc, h, GetCurrentProcess (), &h,
+				   FILE_READ_DATA, 0, 0);
+      if (!res)
+	goto close_proc;
+
+      /* Check object name */
+      status = NtQueryObject (h, ObjectNameInformation, ntfn, 65536, &len);
+      if (!NT_SUCCESS (status))
+	goto close_handle;
+      ntfn->Name.Buffer[ntfn->Name.Length / sizeof (WCHAR)] = L'\0';
+      if (wcscmp (name, ntfn->Name.Buffer) == 0)
+	{
+	  query_hdl_proc = proc;
+	  query_hdl_value = (HANDLE)(intptr_t) shi->Handles[i].HandleValue;
+	  qh = h;
+	  break;
+	}
+close_handle:
+      CloseHandle (h);
+close_proc:
+      CloseHandle (proc);
+    }
+  HeapFree (GetProcessHeap (), 0, shi);
+  return qh;
+}
diff --git a/winsup/cygwin/select.cc b/winsup/cygwin/select.cc
index 33c0c0bb0..15589b711 100644
--- a/winsup/cygwin/select.cc
+++ b/winsup/cygwin/select.cc
@@ -641,10 +641,16 @@ pipe_data_available (int fd, fhandler_base *fh, HANDLE h, bool writing)
       if (fh->get_device () == FH_PIPEW && fpli.WriteQuotaAvailable == 0)
 	{
 	  HANDLE query_hdl = ((fhandler_pipe *) fh)->get_query_handle ();
+	  if (!query_hdl)
+	    query_hdl = ((fhandler_pipe *) fh)->tentative_query_hdl ();
 	  if (!query_hdl)
 	    return 1; /* We cannot know actual write pipe space. */
 	  DWORD nbytes_in_pipe;
-	  if (!PeekNamedPipe (query_hdl, NULL, 0, NULL, &nbytes_in_pipe, NULL))
+	  BOOL res =
+	    PeekNamedPipe (query_hdl, NULL, 0, NULL, &nbytes_in_pipe, NULL);
+	  if (!((fhandler_pipe *) fh)->get_query_handle ())
+	    CloseHandle (query_hdl); /* Close tentative query_hdl */
+	  if (!res)
 	    return 1;
 	  fpli.WriteQuotaAvailable = fpli.InboundQuota - nbytes_in_pipe;
 	}
-- 
2.33.0


^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-20 12:52                                   ` Takashi Yano
@ 2021-09-20 19:14                                     ` Ken Brown
  2021-09-20 21:09                                       ` Ken Brown
  0 siblings, 1 reply; 250+ messages in thread
From: Ken Brown @ 2021-09-20 19:14 UTC (permalink / raw)
  To: cygwin-developers

Hi Takashi,

On 9/20/2021 8:52 AM, Takashi Yano wrote:
> On Mon, 13 Sep 2021 11:07:42 +0200
> Corinna Vinschen wrote:
>> On Sep 11 11:35, Takashi Yano wrote:
>>> Keeping read handle in write pipe (Corinna's query_hdl) causes problem
>>> that write side cannot detect close on read side.
>>> Is it possible to open read handle temporally when pipe_data_available()
>>> is called?
>>
>> 1. You would have to know which process keeps the other side of the pipe.
>> 2. You would have to have the permission to open the other process to
>>     duplicate the pipe into your own process
>> 3. You would have to know the HANDLE value of the read side of your pipe
>>     in that other process.
>>
>> Point 1 is (kind of) doable using GetNamedPipeClientProcessId or
>> GetNamedPipeServerProcessId.  ZIt's not clear how reliable these
>> functions are, given that both pipe sides are created by the same
>> process and then usually inherited by two child processes communicating
>> over that pipe.
>>
>> Point 2 is most of the time the case, especially when talking with
>> native processes.
>>
>> Point 3 requires some sort of IPC.
>>
>> Having said that, I think this is too complicated.
> 
> In the current implementation, there is a potential risk that
> write side fails to detect the closure of read side if more than
> one writers exist and one of them is non-cygwin process.
> 
> Therefore, I had tried to implement the above idea (tentative
> query handle). Finally, I could convince myself to some extent,
> so I attach a patch for that.
> 
> As for point 1 and 3, I used NtQuerySystemInformation() with
> SystemHandleInformation. I also used NtQueryObject() with
> ObjectNameInformation in order to know which handle is the
> other side of the pipe. As for point 2, it is not possible to
> open the process which is running as a service, so the current
> query_hdl is used for such process as an exception.
> 
> Ken, WDYT of this implementation?

This looks like a great idea!  I'll review it in detail later today or tomorrow.

Thanks.

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-20 19:14                                     ` Ken Brown
@ 2021-09-20 21:09                                       ` Ken Brown
  2021-09-20 21:21                                         ` Ken Brown
  2021-09-20 21:27                                         ` Takashi Yano
  0 siblings, 2 replies; 250+ messages in thread
From: Ken Brown @ 2021-09-20 21:09 UTC (permalink / raw)
  To: cygwin-developers

On 9/20/2021 3:14 PM, Ken Brown wrote:
> Hi Takashi,
> 
> On 9/20/2021 8:52 AM, Takashi Yano wrote:
>> On Mon, 13 Sep 2021 11:07:42 +0200
>> Corinna Vinschen wrote:
>>> On Sep 11 11:35, Takashi Yano wrote:
>>>> Keeping read handle in write pipe (Corinna's query_hdl) causes problem
>>>> that write side cannot detect close on read side.
>>>> Is it possible to open read handle temporally when pipe_data_available()
>>>> is called?
>>>
>>> 1. You would have to know which process keeps the other side of the pipe.
>>> 2. You would have to have the permission to open the other process to
>>>     duplicate the pipe into your own process
>>> 3. You would have to know the HANDLE value of the read side of your pipe
>>>     in that other process.
>>>
>>> Point 1 is (kind of) doable using GetNamedPipeClientProcessId or
>>> GetNamedPipeServerProcessId.  ZIt's not clear how reliable these
>>> functions are, given that both pipe sides are created by the same
>>> process and then usually inherited by two child processes communicating
>>> over that pipe.
>>>
>>> Point 2 is most of the time the case, especially when talking with
>>> native processes.
>>>
>>> Point 3 requires some sort of IPC.
>>>
>>> Having said that, I think this is too complicated.
>>
>> In the current implementation, there is a potential risk that
>> write side fails to detect the closure of read side if more than
>> one writers exist and one of them is non-cygwin process.
>>
>> Therefore, I had tried to implement the above idea (tentative
>> query handle). Finally, I could convince myself to some extent,
>> so I attach a patch for that.
>>
>> As for point 1 and 3, I used NtQuerySystemInformation() with
>> SystemHandleInformation. I also used NtQueryObject() with
>> ObjectNameInformation in order to know which handle is the
>> other side of the pipe. As for point 2, it is not possible to
>> open the process which is running as a service, so the current
>> query_hdl is used for such process as an exception.
>>
>> Ken, WDYT of this implementation?
> 
> This looks like a great idea!  I'll review it in detail later today or tomorrow.

I only saw one thing that doesn't look right to me:

> +      /* Check object name */
> +      status = NtQueryObject (h, ObjectNameInformation, ntfn, 65536, &len);
> +      if (!NT_SUCCESS (status))
> +	goto close_handle;
> +      ntfn->Name.Buffer[ntfn->Name.Length / sizeof (WCHAR)] = L'\0';
> +      if (wcscmp (name, ntfn->Name.Buffer) == 0)
> +	{
> +	  query_hdl_proc = proc;
> +	  query_hdl_value = (HANDLE)(intptr_t) shi->Handles[i].HandleValue;
> +	  qh = h;
> +	  break;
> +	}
> +close_handle:
> +      CloseHandle (h);
> +close_proc:
> +      CloseHandle (proc);

Doesn't this mean that query_hdl_proc is not a valid handle any more?  So the 
attempt to reduce overhead at the beginning of tentative_query_hdl() will never 
work.

Other than that, I still think it looks great.  But it's complicated enough that 
I think Corinna should review it too when she returns.  (It also uses Windows 
functions that I have no experience with.)  Nevertheless, I'm tempted to push it 
so that it can get testing, even if Corinna changes it or reverts it later.

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-20 21:09                                       ` Ken Brown
@ 2021-09-20 21:21                                         ` Ken Brown
  2021-09-20 21:27                                         ` Takashi Yano
  1 sibling, 0 replies; 250+ messages in thread
From: Ken Brown @ 2021-09-20 21:21 UTC (permalink / raw)
  To: cygwin-developers

On 9/20/2021 5:09 PM, Ken Brown wrote:
> On 9/20/2021 3:14 PM, Ken Brown wrote:
>> Hi Takashi,
>>
>> On 9/20/2021 8:52 AM, Takashi Yano wrote:
>>> On Mon, 13 Sep 2021 11:07:42 +0200
>>> Corinna Vinschen wrote:
>>>> On Sep 11 11:35, Takashi Yano wrote:
>>>>> Keeping read handle in write pipe (Corinna's query_hdl) causes problem
>>>>> that write side cannot detect close on read side.
>>>>> Is it possible to open read handle temporally when pipe_data_available()
>>>>> is called?
>>>>
>>>> 1. You would have to know which process keeps the other side of the pipe.
>>>> 2. You would have to have the permission to open the other process to
>>>>     duplicate the pipe into your own process
>>>> 3. You would have to know the HANDLE value of the read side of your pipe
>>>>     in that other process.
>>>>
>>>> Point 1 is (kind of) doable using GetNamedPipeClientProcessId or
>>>> GetNamedPipeServerProcessId.  ZIt's not clear how reliable these
>>>> functions are, given that both pipe sides are created by the same
>>>> process and then usually inherited by two child processes communicating
>>>> over that pipe.
>>>>
>>>> Point 2 is most of the time the case, especially when talking with
>>>> native processes.
>>>>
>>>> Point 3 requires some sort of IPC.
>>>>
>>>> Having said that, I think this is too complicated.
>>>
>>> In the current implementation, there is a potential risk that
>>> write side fails to detect the closure of read side if more than
>>> one writers exist and one of them is non-cygwin process.
>>>
>>> Therefore, I had tried to implement the above idea (tentative
>>> query handle). Finally, I could convince myself to some extent,
>>> so I attach a patch for that.
>>>
>>> As for point 1 and 3, I used NtQuerySystemInformation() with
>>> SystemHandleInformation. I also used NtQueryObject() with
>>> ObjectNameInformation in order to know which handle is the
>>> other side of the pipe. As for point 2, it is not possible to
>>> open the process which is running as a service, so the current
>>> query_hdl is used for such process as an exception.
>>>
>>> Ken, WDYT of this implementation?
>>
>> This looks like a great idea!  I'll review it in detail later today or tomorrow.
> 
> I only saw one thing that doesn't look right to me:
> 
>> +      /* Check object name */
>> +      status = NtQueryObject (h, ObjectNameInformation, ntfn, 65536, &len);
>> +      if (!NT_SUCCESS (status))
>> +    goto close_handle;
>> +      ntfn->Name.Buffer[ntfn->Name.Length / sizeof (WCHAR)] = L'\0';
>> +      if (wcscmp (name, ntfn->Name.Buffer) == 0)
>> +    {
>> +      query_hdl_proc = proc;
>> +      query_hdl_value = (HANDLE)(intptr_t) shi->Handles[i].HandleValue;
>> +      qh = h;
>> +      break;
>> +    }
>> +close_handle:
>> +      CloseHandle (h);
>> +close_proc:
>> +      CloseHandle (proc);
> 
> Doesn't this mean that query_hdl_proc is not a valid handle any more?  So the 
> attempt to reduce overhead at the beginning of tentative_query_hdl() will never 
> work.

Never mind.  I see my stupid mistake.

> Other than that, I still think it looks great.  But it's complicated enough that 
> I think Corinna should review it too when she returns.  (It also uses Windows 
> functions that I have no experience with.)  Nevertheless, I'm tempted to push it 
> so that it can get testing, even if Corinna changes it or reverts it later.
> 
> Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-20 21:09                                       ` Ken Brown
  2021-09-20 21:21                                         ` Ken Brown
@ 2021-09-20 21:27                                         ` Takashi Yano
  2021-09-20 21:39                                           ` Ken Brown
  1 sibling, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-20 21:27 UTC (permalink / raw)
  To: cygwin-developers

Hi Ken,

Thanks for reviewing the code.

On Mon, 20 Sep 2021 17:09:48 -0400
Ken Brown wrote:
> On 9/20/2021 3:14 PM, Ken Brown wrote:
> > +      if (wcscmp (name, ntfn->Name.Buffer) == 0)
> > +	{
> > +	  query_hdl_proc = proc;
> > +	  query_hdl_value = (HANDLE)(intptr_t) shi->Handles[i].HandleValue;
> > +	  qh = h;
> > +	  break;
> > +	}
> > +close_handle:
> > +      CloseHandle (h);
> > +close_proc:
> > +      CloseHandle (proc);
> 
> Doesn't this mean that query_hdl_proc is not a valid handle any more?  So the 
> attempt to reduce overhead at the beginning of tentative_query_hdl() will never 
> work.

When the handle which name matches the pipe name is found, 'break'
is used in the 'for' loop, so the CloseHandle(proc) is not called.
Therefore, query_hdl_proc still has valid handle after 'break'.

> Other than that, I still think it looks great.  But it's complicated enough that 
> I think Corinna should review it too when she returns.  (It also uses Windows 
> functions that I have no experience with.)  Nevertheless, I'm tempted to push it 
> so that it can get testing, even if Corinna changes it or reverts it later.

Thanks much.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-20 21:27                                         ` Takashi Yano
@ 2021-09-20 21:39                                           ` Ken Brown
  2021-09-20 22:16                                             ` Takashi Yano
  0 siblings, 1 reply; 250+ messages in thread
From: Ken Brown @ 2021-09-20 21:39 UTC (permalink / raw)
  To: cygwin-developers

On 9/20/2021 5:27 PM, Takashi Yano wrote:
> Hi Ken,
> 
> Thanks for reviewing the code.
> 
> On Mon, 20 Sep 2021 17:09:48 -0400
> Ken Brown wrote:
>> On 9/20/2021 3:14 PM, Ken Brown wrote:
>>> +      if (wcscmp (name, ntfn->Name.Buffer) == 0)
>>> +	{
>>> +	  query_hdl_proc = proc;
>>> +	  query_hdl_value = (HANDLE)(intptr_t) shi->Handles[i].HandleValue;
>>> +	  qh = h;
>>> +	  break;
>>> +	}
>>> +close_handle:
>>> +      CloseHandle (h);
>>> +close_proc:
>>> +      CloseHandle (proc);
>>
>> Doesn't this mean that query_hdl_proc is not a valid handle any more?  So the
>> attempt to reduce overhead at the beginning of tentative_query_hdl() will never
>> work.
> 
> When the handle which name matches the pipe name is found, 'break'
> is used in the 'for' loop, so the CloseHandle(proc) is not called.
> Therefore, query_hdl_proc still has valid handle after 'break'.

Right, I realized this myself.  I guess you wrote your reply before seeing my 
later message in which I said "I see my stupid mistake".
> 
>> Other than that, I still think it looks great.  But it's complicated enough that
>> I think Corinna should review it too when she returns.  (It also uses Windows
>> functions that I have no experience with.)  Nevertheless, I'm tempted to push it
>> so that it can get testing, even if Corinna changes it or reverts it later.

I'll push it tomorrow unless you find something that needs to be changed before 
then.

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-20 21:39                                           ` Ken Brown
@ 2021-09-20 22:16                                             ` Takashi Yano
  2021-09-20 22:46                                               ` Ken Brown
  0 siblings, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-20 22:16 UTC (permalink / raw)
  To: cygwin-developers

[-- Attachment #1: Type: text/plain, Size: 284 bytes --]

On Mon, 20 Sep 2021 17:39:54 -0400
Ken Brown wrote:
> I'll push it tomorrow unless you find something that needs to be changed before 
> then.

typedef of SYSTEM_HANDLE_TABLE_ENTRY_INFO and SYSTEM_HANDLE_INFORMATION
are moved into ntdll.h

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

[-- Attachment #2: v3-0001-Cygwin-pipe-Introduce-tentative-query_hdl.patch --]
[-- Type: application/octet-stream, Size: 9677 bytes --]

From 989d1b7ea1b2448e887ac387ce6982473b439768 Mon Sep 17 00:00:00 2001
From: Takashi Yano <takashi.yano@nifty.ne.jp>
Date: Tue, 21 Sep 2021 07:09:51 +0900
Subject: [PATCH v3] Cygwin: pipe: Introduce tentative query_hdl.

- The commit f79a4611 introduced query_hdl which is the read pipe
  handle kept in the write pipe instance in order to determine if
  the pipe is ready to write in select(). This implementation has
  a potential risk that write side fails to detect the closure of
  read side if more than one writers exist and one of them is non-
  cygwin process.

  With this patch, the strategy of commit f79a4611 is used only if
  the process is running as a service. For normal process, instead
  of keeping query_hdl in the write pipe instance, it is retrieved
  tentatively when select() is called. Actually, we want to use
  tentative query_hdl for all processes, however, it does not work
  for service processes due to OpenProcess() failure.
---
 winsup/cygwin/fhandler.h       |   3 +
 winsup/cygwin/fhandler_pipe.cc | 119 +++++++++++++++++++++++++++++++--
 winsup/cygwin/ntdll.h          |  17 +++++
 winsup/cygwin/select.cc        |   8 ++-
 4 files changed, 141 insertions(+), 6 deletions(-)

diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
index 3471e95b9..d1149d301 100644
--- a/winsup/cygwin/fhandler.h
+++ b/winsup/cygwin/fhandler.h
@@ -1191,6 +1191,8 @@ private:
   pid_t popen_pid;
   HANDLE query_hdl;
   HANDLE hdl_cnt_mtx;
+  HANDLE query_hdl_proc;
+  HANDLE query_hdl_value;
   void release_select_sem (const char *);
 public:
   fhandler_pipe ();
@@ -1249,6 +1251,7 @@ public:
       }
   }
   bool reader_closed ();
+  HANDLE tentative_query_hdl ();
 };
 
 #define CYGWIN_FIFO_PIPE_NAME_LEN     47
diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index 6b41a755f..3cfa78f28 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -19,6 +19,7 @@ details. */
 #include "cygheap.h"
 #include "pinfo.h"
 #include "shared_info.h"
+#include "tls_pbuf.h"
 
 /* This is only to be used for writing.  When reading,
 STATUS_PIPE_EMPTY simply means there's no data to be read. */
@@ -220,8 +221,6 @@ fhandler_pipe::open_setup (int flags)
 	  goto err_close_read_mtx;
 	}
     }
-  if (get_dev () == FH_PIPEW && !query_hdl)
-    set_pipe_non_blocking (is_nonblocking ());
   return true;
 
 err_close_read_mtx:
@@ -267,7 +266,7 @@ fhandler_pipe::release_select_sem (const char *from)
       - get_obj_handle_count (read_mtx);
   else /* Number of select() call */
     n_release = get_obj_handle_count (select_sem)
-      - get_obj_handle_count (query_hdl);
+      - get_obj_handle_count (hdl_cnt_mtx);
   debug_printf("%s(%s) release %d", from,
 	       get_dev () == FH_PIPER ? "PIPER" : "PIPEW", n_release);
   if (n_release)
@@ -667,6 +666,8 @@ fhandler_pipe::close ()
   int ret = fhandler_base::close ();
   ReleaseMutex (hdl_cnt_mtx);
   CloseHandle (hdl_cnt_mtx);
+  if (query_hdl_proc)
+    CloseHandle (query_hdl_proc);
   return ret;
 }
 
@@ -820,6 +821,13 @@ fhandler_pipe::create (LPSECURITY_ATTRIBUTES sa_ptr, PHANDLE r, PHANDLE w,
   return 0;
 }
 
+inline static bool
+is_running_as_service (void)
+{
+  return check_token_membership (well_known_service_sid)
+    || cygheap->user.saved_sid () == well_known_system_sid;
+}
+
 /* The next version of fhandler_pipe::create used to call the previous
    version.  But the read handle created by the latter doesn't have
    FILE_WRITE_ATTRIBUTES access unless the pipe is created with
@@ -874,7 +882,8 @@ fhandler_pipe::create (fhandler_pipe *fhs[2], unsigned psize, int mode)
 			0, sa->bInheritHandle, DUPLICATE_SAME_ACCESS))
     goto err_close_select_sem0;
 
-  if (!DuplicateHandle (GetCurrentProcess (), r,
+  if (is_running_as_service () &&
+      !DuplicateHandle (GetCurrentProcess (), r,
 			GetCurrentProcess (), &fhs[1]->query_hdl,
 			FILE_READ_DATA, sa->bInheritHandle, 0))
     goto err_close_select_sem1;
@@ -893,7 +902,8 @@ fhandler_pipe::create (fhandler_pipe *fhs[2], unsigned psize, int mode)
 err_close_hdl_cnt_mtx0:
   CloseHandle (fhs[0]->hdl_cnt_mtx);
 err_close_query_hdl:
-  CloseHandle (fhs[1]->query_hdl);
+  if (fhs[1]->query_hdl)
+    CloseHandle (fhs[1]->query_hdl);
 err_close_select_sem1:
   CloseHandle (fhs[1]->select_sem);
 err_close_select_sem0:
@@ -946,6 +956,7 @@ nt_create (LPSECURITY_ATTRIBUTES sa_ptr, HANDLE &r, HANDLE &w,
 				 GetCurrentProcessId ());
 
   access = GENERIC_READ | FILE_WRITE_ATTRIBUTES | SYNCHRONIZE;
+  access |= FILE_WRITE_EA; /* Add this right as a marker of cygwin read pipe */
 
   ULONG pipe_type = pipe_byte ? FILE_PIPE_BYTE_STREAM_TYPE
     : FILE_PIPE_MESSAGE_TYPE;
@@ -1112,3 +1123,101 @@ fhandler_pipe::fstatvfs (struct statvfs *sfs)
   set_errno (EBADF);
   return -1;
 }
+
+HANDLE
+fhandler_pipe::tentative_query_hdl ()
+{
+  if (get_dev () != FH_PIPEW)
+    return NULL;
+  /* Try process handle opened and pipe handle value cached first
+     in order to reduce overhead. */
+  if (query_hdl_proc && query_hdl_value)
+    {
+      HANDLE h;
+      if (DuplicateHandle (query_hdl_proc, query_hdl_value,
+			   GetCurrentProcess (), &h, FILE_READ_DATA, 0, 0))
+	return h;
+      CloseHandle (query_hdl_proc);
+      query_hdl_proc = NULL;
+      query_hdl_value = NULL;
+    }
+
+  ULONG len;
+  NTSTATUS status;
+  tmp_pathbuf tp;
+  OBJECT_NAME_INFORMATION *ntfn =(OBJECT_NAME_INFORMATION *) tp.w_get ();
+  status = NtQueryObject (get_handle (), ObjectNameInformation, ntfn,
+			  65536, &len);
+  if (!NT_SUCCESS (status))
+    return NULL; /* Non cygwin pipe? */
+  WCHAR name[MAX_PATH];
+  int namelen = min (ntfn->Name.Length / sizeof (WCHAR), MAX_PATH-1);
+  memcpy (name, ntfn->Name.Buffer, namelen * sizeof (WCHAR));
+  name[namelen] = L'\0';
+  uint64_t key;
+  DWORD pid;
+  LONG id;
+  if (swscanf (name, L"\\Device\\NamedPipe\\%llx-%u-pipe-nt-0x%x",
+	       &key, &pid, &id) != 3)
+    return NULL; /* Non cygwin pipe? */
+
+  SIZE_T n_handle = 65536;
+  SYSTEM_HANDLE_INFORMATION *shi;
+  do
+    {
+      SIZE_T nbytes =
+	sizeof (ULONG) + n_handle * sizeof (SYSTEM_HANDLE_TABLE_ENTRY_INFO);
+      shi = (SYSTEM_HANDLE_INFORMATION *) HeapAlloc (GetProcessHeap (),
+						     0, nbytes);
+      status = NtQuerySystemInformation (SystemHandleInformation,
+					 shi, nbytes, NULL);
+      if (NT_SUCCESS (status))
+	break;
+      HeapFree (GetProcessHeap (), 0, shi);
+      n_handle *= 2;
+    }
+  while (n_handle < (1L<<20));
+  if (!NT_SUCCESS (status))
+    return NULL;
+
+  HANDLE qh = NULL;
+  for (LONG i = (LONG) shi->NumberOfHandles - 1; i >= 0; i--)
+    {
+      /* Check for the peculiarity of cygwin read pipe */
+      DWORD access = FILE_READ_DATA | FILE_READ_EA | FILE_WRITE_EA /* marker */
+	| FILE_READ_ATTRIBUTES | FILE_WRITE_ATTRIBUTES
+	| READ_CONTROL | SYNCHRONIZE;
+      if (shi->Handles[i].GrantedAccess != access)
+	continue;
+
+      /* Retrieve handle */
+      HANDLE proc = OpenProcess (PROCESS_DUP_HANDLE, 0,
+				 shi->Handles[i].UniqueProcessId);
+      if (!proc)
+	continue;
+      HANDLE h = (HANDLE)(intptr_t) shi->Handles[i].HandleValue;
+      BOOL res  = DuplicateHandle (proc, h, GetCurrentProcess (), &h,
+				   FILE_READ_DATA, 0, 0);
+      if (!res)
+	goto close_proc;
+
+      /* Check object name */
+      status = NtQueryObject (h, ObjectNameInformation, ntfn, 65536, &len);
+      if (!NT_SUCCESS (status))
+	goto close_handle;
+      ntfn->Name.Buffer[ntfn->Name.Length / sizeof (WCHAR)] = L'\0';
+      if (wcscmp (name, ntfn->Name.Buffer) == 0)
+	{
+	  query_hdl_proc = proc;
+	  query_hdl_value = (HANDLE)(intptr_t) shi->Handles[i].HandleValue;
+	  qh = h;
+	  break;
+	}
+close_handle:
+      CloseHandle (h);
+close_proc:
+      CloseHandle (proc);
+    }
+  HeapFree (GetProcessHeap (), 0, shi);
+  return qh;
+}
diff --git a/winsup/cygwin/ntdll.h b/winsup/cygwin/ntdll.h
index 4504bdf6d..b45dd702e 100644
--- a/winsup/cygwin/ntdll.h
+++ b/winsup/cygwin/ntdll.h
@@ -620,6 +620,23 @@ typedef struct _SYSTEM_PROCESSOR_PERFORMANCE_INFORMATION
   ULONG InterruptCount;
 } SYSTEM_PROCESSOR_PERFORMANCE_INFORMATION, *PSYSTEM_PROCESSOR_PERFORMANCE_INFORMATION;
 
+typedef struct _SYSTEM_HANDLE_TABLE_ENTRY_INFO
+{
+  USHORT UniqueProcessId;
+  USHORT CreatorBackTraceIndex;
+  UCHAR ObjectTypeIndex;
+  UCHAR HandleAttributes;
+  USHORT HandleValue;
+  PVOID Object;
+  ULONG GrantedAccess;
+} SYSTEM_HANDLE_TABLE_ENTRY_INFO, *PSYSTEM_HANDLE_TABLE_ENTRY_INFO;
+
+typedef struct _SYSTEM_HANDLE_INFORMATION
+{
+  ULONG NumberOfHandles;
+  SYSTEM_HANDLE_TABLE_ENTRY_INFO Handles[1];
+} SYSTEM_HANDLE_INFORMATION, PSYSTEM_HANDLE_INFORMATION;
+
 typedef LONG KPRIORITY;
 
 typedef struct _VM_COUNTERS
diff --git a/winsup/cygwin/select.cc b/winsup/cygwin/select.cc
index 33c0c0bb0..15589b711 100644
--- a/winsup/cygwin/select.cc
+++ b/winsup/cygwin/select.cc
@@ -641,10 +641,16 @@ pipe_data_available (int fd, fhandler_base *fh, HANDLE h, bool writing)
       if (fh->get_device () == FH_PIPEW && fpli.WriteQuotaAvailable == 0)
 	{
 	  HANDLE query_hdl = ((fhandler_pipe *) fh)->get_query_handle ();
+	  if (!query_hdl)
+	    query_hdl = ((fhandler_pipe *) fh)->tentative_query_hdl ();
 	  if (!query_hdl)
 	    return 1; /* We cannot know actual write pipe space. */
 	  DWORD nbytes_in_pipe;
-	  if (!PeekNamedPipe (query_hdl, NULL, 0, NULL, &nbytes_in_pipe, NULL))
+	  BOOL res =
+	    PeekNamedPipe (query_hdl, NULL, 0, NULL, &nbytes_in_pipe, NULL);
+	  if (!((fhandler_pipe *) fh)->get_query_handle ())
+	    CloseHandle (query_hdl); /* Close tentative query_hdl */
+	  if (!res)
 	    return 1;
 	  fpli.WriteQuotaAvailable = fpli.InboundQuota - nbytes_in_pipe;
 	}
-- 
2.33.0


^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-20 22:16                                             ` Takashi Yano
@ 2021-09-20 22:46                                               ` Ken Brown
  2021-09-20 22:50                                                 ` Ken Brown
  0 siblings, 1 reply; 250+ messages in thread
From: Ken Brown @ 2021-09-20 22:46 UTC (permalink / raw)
  To: cygwin-developers

On 9/20/2021 6:16 PM, Takashi Yano wrote:
> On Mon, 20 Sep 2021 17:39:54 -0400
> Ken Brown wrote:
>> I'll push it tomorrow unless you find something that needs to be changed before
>> then.
> 
> typedef of SYSTEM_HANDLE_TABLE_ENTRY_INFO and SYSTEM_HANDLE_INFORMATION
> are moved into ntdll.h

> +typedef struct _SYSTEM_HANDLE_INFORMATION
> +{
> +  ULONG NumberOfHandles;
> +  SYSTEM_HANDLE_TABLE_ENTRY_INFO Handles[1];
> +} SYSTEM_HANDLE_INFORMATION, PSYSTEM_HANDLE_INFORMATION;
> +

You missed a '*' in front of PSYSTEM_HANDLE_INFORMATION.  Also, since you 
defined PSYSTEM_HANDLE_INFORMATION, there are two places where you could use it 
instead of 'SYSTEM_HANDLE_INFORMATION *'.

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-20 22:46                                               ` Ken Brown
@ 2021-09-20 22:50                                                 ` Ken Brown
  2021-09-20 23:22                                                   ` Takashi Yano
  0 siblings, 1 reply; 250+ messages in thread
From: Ken Brown @ 2021-09-20 22:50 UTC (permalink / raw)
  To: cygwin-developers

On 9/20/2021 6:46 PM, Ken Brown wrote:
> On 9/20/2021 6:16 PM, Takashi Yano wrote:
>> On Mon, 20 Sep 2021 17:39:54 -0400
>> Ken Brown wrote:
>>> I'll push it tomorrow unless you find something that needs to be changed before
>>> then.
>>
>> typedef of SYSTEM_HANDLE_TABLE_ENTRY_INFO and SYSTEM_HANDLE_INFORMATION
>> are moved into ntdll.h
> 
>> +typedef struct _SYSTEM_HANDLE_INFORMATION
>> +{
>> +  ULONG NumberOfHandles;
>> +  SYSTEM_HANDLE_TABLE_ENTRY_INFO Handles[1];
>> +} SYSTEM_HANDLE_INFORMATION, PSYSTEM_HANDLE_INFORMATION;
>> +
> 
> You missed a '*' in front of PSYSTEM_HANDLE_INFORMATION.  Also, since you 
> defined PSYSTEM_HANDLE_INFORMATION, there are two places where you could use it 
> instead of 'SYSTEM_HANDLE_INFORMATION *'.

One other thought: I wonder if temporary_query_hdl might be a more suggestive 
name than tentative_query_hdl.

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-20 22:50                                                 ` Ken Brown
@ 2021-09-20 23:22                                                   ` Takashi Yano
  2021-09-21  8:30                                                     ` Takashi Yano
  0 siblings, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-20 23:22 UTC (permalink / raw)
  To: cygwin-developers

[-- Attachment #1: Type: text/plain, Size: 912 bytes --]

On Mon, 20 Sep 2021 18:46:26 -0400
Ken Brown wrote:
> On 9/20/2021 6:16 PM, Takashi Yano wrote:
> > typedef of SYSTEM_HANDLE_TABLE_ENTRY_INFO and SYSTEM_HANDLE_INFORMATION
> > are moved into ntdll.h
> 
> > +typedef struct _SYSTEM_HANDLE_INFORMATION
> > +{
> > +  ULONG NumberOfHandles;
> > +  SYSTEM_HANDLE_TABLE_ENTRY_INFO Handles[1];
> > +} SYSTEM_HANDLE_INFORMATION, PSYSTEM_HANDLE_INFORMATION;
> > +
> 
> You missed a '*' in front of PSYSTEM_HANDLE_INFORMATION.  Also, since you 
> defined PSYSTEM_HANDLE_INFORMATION, there are two places where you could use it 
> instead of 'SYSTEM_HANDLE_INFORMATION *'.

Thanks for pointing out my mistake.

On Mon, 20 Sep 2021 18:50:17 -0400
Ken Brown wrote:
> One other thought: I wonder if temporary_query_hdl might be a more suggestive 
> name than tentative_query_hdl.

And thanks for the suggestion.

I have fixed them.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

[-- Attachment #2: v4-0001-Cygwin-pipe-Introduce-temprary-query_hdl.patch --]
[-- Type: application/octet-stream, Size: 9676 bytes --]

From 1a2f8915fcf56a38f0d44749b559dbfaa84642e8 Mon Sep 17 00:00:00 2001
From: Takashi Yano <takashi.yano@nifty.ne.jp>
Date: Tue, 21 Sep 2021 08:02:43 +0900
Subject: [PATCH v4] Cygwin: pipe: Introduce temprary query_hdl.

- The commit f79a4611 introduced query_hdl which is the read pipe
  handle kept in the write pipe instance in order to determine if
  the pipe is ready to write in select(). This implementation has
  a potential risk that write side fails to detect the closure of
  read side if more than one writers exist and one of them is non-
  cygwin process.

  With this patch, the strategy of commit f79a4611 is used only if
  the process is running as a service. For normal process, instead
  of keeping query_hdl in the write pipe instance, it is retrieved
  temporarily when select() is called. Actually, we want to use
  tenporary query_hdl for all processes, however, it does not work
  for service processes due to OpenProcess() failure.
---
 winsup/cygwin/fhandler.h       |   3 +
 winsup/cygwin/fhandler_pipe.cc | 119 +++++++++++++++++++++++++++++++--
 winsup/cygwin/ntdll.h          |  17 +++++
 winsup/cygwin/select.cc        |   8 ++-
 4 files changed, 141 insertions(+), 6 deletions(-)

diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
index 3471e95b9..2c0586714 100644
--- a/winsup/cygwin/fhandler.h
+++ b/winsup/cygwin/fhandler.h
@@ -1191,6 +1191,8 @@ private:
   pid_t popen_pid;
   HANDLE query_hdl;
   HANDLE hdl_cnt_mtx;
+  HANDLE query_hdl_proc;
+  HANDLE query_hdl_value;
   void release_select_sem (const char *);
 public:
   fhandler_pipe ();
@@ -1249,6 +1251,7 @@ public:
       }
   }
   bool reader_closed ();
+  HANDLE temporary_query_hdl ();
 };
 
 #define CYGWIN_FIFO_PIPE_NAME_LEN     47
diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index 6b41a755f..b1664814f 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -19,6 +19,7 @@ details. */
 #include "cygheap.h"
 #include "pinfo.h"
 #include "shared_info.h"
+#include "tls_pbuf.h"
 
 /* This is only to be used for writing.  When reading,
 STATUS_PIPE_EMPTY simply means there's no data to be read. */
@@ -220,8 +221,6 @@ fhandler_pipe::open_setup (int flags)
 	  goto err_close_read_mtx;
 	}
     }
-  if (get_dev () == FH_PIPEW && !query_hdl)
-    set_pipe_non_blocking (is_nonblocking ());
   return true;
 
 err_close_read_mtx:
@@ -267,7 +266,7 @@ fhandler_pipe::release_select_sem (const char *from)
       - get_obj_handle_count (read_mtx);
   else /* Number of select() call */
     n_release = get_obj_handle_count (select_sem)
-      - get_obj_handle_count (query_hdl);
+      - get_obj_handle_count (hdl_cnt_mtx);
   debug_printf("%s(%s) release %d", from,
 	       get_dev () == FH_PIPER ? "PIPER" : "PIPEW", n_release);
   if (n_release)
@@ -667,6 +666,8 @@ fhandler_pipe::close ()
   int ret = fhandler_base::close ();
   ReleaseMutex (hdl_cnt_mtx);
   CloseHandle (hdl_cnt_mtx);
+  if (query_hdl_proc)
+    CloseHandle (query_hdl_proc);
   return ret;
 }
 
@@ -820,6 +821,13 @@ fhandler_pipe::create (LPSECURITY_ATTRIBUTES sa_ptr, PHANDLE r, PHANDLE w,
   return 0;
 }
 
+inline static bool
+is_running_as_service (void)
+{
+  return check_token_membership (well_known_service_sid)
+    || cygheap->user.saved_sid () == well_known_system_sid;
+}
+
 /* The next version of fhandler_pipe::create used to call the previous
    version.  But the read handle created by the latter doesn't have
    FILE_WRITE_ATTRIBUTES access unless the pipe is created with
@@ -874,7 +882,8 @@ fhandler_pipe::create (fhandler_pipe *fhs[2], unsigned psize, int mode)
 			0, sa->bInheritHandle, DUPLICATE_SAME_ACCESS))
     goto err_close_select_sem0;
 
-  if (!DuplicateHandle (GetCurrentProcess (), r,
+  if (is_running_as_service () &&
+      !DuplicateHandle (GetCurrentProcess (), r,
 			GetCurrentProcess (), &fhs[1]->query_hdl,
 			FILE_READ_DATA, sa->bInheritHandle, 0))
     goto err_close_select_sem1;
@@ -893,7 +902,8 @@ fhandler_pipe::create (fhandler_pipe *fhs[2], unsigned psize, int mode)
 err_close_hdl_cnt_mtx0:
   CloseHandle (fhs[0]->hdl_cnt_mtx);
 err_close_query_hdl:
-  CloseHandle (fhs[1]->query_hdl);
+  if (fhs[1]->query_hdl)
+    CloseHandle (fhs[1]->query_hdl);
 err_close_select_sem1:
   CloseHandle (fhs[1]->select_sem);
 err_close_select_sem0:
@@ -946,6 +956,7 @@ nt_create (LPSECURITY_ATTRIBUTES sa_ptr, HANDLE &r, HANDLE &w,
 				 GetCurrentProcessId ());
 
   access = GENERIC_READ | FILE_WRITE_ATTRIBUTES | SYNCHRONIZE;
+  access |= FILE_WRITE_EA; /* Add this right as a marker of cygwin read pipe */
 
   ULONG pipe_type = pipe_byte ? FILE_PIPE_BYTE_STREAM_TYPE
     : FILE_PIPE_MESSAGE_TYPE;
@@ -1112,3 +1123,101 @@ fhandler_pipe::fstatvfs (struct statvfs *sfs)
   set_errno (EBADF);
   return -1;
 }
+
+HANDLE
+fhandler_pipe::temporary_query_hdl ()
+{
+  if (get_dev () != FH_PIPEW)
+    return NULL;
+  /* Try process handle opened and pipe handle value cached first
+     in order to reduce overhead. */
+  if (query_hdl_proc && query_hdl_value)
+    {
+      HANDLE h;
+      if (DuplicateHandle (query_hdl_proc, query_hdl_value,
+			   GetCurrentProcess (), &h, FILE_READ_DATA, 0, 0))
+	return h;
+      CloseHandle (query_hdl_proc);
+      query_hdl_proc = NULL;
+      query_hdl_value = NULL;
+    }
+
+  ULONG len;
+  NTSTATUS status;
+  tmp_pathbuf tp;
+  OBJECT_NAME_INFORMATION *ntfn =(OBJECT_NAME_INFORMATION *) tp.w_get ();
+  status = NtQueryObject (get_handle (), ObjectNameInformation, ntfn,
+			  65536, &len);
+  if (!NT_SUCCESS (status))
+    return NULL; /* Non cygwin pipe? */
+  WCHAR name[MAX_PATH];
+  int namelen = min (ntfn->Name.Length / sizeof (WCHAR), MAX_PATH-1);
+  memcpy (name, ntfn->Name.Buffer, namelen * sizeof (WCHAR));
+  name[namelen] = L'\0';
+  uint64_t key;
+  DWORD pid;
+  LONG id;
+  if (swscanf (name, L"\\Device\\NamedPipe\\%llx-%u-pipe-nt-0x%x",
+	       &key, &pid, &id) != 3)
+    return NULL; /* Non cygwin pipe? */
+
+  SIZE_T n_handle = 65536;
+  PSYSTEM_HANDLE_INFORMATION shi;
+  do
+    {
+      SIZE_T nbytes =
+	sizeof (ULONG) + n_handle * sizeof (SYSTEM_HANDLE_TABLE_ENTRY_INFO);
+      shi = (PSYSTEM_HANDLE_INFORMATION) HeapAlloc (GetProcessHeap (),
+						     0, nbytes);
+      status = NtQuerySystemInformation (SystemHandleInformation,
+					 shi, nbytes, NULL);
+      if (NT_SUCCESS (status))
+	break;
+      HeapFree (GetProcessHeap (), 0, shi);
+      n_handle *= 2;
+    }
+  while (n_handle < (1L<<20));
+  if (!NT_SUCCESS (status))
+    return NULL;
+
+  HANDLE qh = NULL;
+  for (LONG i = (LONG) shi->NumberOfHandles - 1; i >= 0; i--)
+    {
+      /* Check for the peculiarity of cygwin read pipe */
+      DWORD access = FILE_READ_DATA | FILE_READ_EA | FILE_WRITE_EA /* marker */
+	| FILE_READ_ATTRIBUTES | FILE_WRITE_ATTRIBUTES
+	| READ_CONTROL | SYNCHRONIZE;
+      if (shi->Handles[i].GrantedAccess != access)
+	continue;
+
+      /* Retrieve handle */
+      HANDLE proc = OpenProcess (PROCESS_DUP_HANDLE, 0,
+				 shi->Handles[i].UniqueProcessId);
+      if (!proc)
+	continue;
+      HANDLE h = (HANDLE)(intptr_t) shi->Handles[i].HandleValue;
+      BOOL res  = DuplicateHandle (proc, h, GetCurrentProcess (), &h,
+				   FILE_READ_DATA, 0, 0);
+      if (!res)
+	goto close_proc;
+
+      /* Check object name */
+      status = NtQueryObject (h, ObjectNameInformation, ntfn, 65536, &len);
+      if (!NT_SUCCESS (status))
+	goto close_handle;
+      ntfn->Name.Buffer[ntfn->Name.Length / sizeof (WCHAR)] = L'\0';
+      if (wcscmp (name, ntfn->Name.Buffer) == 0)
+	{
+	  query_hdl_proc = proc;
+	  query_hdl_value = (HANDLE)(intptr_t) shi->Handles[i].HandleValue;
+	  qh = h;
+	  break;
+	}
+close_handle:
+      CloseHandle (h);
+close_proc:
+      CloseHandle (proc);
+    }
+  HeapFree (GetProcessHeap (), 0, shi);
+  return qh;
+}
diff --git a/winsup/cygwin/ntdll.h b/winsup/cygwin/ntdll.h
index 4504bdf6d..e8c3c45c5 100644
--- a/winsup/cygwin/ntdll.h
+++ b/winsup/cygwin/ntdll.h
@@ -620,6 +620,23 @@ typedef struct _SYSTEM_PROCESSOR_PERFORMANCE_INFORMATION
   ULONG InterruptCount;
 } SYSTEM_PROCESSOR_PERFORMANCE_INFORMATION, *PSYSTEM_PROCESSOR_PERFORMANCE_INFORMATION;
 
+typedef struct _SYSTEM_HANDLE_TABLE_ENTRY_INFO
+{
+  USHORT UniqueProcessId;
+  USHORT CreatorBackTraceIndex;
+  UCHAR ObjectTypeIndex;
+  UCHAR HandleAttributes;
+  USHORT HandleValue;
+  PVOID Object;
+  ULONG GrantedAccess;
+} SYSTEM_HANDLE_TABLE_ENTRY_INFO, *PSYSTEM_HANDLE_TABLE_ENTRY_INFO;
+
+typedef struct _SYSTEM_HANDLE_INFORMATION
+{
+  ULONG NumberOfHandles;
+  SYSTEM_HANDLE_TABLE_ENTRY_INFO Handles[1];
+} SYSTEM_HANDLE_INFORMATION, *PSYSTEM_HANDLE_INFORMATION;
+
 typedef LONG KPRIORITY;
 
 typedef struct _VM_COUNTERS
diff --git a/winsup/cygwin/select.cc b/winsup/cygwin/select.cc
index 33c0c0bb0..f27a51683 100644
--- a/winsup/cygwin/select.cc
+++ b/winsup/cygwin/select.cc
@@ -641,10 +641,16 @@ pipe_data_available (int fd, fhandler_base *fh, HANDLE h, bool writing)
       if (fh->get_device () == FH_PIPEW && fpli.WriteQuotaAvailable == 0)
 	{
 	  HANDLE query_hdl = ((fhandler_pipe *) fh)->get_query_handle ();
+	  if (!query_hdl)
+	    query_hdl = ((fhandler_pipe *) fh)->temporary_query_hdl ();
 	  if (!query_hdl)
 	    return 1; /* We cannot know actual write pipe space. */
 	  DWORD nbytes_in_pipe;
-	  if (!PeekNamedPipe (query_hdl, NULL, 0, NULL, &nbytes_in_pipe, NULL))
+	  BOOL res =
+	    PeekNamedPipe (query_hdl, NULL, 0, NULL, &nbytes_in_pipe, NULL);
+	  if (!((fhandler_pipe *) fh)->get_query_handle ())
+	    CloseHandle (query_hdl); /* Close temporary query_hdl */
+	  if (!res)
 	    return 1;
 	  fpli.WriteQuotaAvailable = fpli.InboundQuota - nbytes_in_pipe;
 	}
-- 
2.33.0


^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-20 23:22                                                   ` Takashi Yano
@ 2021-09-21  8:30                                                     ` Takashi Yano
  2021-09-21  9:26                                                       ` Mark Geisert
  2021-09-21 13:31                                                       ` Ken Brown
  0 siblings, 2 replies; 250+ messages in thread
From: Takashi Yano @ 2021-09-21  8:30 UTC (permalink / raw)
  To: cygwin-developers

[-- Attachment #1: Type: text/plain, Size: 337 bytes --]

On Mon, 20 Sep 2021 17:39:54 -0400
Ken Brown wrote:
> I'll push it tomorrow unless you find something that needs to be changed before 
> then.

* Add name check when cached information is used.
* Confirm Name.Buffer returned by NtQueryObject() is not NULL.
* Add error check for HeapAlloc().

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

[-- Attachment #2: v5-0001-Cygwin-pipe-Introduce-temprary-query_hdl.patch --]
[-- Type: application/octet-stream, Size: 10355 bytes --]

From a4008364648966bebd3d2e6ea8cd74397508df39 Mon Sep 17 00:00:00 2001
From: Takashi Yano <takashi.yano@nifty.ne.jp>
Date: Tue, 21 Sep 2021 16:37:20 +0900
Subject: [PATCH v5]     Cygwin: pipe: Introduce temprary query_hdl.

    - The commit f79a4611 introduced query_hdl which is the read pipe
      handle kept in the write pipe instance in order to determine if
      the pipe is ready to write in select(). This implementation has
      a potential risk that write side fails to detect the closure of
      read side if more than one writers exist and one of them is non-
      cygwin process.

      With this patch, the strategy of commit f79a4611 is used only if
      the process is running as a service. For normal process, instead
      of keeping query_hdl in the write pipe instance, it is retrieved
      temporarily when select() is called. Actually, we want to use
      tenporary query_hdl for all processes, however, it does not work
      for service processes due to OpenProcess() failure.
---
 winsup/cygwin/fhandler.h       |   5 ++
 winsup/cygwin/fhandler_pipe.cc | 136 +++++++++++++++++++++++++++++++--
 winsup/cygwin/ntdll.h          |  17 +++++
 winsup/cygwin/select.cc        |   8 +-
 4 files changed, 160 insertions(+), 6 deletions(-)

diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
index 3471e95b9..81d5d0a43 100644
--- a/winsup/cygwin/fhandler.h
+++ b/winsup/cygwin/fhandler.h
@@ -1191,6 +1191,10 @@ private:
   pid_t popen_pid;
   HANDLE query_hdl;
   HANDLE hdl_cnt_mtx;
+  HANDLE query_hdl_proc;
+  HANDLE query_hdl_value;
+  DWORD pipename_pid;
+  LONG pipename_id;
   void release_select_sem (const char *);
 public:
   fhandler_pipe ();
@@ -1249,6 +1253,7 @@ public:
       }
   }
   bool reader_closed ();
+  HANDLE temporary_query_hdl ();
 };
 
 #define CYGWIN_FIFO_PIPE_NAME_LEN     47
diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index 6b41a755f..e94dca6ac 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -19,6 +19,7 @@ details. */
 #include "cygheap.h"
 #include "pinfo.h"
 #include "shared_info.h"
+#include "tls_pbuf.h"
 
 /* This is only to be used for writing.  When reading,
 STATUS_PIPE_EMPTY simply means there's no data to be read. */
@@ -220,8 +221,6 @@ fhandler_pipe::open_setup (int flags)
 	  goto err_close_read_mtx;
 	}
     }
-  if (get_dev () == FH_PIPEW && !query_hdl)
-    set_pipe_non_blocking (is_nonblocking ());
   return true;
 
 err_close_read_mtx:
@@ -267,7 +266,7 @@ fhandler_pipe::release_select_sem (const char *from)
       - get_obj_handle_count (read_mtx);
   else /* Number of select() call */
     n_release = get_obj_handle_count (select_sem)
-      - get_obj_handle_count (query_hdl);
+      - get_obj_handle_count (hdl_cnt_mtx);
   debug_printf("%s(%s) release %d", from,
 	       get_dev () == FH_PIPER ? "PIPER" : "PIPEW", n_release);
   if (n_release)
@@ -667,6 +666,8 @@ fhandler_pipe::close ()
   int ret = fhandler_base::close ();
   ReleaseMutex (hdl_cnt_mtx);
   CloseHandle (hdl_cnt_mtx);
+  if (query_hdl_proc)
+    CloseHandle (query_hdl_proc);
   return ret;
 }
 
@@ -820,6 +821,13 @@ fhandler_pipe::create (LPSECURITY_ATTRIBUTES sa_ptr, PHANDLE r, PHANDLE w,
   return 0;
 }
 
+inline static bool
+is_running_as_service (void)
+{
+  return check_token_membership (well_known_service_sid)
+    || cygheap->user.saved_sid () == well_known_system_sid;
+}
+
 /* The next version of fhandler_pipe::create used to call the previous
    version.  But the read handle created by the latter doesn't have
    FILE_WRITE_ATTRIBUTES access unless the pipe is created with
@@ -874,7 +882,8 @@ fhandler_pipe::create (fhandler_pipe *fhs[2], unsigned psize, int mode)
 			0, sa->bInheritHandle, DUPLICATE_SAME_ACCESS))
     goto err_close_select_sem0;
 
-  if (!DuplicateHandle (GetCurrentProcess (), r,
+  if (is_running_as_service () &&
+      !DuplicateHandle (GetCurrentProcess (), r,
 			GetCurrentProcess (), &fhs[1]->query_hdl,
 			FILE_READ_DATA, sa->bInheritHandle, 0))
     goto err_close_select_sem1;
@@ -893,7 +902,8 @@ fhandler_pipe::create (fhandler_pipe *fhs[2], unsigned psize, int mode)
 err_close_hdl_cnt_mtx0:
   CloseHandle (fhs[0]->hdl_cnt_mtx);
 err_close_query_hdl:
-  CloseHandle (fhs[1]->query_hdl);
+  if (fhs[1]->query_hdl)
+    CloseHandle (fhs[1]->query_hdl);
 err_close_select_sem1:
   CloseHandle (fhs[1]->select_sem);
 err_close_select_sem0:
@@ -946,6 +956,7 @@ nt_create (LPSECURITY_ATTRIBUTES sa_ptr, HANDLE &r, HANDLE &w,
 				 GetCurrentProcessId ());
 
   access = GENERIC_READ | FILE_WRITE_ATTRIBUTES | SYNCHRONIZE;
+  access |= FILE_WRITE_EA; /* Add this right as a marker of cygwin read pipe */
 
   ULONG pipe_type = pipe_byte ? FILE_PIPE_BYTE_STREAM_TYPE
     : FILE_PIPE_MESSAGE_TYPE;
@@ -1112,3 +1123,118 @@ fhandler_pipe::fstatvfs (struct statvfs *sfs)
   set_errno (EBADF);
   return -1;
 }
+
+HANDLE
+fhandler_pipe::temporary_query_hdl ()
+{
+  if (get_dev () != FH_PIPEW)
+    return NULL;
+
+  ULONG len;
+  NTSTATUS status;
+  tmp_pathbuf tp;
+  OBJECT_NAME_INFORMATION *ntfn = (OBJECT_NAME_INFORMATION *) tp.w_get ();
+  uint64_t key;
+
+  /* Try process handle opened and pipe handle value cached first
+     in order to reduce overhead. */
+  if (query_hdl_proc && query_hdl_value)
+    {
+      HANDLE h;
+      if (!DuplicateHandle (query_hdl_proc, query_hdl_value,
+			   GetCurrentProcess (), &h, FILE_READ_DATA, 0, 0))
+	goto cache_err;
+      /* Check name */
+      status = NtQueryObject (h, ObjectNameInformation, ntfn, 65536, &len);
+      if (!NT_SUCCESS (status) || !ntfn->Name.Buffer)
+	goto hdl_err;
+      ntfn->Name.Buffer[ntfn->Name.Length / sizeof (WCHAR)] = L'\0';
+      DWORD pid;
+      LONG id;
+      if (swscanf (ntfn->Name.Buffer,
+		   L"\\Device\\NamedPipe\\%llx-%u-pipe-nt-0x%x",
+		   &key, &pid, &id) == 3 &&
+	  pid == pipename_pid && id == pipename_id)
+	return h;
+hdl_err:
+      CloseHandle (h);
+cache_err:
+      CloseHandle (query_hdl_proc);
+      query_hdl_proc = NULL;
+      query_hdl_value = NULL;
+    }
+
+  status = NtQueryObject (get_handle (), ObjectNameInformation, ntfn,
+			  65536, &len);
+  if (!NT_SUCCESS (status) || !ntfn->Name.Buffer)
+    return NULL; /* Non cygwin pipe? */
+  WCHAR name[MAX_PATH];
+  int namelen = min (ntfn->Name.Length / sizeof (WCHAR), MAX_PATH-1);
+  memcpy (name, ntfn->Name.Buffer, namelen * sizeof (WCHAR));
+  name[namelen] = L'\0';
+  if (swscanf (name, L"\\Device\\NamedPipe\\%llx-%u-pipe-nt-0x%x",
+	       &key, &pipename_pid, &pipename_id) != 3)
+    return NULL; /* Non cygwin pipe? */
+
+  SIZE_T n_handle = 65536;
+  PSYSTEM_HANDLE_INFORMATION shi;
+  do
+    {
+      SIZE_T nbytes =
+	sizeof (ULONG) + n_handle * sizeof (SYSTEM_HANDLE_TABLE_ENTRY_INFO);
+      shi = (PSYSTEM_HANDLE_INFORMATION) HeapAlloc (GetProcessHeap (),
+						     0, nbytes);
+      if (!shi)
+	return NULL;
+      status = NtQuerySystemInformation (SystemHandleInformation,
+					 shi, nbytes, NULL);
+      if (NT_SUCCESS (status))
+	break;
+      HeapFree (GetProcessHeap (), 0, shi);
+      n_handle *= 2;
+    }
+  while (n_handle < (1L<<20));
+  if (!NT_SUCCESS (status))
+    return NULL;
+
+  HANDLE qh = NULL;
+  for (LONG i = (LONG) shi->NumberOfHandles - 1; i >= 0; i--)
+    {
+      /* Check for the peculiarity of cygwin read pipe */
+      DWORD access = FILE_READ_DATA | FILE_READ_EA | FILE_WRITE_EA /* marker */
+	| FILE_READ_ATTRIBUTES | FILE_WRITE_ATTRIBUTES
+	| READ_CONTROL | SYNCHRONIZE;
+      if (shi->Handles[i].GrantedAccess != access)
+	continue;
+
+      /* Retrieve handle */
+      HANDLE proc = OpenProcess (PROCESS_DUP_HANDLE, 0,
+				 shi->Handles[i].UniqueProcessId);
+      if (!proc)
+	continue;
+      HANDLE h = (HANDLE)(intptr_t) shi->Handles[i].HandleValue;
+      BOOL res  = DuplicateHandle (proc, h, GetCurrentProcess (), &h,
+				   FILE_READ_DATA, 0, 0);
+      if (!res)
+	goto close_proc;
+
+      /* Check object name */
+      status = NtQueryObject (h, ObjectNameInformation, ntfn, 65536, &len);
+      if (!NT_SUCCESS (status) || !ntfn->Name.Buffer)
+	goto close_handle;
+      ntfn->Name.Buffer[ntfn->Name.Length / sizeof (WCHAR)] = L'\0';
+      if (wcscmp (name, ntfn->Name.Buffer) == 0)
+	{
+	  query_hdl_proc = proc;
+	  query_hdl_value = (HANDLE)(intptr_t) shi->Handles[i].HandleValue;
+	  qh = h;
+	  break;
+	}
+close_handle:
+      CloseHandle (h);
+close_proc:
+      CloseHandle (proc);
+    }
+  HeapFree (GetProcessHeap (), 0, shi);
+  return qh;
+}
diff --git a/winsup/cygwin/ntdll.h b/winsup/cygwin/ntdll.h
index 4504bdf6d..e8c3c45c5 100644
--- a/winsup/cygwin/ntdll.h
+++ b/winsup/cygwin/ntdll.h
@@ -620,6 +620,23 @@ typedef struct _SYSTEM_PROCESSOR_PERFORMANCE_INFORMATION
   ULONG InterruptCount;
 } SYSTEM_PROCESSOR_PERFORMANCE_INFORMATION, *PSYSTEM_PROCESSOR_PERFORMANCE_INFORMATION;
 
+typedef struct _SYSTEM_HANDLE_TABLE_ENTRY_INFO
+{
+  USHORT UniqueProcessId;
+  USHORT CreatorBackTraceIndex;
+  UCHAR ObjectTypeIndex;
+  UCHAR HandleAttributes;
+  USHORT HandleValue;
+  PVOID Object;
+  ULONG GrantedAccess;
+} SYSTEM_HANDLE_TABLE_ENTRY_INFO, *PSYSTEM_HANDLE_TABLE_ENTRY_INFO;
+
+typedef struct _SYSTEM_HANDLE_INFORMATION
+{
+  ULONG NumberOfHandles;
+  SYSTEM_HANDLE_TABLE_ENTRY_INFO Handles[1];
+} SYSTEM_HANDLE_INFORMATION, *PSYSTEM_HANDLE_INFORMATION;
+
 typedef LONG KPRIORITY;
 
 typedef struct _VM_COUNTERS
diff --git a/winsup/cygwin/select.cc b/winsup/cygwin/select.cc
index 33c0c0bb0..a2868abd0 100644
--- a/winsup/cygwin/select.cc
+++ b/winsup/cygwin/select.cc
@@ -641,10 +641,16 @@ pipe_data_available (int fd, fhandler_base *fh, HANDLE h, bool writing)
       if (fh->get_device () == FH_PIPEW && fpli.WriteQuotaAvailable == 0)
 	{
 	  HANDLE query_hdl = ((fhandler_pipe *) fh)->get_query_handle ();
+	  if (!query_hdl)
+	    query_hdl = ((fhandler_pipe *) fh)->temporary_query_hdl ();
 	  if (!query_hdl)
 	    return 1; /* We cannot know actual write pipe space. */
 	  DWORD nbytes_in_pipe;
-	  if (!PeekNamedPipe (query_hdl, NULL, 0, NULL, &nbytes_in_pipe, NULL))
+	  BOOL res =
+	    PeekNamedPipe (query_hdl, NULL, 0, NULL, &nbytes_in_pipe, NULL);
+	  if (!((fhandler_pipe *) fh)->get_query_handle ())
+	    CloseHandle (query_hdl); /* Close temporary query_hdl */
+	  if (!res)
 	    return 1;
 	  fpli.WriteQuotaAvailable = fpli.InboundQuota - nbytes_in_pipe;
 	}
-- 
2.33.0


^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-21  8:30                                                     ` Takashi Yano
@ 2021-09-21  9:26                                                       ` Mark Geisert
  2021-09-21 10:10                                                         ` Takashi Yano
  2021-09-21 13:31                                                       ` Ken Brown
  1 sibling, 1 reply; 250+ messages in thread
From: Mark Geisert @ 2021-09-21  9:26 UTC (permalink / raw)
  To: Takashi Yano; +Cc: cygwin-developers


Sep 21, 2021 1:31:28 AM Takashi Yano <takashi.yano@nifty.ne.jp>:

> On Mon, 20 Sep 2021 17:39:54 -0400
> Ken Brown wrote:
>> I'll push it tomorrow unless you find something that needs to be changed before
>> then.
> 
> * Add name check when cached information is used.
> * Confirm Name.Buffer returned by NtQueryObject() is not NULL.
> * Add error check for HeapAlloc().
> 
> -- 
> Takashi Yano <takashi.yano@nifty.ne.jp>
Sorry if this is a dumb question, but why HeapAlloc? If the buffer is only needed temporarily wouldn't alloca work? Or malloc?

.mark

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-21  9:26                                                       ` Mark Geisert
@ 2021-09-21 10:10                                                         ` Takashi Yano
  2021-09-21 21:10                                                           ` Mark Geisert
  0 siblings, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-21 10:10 UTC (permalink / raw)
  To: cygwin-developers

On Tue, 21 Sep 2021 09:26:09 +0000 (UTC)
Mark Geisert wrote:
> Sep 21, 2021 1:31:28 AM Takashi Yano <takashi.yano@nifty.ne.jp>:
> > On Mon, 20 Sep 2021 17:39:54 -0400
> > Ken Brown wrote:
> >> I'll push it tomorrow unless you find something that needs to be changed before
> >> then.
> > 
> > * Add name check when cached information is used.
> > * Confirm Name.Buffer returned by NtQueryObject() is not NULL.
> > * Add error check for HeapAlloc().
> > 
> Sorry if this is a dumb question, but why HeapAlloc? If the buffer is only needed temporarily wouldn't alloca work? Or malloc?

This function (temporary_query_hdl()) is called within a  system call,
so calling another system call from it is not good idea I think.
For example, errno may be changed by malloc().

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-21  8:30                                                     ` Takashi Yano
  2021-09-21  9:26                                                       ` Mark Geisert
@ 2021-09-21 13:31                                                       ` Ken Brown
  2021-09-21 15:36                                                         ` Takashi Yano
  1 sibling, 1 reply; 250+ messages in thread
From: Ken Brown @ 2021-09-21 13:31 UTC (permalink / raw)
  To: cygwin-developers

On 9/21/2021 4:30 AM, Takashi Yano wrote:
> On Mon, 20 Sep 2021 17:39:54 -0400
> Ken Brown wrote:
>> I'll push it tomorrow unless you find something that needs to be changed before
>> then.
> 
> * Add name check when cached information is used.
> * Confirm Name.Buffer returned by NtQueryObject() is not NULL.
> * Add error check for HeapAlloc().

Still looks good.  One small question: Is there a reason you cache the pipe 
name's pid and id but not the key?  Wouldn't you get a more complete name check 
by also using the key?

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-21 13:31                                                       ` Ken Brown
@ 2021-09-21 15:36                                                         ` Takashi Yano
  2021-09-21 18:51                                                           ` Ken Brown
  0 siblings, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-21 15:36 UTC (permalink / raw)
  To: cygwin-developers

[-- Attachment #1: Type: text/plain, Size: 907 bytes --]

On Tue, 21 Sep 2021 09:31:45 -0400
Ken Brown wrote:
> On 9/21/2021 4:30 AM, Takashi Yano wrote:
> > On Mon, 20 Sep 2021 17:39:54 -0400
> > Ken Brown wrote:
> >> I'll push it tomorrow unless you find something that needs to be changed before
> >> then.
> > 
> > * Add name check when cached information is used.
> > * Confirm Name.Buffer returned by NtQueryObject() is not NULL.
> > * Add error check for HeapAlloc().
> 
> Still looks good.  One small question: Is there a reason you cache the pipe 
> name's pid and id but not the key?  Wouldn't you get a more complete name check 
> by also using the key?

key always has same value, so I did not think it is necessary to
be checked. However, if someone has multiple cygwin installations,
pipe name may be confilict without keys.
Therefore, I have added key check as well as pid and id.

Thanks for the advice.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

[-- Attachment #2: v6-0001-Cygwin-pipe-Introduce-temprary-query_hdl.patch --]
[-- Type: application/octet-stream, Size: 10365 bytes --]

From 3f68e8d0e8992bba6663f2b6b3868ea9d3598518 Mon Sep 17 00:00:00 2001
From: Takashi Yano <takashi.yano@nifty.ne.jp>
Date: Wed, 22 Sep 2021 00:30:34 +0900
Subject: [PATCH v6] Cygwin: pipe: Introduce temprary query_hdl.

- The commit f79a4611 introduced query_hdl which is the read pipe
  handle kept in the write pipe instance in order to determine if
  the pipe is ready to write in select(). This implementation has
  a potential risk that write side fails to detect the closure of
  read side if more than one writers exist and one of them is non-
  cygwin process.

  With this patch, the strategy of commit f79a4611 is used only if
  the process is running as a service. For normal process, instead
  of keeping query_hdl in the write pipe instance, it is retrieved
  temporarily when select() is called. Actually, we want to use
  tenporary query_hdl for all processes, however, it does not work
  for service processes due to OpenProcess() failure.
---
 winsup/cygwin/fhandler.h       |   6 ++
 winsup/cygwin/fhandler_pipe.cc | 136 +++++++++++++++++++++++++++++++--
 winsup/cygwin/ntdll.h          |  17 +++++
 winsup/cygwin/select.cc        |   8 +-
 4 files changed, 161 insertions(+), 6 deletions(-)

diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
index 3471e95b9..0061d4830 100644
--- a/winsup/cygwin/fhandler.h
+++ b/winsup/cygwin/fhandler.h
@@ -1191,6 +1191,11 @@ private:
   pid_t popen_pid;
   HANDLE query_hdl;
   HANDLE hdl_cnt_mtx;
+  HANDLE query_hdl_proc;
+  HANDLE query_hdl_value;
+  uint64_t pipename_key;
+  DWORD pipename_pid;
+  LONG pipename_id;
   void release_select_sem (const char *);
 public:
   fhandler_pipe ();
@@ -1249,6 +1254,7 @@ public:
       }
   }
   bool reader_closed ();
+  HANDLE temporary_query_hdl ();
 };
 
 #define CYGWIN_FIFO_PIPE_NAME_LEN     47
diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index 6b41a755f..78e2f90d8 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -19,6 +19,7 @@ details. */
 #include "cygheap.h"
 #include "pinfo.h"
 #include "shared_info.h"
+#include "tls_pbuf.h"
 
 /* This is only to be used for writing.  When reading,
 STATUS_PIPE_EMPTY simply means there's no data to be read. */
@@ -220,8 +221,6 @@ fhandler_pipe::open_setup (int flags)
 	  goto err_close_read_mtx;
 	}
     }
-  if (get_dev () == FH_PIPEW && !query_hdl)
-    set_pipe_non_blocking (is_nonblocking ());
   return true;
 
 err_close_read_mtx:
@@ -267,7 +266,7 @@ fhandler_pipe::release_select_sem (const char *from)
       - get_obj_handle_count (read_mtx);
   else /* Number of select() call */
     n_release = get_obj_handle_count (select_sem)
-      - get_obj_handle_count (query_hdl);
+      - get_obj_handle_count (hdl_cnt_mtx);
   debug_printf("%s(%s) release %d", from,
 	       get_dev () == FH_PIPER ? "PIPER" : "PIPEW", n_release);
   if (n_release)
@@ -667,6 +666,8 @@ fhandler_pipe::close ()
   int ret = fhandler_base::close ();
   ReleaseMutex (hdl_cnt_mtx);
   CloseHandle (hdl_cnt_mtx);
+  if (query_hdl_proc)
+    CloseHandle (query_hdl_proc);
   return ret;
 }
 
@@ -820,6 +821,13 @@ fhandler_pipe::create (LPSECURITY_ATTRIBUTES sa_ptr, PHANDLE r, PHANDLE w,
   return 0;
 }
 
+inline static bool
+is_running_as_service (void)
+{
+  return check_token_membership (well_known_service_sid)
+    || cygheap->user.saved_sid () == well_known_system_sid;
+}
+
 /* The next version of fhandler_pipe::create used to call the previous
    version.  But the read handle created by the latter doesn't have
    FILE_WRITE_ATTRIBUTES access unless the pipe is created with
@@ -874,7 +882,8 @@ fhandler_pipe::create (fhandler_pipe *fhs[2], unsigned psize, int mode)
 			0, sa->bInheritHandle, DUPLICATE_SAME_ACCESS))
     goto err_close_select_sem0;
 
-  if (!DuplicateHandle (GetCurrentProcess (), r,
+  if (is_running_as_service () &&
+      !DuplicateHandle (GetCurrentProcess (), r,
 			GetCurrentProcess (), &fhs[1]->query_hdl,
 			FILE_READ_DATA, sa->bInheritHandle, 0))
     goto err_close_select_sem1;
@@ -893,7 +902,8 @@ fhandler_pipe::create (fhandler_pipe *fhs[2], unsigned psize, int mode)
 err_close_hdl_cnt_mtx0:
   CloseHandle (fhs[0]->hdl_cnt_mtx);
 err_close_query_hdl:
-  CloseHandle (fhs[1]->query_hdl);
+  if (fhs[1]->query_hdl)
+    CloseHandle (fhs[1]->query_hdl);
 err_close_select_sem1:
   CloseHandle (fhs[1]->select_sem);
 err_close_select_sem0:
@@ -946,6 +956,7 @@ nt_create (LPSECURITY_ATTRIBUTES sa_ptr, HANDLE &r, HANDLE &w,
 				 GetCurrentProcessId ());
 
   access = GENERIC_READ | FILE_WRITE_ATTRIBUTES | SYNCHRONIZE;
+  access |= FILE_WRITE_EA; /* Add this right as a marker of cygwin read pipe */
 
   ULONG pipe_type = pipe_byte ? FILE_PIPE_BYTE_STREAM_TYPE
     : FILE_PIPE_MESSAGE_TYPE;
@@ -1112,3 +1123,118 @@ fhandler_pipe::fstatvfs (struct statvfs *sfs)
   set_errno (EBADF);
   return -1;
 }
+
+HANDLE
+fhandler_pipe::temporary_query_hdl ()
+{
+  if (get_dev () != FH_PIPEW)
+    return NULL;
+
+  ULONG len;
+  NTSTATUS status;
+  tmp_pathbuf tp;
+  OBJECT_NAME_INFORMATION *ntfn = (OBJECT_NAME_INFORMATION *) tp.w_get ();
+
+  /* Try process handle opened and pipe handle value cached first
+     in order to reduce overhead. */
+  if (query_hdl_proc && query_hdl_value)
+    {
+      HANDLE h;
+      if (!DuplicateHandle (query_hdl_proc, query_hdl_value,
+			   GetCurrentProcess (), &h, FILE_READ_DATA, 0, 0))
+	goto cache_err;
+      /* Check name */
+      status = NtQueryObject (h, ObjectNameInformation, ntfn, 65536, &len);
+      if (!NT_SUCCESS (status) || !ntfn->Name.Buffer)
+	goto hdl_err;
+      ntfn->Name.Buffer[ntfn->Name.Length / sizeof (WCHAR)] = L'\0';
+      uint64_t key;
+      DWORD pid;
+      LONG id;
+      if (swscanf (ntfn->Name.Buffer,
+		   L"\\Device\\NamedPipe\\%llx-%u-pipe-nt-0x%x",
+		   &key, &pid, &id) == 3 &&
+	  key == pipename_key && pid == pipename_pid && id == pipename_id)
+	return h;
+hdl_err:
+      CloseHandle (h);
+cache_err:
+      CloseHandle (query_hdl_proc);
+      query_hdl_proc = NULL;
+      query_hdl_value = NULL;
+    }
+
+  status = NtQueryObject (get_handle (), ObjectNameInformation, ntfn,
+			  65536, &len);
+  if (!NT_SUCCESS (status) || !ntfn->Name.Buffer)
+    return NULL; /* Non cygwin pipe? */
+  WCHAR name[MAX_PATH];
+  int namelen = min (ntfn->Name.Length / sizeof (WCHAR), MAX_PATH-1);
+  memcpy (name, ntfn->Name.Buffer, namelen * sizeof (WCHAR));
+  name[namelen] = L'\0';
+  if (swscanf (name, L"\\Device\\NamedPipe\\%llx-%u-pipe-nt-0x%x",
+	       &pipename_key, &pipename_pid, &pipename_id) != 3)
+    return NULL; /* Non cygwin pipe? */
+
+  SIZE_T n_handle = 65536;
+  PSYSTEM_HANDLE_INFORMATION shi;
+  do
+    {
+      SIZE_T nbytes =
+	sizeof (ULONG) + n_handle * sizeof (SYSTEM_HANDLE_TABLE_ENTRY_INFO);
+      shi = (PSYSTEM_HANDLE_INFORMATION) HeapAlloc (GetProcessHeap (),
+						     0, nbytes);
+      if (!shi)
+	return NULL;
+      status = NtQuerySystemInformation (SystemHandleInformation,
+					 shi, nbytes, NULL);
+      if (NT_SUCCESS (status))
+	break;
+      HeapFree (GetProcessHeap (), 0, shi);
+      n_handle *= 2;
+    }
+  while (n_handle < (1L<<20));
+  if (!NT_SUCCESS (status))
+    return NULL;
+
+  HANDLE qh = NULL;
+  for (LONG i = (LONG) shi->NumberOfHandles - 1; i >= 0; i--)
+    {
+      /* Check for the peculiarity of cygwin read pipe */
+      DWORD access = FILE_READ_DATA | FILE_READ_EA | FILE_WRITE_EA /* marker */
+	| FILE_READ_ATTRIBUTES | FILE_WRITE_ATTRIBUTES
+	| READ_CONTROL | SYNCHRONIZE;
+      if (shi->Handles[i].GrantedAccess != access)
+	continue;
+
+      /* Retrieve handle */
+      HANDLE proc = OpenProcess (PROCESS_DUP_HANDLE, 0,
+				 shi->Handles[i].UniqueProcessId);
+      if (!proc)
+	continue;
+      HANDLE h = (HANDLE)(intptr_t) shi->Handles[i].HandleValue;
+      BOOL res  = DuplicateHandle (proc, h, GetCurrentProcess (), &h,
+				   FILE_READ_DATA, 0, 0);
+      if (!res)
+	goto close_proc;
+
+      /* Check object name */
+      status = NtQueryObject (h, ObjectNameInformation, ntfn, 65536, &len);
+      if (!NT_SUCCESS (status) || !ntfn->Name.Buffer)
+	goto close_handle;
+      ntfn->Name.Buffer[ntfn->Name.Length / sizeof (WCHAR)] = L'\0';
+      if (wcscmp (name, ntfn->Name.Buffer) == 0)
+	{
+	  query_hdl_proc = proc;
+	  query_hdl_value = (HANDLE)(intptr_t) shi->Handles[i].HandleValue;
+	  qh = h;
+	  break;
+	}
+close_handle:
+      CloseHandle (h);
+close_proc:
+      CloseHandle (proc);
+    }
+  HeapFree (GetProcessHeap (), 0, shi);
+  return qh;
+}
diff --git a/winsup/cygwin/ntdll.h b/winsup/cygwin/ntdll.h
index 4504bdf6d..e8c3c45c5 100644
--- a/winsup/cygwin/ntdll.h
+++ b/winsup/cygwin/ntdll.h
@@ -620,6 +620,23 @@ typedef struct _SYSTEM_PROCESSOR_PERFORMANCE_INFORMATION
   ULONG InterruptCount;
 } SYSTEM_PROCESSOR_PERFORMANCE_INFORMATION, *PSYSTEM_PROCESSOR_PERFORMANCE_INFORMATION;
 
+typedef struct _SYSTEM_HANDLE_TABLE_ENTRY_INFO
+{
+  USHORT UniqueProcessId;
+  USHORT CreatorBackTraceIndex;
+  UCHAR ObjectTypeIndex;
+  UCHAR HandleAttributes;
+  USHORT HandleValue;
+  PVOID Object;
+  ULONG GrantedAccess;
+} SYSTEM_HANDLE_TABLE_ENTRY_INFO, *PSYSTEM_HANDLE_TABLE_ENTRY_INFO;
+
+typedef struct _SYSTEM_HANDLE_INFORMATION
+{
+  ULONG NumberOfHandles;
+  SYSTEM_HANDLE_TABLE_ENTRY_INFO Handles[1];
+} SYSTEM_HANDLE_INFORMATION, *PSYSTEM_HANDLE_INFORMATION;
+
 typedef LONG KPRIORITY;
 
 typedef struct _VM_COUNTERS
diff --git a/winsup/cygwin/select.cc b/winsup/cygwin/select.cc
index 33c0c0bb0..a2868abd0 100644
--- a/winsup/cygwin/select.cc
+++ b/winsup/cygwin/select.cc
@@ -641,10 +641,16 @@ pipe_data_available (int fd, fhandler_base *fh, HANDLE h, bool writing)
       if (fh->get_device () == FH_PIPEW && fpli.WriteQuotaAvailable == 0)
 	{
 	  HANDLE query_hdl = ((fhandler_pipe *) fh)->get_query_handle ();
+	  if (!query_hdl)
+	    query_hdl = ((fhandler_pipe *) fh)->temporary_query_hdl ();
 	  if (!query_hdl)
 	    return 1; /* We cannot know actual write pipe space. */
 	  DWORD nbytes_in_pipe;
-	  if (!PeekNamedPipe (query_hdl, NULL, 0, NULL, &nbytes_in_pipe, NULL))
+	  BOOL res =
+	    PeekNamedPipe (query_hdl, NULL, 0, NULL, &nbytes_in_pipe, NULL);
+	  if (!((fhandler_pipe *) fh)->get_query_handle ())
+	    CloseHandle (query_hdl); /* Close temporary query_hdl */
+	  if (!res)
 	    return 1;
 	  fpli.WriteQuotaAvailable = fpli.InboundQuota - nbytes_in_pipe;
 	}
-- 
2.33.0


^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-21 15:36                                                         ` Takashi Yano
@ 2021-09-21 18:51                                                           ` Ken Brown
  2021-09-23  8:26                                                             ` Takashi Yano
  0 siblings, 1 reply; 250+ messages in thread
From: Ken Brown @ 2021-09-21 18:51 UTC (permalink / raw)
  To: cygwin-developers

On 9/21/2021 11:36 AM, Takashi Yano wrote:
> On Tue, 21 Sep 2021 09:31:45 -0400
> Ken Brown wrote:
>> On 9/21/2021 4:30 AM, Takashi Yano wrote:
>>> On Mon, 20 Sep 2021 17:39:54 -0400
>>> Ken Brown wrote:
>>>> I'll push it tomorrow unless you find something that needs to be changed before
>>>> then.
>>>
>>> * Add name check when cached information is used.
>>> * Confirm Name.Buffer returned by NtQueryObject() is not NULL.
>>> * Add error check for HeapAlloc().
>>
>> Still looks good.  One small question: Is there a reason you cache the pipe
>> name's pid and id but not the key?  Wouldn't you get a more complete name check
>> by also using the key?
> 
> key always has same value, so I did not think it is necessary to
> be checked. However, if someone has multiple cygwin installations,
> pipe name may be confilict without keys.
> Therefore, I have added key check as well as pid and id.
> 
> Thanks for the advice.

Thanks again for the patch.  I've pushed it to master.  I'll probably make 
another test release at the end of the week.

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-21 10:10                                                         ` Takashi Yano
@ 2021-09-21 21:10                                                           ` Mark Geisert
  0 siblings, 0 replies; 250+ messages in thread
From: Mark Geisert @ 2021-09-21 21:10 UTC (permalink / raw)
  To: cygwin-developers

Takashi Yano wrote:
> On Tue, 21 Sep 2021 09:26:09 +0000 (UTC)
> Mark Geisert wrote:
>> Sep 21, 2021 1:31:28 AM Takashi Yano <takashi.yano-fFkzxiWP14B3+QwDJ9on6Q@public.gmane.org>:
>>> On Mon, 20 Sep 2021 17:39:54 -0400
>>> Ken Brown wrote:
>>>> I'll push it tomorrow unless you find something that needs to be changed before
>>>> then.
>>>
>>> * Add name check when cached information is used.
>>> * Confirm Name.Buffer returned by NtQueryObject() is not NULL.
>>> * Add error check for HeapAlloc().
>>>
>> Sorry if this is a dumb question, but why HeapAlloc? If the buffer is only needed temporarily wouldn't alloca work? Or malloc?
> 
> This function (temporary_query_hdl()) is called within a  system call,
> so calling another system call from it is not good idea I think.
> For example, errno may be changed by malloc().

I've now had a chance to look at the surrounding code and I have no big issue with 
it.  It's just uncommon to see HeapAlloc/HeapFree used within the Cygwin DLL 
because blocks allocated with it won't be free-able in the child after a fork. 
But for short-term use guaranteed to be localized in-process like what you're 
doing I guess it's OK.  Carry on!

..mark

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-21 18:51                                                           ` Ken Brown
@ 2021-09-23  8:26                                                             ` Takashi Yano
  2021-09-23 13:03                                                               ` Ken Brown
  0 siblings, 1 reply; 250+ messages in thread
From: Takashi Yano @ 2021-09-23  8:26 UTC (permalink / raw)
  To: cygwin-developers

[-- Attachment #1: Type: text/plain, Size: 260 bytes --]

On Tue, 21 Sep 2021 14:51:46 -0400
Ken Brown wrote:
> Thanks again for the patch.  I've pushed it to master.  I'll probably make 
> another test release at the end of the week.

* Make temporary_query_hdl() faster.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

[-- Attachment #2: 0001-Cygwin-pipe-Use-ProcessHandleInformation-if-availabl.patch --]
[-- Type: application/octet-stream, Size: 11819 bytes --]

From db1948077e797cc3fb0d721b52f940a0f99426da Mon Sep 17 00:00:00 2001
From: Takashi Yano <takashi.yano@nifty.ne.jp>
Date: Thu, 23 Sep 2021 16:47:44 +0900
Subject: [PATCH] Cygwin: pipe: Use ProcessHandleInformation if available.

- The commit b531d6b0 introduced temporary_query_hdl() which uses
  SystemHandleInformation. With this patch, ProcessHandleInformation
  rather than SystemHandleInformation is used if it is available.
  This request is faster, however, is only available since Windows 8,
  therefore, SystemHandleInformation is used for Windows Vista and 7
  as before.
---
 winsup/cygwin/fhandler.h       |   2 +
 winsup/cygwin/fhandler_pipe.cc | 132 +++++++++++++++++++++++++++++++--
 winsup/cygwin/ntdll.h          |  21 +++++-
 winsup/cygwin/wincap.cc        |  11 +++
 winsup/cygwin/wincap.h         |   2 +
 5 files changed, 159 insertions(+), 9 deletions(-)

diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
index 0061d4830..c033f7816 100644
--- a/winsup/cygwin/fhandler.h
+++ b/winsup/cygwin/fhandler.h
@@ -1197,6 +1197,8 @@ private:
   DWORD pipename_pid;
   LONG pipename_id;
   void release_select_sem (const char *);
+  HANDLE get_query_hdl_per_process (WCHAR *, OBJECT_NAME_INFORMATION *);
+  HANDLE get_query_hdl_per_system (WCHAR *, OBJECT_NAME_INFORMATION *);
 public:
   fhandler_pipe ();
 
diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index 78e2f90d8..e2df69c92 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -20,6 +20,7 @@ details. */
 #include "pinfo.h"
 #include "shared_info.h"
 #include "tls_pbuf.h"
+#include <psapi.h>
 
 /* This is only to be used for writing.  When reading,
 STATUS_PIPE_EMPTY simply means there's no data to be read. */
@@ -1176,14 +1177,128 @@ cache_err:
 	       &pipename_key, &pipename_pid, &pipename_id) != 3)
     return NULL; /* Non cygwin pipe? */
 
+  if (wincap.has_query_process_handle_info ())
+    return get_query_hdl_per_process (name, ntfn); /* Since Win8 */
+  else
+    return get_query_hdl_per_system (name, ntfn); /* Vista or Win7 */
+}
+
+/* This function is faster than get_query_hdl_per_system(), however,
+   only works since Windows 8 because ProcessHandleInformation is not
+   suppoted by NtQueryInformationProcess() before Windows 8. */
+HANDLE
+fhandler_pipe::get_query_hdl_per_process (WCHAR *name,
+					  OBJECT_NAME_INFORMATION *ntfn)
+{
+  ULONG len;
+  BOOL res;
+  DWORD n_process = 256;
+  DWORD *proc_pids;
+  do
+    { /* Enumerate processes */
+      DWORD nbytes = n_process * sizeof (DWORD);
+      proc_pids = (DWORD *) HeapAlloc (GetProcessHeap (), 0, nbytes);
+      if (!proc_pids)
+	return NULL;
+      res = EnumProcesses (proc_pids, nbytes, &len);
+      if (res)
+	break;
+      HeapFree (GetProcessHeap (), 0, proc_pids);
+      n_process *= 2;
+    }
+  while (n_process < (1L<<20));
+  if (!res)
+    return NULL;
+  n_process = len / sizeof (DWORD);
+
+  for (LONG i = (LONG) n_process - 1; i >= 0; i--)
+    {
+      HANDLE proc = OpenProcess (PROCESS_DUP_HANDLE
+				 | PROCESS_QUERY_INFORMATION,
+				 0, proc_pids[i]);
+      if (!proc)
+	continue;
+
+      /* Retrieve process handles */
+      NTSTATUS status;
+      DWORD n_handle = 256;
+      PPROCESS_HANDLE_SNAPSHOT_INFORMATION phi;
+      do
+	{
+	  DWORD nbytes = 2 * sizeof (ULONG_PTR) +
+	    n_handle * sizeof (PROCESS_HANDLE_TABLE_ENTRY_INFO);
+	  phi = (PPROCESS_HANDLE_SNAPSHOT_INFORMATION)
+	    HeapAlloc (GetProcessHeap (), 0, nbytes);
+	  if (!phi)
+	    goto close_proc;
+	  status = NtQueryInformationProcess (proc, ProcessHandleInformation,
+					      phi, nbytes, &len);
+	  if (NT_SUCCESS (status))
+	    break;
+	  HeapFree (GetProcessHeap (), 0, phi);
+	  n_handle *= 2;
+	}
+      while (n_handle < (1L<<20) && status == STATUS_INFO_LENGTH_MISMATCH);
+      if (!NT_SUCCESS (status))
+	goto close_proc;
+
+      for (ULONG j = 0; j < phi->NumberOfHandles; j++)
+	{
+	  /* Check for the peculiarity of cygwin read pipe */
+	  const ULONG access = FILE_READ_DATA | FILE_READ_EA
+	    | FILE_WRITE_EA /* marker */
+	    | FILE_READ_ATTRIBUTES | FILE_WRITE_ATTRIBUTES
+	    | READ_CONTROL | SYNCHRONIZE;
+	  if (phi->Handles[j].GrantedAccess != access)
+	    continue;
+
+	  /* Retrieve handle */
+	  HANDLE h = (HANDLE)(intptr_t) phi->Handles[j].HandleValue;
+	  res = DuplicateHandle (proc, h, GetCurrentProcess (), &h,
+				 FILE_READ_DATA, 0, 0);
+	  if (!res)
+	    continue;
+
+	  /* Check object name */
+	  status = NtQueryObject (h, ObjectNameInformation,
+				  ntfn, 65536, &len);
+	  if (!NT_SUCCESS (status) || !ntfn->Name.Buffer)
+	    goto close_handle;
+	  ntfn->Name.Buffer[ntfn->Name.Length / sizeof (WCHAR)] = L'\0';
+	  if (wcscmp (name, ntfn->Name.Buffer) == 0)
+	    {
+	      query_hdl_proc = proc;
+	      query_hdl_value = (HANDLE)(intptr_t) phi->Handles[j].HandleValue;
+	      HeapFree (GetProcessHeap (), 0, phi);
+	      HeapFree (GetProcessHeap (), 0, proc_pids);
+	      return h;
+	    }
+close_handle:
+	  CloseHandle (h);
+	}
+      HeapFree (GetProcessHeap (), 0, phi);
+close_proc:
+      CloseHandle (proc);
+    }
+  HeapFree (GetProcessHeap (), 0, proc_pids);
+  return NULL;
+}
+
+/* This function is slower than get_query_hdl_per_process(), however,
+   works even before Windows 8. */
+HANDLE
+fhandler_pipe::get_query_hdl_per_system (WCHAR *name,
+					 OBJECT_NAME_INFORMATION *ntfn)
+{
+  NTSTATUS status;
   SIZE_T n_handle = 65536;
   PSYSTEM_HANDLE_INFORMATION shi;
   do
-    {
+    { /* Enumerate handles */
       SIZE_T nbytes =
 	sizeof (ULONG) + n_handle * sizeof (SYSTEM_HANDLE_TABLE_ENTRY_INFO);
       shi = (PSYSTEM_HANDLE_INFORMATION) HeapAlloc (GetProcessHeap (),
-						     0, nbytes);
+						    0, nbytes);
       if (!shi)
 	return NULL;
       status = NtQuerySystemInformation (SystemHandleInformation,
@@ -1193,15 +1308,15 @@ cache_err:
       HeapFree (GetProcessHeap (), 0, shi);
       n_handle *= 2;
     }
-  while (n_handle < (1L<<20));
+  while (n_handle < (1L<<23) && status == STATUS_INFO_LENGTH_MISMATCH);
   if (!NT_SUCCESS (status))
     return NULL;
 
-  HANDLE qh = NULL;
   for (LONG i = (LONG) shi->NumberOfHandles - 1; i >= 0; i--)
     {
       /* Check for the peculiarity of cygwin read pipe */
-      DWORD access = FILE_READ_DATA | FILE_READ_EA | FILE_WRITE_EA /* marker */
+      const ULONG access = FILE_READ_DATA | FILE_READ_EA
+	| FILE_WRITE_EA /* marker */
 	| FILE_READ_ATTRIBUTES | FILE_WRITE_ATTRIBUTES
 	| READ_CONTROL | SYNCHRONIZE;
       if (shi->Handles[i].GrantedAccess != access)
@@ -1219,6 +1334,7 @@ cache_err:
 	goto close_proc;
 
       /* Check object name */
+      ULONG len;
       status = NtQueryObject (h, ObjectNameInformation, ntfn, 65536, &len);
       if (!NT_SUCCESS (status) || !ntfn->Name.Buffer)
 	goto close_handle;
@@ -1227,8 +1343,8 @@ cache_err:
 	{
 	  query_hdl_proc = proc;
 	  query_hdl_value = (HANDLE)(intptr_t) shi->Handles[i].HandleValue;
-	  qh = h;
-	  break;
+	  HeapFree (GetProcessHeap (), 0, shi);
+	  return h;
 	}
 close_handle:
       CloseHandle (h);
@@ -1236,5 +1352,5 @@ close_proc:
       CloseHandle (proc);
     }
   HeapFree (GetProcessHeap (), 0, shi);
-  return qh;
+  return NULL;
 }
diff --git a/winsup/cygwin/ntdll.h b/winsup/cygwin/ntdll.h
index e8c3c45c5..0510d833b 100644
--- a/winsup/cygwin/ntdll.h
+++ b/winsup/cygwin/ntdll.h
@@ -842,9 +842,28 @@ typedef enum _PROCESSINFOCLASS
   ProcessSessionInformation = 24,
   ProcessWow64Information = 26,
   ProcessImageFileName = 27,
-  ProcessDebugFlags = 31
+  ProcessDebugFlags = 31,
+  ProcessHandleInformation = 51 /* Since Win8 */
 } PROCESSINFOCLASS;
 
+typedef struct _PROCESS_HANDLE_TABLE_ENTRY_INFO
+{
+  HANDLE HandleValue;
+  ULONG_PTR HandleCount;
+  ULONG_PTR PointerCount;
+  ULONG GrantedAccess;
+  ULONG ObjectTypeIndex;
+  ULONG HandleAttributes;
+  ULONG Reserved;
+} PROCESS_HANDLE_TABLE_ENTRY_INFO, *PPROCESS_HANDLE_TABLE_ENTRY_INFO;
+
+typedef struct _PROCESS_HANDLE_SNAPSHOT_INFORMATION
+{
+  ULONG_PTR NumberOfHandles;
+  ULONG_PTR Reserved;
+  PROCESS_HANDLE_TABLE_ENTRY_INFO Handles[1];
+} PROCESS_HANDLE_SNAPSHOT_INFORMATION, *PPROCESS_HANDLE_SNAPSHOT_INFORMATION;
+
 typedef struct _DEBUG_BUFFER
 {
   HANDLE SectionHandle;
diff --git a/winsup/cygwin/wincap.cc b/winsup/cygwin/wincap.cc
index 635e0892b..6c79d8710 100644
--- a/winsup/cygwin/wincap.cc
+++ b/winsup/cygwin/wincap.cc
@@ -50,6 +50,7 @@ wincaps wincap_vista __attribute__((section (".cygwin_dll_common"), shared)) = {
     has_tcp_fastopen:false,
     has_linux_tcp_keepalive_sockopts:false,
     has_tcp_maxrtms:false,
+    has_query_process_handle_info:false,
   },
 };
 
@@ -85,6 +86,7 @@ wincaps wincap_7 __attribute__((section (".cygwin_dll_common"), shared)) = {
     has_tcp_fastopen:false,
     has_linux_tcp_keepalive_sockopts:false,
     has_tcp_maxrtms:false,
+    has_query_process_handle_info:false,
   },
 };
 
@@ -120,6 +122,7 @@ wincaps wincap_8 __attribute__((section (".cygwin_dll_common"), shared)) = {
     has_tcp_fastopen:false,
     has_linux_tcp_keepalive_sockopts:false,
     has_tcp_maxrtms:false,
+    has_query_process_handle_info:true,
   },
 };
 
@@ -155,6 +158,7 @@ wincaps wincap_8_1 __attribute__((section (".cygwin_dll_common"), shared)) = {
     has_tcp_fastopen:false,
     has_linux_tcp_keepalive_sockopts:false,
     has_tcp_maxrtms:false,
+    has_query_process_handle_info:true,
   },
 };
 
@@ -190,6 +194,7 @@ wincaps  wincap_10_1507 __attribute__((section (".cygwin_dll_common"), shared))
     has_tcp_fastopen:false,
     has_linux_tcp_keepalive_sockopts:false,
     has_tcp_maxrtms:false,
+    has_query_process_handle_info:true,
   },
 };
 
@@ -225,6 +230,7 @@ wincaps  wincap_10_1607 __attribute__((section (".cygwin_dll_common"), shared))
     has_tcp_fastopen:true,
     has_linux_tcp_keepalive_sockopts:false,
     has_tcp_maxrtms:true,
+    has_query_process_handle_info:true,
   },
 };
 
@@ -260,6 +266,7 @@ wincaps wincap_10_1703 __attribute__((section (".cygwin_dll_common"), shared)) =
     has_tcp_fastopen:true,
     has_linux_tcp_keepalive_sockopts:false,
     has_tcp_maxrtms:true,
+    has_query_process_handle_info:true,
   },
 };
 
@@ -295,6 +302,7 @@ wincaps wincap_10_1709 __attribute__((section (".cygwin_dll_common"), shared)) =
     has_tcp_fastopen:true,
     has_linux_tcp_keepalive_sockopts:true,
     has_tcp_maxrtms:true,
+    has_query_process_handle_info:true,
   },
 };
 
@@ -330,6 +338,7 @@ wincaps wincap_10_1803 __attribute__((section (".cygwin_dll_common"), shared)) =
     has_tcp_fastopen:true,
     has_linux_tcp_keepalive_sockopts:true,
     has_tcp_maxrtms:true,
+    has_query_process_handle_info:true,
   },
 };
 
@@ -365,6 +374,7 @@ wincaps wincap_10_1809 __attribute__((section (".cygwin_dll_common"), shared)) =
     has_tcp_fastopen:true,
     has_linux_tcp_keepalive_sockopts:true,
     has_tcp_maxrtms:true,
+    has_query_process_handle_info:true,
   },
 };
 
@@ -400,6 +410,7 @@ wincaps wincap_10_1903 __attribute__((section (".cygwin_dll_common"), shared)) =
     has_tcp_fastopen:true,
     has_linux_tcp_keepalive_sockopts:true,
     has_tcp_maxrtms:true,
+    has_query_process_handle_info:true,
   },
 };
 
diff --git a/winsup/cygwin/wincap.h b/winsup/cygwin/wincap.h
index 6be2ca2a1..7249b9518 100644
--- a/winsup/cygwin/wincap.h
+++ b/winsup/cygwin/wincap.h
@@ -44,6 +44,7 @@ struct wincaps
     unsigned has_tcp_fastopen					: 1;
     unsigned has_linux_tcp_keepalive_sockopts			: 1;
     unsigned has_tcp_maxrtms					: 1;
+    unsigned has_query_process_handle_info			: 1;
   };
 };
 
@@ -111,6 +112,7 @@ public:
   bool	IMPLEMENT (has_tcp_fastopen)
   bool	IMPLEMENT (has_linux_tcp_keepalive_sockopts)
   bool	IMPLEMENT (has_tcp_maxrtms)
+  bool	IMPLEMENT (has_query_process_handle_info)
 
   void disable_case_sensitive_dirs ()
   {
-- 
2.33.0


^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-23  8:26                                                             ` Takashi Yano
@ 2021-09-23 13:03                                                               ` Ken Brown
  2021-09-23 15:03                                                                 ` Takashi Yano
  0 siblings, 1 reply; 250+ messages in thread
From: Ken Brown @ 2021-09-23 13:03 UTC (permalink / raw)
  To: cygwin-developers

On 9/23/2021 4:26 AM, Takashi Yano wrote:
> * Make temporary_query_hdl() faster.

Thanks!  Again, I'm not familiar with the Windows functions that you used, so 
Corinna should review this too when she returns.

I have one question:

The documentation at 
https://docs.microsoft.com/en-us/windows/win32/api/psapi/nf-psapi-enumprocesses 
says, "There is no indication given when the buffer is too small to store all 
process identifiers. Therefore, if lpcbNeeded equals cb, consider retrying the 
call with a larger array."

Does this mean that EnumProcesses could return TRUE even if the buffer is too small?

Other than that, LGTM.

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-23 13:03                                                               ` Ken Brown
@ 2021-09-23 15:03                                                                 ` Takashi Yano
  2021-09-23 16:29                                                                   ` Ken Brown
  2021-10-18 10:51                                                                   ` Corinna Vinschen
  0 siblings, 2 replies; 250+ messages in thread
From: Takashi Yano @ 2021-09-23 15:03 UTC (permalink / raw)
  To: cygwin-developers

[-- Attachment #1: Type: text/plain, Size: 1205 bytes --]

On Thu, 23 Sep 2021 09:03:13 -0400
Ken Brown wrote:
> On 9/23/2021 4:26 AM, Takashi Yano wrote:
> > * Make temporary_query_hdl() faster.
> 
> Thanks!  Again, I'm not familiar with the Windows functions that you used, so 
> Corinna should review this too when she returns.
> 
> I have one question:
> 
> The documentation at 
> https://docs.microsoft.com/en-us/windows/win32/api/psapi/nf-psapi-enumprocesses 
> says, "There is no indication given when the buffer is too small to store all 
> process identifiers. Therefore, if lpcbNeeded equals cb, consider retrying the 
> call with a larger array."
> 
> Does this mean that EnumProcesses could return TRUE even if the buffer is too small?

Ah, right. Thanks for pointing out that mistake.
I have fixed that.

P.S.
I measured the response of select().
The first time and second time response is measured.

Only the first time takes long response time because
the cached info is used in the second time. This patch
improves the response much. Also, caching info is very
effective.

First time, Second time
Without this patch:
19.427600 [msec], 0.113900 [msec]

With this patch:
3.344600 [msec], 0.115000 [msec]

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

[-- Attachment #2: v2-0001-Cygwin-pipe-Use-ProcessHandleInformation-if-avail.patch --]
[-- Type: application/octet-stream, Size: 11858 bytes --]

From 11741ec6de92022ec30bd57a45b6d66f7cb8ebe0 Mon Sep 17 00:00:00 2001
From: Takashi Yano <takashi.yano@nifty.ne.jp>
Date: Thu, 23 Sep 2021 23:33:55 +0900
Subject: [PATCH v2] Cygwin: pipe: Use ProcessHandleInformation if available.

- The commit b531d6b0 introduced temporary_query_hdl() which uses
  SystemHandleInformation. With this patch, ProcessHandleInformation
  rather than SystemHandleInformation is used if it is available.
  This request is faster, however, is only available since Windows 8,
  therefore, SystemHandleInformation is used for Windows Vista and 7
  as before.
---
 winsup/cygwin/fhandler.h       |   2 +
 winsup/cygwin/fhandler_pipe.cc | 133 +++++++++++++++++++++++++++++++--
 winsup/cygwin/ntdll.h          |  21 +++++-
 winsup/cygwin/wincap.cc        |  11 +++
 winsup/cygwin/wincap.h         |   2 +
 5 files changed, 160 insertions(+), 9 deletions(-)

diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h
index 0061d4830..c033f7816 100644
--- a/winsup/cygwin/fhandler.h
+++ b/winsup/cygwin/fhandler.h
@@ -1197,6 +1197,8 @@ private:
   DWORD pipename_pid;
   LONG pipename_id;
   void release_select_sem (const char *);
+  HANDLE get_query_hdl_per_process (WCHAR *, OBJECT_NAME_INFORMATION *);
+  HANDLE get_query_hdl_per_system (WCHAR *, OBJECT_NAME_INFORMATION *);
 public:
   fhandler_pipe ();
 
diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index 78e2f90d8..dd8b4f317 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -20,6 +20,7 @@ details. */
 #include "pinfo.h"
 #include "shared_info.h"
 #include "tls_pbuf.h"
+#include <psapi.h>
 
 /* This is only to be used for writing.  When reading,
 STATUS_PIPE_EMPTY simply means there's no data to be read. */
@@ -1176,14 +1177,129 @@ cache_err:
 	       &pipename_key, &pipename_pid, &pipename_id) != 3)
     return NULL; /* Non cygwin pipe? */
 
+  if (wincap.has_query_process_handle_info ())
+    return get_query_hdl_per_process (name, ntfn); /* Since Win8 */
+  else
+    return get_query_hdl_per_system (name, ntfn); /* Vista or Win7 */
+}
+
+/* This function is faster than get_query_hdl_per_system(), however,
+   only works since Windows 8 because ProcessHandleInformation is not
+   suppoted by NtQueryInformationProcess() before Windows 8. */
+HANDLE
+fhandler_pipe::get_query_hdl_per_process (WCHAR *name,
+					  OBJECT_NAME_INFORMATION *ntfn)
+{
+  ULONG len;
+  BOOL res;
+  DWORD n_process = 256;
+  DWORD *proc_pids;
+  do
+    { /* Enumerate processes */
+      DWORD nbytes = n_process * sizeof (DWORD);
+      proc_pids = (DWORD *) HeapAlloc (GetProcessHeap (), 0, nbytes);
+      if (!proc_pids)
+	return NULL;
+      res = EnumProcesses (proc_pids, nbytes, &len);
+      if (res && len < nbytes)
+	break;
+      res = FALSE;
+      HeapFree (GetProcessHeap (), 0, proc_pids);
+      n_process *= 2;
+    }
+  while (n_process < (1L<<20));
+  if (!res)
+    return NULL;
+  n_process = len / sizeof (DWORD);
+
+  for (LONG i = (LONG) n_process - 1; i >= 0; i--)
+    {
+      HANDLE proc = OpenProcess (PROCESS_DUP_HANDLE
+				 | PROCESS_QUERY_INFORMATION,
+				 0, proc_pids[i]);
+      if (!proc)
+	continue;
+
+      /* Retrieve process handles */
+      NTSTATUS status;
+      DWORD n_handle = 256;
+      PPROCESS_HANDLE_SNAPSHOT_INFORMATION phi;
+      do
+	{
+	  DWORD nbytes = 2 * sizeof (ULONG_PTR) +
+	    n_handle * sizeof (PROCESS_HANDLE_TABLE_ENTRY_INFO);
+	  phi = (PPROCESS_HANDLE_SNAPSHOT_INFORMATION)
+	    HeapAlloc (GetProcessHeap (), 0, nbytes);
+	  if (!phi)
+	    goto close_proc;
+	  status = NtQueryInformationProcess (proc, ProcessHandleInformation,
+					      phi, nbytes, &len);
+	  if (NT_SUCCESS (status))
+	    break;
+	  HeapFree (GetProcessHeap (), 0, phi);
+	  n_handle *= 2;
+	}
+      while (n_handle < (1L<<20) && status == STATUS_INFO_LENGTH_MISMATCH);
+      if (!NT_SUCCESS (status))
+	goto close_proc;
+
+      for (ULONG j = 0; j < phi->NumberOfHandles; j++)
+	{
+	  /* Check for the peculiarity of cygwin read pipe */
+	  const ULONG access = FILE_READ_DATA | FILE_READ_EA
+	    | FILE_WRITE_EA /* marker */
+	    | FILE_READ_ATTRIBUTES | FILE_WRITE_ATTRIBUTES
+	    | READ_CONTROL | SYNCHRONIZE;
+	  if (phi->Handles[j].GrantedAccess != access)
+	    continue;
+
+	  /* Retrieve handle */
+	  HANDLE h = (HANDLE)(intptr_t) phi->Handles[j].HandleValue;
+	  res = DuplicateHandle (proc, h, GetCurrentProcess (), &h,
+				 FILE_READ_DATA, 0, 0);
+	  if (!res)
+	    continue;
+
+	  /* Check object name */
+	  status = NtQueryObject (h, ObjectNameInformation,
+				  ntfn, 65536, &len);
+	  if (!NT_SUCCESS (status) || !ntfn->Name.Buffer)
+	    goto close_handle;
+	  ntfn->Name.Buffer[ntfn->Name.Length / sizeof (WCHAR)] = L'\0';
+	  if (wcscmp (name, ntfn->Name.Buffer) == 0)
+	    {
+	      query_hdl_proc = proc;
+	      query_hdl_value = (HANDLE)(intptr_t) phi->Handles[j].HandleValue;
+	      HeapFree (GetProcessHeap (), 0, phi);
+	      HeapFree (GetProcessHeap (), 0, proc_pids);
+	      return h;
+	    }
+close_handle:
+	  CloseHandle (h);
+	}
+      HeapFree (GetProcessHeap (), 0, phi);
+close_proc:
+      CloseHandle (proc);
+    }
+  HeapFree (GetProcessHeap (), 0, proc_pids);
+  return NULL;
+}
+
+/* This function is slower than get_query_hdl_per_process(), however,
+   works even before Windows 8. */
+HANDLE
+fhandler_pipe::get_query_hdl_per_system (WCHAR *name,
+					 OBJECT_NAME_INFORMATION *ntfn)
+{
+  NTSTATUS status;
   SIZE_T n_handle = 65536;
   PSYSTEM_HANDLE_INFORMATION shi;
   do
-    {
+    { /* Enumerate handles */
       SIZE_T nbytes =
 	sizeof (ULONG) + n_handle * sizeof (SYSTEM_HANDLE_TABLE_ENTRY_INFO);
       shi = (PSYSTEM_HANDLE_INFORMATION) HeapAlloc (GetProcessHeap (),
-						     0, nbytes);
+						    0, nbytes);
       if (!shi)
 	return NULL;
       status = NtQuerySystemInformation (SystemHandleInformation,
@@ -1193,15 +1309,15 @@ cache_err:
       HeapFree (GetProcessHeap (), 0, shi);
       n_handle *= 2;
     }
-  while (n_handle < (1L<<20));
+  while (n_handle < (1L<<23) && status == STATUS_INFO_LENGTH_MISMATCH);
   if (!NT_SUCCESS (status))
     return NULL;
 
-  HANDLE qh = NULL;
   for (LONG i = (LONG) shi->NumberOfHandles - 1; i >= 0; i--)
     {
       /* Check for the peculiarity of cygwin read pipe */
-      DWORD access = FILE_READ_DATA | FILE_READ_EA | FILE_WRITE_EA /* marker */
+      const ULONG access = FILE_READ_DATA | FILE_READ_EA
+	| FILE_WRITE_EA /* marker */
 	| FILE_READ_ATTRIBUTES | FILE_WRITE_ATTRIBUTES
 	| READ_CONTROL | SYNCHRONIZE;
       if (shi->Handles[i].GrantedAccess != access)
@@ -1219,6 +1335,7 @@ cache_err:
 	goto close_proc;
 
       /* Check object name */
+      ULONG len;
       status = NtQueryObject (h, ObjectNameInformation, ntfn, 65536, &len);
       if (!NT_SUCCESS (status) || !ntfn->Name.Buffer)
 	goto close_handle;
@@ -1227,8 +1344,8 @@ cache_err:
 	{
 	  query_hdl_proc = proc;
 	  query_hdl_value = (HANDLE)(intptr_t) shi->Handles[i].HandleValue;
-	  qh = h;
-	  break;
+	  HeapFree (GetProcessHeap (), 0, shi);
+	  return h;
 	}
 close_handle:
       CloseHandle (h);
@@ -1236,5 +1353,5 @@ close_proc:
       CloseHandle (proc);
     }
   HeapFree (GetProcessHeap (), 0, shi);
-  return qh;
+  return NULL;
 }
diff --git a/winsup/cygwin/ntdll.h b/winsup/cygwin/ntdll.h
index e8c3c45c5..0510d833b 100644
--- a/winsup/cygwin/ntdll.h
+++ b/winsup/cygwin/ntdll.h
@@ -842,9 +842,28 @@ typedef enum _PROCESSINFOCLASS
   ProcessSessionInformation = 24,
   ProcessWow64Information = 26,
   ProcessImageFileName = 27,
-  ProcessDebugFlags = 31
+  ProcessDebugFlags = 31,
+  ProcessHandleInformation = 51 /* Since Win8 */
 } PROCESSINFOCLASS;
 
+typedef struct _PROCESS_HANDLE_TABLE_ENTRY_INFO
+{
+  HANDLE HandleValue;
+  ULONG_PTR HandleCount;
+  ULONG_PTR PointerCount;
+  ULONG GrantedAccess;
+  ULONG ObjectTypeIndex;
+  ULONG HandleAttributes;
+  ULONG Reserved;
+} PROCESS_HANDLE_TABLE_ENTRY_INFO, *PPROCESS_HANDLE_TABLE_ENTRY_INFO;
+
+typedef struct _PROCESS_HANDLE_SNAPSHOT_INFORMATION
+{
+  ULONG_PTR NumberOfHandles;
+  ULONG_PTR Reserved;
+  PROCESS_HANDLE_TABLE_ENTRY_INFO Handles[1];
+} PROCESS_HANDLE_SNAPSHOT_INFORMATION, *PPROCESS_HANDLE_SNAPSHOT_INFORMATION;
+
 typedef struct _DEBUG_BUFFER
 {
   HANDLE SectionHandle;
diff --git a/winsup/cygwin/wincap.cc b/winsup/cygwin/wincap.cc
index 635e0892b..6c79d8710 100644
--- a/winsup/cygwin/wincap.cc
+++ b/winsup/cygwin/wincap.cc
@@ -50,6 +50,7 @@ wincaps wincap_vista __attribute__((section (".cygwin_dll_common"), shared)) = {
     has_tcp_fastopen:false,
     has_linux_tcp_keepalive_sockopts:false,
     has_tcp_maxrtms:false,
+    has_query_process_handle_info:false,
   },
 };
 
@@ -85,6 +86,7 @@ wincaps wincap_7 __attribute__((section (".cygwin_dll_common"), shared)) = {
     has_tcp_fastopen:false,
     has_linux_tcp_keepalive_sockopts:false,
     has_tcp_maxrtms:false,
+    has_query_process_handle_info:false,
   },
 };
 
@@ -120,6 +122,7 @@ wincaps wincap_8 __attribute__((section (".cygwin_dll_common"), shared)) = {
     has_tcp_fastopen:false,
     has_linux_tcp_keepalive_sockopts:false,
     has_tcp_maxrtms:false,
+    has_query_process_handle_info:true,
   },
 };
 
@@ -155,6 +158,7 @@ wincaps wincap_8_1 __attribute__((section (".cygwin_dll_common"), shared)) = {
     has_tcp_fastopen:false,
     has_linux_tcp_keepalive_sockopts:false,
     has_tcp_maxrtms:false,
+    has_query_process_handle_info:true,
   },
 };
 
@@ -190,6 +194,7 @@ wincaps  wincap_10_1507 __attribute__((section (".cygwin_dll_common"), shared))
     has_tcp_fastopen:false,
     has_linux_tcp_keepalive_sockopts:false,
     has_tcp_maxrtms:false,
+    has_query_process_handle_info:true,
   },
 };
 
@@ -225,6 +230,7 @@ wincaps  wincap_10_1607 __attribute__((section (".cygwin_dll_common"), shared))
     has_tcp_fastopen:true,
     has_linux_tcp_keepalive_sockopts:false,
     has_tcp_maxrtms:true,
+    has_query_process_handle_info:true,
   },
 };
 
@@ -260,6 +266,7 @@ wincaps wincap_10_1703 __attribute__((section (".cygwin_dll_common"), shared)) =
     has_tcp_fastopen:true,
     has_linux_tcp_keepalive_sockopts:false,
     has_tcp_maxrtms:true,
+    has_query_process_handle_info:true,
   },
 };
 
@@ -295,6 +302,7 @@ wincaps wincap_10_1709 __attribute__((section (".cygwin_dll_common"), shared)) =
     has_tcp_fastopen:true,
     has_linux_tcp_keepalive_sockopts:true,
     has_tcp_maxrtms:true,
+    has_query_process_handle_info:true,
   },
 };
 
@@ -330,6 +338,7 @@ wincaps wincap_10_1803 __attribute__((section (".cygwin_dll_common"), shared)) =
     has_tcp_fastopen:true,
     has_linux_tcp_keepalive_sockopts:true,
     has_tcp_maxrtms:true,
+    has_query_process_handle_info:true,
   },
 };
 
@@ -365,6 +374,7 @@ wincaps wincap_10_1809 __attribute__((section (".cygwin_dll_common"), shared)) =
     has_tcp_fastopen:true,
     has_linux_tcp_keepalive_sockopts:true,
     has_tcp_maxrtms:true,
+    has_query_process_handle_info:true,
   },
 };
 
@@ -400,6 +410,7 @@ wincaps wincap_10_1903 __attribute__((section (".cygwin_dll_common"), shared)) =
     has_tcp_fastopen:true,
     has_linux_tcp_keepalive_sockopts:true,
     has_tcp_maxrtms:true,
+    has_query_process_handle_info:true,
   },
 };
 
diff --git a/winsup/cygwin/wincap.h b/winsup/cygwin/wincap.h
index 6be2ca2a1..7249b9518 100644
--- a/winsup/cygwin/wincap.h
+++ b/winsup/cygwin/wincap.h
@@ -44,6 +44,7 @@ struct wincaps
     unsigned has_tcp_fastopen					: 1;
     unsigned has_linux_tcp_keepalive_sockopts			: 1;
     unsigned has_tcp_maxrtms					: 1;
+    unsigned has_query_process_handle_info			: 1;
   };
 };
 
@@ -111,6 +112,7 @@ public:
   bool	IMPLEMENT (has_tcp_fastopen)
   bool	IMPLEMENT (has_linux_tcp_keepalive_sockopts)
   bool	IMPLEMENT (has_tcp_maxrtms)
+  bool	IMPLEMENT (has_query_process_handle_info)
 
   void disable_case_sensitive_dirs ()
   {
-- 
2.33.0


^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-23 15:03                                                                 ` Takashi Yano
@ 2021-09-23 16:29                                                                   ` Ken Brown
  2021-10-18 10:51                                                                   ` Corinna Vinschen
  1 sibling, 0 replies; 250+ messages in thread
From: Ken Brown @ 2021-09-23 16:29 UTC (permalink / raw)
  To: cygwin-developers

On 9/23/2021 11:03 AM, Takashi Yano wrote:
> On Thu, 23 Sep 2021 09:03:13 -0400
> Ken Brown wrote:
>> On 9/23/2021 4:26 AM, Takashi Yano wrote:
>>> * Make temporary_query_hdl() faster.
>>
>> Thanks!  Again, I'm not familiar with the Windows functions that you used, so
>> Corinna should review this too when she returns.
>>
>> I have one question:
>>
>> The documentation at
>> https://docs.microsoft.com/en-us/windows/win32/api/psapi/nf-psapi-enumprocesses
>> says, "There is no indication given when the buffer is too small to store all
>> process identifiers. Therefore, if lpcbNeeded equals cb, consider retrying the
>> call with a larger array."
>>
>> Does this mean that EnumProcesses could return TRUE even if the buffer is too small?
> 
> Ah, right. Thanks for pointing out that mistake.
> I have fixed that.
> 
> P.S.
> I measured the response of select().
> The first time and second time response is measured.
> 
> Only the first time takes long response time because
> the cached info is used in the second time. This patch
> improves the response much. Also, caching info is very
> effective.
> 
> First time, Second time
> Without this patch:
> 19.427600 [msec], 0.113900 [msec]
> 
> With this patch:
> 3.344600 [msec], 0.115000 [msec]

That's great!  I'll push the patch shortly, and I'll make a new test release in 
about 24 hours if nothing else comes up.

Thanks.

Ken

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-09-23 15:03                                                                 ` Takashi Yano
  2021-09-23 16:29                                                                   ` Ken Brown
@ 2021-10-18 10:51                                                                   ` Corinna Vinschen
  2021-10-18 12:02                                                                     ` Takashi Yano
  1 sibling, 1 reply; 250+ messages in thread
From: Corinna Vinschen @ 2021-10-18 10:51 UTC (permalink / raw)
  To: cygwin-developers

Hi Takashi, Hi Ken,

first of all, thanks for keeping at it!

On Sep 24 00:03, Takashi Yano wrote:
> On Thu, 23 Sep 2021 09:03:13 -0400
> Ken Brown wrote:
> > On 9/23/2021 4:26 AM, Takashi Yano wrote:
> > > * Make temporary_query_hdl() faster.
> > 
> > Thanks!  Again, I'm not familiar with the Windows functions that you used, so 
> > Corinna should review this too when she returns.
> > 
> > I have one question:
> > 
> > The documentation at 
> > https://docs.microsoft.com/en-us/windows/win32/api/psapi/nf-psapi-enumprocesses 
> > says, "There is no indication given when the buffer is too small to store all 
> > process identifiers. Therefore, if lpcbNeeded equals cb, consider retrying the 
> > call with a larger array."
> > 
> > Does this mean that EnumProcesses could return TRUE even if the buffer is too small?
> 
> Ah, right. Thanks for pointing out that mistake.
> I have fixed that.

The patch looks basically ok, but we shouldn't use PSAPI calls inside
Cygwin.  I'm a bit fuzzy on the details, but we had some problems with
PSAPI way back when.

One of them, prominently, is the fact that the entry points have changed
in the past.  EnumProcess from psapi.dll is now k32_EnumProcess from
kernel32.dll or whatever, since the headers are using PSAPI_VERSION 2 as
default.  Linking against that will break Vista.  If you change this to
PSAPI_VERSION 1, OTOH, you will have to add the psapi symbols to
autoload.cc.

Why don't you just use winpids, as in:

  winpids pids;
  pids.set (true);
  for (int i = 0; i < pids.npids; ++i)
    ...
  pids.reset ();

This will use NtQuerySystemInformation (SystemProcessInformation) under
the hood.  Of course, winpids adds a bit of unnecessary overhead by
checking for procinfo stuff, so maybe you may want to use
NtQuerySystemInformation (SystemProcessInformation) directly and
just copy/paste the minimally required code from the `else' branch in
winpids::enum_processes.


Thanks,
Corinna

^ permalink raw reply	[flat|nested] 250+ messages in thread

* Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
  2021-10-18 10:51                                                                   ` Corinna Vinschen
@ 2021-10-18 12:02                                                                     ` Takashi Yano
  0 siblings, 0 replies; 250+ messages in thread
From: Takashi Yano @ 2021-10-18 12:02 UTC (permalink / raw)
  To: cygwin-developers

Hi Corinna,

On Mon, 18 Oct 2021 12:51:37 +0200
Corinna Vinschen wrote:
> The patch looks basically ok, but we shouldn't use PSAPI calls inside
> Cygwin.  I'm a bit fuzzy on the details, but we had some problems with
> PSAPI way back when.
> 
> One of them, prominently, is the fact that the entry points have changed
> in the past.  EnumProcess from psapi.dll is now k32_EnumProcess from
> kernel32.dll or whatever, since the headers are using PSAPI_VERSION 2 as
> default.  Linking against that will break Vista.  If you change this to
> PSAPI_VERSION 1, OTOH, you will have to add the psapi symbols to
> autoload.cc.
> 
> Why don't you just use winpids, as in:
> 
>   winpids pids;
>   pids.set (true);
>   for (int i = 0; i < pids.npids; ++i)
>     ...
>   pids.reset ();
> 
> This will use NtQuerySystemInformation (SystemProcessInformation) under
> the hood.  Of course, winpids adds a bit of unnecessary overhead by
> checking for procinfo stuff, so maybe you may want to use
> NtQuerySystemInformation (SystemProcessInformation) directly and
> just copy/paste the minimally required code from the `else' branch in
> winpids::enum_processes.

Thanks for reviewing the code.

I will make a patch which replaces EnumProcesses() with
NtQuerySystemInformation (SystemProcessInformation).

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

^ permalink raw reply	[flat|nested] 250+ messages in thread

end of thread, other threads:[~2021-10-18 12:02 UTC | newest]

Thread overview: 250+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
     [not found] <41A583E1-C8E7-42AB-9F24-EEC33A41EC60@house.org>
     [not found] ` <20210825201845.07b6400b79dc5558a7761efe@nifty.ne.jp>
     [not found]   ` <f8106fe7-a2b8-d106-3061-4d888124f4b0@cornell.edu>
     [not found]     ` <20210826062934.54f2f2216021c095bb7ba13b@nifty.ne.jp>
     [not found]       ` <d0a8c57d-1ed1-6b4f-c6e7-cbe0e2ec8a1c@cornell.edu>
     [not found]         ` <3b560051-ab27-f392-ca4b-d1fd9b5733b0@cornell.edu>
     [not found]           ` <20210827202440.47706fc2fc07c5e9a1bc0047@nifty.ne.jp>
     [not found]             ` <4f2cb5f3-ce9c-c617-f65f-841a5eca096e@cornell.edu>
     [not found]               ` <20210828022111.91ef5b4ff24f6da9fadb489e@nifty.ne.jp>
     [not found]                 ` <YSn3L0W1M527utK0@calimero.vinschen.de>
     [not found]                   ` <20210828184102.f2206a8a9e5fe5cf24bf5e45@nifty.ne.jp>
     [not found]                     ` <YSok0PoCQn2TPPrn@calimero.vinschen.de>
     [not found]                       ` <20210829004346.c2f80469abc3a07fd4b2918d@nifty.ne.jp>
     [not found]                         ` <e8caa02f-be85-33bc-3f09-347c1cdb0923@cornell.edu>
     [not found]                           ` <20210829174124.0c1ae6c16a3e8da1f490abc7@nifty.ne.jp>
     [not found]                             ` <6e9bb35e-6f4f-cf78-e515-549da487b5ef@cornell.edu>
2021-08-30  7:57                               ` cygrunsrv + sshd + rsync = 20 times too slow -- throttled? Corinna Vinschen
     [not found]                     ` <20210829180729.48b4e877f773cb3980c5766d@nifty.ne.jp>
     [not found]                       ` <789f056a-f164-d71d-1dc9-230f5a41846d@cornell.edu>
2021-08-30  8:27                         ` Corinna Vinschen
2021-08-30 13:00                           ` Corinna Vinschen
2021-08-30 13:20                             ` Corinna Vinschen
2021-08-30 13:41                               ` Ken Brown
2021-08-30 14:12                                 ` Corinna Vinschen
2021-08-30 14:52                                   ` Ken Brown
2021-08-30 15:15                                     ` Corinna Vinschen
     [not found]                         ` <20210830043756.8aa0ada77db0bfbbe3889f62@nifty.ne.jp>
     [not found]                           ` <47e5dd74-b940-f305-fd5a-c6c9d8f41305@cornell.edu>
2021-08-30  8:48                             ` Corinna Vinschen
     [not found]                       ` <c62d18df-ab7a-7233-62ee-29a8eced5353@cornell.edu>
     [not found]                         ` <20210830091314.f9a2cb71794d0f68cdb5eba7@nifty.ne.jp>
     [not found]                           ` <20210830092259.52f7d54fc3fa340738373af4@nifty.ne.jp>
     [not found]                             ` <d217ef03-7858-5e22-0aa6-f0507eedd9da@cornell.edu>
     [not found]                               ` <20210830170204.fa91eaf110f310f13b67abc3@nifty.ne.jp>
2021-08-30 10:20                                 ` Corinna Vinschen
2021-08-30 10:38                                   ` Corinna Vinschen
2021-08-30 12:04                                   ` Takashi Yano
2021-08-30 12:55                                     ` Corinna Vinschen
2021-08-30 13:31                                       ` Corinna Vinschen
2021-08-31  8:50                                         ` Takashi Yano
2021-08-30 13:51                                       ` Ken Brown
2021-08-30 15:00                                         ` Ken Brown
2021-08-30 15:19                                           ` Corinna Vinschen
2021-08-30 15:43                                             ` Ken Brown
2021-08-31  9:43                                               ` Corinna Vinschen
2021-08-31  8:52                                             ` Takashi Yano
2021-08-31  9:04                                               ` Corinna Vinschen
2021-08-31 11:05                                                 ` Takashi Yano
2021-08-31 15:20                                                   ` Corinna Vinschen
2021-09-01  2:39                                                     ` Takashi Yano
2021-09-01  8:03                                                       ` Corinna Vinschen
2021-09-01  8:13                                                         ` Corinna Vinschen
2021-08-30 13:36                               ` Ken Brown
2021-08-30 14:05                                 ` Corinna Vinschen
2021-08-30 15:53                                   ` Corinna Vinschen
2021-08-30 17:00                                     ` Corinna Vinschen
2021-08-30 17:11                                       ` Corinna Vinschen
2021-08-30 18:59                                       ` Ken Brown
2021-08-30 19:12                                         ` Ken Brown
2021-08-30 20:21                                         ` Corinna Vinschen
2021-08-30 20:14                                       ` Corinna Vinschen
2021-08-30 20:47                                         ` Ken Brown
2021-08-31  8:55                                         ` Takashi Yano
2021-08-31  9:08                                           ` Corinna Vinschen
2021-08-31  9:25                                             ` Takashi Yano
2021-08-31 10:05                                               ` Corinna Vinschen
2021-08-31 10:18                                                 ` Corinna Vinschen
2021-08-31 11:45                                                   ` Takashi Yano
2021-08-31 12:31                                                     ` Takashi Yano
2021-08-31 15:08                                                       ` Corinna Vinschen
2021-08-31 12:33                                                     ` Ken Brown
2021-08-31 15:18                                                       ` Corinna Vinschen
2021-08-31 15:27                                                         ` Corinna Vinschen
2021-08-31 15:50                                                           ` Corinna Vinschen
2021-08-31 16:19                                                             ` Ken Brown
2021-08-31 16:38                                                               ` Ken Brown
2021-08-31 17:30                                                                 ` Corinna Vinschen
2021-08-31 18:54                                                                   ` Ken Brown
2021-08-31 19:51                                                                     ` Corinna Vinschen
2021-08-31 23:02                                                             ` Takashi Yano
2021-09-01  0:16                                                               ` Takashi Yano
2021-09-01  8:07                                                                 ` Corinna Vinschen
2021-09-01  8:23                                                                   ` Takashi Yano
2021-09-01  8:46                                                                     ` Corinna Vinschen
2021-09-01 12:56                                                                       ` Ken Brown
2021-09-01 13:52                                                                         ` Corinna Vinschen
2021-09-01 23:02                                                                           ` Ken Brown
2021-09-02  8:17                                                                             ` Corinna Vinschen
2021-09-02 13:01                                                                               ` Ken Brown
2021-09-02 19:00                                                                                 ` Corinna Vinschen
2021-09-02 19:34                                                                                   ` Ken Brown
2021-09-02 19:35                                                                                   ` Corinna Vinschen
2021-09-02 20:19                                                                                     ` Ken Brown
2021-09-03  9:12                                                                                       ` Corinna Vinschen
2021-09-03 19:00                                                                                         ` Ken Brown
2021-09-03 19:53                                                                                           ` Ken Brown
2021-09-03 19:54                                                                                           ` Corinna Vinschen
2021-09-03 20:05                                                                                             ` Ken Brown
2021-09-03 10:00                                                                                     ` Takashi Yano
2021-09-03 10:13                                                                                       ` Takashi Yano
2021-09-03 11:31                                                                                         ` Corinna Vinschen
2021-09-03 11:41                                                                                           ` Corinna Vinschen
2021-09-03 12:13                                                                                             ` Ken Brown
2021-09-03 15:00                                                                                               ` Corinna Vinschen
2021-09-03 15:14                                                                                                 ` Ken Brown
2021-09-03 15:17                                                                                                   ` Corinna Vinschen
2021-09-03 12:22                                                                                             ` Takashi Yano
2021-09-03 13:27                                                                                               ` Ken Brown
2021-09-03 15:37                                                                                               ` Corinna Vinschen
2021-09-04 12:02                                                                                                 ` Takashi Yano
2021-09-04 12:37                                                                                                   ` Takashi Yano
2021-09-04 14:04                                                                                                     ` Ken Brown
2021-09-04 23:15                                                                                                       ` Takashi Yano
2021-09-05 13:40                                                                                                         ` Takashi Yano
2021-09-05 13:50                                                                                                           ` Takashi Yano
2021-09-05 18:47                                                                                                             ` Ken Brown
2021-09-05 19:42                                                                                                               ` Takashi Yano
2021-09-05 20:09                                                                                                               ` Takashi Yano
2021-09-05 20:27                                                                                                                 ` Ken Brown
2021-09-06  8:13                                                                                                                 ` Corinna Vinschen
2021-09-06 11:16                                                                                                                   ` Takashi Yano
2021-09-06 12:49                                                                                                                     ` Corinna Vinschen
2021-09-06 13:16                                                                                                                       ` Takashi Yano
2021-09-06 16:08                                                                                                                         ` Corinna Vinschen
2021-09-06 23:39                                                                                                                           ` Takashi Yano
2021-09-07  9:14                                                                                                                             ` Corinna Vinschen
2021-09-07 11:03                                                                                                                               ` Takashi Yano
2021-09-07 16:14                                                                                                                       ` Ken Brown
2021-09-07 18:26                                                                                                                         ` Corinna Vinschen
2021-09-03 10:38                                                                                       ` Takashi Yano
2021-09-08 11:32                                                                                     ` Takashi Yano
2021-09-08 11:55                                                                                       ` Corinna Vinschen
2021-09-08 12:33                                                                                         ` Takashi Yano
2021-09-08 17:43                                                                                         ` Ken Brown
2021-09-08 18:28                                                                                           ` Corinna Vinschen
2021-09-02  8:15                                                                       ` Takashi Yano
2021-09-02 18:54                                                                         ` Corinna Vinschen
2021-09-07  3:26             ` Takashi Yano
2021-09-07 10:50               ` Takashi Yano
2021-09-08  0:07                 ` Takashi Yano
2021-09-08  4:11                   ` Takashi Yano
2021-09-08  9:01                     ` Takashi Yano
2021-09-08  9:01                     ` Corinna Vinschen
2021-09-08  9:26                       ` Corinna Vinschen
2021-09-08  9:45                         ` Takashi Yano
2021-09-08 10:04                           ` Corinna Vinschen
2021-09-08 10:45                             ` Takashi Yano
2021-09-08 10:51                               ` Corinna Vinschen
2021-09-09  3:21                                 ` Takashi Yano
2021-09-09  9:37                                   ` Corinna Vinschen
2021-09-09 10:55                                     ` Takashi Yano
2021-09-09 11:41                                       ` Corinna Vinschen
2021-09-08  9:37                       ` Takashi Yano
2021-09-09  3:41               ` Takashi Yano
2021-09-09  8:05                 ` Takashi Yano
2021-09-09 12:19                   ` Takashi Yano
2021-09-09 12:42                     ` Takashi Yano
2021-09-09 21:53                       ` Takashi Yano
2021-09-10  3:41                         ` Takashi Yano
2021-09-10 10:57                       ` Ken Brown
2021-09-10 15:17                         ` Ken Brown
2021-09-10 15:26                           ` Corinna Vinschen
2021-09-10 22:57                           ` Takashi Yano
2021-09-11  2:17                             ` Ken Brown
2021-09-11  2:35                               ` Takashi Yano
2021-09-11 13:12                                 ` Ken Brown
2021-09-12  6:23                                   ` Takashi Yano
2021-09-12 14:39                                     ` Ken Brown
2021-09-13  9:11                                       ` Corinna Vinschen
2021-09-13 12:30                                         ` Ken Brown
2021-09-12  8:48                                   ` Takashi Yano
2021-09-12 11:04                                     ` Takashi Yano
2021-09-12 15:10                                       ` Ken Brown
2021-09-12 21:46                                         ` Ken Brown
2021-09-12 23:54                                           ` Takashi Yano
2021-09-13  2:19                                             ` Ken Brown
2021-09-13  8:40                                             ` Takashi Yano
2021-09-13 12:51                                               ` Ken Brown
2021-09-13 17:05                                                 ` Ken Brown
2021-09-13  9:42                                           ` Corinna Vinschen
2021-09-13 13:03                                             ` Ken Brown
2021-09-13 18:39                                               ` Takashi Yano
2021-09-12 23:41                                         ` Takashi Yano
2021-09-13 17:42                                       ` Ken Brown
2021-09-13 18:54                                         ` Takashi Yano
2021-09-13 18:32                                       ` Corinna Vinschen
2021-09-13 19:37                                         ` Takashi Yano
2021-09-13 20:15                                           ` Corinna Vinschen
2021-09-14  8:07                                             ` Takashi Yano
2021-09-14  8:47                                               ` Corinna Vinschen
2021-09-14 12:38                                                 ` Ken Brown
2021-09-14 14:15                                                   ` Corinna Vinschen
2021-09-14  8:08                                           ` Takashi Yano
2021-09-14  9:03                                             ` Corinna Vinschen
2021-09-14  9:56                                               ` Takashi Yano
2021-09-14 10:19                                                 ` Takashi Yano
2021-09-14 11:03                                                   ` Corinna Vinschen
2021-09-14 12:05                                                     ` Takashi Yano
2021-09-14 14:17                                                       ` Corinna Vinschen
2021-09-14 22:14                                                       ` Ken Brown
2021-09-15  0:21                                                         ` Takashi Yano
2021-09-15  0:44                                                           ` Takashi Yano
2021-09-15  0:59                                                             ` Takashi Yano
2021-09-15  9:57                                                               ` Corinna Vinschen
2021-09-15 10:48                                                                 ` Takashi Yano
2021-09-15 10:58                                                                   ` Takashi Yano
2021-09-15 11:34                                                                     ` Corinna Vinschen
2021-09-15 11:40                                                                       ` Corinna Vinschen
2021-09-15 11:13                                                                   ` Corinna Vinschen
2021-09-15 11:41                                                                     ` Ken Brown
2021-09-15 11:49                                                                       ` Corinna Vinschen
2021-09-15 11:54                                                                     ` Takashi Yano
2021-09-15 12:20                                                                       ` Corinna Vinschen
2021-09-15 13:04                                                                         ` Takashi Yano
2021-09-15 13:42                                                                           ` Corinna Vinschen
2021-09-15 16:22                                                                             ` Ken Brown
2021-09-15 17:09                                                                               ` Ken Brown
2021-09-16  0:22                                                                                 ` Takashi Yano
2021-09-16  2:28                                                                                   ` Ken Brown
2021-09-16  9:09                                                                                 ` Takashi Yano
2021-09-16 13:02                                                                                   ` Takashi Yano
2021-09-16 13:25                                                                                     ` Corinna Vinschen
2021-09-16 14:27                                                                                       ` Takashi Yano
2021-09-16 15:01                                                                                         ` Corinna Vinschen
2021-09-16 15:46                                                                                           ` Ken Brown
2021-09-16 16:02                                                                                             ` Ken Brown
2021-09-16 19:42                                                                                               ` Takashi Yano
2021-09-16 20:28                                                                                                 ` Ken Brown
2021-09-16 19:48                                                                                               ` Ken Brown
2021-09-16 20:01                                                                                                 ` Takashi Yano
2021-09-17  2:25                                                                                                   ` Ken Brown
2021-09-17  8:31                                                                                                     ` Takashi Yano
2021-09-17 11:16                                                                                                       ` Ken Brown
2021-09-17 16:23                                                                                                         ` Takashi Yano
2021-09-17 17:08                                                                                                           ` Ken Brown
2021-09-17 17:39                                                                                                             ` Jon Turney
2021-09-17 17:43                                                                                                             ` Takashi Yano
2021-09-17 19:53                                                                                                               ` Ken Brown
2021-09-18  1:30                                                                                                                 ` Takashi Yano
2021-09-18  2:07                                                                                                                   ` Ken Brown
2021-09-18  2:10                                                                                                                     ` Ken Brown
2021-09-18  8:03                                                                                                                       ` Takashi Yano
2021-09-18 11:12                                                                                                                         ` Ken Brown
2021-09-18 11:35                                                                                                                           ` Takashi Yano
2021-09-18 14:11                                                                                                                             ` Jon Turney
2021-09-18 13:44                                                                                           ` Ken Brown
2021-09-19  1:31                                                                                             ` Takashi Yano
2021-09-19 14:35                                                                                               ` Ken Brown
2021-09-20  9:29                                                                                                 ` Takashi Yano
2021-09-16  0:13                                                                               ` Takashi Yano
2021-09-16  2:26                                                                                 ` Ken Brown
2021-09-13  9:07                                 ` Corinna Vinschen
2021-09-20 12:52                                   ` Takashi Yano
2021-09-20 19:14                                     ` Ken Brown
2021-09-20 21:09                                       ` Ken Brown
2021-09-20 21:21                                         ` Ken Brown
2021-09-20 21:27                                         ` Takashi Yano
2021-09-20 21:39                                           ` Ken Brown
2021-09-20 22:16                                             ` Takashi Yano
2021-09-20 22:46                                               ` Ken Brown
2021-09-20 22:50                                                 ` Ken Brown
2021-09-20 23:22                                                   ` Takashi Yano
2021-09-21  8:30                                                     ` Takashi Yano
2021-09-21  9:26                                                       ` Mark Geisert
2021-09-21 10:10                                                         ` Takashi Yano
2021-09-21 21:10                                                           ` Mark Geisert
2021-09-21 13:31                                                       ` Ken Brown
2021-09-21 15:36                                                         ` Takashi Yano
2021-09-21 18:51                                                           ` Ken Brown
2021-09-23  8:26                                                             ` Takashi Yano
2021-09-23 13:03                                                               ` Ken Brown
2021-09-23 15:03                                                                 ` Takashi Yano
2021-09-23 16:29                                                                   ` Ken Brown
2021-10-18 10:51                                                                   ` Corinna Vinschen
2021-10-18 12:02                                                                     ` Takashi Yano

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).