Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Ken Brown <kbrown@cornell.edu>]

On 8/30/2021 2:59 PM, Ken Brown wrote:
> On 8/30/2021 1:00 PM, Corinna Vinschen wrote:
>> On Aug 30 17:53, Corinna Vinschen wrote:
>>> On Aug 30 16:05, Corinna Vinschen wrote:
>>>> On Aug 30 09:36, Ken Brown wrote:
>>>>> BTW, when I was working on the pipe approach to AF_UNIX sockets
>>>>> (topic/af_unix branch), I had occasion to step through
>>>>> select.cc:pipe_data_available in gdb, and the use of fpli.OutboundQuota -
>>>>> fpli.ReadDataAvailable definitely seemed wrong to me.  So when I wrote
>>>>> peek_socket_unix on that branch, I used fpli.WriteQuotaAvailable, as Takashi
>>>>> is suggesting now.
>>>>
>>>> If that's working reliable these days (keeping fingers crossed for W7),
>>>> it's ok if we use that.  We may want to check if the above observation
>>>> in terms on WriteQuotaAvailable on a pipe with pending read is still an
>>>> issue.
>>>
>>> Ok, I wrote a small testcase.  It creates a named pipe, reads from the
>>> pipe, then, later, writes to the pipe.  Interlaced with these calls, it
>>> calls NtQueryInformationFile(FilePipeLocalInformation) on the write side
>>> of the pipe.  Kind of like this:
>>>
>>>    CreatePipe
>>>    NtQueryInformationFile
>>>    ReadFile
>>>    NtQueryInformationFile
>>>    WriteFile
>>>    NtQueryInformationFile
>>>
>>> Here's the result:
>>>
>>> Before ReadFile:
>>>
>>>    InboundQuota: 65536
>>>    ReadDataAvailable: 0
>>>    OutboundQuota: 65536
>>>    WriteQuotaAvailable: 65536
>>>
>>> While ReadFile is running:
>>>
>>>    InboundQuota: 65536
>>>    ReadDataAvailable: 0
>>>    OutboundQuota: 65536
>>>    WriteQuotaAvailable: 65494    !!!
>>>
>>> After WriteFile and ReadFile succeeded:
>>>
>>>    InboundQuota: 65536
>>>    ReadDataAvailable: 0
>>>    OutboundQuota: 65536
>>>    WriteQuotaAvailable: 65536
>>>
>>> That means, while a reader on the reader side is waiting for data, the
>>> WriteQuotaAvailable on the write side is decremented by the amount of
>>> data requested by the reader (42 bytes in my case), just as outlined in that
>>> mail from 2004.  And this is on W10 now.
>>>
>>> What to do with this information?  TBD.
>>
>> Ok, let's discuss this.  I added more code to my testcase and here's
>> what I see.  I dropped all data from the output which doesn't change.
>>
>> What I'm trying to get a grip on are the dependencies here.
>>
>> After creating the pipe:
>>
>>    read side: ReadDataAvailable: 0
>>    write side: WriteQuotaAvailable: 65536
>>
>> After writing 20 bytes...
>>
>>    read side: ReadDataAvailable: 20
>>    write side: WriteQuotaAvailable: 65516
>>
>> After writing 40 more bytes...
>>
>>    read side: ReadDataAvailable: 60
>>    write side: WriteQuotaAvailable: 65476
>>
>> After reading 42 bytes...
>>
>>    read side: ReadDataAvailable: 18
>>    write side: WriteQuotaAvailable: 65518
>>
>> After writing 20 bytes...
>>
>>    read side: ReadDataAvailable: 38
>>    write side: WriteQuotaAvailable: 65498
>>
>> *While* reading 42 bytes with an empty buffer...
>>
>>    read side: ReadDataAvailable: 0
>>    write side: WriteQuotaAvailable: 65494
>>
>> Another important fun fact:  Assuming the read and write buffer sizes
>> are differently specified.  I called CreateNamedPipe with an outbuffer
>> size of 32K and an inbuffer size of 64K:
>>
>> After creating the pipe:
>>
>>    read side:
>>      InboundQuota: 65536
>>      ReadDataAvailable: 0
>>      OutboundQuota: 32768
>>      WriteQuotaAvailable: 32768
>>    write side:
>>      InboundQuota: 65536
>>      ReadDataAvailable: 0
>>      OutboundQuota: 32768
>>      WriteQuotaAvailable: 65536    !!!
>>
>> This last data point shows that:
>>
>> - InboundQuota and OutboundQuota are always constant values and
>>    do not depend on the side the information has been queried on.
>>    That certainly makes sense.
>>
>> - WriteQuotaAvailable does not depend on the OutboundQuota, but on
>>    the InboundQuota, and very likely on the InboundQuota of the read
>>    side.  The OutboundQuota *probably* only makes sense when using
>>    named pipes with remote clients, which we never do anyway.
>>
>> The preceeding output shows that ReadDataAvailable on the read side and
>> WriteQuotaAvailable on the write side are connected.  If we write 20
>> bytes, ReadDataAvailable is incremented by 20 and WriteQuotaAvailable is
>> decremented by 20.
>>
>> So: write.WriteQuotaAvailable == InboundQuota - read.ReadDataAvailable.
>>
>> Except when a ReadFile is pending on the read side.  It's as if the
>> running ReadFile already reserved write quota.  So the write side
>> WriteQuotaAvailable is the number of bytes we can write without blocking,
>> after all pending ReadFiles have been satisfied.
>>
>> Unfortunately that doesn't really make sense when looked at it from the
>> user space.
>>
>> What that means in the first place is that WriteQuotaAvailable on the
>> write side is unreliable.  What we really need is InboundQuota -
>> read.ReadDataAvailable.  The problem with that is that the write side
>> usually has no access to the read side of the pipe.
> 
> For the purposes of select.cc:pipe_data_available, we only need to know whether 
> InboundQuota - read.ReadDataAvailable is positive.  If WriteQuotaAvailable is 
> positive, then we're OK, even though its precise value might be too small.  But 
> if WriteQuotaAvailable == 0, we don't know whether the buffer is actually full 
> or there's a pending ReadFile.  It's only in this case that we need access to 
> the read side.
> 
> What if we reverse the roles of the read and write sides of the pipe, so that 
> the write side is the server and the read side is the client.  We can then try 
> to use ImpersonateNamedPipeClient to get information about the read side when 
> WriteQuotaAvailable == 0.  If we succeed, then we can determine whether or not 
> there's space in the buffer.  If we fail, we simply report, possibly 
> incorrectly, that there is space.  This is no worse than the current situation 
> (on the master branch), in which we use OutboundQuota - ReadDataAvailable, which 
> is always positive.

I should add that I know absolutely nothing about ImpersonateNamedPipeClient, so 
this might be complete nonsense.

Ken