cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Chris Roehrig]

I have a network of Windows, Linux and Mac machines and I use rsync to synchronize various directories between them.

I'm trying to figure out why my rsync transfers are so slow (<4 MB/s) only when the remote endpoint is Cygwin rsync over sshd (with both a Linux or Cygwin rsync client).   In all other scenarios, I get the full 100MB/s as expected from gigabit ethernet.  This has been an ongoing problem for me for a couple of years over several Windows and Cygwin versions, and I'd like to try to fix it.    

If I run rsync --daemon --no-detach under mintty in the foreground on the remote Windows endpoint,  I get the full 100 MB/s transfers, so it seems like it has something to do with rsync.exe running in the background under the cygrunsrv+sshd service (which was installed normally using ssh-host-config).

If I do:
	pv /dev/zero | ssh $WINHOST "cat > /dev/null"
or even
	pv /dev/urandom | ssh $WINHOST md5sum
I also get the full 100 MB/s transfers, so it doesn't look like sshd itself is being throttled by bandwidth or CPU.

The machines have less than 15% CPU utilization while transferring, with each of the 4 cores less than 30%, so it doesn't look to be CPU issue.
In Task Manager, sshd.exe and rsync.exe seem to be running normally using only few percent CPU, and show Power Throttling=Disabled, Priority=Normal.   Setting their Priority to High doesn't seem to change things.

Looking in Resource Monitor on the remote endpoint, the network usage is pretty much a flat horizontal line at about 18 Mbps (2.5 MB/s), so it sure looks to me as if rsync is somehow being bandwidth-throttled\b when run in the background under cygsshd.

It's almost as if rsync has an implicit --bwlimit override when it is run from cygrunsrv+sshd (I've tried --bwlimit=0 on the client which makes no difference).


Any ideas?    Not sure where to go from here. 

Thanks,
-- Chris


Windows 10 v2004 (64-bit)
CYGWIN_NT-10.0 .... 3.2.0(0.340/5/3) 2021-03-29 08:42 x86_64 Cygwin
rsync version 3.2.4dev protocol version 31
Linux Mint 20

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Mark Geisert]

Chris Roehrig wrote:
> I have a network of Windows, Linux and Mac machines and I use rsync to synchronize various directories between them.
> 
> I'm trying to figure out why my rsync transfers are so slow (<4 MB/s) only when the remote endpoint is Cygwin rsync over sshd (with both a Linux or Cygwin rsync client).   In all other scenarios, I get the full 100MB/s as expected from gigabit ethernet.  This has been an ongoing problem for me for a couple of years over several Windows and Cygwin versions, and I'd like to try to fix it.
> 
> If I run rsync --daemon --no-detach under mintty in the foreground on the remote Windows endpoint,  I get the full 100 MB/s transfers, so it seems like it has something to do with rsync.exe running in the background under the cygrunsrv+sshd service (which was installed normally using ssh-host-config).
> 
> If I do:
> 	pv /dev/zero | ssh $WINHOST "cat > /dev/null"
> or even
> 	pv /dev/urandom | ssh $WINHOST md5sum
> I also get the full 100 MB/s transfers, so it doesn't look like sshd itself is being throttled by bandwidth or CPU.
> 
> The machines have less than 15% CPU utilization while transferring, with each of the 4 cores less than 30%, so it doesn't look to be CPU issue.
> In Task Manager, sshd.exe and rsync.exe seem to be running normally using only few percent CPU, and show Power Throttling=Disabled, Priority=Normal.   Setting their Priority to High doesn't seem to change things.
> 
> Looking in Resource Monitor on the remote endpoint, the network usage is pretty much a flat horizontal line at about 18 Mbps (2.5 MB/s), so it sure looks to me as if rsync is somehow being bandwidth-throttled\b when run in the background under cygsshd.
> 
> It's almost as if rsync has an implicit --bwlimit override when it is run from cygrunsrv+sshd (I've tried --bwlimit=0 on the client which makes no difference).
> 
> 
> Any ideas?    Not sure where to go from here.

You're not the first to report this.  I don't have any quick answer.  But can you 
give one or two simple examples of commands that give slow transfers in your 
environment?  Simple like your 'pv' examples, if possible, using whatever method 
that works.

..mark

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Chris Roehrig]

Here's my test in a nutshell:

# ON WINDOWS: install Cygwin and enable cygsshd
	ssh-host-config -y
	# set up authorized keys, etc to make things easier

# LINUX: Create a 2GB random file on Linux:
	dd if=/dev/urandom of=/tmp/bigfile.bin bs=1M count=2000

# WINDOWS: rsync "pull" to cygwin
	LINUXHOST=mylinuxhost
	rsync -Pva $LINUXHOST:/tmp/bigfile.bin /tmp/            # 100MB/s full speed
	rm /tmp/bigfile.bin

# LINUX: rsync "push" to cygwin (from Linux machine)
	WINHOST=mywindowshost
	rsync -Pva /tmp/bigfile.bin $WINHOST:/tmp/              # slow, dropping to 4MB/s


I get the same results transferring the other direction (WIN -> LINUX):   The Cygwin remote endpoint is always slow.





On Tue Aug 24 2021, at 4:43 PM, Mark Geisert <mark@maxrnd.com> wrote:

> Chris Roehrig wrote:
>> I have a network of Windows, Linux and Mac machines and I use rsync to synchronize various directories between them.
>> I'm trying to figure out why my rsync transfers are so slow (<4 MB/s) only when the remote endpoint is Cygwin rsync over sshd (with both a Linux or Cygwin rsync client).   In all other scenarios, I get the full 100MB/s as expected from gigabit ethernet.  This has been an ongoing problem for me for a couple of years over several Windows and Cygwin versions, and I'd like to try to fix it.
>> If I run rsync --daemon --no-detach under mintty in the foreground on the remote Windows endpoint,  I get the full 100 MB/s transfers, so it seems like it has something to do with rsync.exe running in the background under the cygrunsrv+sshd service (which was installed normally using ssh-host-config).
>> If I do:
>> 	pv /dev/zero | ssh $WINHOST "cat > /dev/null"
>> or even
>> 	pv /dev/urandom | ssh $WINHOST md5sum
>> I also get the full 100 MB/s transfers, so it doesn't look like sshd itself is being throttled by bandwidth or CPU.
>> The machines have less than 15% CPU utilization while transferring, with each of the 4 cores less than 30%, so it doesn't look to be CPU issue.
>> In Task Manager, sshd.exe and rsync.exe seem to be running normally using only few percent CPU, and show Power Throttling=Disabled, Priority=Normal.   Setting their Priority to High doesn't seem to change things.
>> Looking in Resource Monitor on the remote endpoint, the network usage is pretty much a flat horizontal line at about 18 Mbps (2.5 MB/s), so it sure looks to me as if rsync is somehow being bandwidth-throttled\b when run in the background under cygsshd.
>> It's almost as if rsync has an implicit --bwlimit override when it is run from cygrunsrv+sshd (I've tried --bwlimit=0 on the client which makes no difference).
>> Any ideas?    Not sure where to go from here.
> 
> You're not the first to report this.  I don't have any quick answer.  But can you give one or two simple examples of commands that give slow transfers in your environment?  Simple like your 'pv' examples, if possible, using whatever method that works.
> 
> ..mark
> 
> -- 
> Problem reports:      https://cygwin.com/problems.html
> FAQ:                  https://cygwin.com/faq/
> Documentation:        https://cygwin.com/docs.html
> Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Chris Roehrig]

Here's the other direction with rsync also in --daemon mode:

# ON WINDOWS (from a mintty terminal):
	dd if=/dev/urandom of=/tmp/bigfile.bin bs=1M count=2000	# create 2GB file
	printf "[TMP]\npath = /tmp\n" > /etc/rsyncd.conf				# create rsyncd.conf
	rsync --daemon --no-detach								# run rsync in daemon mode
	
# ON LINUX:
rm -f /tmp/bigfile.bin; rsync -Pva $WINHOST:/tmp/bigfile.bin  /tmp/				# 4MB/s
rm -f /tmp/bigfile.bin; rsync -Pva rsync://$WINHOST/TMP/bigfile.bin  /tmp/		# 100MB/s


I was thinking that there might be some Windows Policy that rate-limits some syscalls for background process, but I also tried:
# ON WINDOWS (Administrator mintty):
	cygrunsrv -E cygsshd			# exit sshd service
	/usr/sbin/sshd -D				# run sshd in "foreground" (GUI app)
and I still get 4MB/s.

Maybe some IPC rate-limit issue between sshd and rsync (but that isn't there between e.g. sshd and md5sum... ?)


On Tue Aug 24 2021, at 7:02 PM, NightStrike <nightstrike@gmail.com> wrote:
> Older versions of windows had a setting to optimize the OS for either background services or foreground applications. One of the things this did was throttle network usage. I don't know if windows 10 has the same setting though.

I read about a NetworkThrottlingIndex that limits "non-multimedia" packets to 10 per millisecond which can be disabled by setting:
	[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Multimedia\SystemProfile]
	"NetworkThrottlingIndex"=dword:ffffffff
but this made no difference to my tests.



On Tue Aug 24 2021, at 5:35 PM, Chris Roehrig <croehrig@house.org> wrote:

> Here's my test in a nutshell:
> 
> # ON WINDOWS: install Cygwin and enable cygsshd
> 	ssh-host-config -y
> 	# set up authorized keys, etc to make things easier
> 
> # LINUX: Create a 2GB random file on Linux:
> 	dd if=/dev/urandom of=/tmp/bigfile.bin bs=1M count=2000
> 
> # WINDOWS: rsync "pull" to cygwin
> 	LINUXHOST=mylinuxhost
> 	rsync -Pva $LINUXHOST:/tmp/bigfile.bin /tmp/            # 100MB/s full speed
> 	rm /tmp/bigfile.bin
> 
> # LINUX: rsync "push" to cygwin (from Linux machine)
> 	WINHOST=mywindowshost
> 	rsync -Pva /tmp/bigfile.bin $WINHOST:/tmp/              # slow, dropping to 4MB/s
> 
> 
> I get the same results transferring the other direction (WIN -> LINUX):   The Cygwin remote endpoint is always slow.
> 
> 
> 
> 
> 
> On Tue Aug 24 2021, at 4:43 PM, Mark Geisert <mark@maxrnd.com> wrote:
> 
>> Chris Roehrig wrote:
>>> I have a network of Windows, Linux and Mac machines and I use rsync to synchronize various directories between them.
>>> I'm trying to figure out why my rsync transfers are so slow (<4 MB/s) only when the remote endpoint is Cygwin rsync over sshd (with both a Linux or Cygwin rsync client).   In all other scenarios, I get the full 100MB/s as expected from gigabit ethernet.  This has been an ongoing problem for me for a couple of years over several Windows and Cygwin versions, and I'd like to try to fix it.
>>> If I run rsync --daemon --no-detach under mintty in the foreground on the remote Windows endpoint,  I get the full 100 MB/s transfers, so it seems like it has something to do with rsync.exe running in the background under the cygrunsrv+sshd service (which was installed normally using ssh-host-config).
>>> If I do:
>>> 	pv /dev/zero | ssh $WINHOST "cat > /dev/null"
>>> or even
>>> 	pv /dev/urandom | ssh $WINHOST md5sum
>>> I also get the full 100 MB/s transfers, so it doesn't look like sshd itself is being throttled by bandwidth or CPU.
>>> The machines have less than 15% CPU utilization while transferring, with each of the 4 cores less than 30%, so it doesn't look to be CPU issue.
>>> In Task Manager, sshd.exe and rsync.exe seem to be running normally using only few percent CPU, and show Power Throttling=Disabled, Priority=Normal.   Setting their Priority to High doesn't seem to change things.
>>> Looking in Resource Monitor on the remote endpoint, the network usage is pretty much a flat horizontal line at about 18 Mbps (2.5 MB/s), so it sure looks to me as if rsync is somehow being bandwidth-throttled\b when run in the background under cygsshd.
>>> It's almost as if rsync has an implicit --bwlimit override when it is run from cygrunsrv+sshd (I've tried --bwlimit=0 on the client which makes no difference).
>>> Any ideas?    Not sure where to go from here.
>> 
>> You're not the first to report this.  I don't have any quick answer.  But can you give one or two simple examples of commands that give slow transfers in your environment?  Simple like your 'pv' examples, if possible, using whatever method that works.
>> 
>> ..mark
>> 
>> -- 
>> Problem reports:      https://cygwin.com/problems.html
>> FAQ:                  https://cygwin.com/faq/
>> Documentation:        https://cygwin.com/docs.html
>> Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple
> 
> 
> -- 
> Problem reports:      https://cygwin.com/problems.html
> FAQ:                  https://cygwin.com/faq/
> Documentation:        https://cygwin.com/docs.html
> Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[NightStrike]

On Tue, Aug 24, 2021, 09:50 Chris Roehrig <croehrig@house.org> wrote:

> I have a network of Windows, Linux and Mac machines and I use rsync to
> synchronize various directories between them.
>
> I'm trying to figure out why my rsync transfers are so slow (<4 MB/s) only
> when the remote endpoint is Cygwin rsync over sshd (with both a Linux or
> Cygwin rsync client).   In all other scenarios, I get the full 100MB/s as
> expected from gigabit ethernet.  This has been an ongoing problem for me
> for a couple of years over several Windows and Cygwin versions, and I'd
> like to try to fix it.
>
> If I run rsync --daemon --no-detach under mintty in the foreground on the
> remote Windows endpoint,  I get the full 100 MB/s transfers, so it seems
> like it has something to do with rsync.exe running in the background under
> the cygrunsrv+sshd service (which was installed normally using
> ssh-host-config).
>
> If I do:
>         pv /dev/zero | ssh $WINHOST "cat > /dev/null"
> or even
>         pv /dev/urandom | ssh $WINHOST md5sum
> I also get the full 100 MB/s transfers, so it doesn't look like sshd
> itself is being throttled by bandwidth or CPU.
>
> The machines have less than 15% CPU utilization while transferring, with
> each of the 4 cores less than 30%, so it doesn't look to be CPU issue.
> In Task Manager, sshd.exe and rsync.exe seem to be running normally using
> only few percent CPU, and show Power Throttling=Disabled, Priority=Normal.
>  Setting their Priority to High doesn't seem to change things.
>
> Looking in Resource Monitor on the remote endpoint, the network usage is
> pretty much a flat horizontal line at about 18 Mbps (2.5 MB/s), so it sure
> looks to me as if rsync is somehow being bandwidth-throttled  when run in
> the background under cygsshd.
>
> It's almost as if rsync has an implicit --bwlimit override when it is run
> from cygrunsrv+sshd (I've tried --bwlimit=0 on the client which makes no
> difference).
>
>
> Any ideas?    Not sure where to go from here.
>
> Thanks,
> -- Chris
>
>
> Windows 10 v2004 (64-bit)
> CYGWIN_NT-10.0 .... 3.2.0(0.340/5/3) 2021-03-29 08:42 x86_64 Cygwin
> rsync version 3.2.4dev protocol version 31
> Linux Mint 20
>

Older versions of windows had a setting to optimize the OS for either
background services or foreground applications. One of the things this did
was throttle network usage. I don't know if windows 10 has the same setting
though.

>

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Mark Geisert]

NightStrike via Cygwin wrote:
> Older versions of windows had a setting to optimize the OS for either
> background services or foreground applications. One of the things this did
> was throttle network usage. I don't know if windows 10 has the same setting
> though.

Yes, it does.  Getting to it is a pain.  Search the "Settings" thingy for 
"Advanced system settings".  That will bring up the System Properties dialog you 
used to be able to get to from the Control Panel.  Click the Advanced tab.  Under 
Performance, click Settings... .  Click the Advanced tab there.  Right in that 
dialog there's a Processor Scheduling box where you select for best performance of 
"Programs" or "Background services".

It would be interesting to hear what it's currently set to before you try changing 
it.  Try to keep all your testcases the same so we're dealing with apples to apples.

..mark

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Chris Roehrig]

It was set to "Programs".    Changing it to "Background services"  didn't make a difference.

TCPOptimizer can adjust 2 registry entries that I think are related to that Windows Setting:

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Multimedia\SystemProfile]
	"NetworkThrottlingIndex"=dword:0000000a
	"SystemResponsiveness"=dword:00000014

	The first rate-limits IP packets; the second reserves a percent (0-100) of CPU scheduling for background tasks.   Changing either of those values made no difference.      I think the CPU scheduling reservation only applies when there's CPU contention (not on an idle system) so I wouldn't expect it to make a difference when CPU/core usage is low.



On Tue Aug 24 2021, at 8:20 PM, Mark Geisert <mark@maxrnd.com> wrote:

> NightStrike via Cygwin wrote:
>> Older versions of windows had a setting to optimize the OS for either
>> background services or foreground applications. One of the things this did
>> was throttle network usage. I don't know if windows 10 has the same setting
>> though.
> 
> Yes, it does.  Getting to it is a pain.  Search the "Settings" thingy for "Advanced system settings".  That will bring up the System Properties dialog you used to be able to get to from the Control Panel.  Click the Advanced tab.  Under Performance, click Settings... .  Click the Advanced tab there.  Right in that dialog there's a Processor Scheduling box where you select for best performance of "Programs" or "Background services".
> 
> It would be interesting to hear what it's currently set to before you try changing it.  Try to keep all your testcases the same so we're dealing with apples to apples.
> 
> ..mark
> 
> -- 
> Problem reports:      https://cygwin.com/problems.html
> FAQ:                  https://cygwin.com/faq/
> Documentation:        https://cygwin.com/docs.html
> Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Takashi Yano]

On Tue, 24 Aug 2021 12:49:52 -0700
Chris Roehrig wrote:
> I have a network of Windows, Linux and Mac machines and I use rsync to synchronize various directories between them.
> 
> I'm trying to figure out why my rsync transfers are so slow (<4 MB/s) only when the remote endpoint is Cygwin rsync over sshd (with both a Linux or Cygwin rsync client).   In all other scenarios, I get the full 100MB/s as expected from gigabit ethernet.  This has been an ongoing problem for me for a couple of years over several Windows and Cygwin versions, and I'd like to try to fix it.    
> 
> If I run rsync --daemon --no-detach under mintty in the foreground on the remote Windows endpoint,  I get the full 100 MB/s transfers, so it seems like it has something to do with rsync.exe running in the background under the cygrunsrv+sshd service (which was installed normally using ssh-host-config).
> 
> If I do:
> 	pv /dev/zero | ssh $WINHOST "cat > /dev/null"
> or even
> 	pv /dev/urandom | ssh $WINHOST md5sum
> I also get the full 100 MB/s transfers, so it doesn't look like sshd itself is being throttled by bandwidth or CPU.
> 
> The machines have less than 15% CPU utilization while transferring, with each of the 4 cores less than 30%, so it doesn't look to be CPU issue.
> In Task Manager, sshd.exe and rsync.exe seem to be running normally using only few percent CPU, and show Power Throttling=Disabled, Priority=Normal.   Setting their Priority to High doesn't seem to change things.
> 
> Looking in Resource Monitor on the remote endpoint, the network usage is pretty much a flat horizontal line at about 18 Mbps (2.5 MB/s), so it sure looks to me as if rsync is somehow being bandwidth-throttled\b when run in the background under cygsshd.
> 
> It's almost as if rsync has an implicit --bwlimit override when it is run from cygrunsrv+sshd (I've tried --bwlimit=0 on the client which makes no difference).
> 
> 
> Any ideas?    Not sure where to go from here. 

In cygwin, just scp is very slow.

The transfer speed in my environment is as follows.
The tests were done with 100MB of test.dat file.

(1-1) From cygwin-PC,
[yano@cygwin-PC ~]$ scp test.dat yano@linux-server:.
yano@linux-server's password:
test.dat                                      100%  100MB   4.0MB/s   00:24
[yano@cygwin-PC ~]$ scp yano@linux-server:test.dat .
yano@linux-server's password:
test.dat                                      100%  100MB   8.0MB/s   00:12

(1-2) From linux-server,
yano@linux-server:~$ scp yano@cygwin-PC:test.dat .
yano@cygwin-PC's password:
test.dat                                      100%  100MB   4.0MB/s   00:24
yano@linux-server:~$ scp test.dat yano@cygwin-PC:.
yano@cygwin-PC's password:
test.dat                                      100%  100MB   4.1MB/s   00:24


I looked into this problem, and noticed that this is caused
by cygwin pipe implementation. Pipe in cygwin is configured
with FILE_FLAG_OVERLAPPED.

If the pipe is configured without FILE_FLAG_OVERLAPPED,
the transfer speed is much improved as follows.


(2-1) From cygwin-PC,
[yano@cygwin-PC ~]$ scp test.dat yano@linux-server:.
yano@linux-server's password:
test.dat                                      100%  100MB  85.5MB/s   00:01
[yano@cygwin-PC ~]$ scp yano@linux-server:test.dat .
yano@linux-server's password:
test.dat                                      100%  100MB  69.7MB/s   00:01

(2-2) From linux-server,
yano@linux-server:~$ scp yano@cygwin-PC:test.dat .
yano@cygwin-PC's password:
test.dat                                      100%  100MB  80.1MB/s   00:01
yano@linux-server:~$ scp test.dat yano@cygwin-PC:.
yano@cygwin-PC's password:
test.dat                                      100%  100MB  57.7MB/s   00:01

I am not sure why this happens and how to fix this.
Any idea?

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Chris Roehrig]

The FILE_FLAG_OVERLAPPED avenue looks promising.     I get exactly the same results as you using 'scp':   4MB/s in either direction (with either remote endpoint.)

I guess the next step is to install a build environment and build rsync and sshd ...





On Wed Aug 25 2021, at 4:18 AM, Takashi Yano via Cygwin <cygwin@cygwin.com> wrote:

> On Tue, 24 Aug 2021 12:49:52 -0700
> Chris Roehrig wrote:
>> I have a network of Windows, Linux and Mac machines and I use rsync to synchronize various directories between them.
>> 
>> I'm trying to figure out why my rsync transfers are so slow (<4 MB/s) only when the remote endpoint is Cygwin rsync over sshd (with both a Linux or Cygwin rsync client).   In all other scenarios, I get the full 100MB/s as expected from gigabit ethernet.  This has been an ongoing problem for me for a couple of years over several Windows and Cygwin versions, and I'd like to try to fix it.    
>> 
>> If I run rsync --daemon --no-detach under mintty in the foreground on the remote Windows endpoint,  I get the full 100 MB/s transfers, so it seems like it has something to do with rsync.exe running in the background under the cygrunsrv+sshd service (which was installed normally using ssh-host-config).
>> 
>> If I do:
>> 	pv /dev/zero | ssh $WINHOST "cat > /dev/null"
>> or even
>> 	pv /dev/urandom | ssh $WINHOST md5sum
>> I also get the full 100 MB/s transfers, so it doesn't look like sshd itself is being throttled by bandwidth or CPU.
>> 
>> The machines have less than 15% CPU utilization while transferring, with each of the 4 cores less than 30%, so it doesn't look to be CPU issue.
>> In Task Manager, sshd.exe and rsync.exe seem to be running normally using only few percent CPU, and show Power Throttling=Disabled, Priority=Normal.   Setting their Priority to High doesn't seem to change things.
>> 
>> Looking in Resource Monitor on the remote endpoint, the network usage is pretty much a flat horizontal line at about 18 Mbps (2.5 MB/s), so it sure looks to me as if rsync is somehow being bandwidth-throttled\b when run in the background under cygsshd.
>> 
>> It's almost as if rsync has an implicit --bwlimit override when it is run from cygrunsrv+sshd (I've tried --bwlimit=0 on the client which makes no difference).
>> 
>> 
>> Any ideas?    Not sure where to go from here. 
> 
> In cygwin, just scp is very slow.
> 
> The transfer speed in my environment is as follows.
> The tests were done with 100MB of test.dat file.
> 
> (1-1) From cygwin-PC,
> [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:.
> yano@linux-server's password:
> test.dat                                      100%  100MB   4.0MB/s   00:24
> [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat .
> yano@linux-server's password:
> test.dat                                      100%  100MB   8.0MB/s   00:12
> 
> (1-2) From linux-server,
> yano@linux-server:~$ scp yano@cygwin-PC:test.dat .
> yano@cygwin-PC's password:
> test.dat                                      100%  100MB   4.0MB/s   00:24
> yano@linux-server:~$ scp test.dat yano@cygwin-PC:.
> yano@cygwin-PC's password:
> test.dat                                      100%  100MB   4.1MB/s   00:24
> 
> 
> I looked into this problem, and noticed that this is caused
> by cygwin pipe implementation. Pipe in cygwin is configured
> with FILE_FLAG_OVERLAPPED.
> 
> If the pipe is configured without FILE_FLAG_OVERLAPPED,
> the transfer speed is much improved as follows.
> 
> 
> (2-1) From cygwin-PC,
> [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:.
> yano@linux-server's password:
> test.dat                                      100%  100MB  85.5MB/s   00:01
> [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat .
> yano@linux-server's password:
> test.dat                                      100%  100MB  69.7MB/s   00:01
> 
> (2-2) From linux-server,
> yano@linux-server:~$ scp yano@cygwin-PC:test.dat .
> yano@cygwin-PC's password:
> test.dat                                      100%  100MB  80.1MB/s   00:01
> yano@linux-server:~$ scp test.dat yano@cygwin-PC:.
> yano@cygwin-PC's password:
> test.dat                                      100%  100MB  57.7MB/s   00:01
> 
> I am not sure why this happens and how to fix this.
> Any idea?
> 
> -- 
> Takashi Yano <takashi.yano@nifty.ne.jp>
> 
> -- 
> Problem reports:      https://cygwin.com/problems.html
> FAQ:                  https://cygwin.com/faq/
> Documentation:        https://cygwin.com/docs.html
> Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Ken Brown]

On 8/25/2021 7:18 AM, Takashi Yano via Cygwin wrote:
> On Tue, 24 Aug 2021 12:49:52 -0700
> Chris Roehrig wrote:
>> I have a network of Windows, Linux and Mac machines and I use rsync to synchronize various directories between them.
>>
>> I'm trying to figure out why my rsync transfers are so slow (<4 MB/s) only when the remote endpoint is Cygwin rsync over sshd (with both a Linux or Cygwin rsync client).   In all other scenarios, I get the full 100MB/s as expected from gigabit ethernet.  This has been an ongoing problem for me for a couple of years over several Windows and Cygwin versions, and I'd like to try to fix it.
>>
>> If I run rsync --daemon --no-detach under mintty in the foreground on the remote Windows endpoint,  I get the full 100 MB/s transfers, so it seems like it has something to do with rsync.exe running in the background under the cygrunsrv+sshd service (which was installed normally using ssh-host-config).
>>
>> If I do:
>> 	pv /dev/zero | ssh $WINHOST "cat > /dev/null"
>> or even
>> 	pv /dev/urandom | ssh $WINHOST md5sum
>> I also get the full 100 MB/s transfers, so it doesn't look like sshd itself is being throttled by bandwidth or CPU.
>>
>> The machines have less than 15% CPU utilization while transferring, with each of the 4 cores less than 30%, so it doesn't look to be CPU issue.
>> In Task Manager, sshd.exe and rsync.exe seem to be running normally using only few percent CPU, and show Power Throttling=Disabled, Priority=Normal.   Setting their Priority to High doesn't seem to change things.
>>
>> Looking in Resource Monitor on the remote endpoint, the network usage is pretty much a flat horizontal line at about 18 Mbps (2.5 MB/s), so it sure looks to me as if rsync is somehow being bandwidth-throttled\b when run in the background under cygsshd.
>>
>> It's almost as if rsync has an implicit --bwlimit override when it is run from cygrunsrv+sshd (I've tried --bwlimit=0 on the client which makes no difference).
>>
>>
>> Any ideas?    Not sure where to go from here.
> 
> In cygwin, just scp is very slow.
> 
> The transfer speed in my environment is as follows.
> The tests were done with 100MB of test.dat file.
> 
> (1-1) From cygwin-PC,
> [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:.
> yano@linux-server's password:
> test.dat                                      100%  100MB   4.0MB/s   00:24
> [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat .
> yano@linux-server's password:
> test.dat                                      100%  100MB   8.0MB/s   00:12
> 
> (1-2) From linux-server,
> yano@linux-server:~$ scp yano@cygwin-PC:test.dat .
> yano@cygwin-PC's password:
> test.dat                                      100%  100MB   4.0MB/s   00:24
> yano@linux-server:~$ scp test.dat yano@cygwin-PC:.
> yano@cygwin-PC's password:
> test.dat                                      100%  100MB   4.1MB/s   00:24
> 
> 
> I looked into this problem, and noticed that this is caused
> by cygwin pipe implementation. Pipe in cygwin is configured
> with FILE_FLAG_OVERLAPPED.
> 
> If the pipe is configured without FILE_FLAG_OVERLAPPED,
> the transfer speed is much improved as follows.
> 
> 
> (2-1) From cygwin-PC,
> [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:.
> yano@linux-server's password:
> test.dat                                      100%  100MB  85.5MB/s   00:01
> [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat .
> yano@linux-server's password:
> test.dat                                      100%  100MB  69.7MB/s   00:01
> 
> (2-2) From linux-server,
> yano@linux-server:~$ scp yano@cygwin-PC:test.dat .
> yano@cygwin-PC's password:
> test.dat                                      100%  100MB  80.1MB/s   00:01
> yano@linux-server:~$ scp test.dat yano@cygwin-PC:.
> yano@cygwin-PC's password:
> test.dat                                      100%  100MB  57.7MB/s   00:01
> 
> I am not sure why this happens and how to fix this.

A couple years ago I had an idea for changing the pipe implementation to avoid 
overlapped I/O:

   https://cygwin.com/pipermail/cygwin-patches/2019q2/009393.html
   https://cygwin.com/pipermail/cygwin-patches/2019q2/009423.html

I never followed up on it.  But if you think it might help with this problem, I 
could dust it off and try to finish it.

Ken

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Chris Roehrig]

On Wed Aug 25 2021, at 10:52 AM, Ken Brown via Cygwin <cygwin@cygwin.com> wrote:
> A couple years ago I had an idea for changing the pipe implementation to avoid overlapped I/O:
> 
>  https://cygwin.com/pipermail/cygwin-patches/2019q2/009393.html
>  https://cygwin.com/pipermail/cygwin-patches/2019q2/009423.html
> 
> I never followed up on it.  But if you think it might help with this problem, I could dust it off and try to finish it.
> 
> Ken

I'm not familiar enough with the innards of rsync, sshd or cygwin to know how this would work.
Is it possible to have a new CYGWIN environment option to switch the pipe behaviour without requiring changes to the ssh or rsync source code (and without breaking any existing stuff)?

- Chris



> On 8/25/2021 7:18 AM, Takashi Yano via Cygwin wrote:
>> On Tue, 24 Aug 2021 12:49:52 -0700
>> Chris Roehrig wrote:
>>> I have a network of Windows, Linux and Mac machines and I use rsync to synchronize various directories between them.
>>> 
>>> I'm trying to figure out why my rsync transfers are so slow (<4 MB/s) only when the remote endpoint is Cygwin rsync over sshd (with both a Linux or Cygwin rsync client).   In all other scenarios, I get the full 100MB/s as expected from gigabit ethernet.  This has been an ongoing problem for me for a couple of years over several Windows and Cygwin versions, and I'd like to try to fix it.
>>> 
>>> If I run rsync --daemon --no-detach under mintty in the foreground on the remote Windows endpoint,  I get the full 100 MB/s transfers, so it seems like it has something to do with rsync.exe running in the background under the cygrunsrv+sshd service (which was installed normally using ssh-host-config).
>>> 
>>> If I do:
>>> 	pv /dev/zero | ssh $WINHOST "cat > /dev/null"
>>> or even
>>> 	pv /dev/urandom | ssh $WINHOST md5sum
>>> I also get the full 100 MB/s transfers, so it doesn't look like sshd itself is being throttled by bandwidth or CPU.
>>> 
>>> The machines have less than 15% CPU utilization while transferring, with each of the 4 cores less than 30%, so it doesn't look to be CPU issue.
>>> In Task Manager, sshd.exe and rsync.exe seem to be running normally using only few percent CPU, and show Power Throttling=Disabled, Priority=Normal.   Setting their Priority to High doesn't seem to change things.
>>> 
>>> Looking in Resource Monitor on the remote endpoint, the network usage is pretty much a flat horizontal line at about 18 Mbps (2.5 MB/s), so it sure looks to me as if rsync is somehow being bandwidth-throttled\b when run in the background under cygsshd.
>>> 
>>> It's almost as if rsync has an implicit --bwlimit override when it is run from cygrunsrv+sshd (I've tried --bwlimit=0 on the client which makes no difference).
>>> 
>>> 
>>> Any ideas?    Not sure where to go from here.
>> In cygwin, just scp is very slow.
>> The transfer speed in my environment is as follows.
>> The tests were done with 100MB of test.dat file.
>> (1-1) From cygwin-PC,
>> [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:.
>> yano@linux-server's password:
>> test.dat                                      100%  100MB   4.0MB/s   00:24
>> [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat .
>> yano@linux-server's password:
>> test.dat                                      100%  100MB   8.0MB/s   00:12
>> (1-2) From linux-server,
>> yano@linux-server:~$ scp yano@cygwin-PC:test.dat .
>> yano@cygwin-PC's password:
>> test.dat                                      100%  100MB   4.0MB/s   00:24
>> yano@linux-server:~$ scp test.dat yano@cygwin-PC:.
>> yano@cygwin-PC's password:
>> test.dat                                      100%  100MB   4.1MB/s   00:24
>> I looked into this problem, and noticed that this is caused
>> by cygwin pipe implementation. Pipe in cygwin is configured
>> with FILE_FLAG_OVERLAPPED.
>> If the pipe is configured without FILE_FLAG_OVERLAPPED,
>> the transfer speed is much improved as follows.
>> (2-1) From cygwin-PC,
>> [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:.
>> yano@linux-server's password:
>> test.dat                                      100%  100MB  85.5MB/s   00:01
>> [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat .
>> yano@linux-server's password:
>> test.dat                                      100%  100MB  69.7MB/s   00:01
>> (2-2) From linux-server,
>> yano@linux-server:~$ scp yano@cygwin-PC:test.dat .
>> yano@cygwin-PC's password:
>> test.dat                                      100%  100MB  80.1MB/s   00:01
>> yano@linux-server:~$ scp test.dat yano@cygwin-PC:.
>> yano@cygwin-PC's password:
>> test.dat                                      100%  100MB  57.7MB/s   00:01
>> I am not sure why this happens and how to fix this.
> 
> 
> 
> -- 
> Problem reports:      https://cygwin.com/problems.html
> FAQ:                  https://cygwin.com/faq/
> Documentation:        https://cygwin.com/docs.html
> Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Chris Roehrig]

It looks like there was a previous (2013) patch and attempt to add a pipe_nooverlap CYGWIN option that was rejected by the maintainers:

https://stackoverflow.com/questions/10385424/good-alternatives-to-cygwin-cygwin-doesnt-support-natively-support-win32-app

Is this something that can be revisited?    It seems to be affecting pure-UNIX situations (in my case) which is Cygwin's core audience.

- Chris


On Wed Aug 25 2021, at 11:18 AM, Chris Roehrig <croehrig@house.org> wrote:

> On Wed Aug 25 2021, at 10:52 AM, Ken Brown via Cygwin <cygwin@cygwin.com> wrote:
>> A couple years ago I had an idea for changing the pipe implementation to avoid overlapped I/O:
>> 
>> https://cygwin.com/pipermail/cygwin-patches/2019q2/009393.html
>> https://cygwin.com/pipermail/cygwin-patches/2019q2/009423.html
>> 
>> I never followed up on it.  But if you think it might help with this problem, I could dust it off and try to finish it.
>> 
>> Ken
> 
> I'm not familiar enough with the innards of rsync, sshd or cygwin to know how this would work.
> Is it possible to have a new CYGWIN environment option to switch the pipe behaviour without requiring changes to the ssh or rsync source code (and without breaking any existing stuff)?
> 
> - Chris
> 
> 
> 
>> On 8/25/2021 7:18 AM, Takashi Yano via Cygwin wrote:
>>> On Tue, 24 Aug 2021 12:49:52 -0700
>>> Chris Roehrig wrote:
>>>> I have a network of Windows, Linux and Mac machines and I use rsync to synchronize various directories between them.
>>>> 
>>>> I'm trying to figure out why my rsync transfers are so slow (<4 MB/s) only when the remote endpoint is Cygwin rsync over sshd (with both a Linux or Cygwin rsync client).   In all other scenarios, I get the full 100MB/s as expected from gigabit ethernet.  This has been an ongoing problem for me for a couple of years over several Windows and Cygwin versions, and I'd like to try to fix it.
>>>> 
>>>> If I run rsync --daemon --no-detach under mintty in the foreground on the remote Windows endpoint,  I get the full 100 MB/s transfers, so it seems like it has something to do with rsync.exe running in the background under the cygrunsrv+sshd service (which was installed normally using ssh-host-config).
>>>> 
>>>> If I do:
>>>> 	pv /dev/zero | ssh $WINHOST "cat > /dev/null"
>>>> or even
>>>> 	pv /dev/urandom | ssh $WINHOST md5sum
>>>> I also get the full 100 MB/s transfers, so it doesn't look like sshd itself is being throttled by bandwidth or CPU.
>>>> 
>>>> The machines have less than 15% CPU utilization while transferring, with each of the 4 cores less than 30%, so it doesn't look to be CPU issue.
>>>> In Task Manager, sshd.exe and rsync.exe seem to be running normally using only few percent CPU, and show Power Throttling=Disabled, Priority=Normal.   Setting their Priority to High doesn't seem to change things.
>>>> 
>>>> Looking in Resource Monitor on the remote endpoint, the network usage is pretty much a flat horizontal line at about 18 Mbps (2.5 MB/s), so it sure looks to me as if rsync is somehow being bandwidth-throttled\b when run in the background under cygsshd.
>>>> 
>>>> It's almost as if rsync has an implicit --bwlimit override when it is run from cygrunsrv+sshd (I've tried --bwlimit=0 on the client which makes no difference).
>>>> 
>>>> 
>>>> Any ideas?    Not sure where to go from here.
>>> In cygwin, just scp is very slow.
>>> The transfer speed in my environment is as follows.
>>> The tests were done with 100MB of test.dat file.
>>> (1-1) From cygwin-PC,
>>> [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:.
>>> yano@linux-server's password:
>>> test.dat                                      100%  100MB   4.0MB/s   00:24
>>> [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat .
>>> yano@linux-server's password:
>>> test.dat                                      100%  100MB   8.0MB/s   00:12
>>> (1-2) From linux-server,
>>> yano@linux-server:~$ scp yano@cygwin-PC:test.dat .
>>> yano@cygwin-PC's password:
>>> test.dat                                      100%  100MB   4.0MB/s   00:24
>>> yano@linux-server:~$ scp test.dat yano@cygwin-PC:.
>>> yano@cygwin-PC's password:
>>> test.dat                                      100%  100MB   4.1MB/s   00:24
>>> I looked into this problem, and noticed that this is caused
>>> by cygwin pipe implementation. Pipe in cygwin is configured
>>> with FILE_FLAG_OVERLAPPED.
>>> If the pipe is configured without FILE_FLAG_OVERLAPPED,
>>> the transfer speed is much improved as follows.
>>> (2-1) From cygwin-PC,
>>> [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:.
>>> yano@linux-server's password:
>>> test.dat                                      100%  100MB  85.5MB/s   00:01
>>> [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat .
>>> yano@linux-server's password:
>>> test.dat                                      100%  100MB  69.7MB/s   00:01
>>> (2-2) From linux-server,
>>> yano@linux-server:~$ scp yano@cygwin-PC:test.dat .
>>> yano@cygwin-PC's password:
>>> test.dat                                      100%  100MB  80.1MB/s   00:01
>>> yano@linux-server:~$ scp test.dat yano@cygwin-PC:.
>>> yano@cygwin-PC's password:
>>> test.dat                                      100%  100MB  57.7MB/s   00:01
>>> I am not sure why this happens and how to fix this.
>> 
>> 
>> 
>> -- 
>> Problem reports:      https://cygwin.com/problems.html
>> FAQ:                  https://cygwin.com/faq/
>> Documentation:        https://cygwin.com/docs.html
>> Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple
> 
> 
> -- 
> Problem reports:      https://cygwin.com/problems.html
> FAQ:                  https://cygwin.com/faq/
> Documentation:        https://cygwin.com/docs.html
> Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Ken Brown]

On 8/25/2021 2:18 PM, Chris Roehrig wrote:
> On Wed Aug 25 2021, at 10:52 AM, Ken Brown via Cygwin <cygwin@cygwin.com> wrote:
>> A couple years ago I had an idea for changing the pipe implementation to avoid overlapped I/O:
>>
>>   https://cygwin.com/pipermail/cygwin-patches/2019q2/009393.html
>>   https://cygwin.com/pipermail/cygwin-patches/2019q2/009423.html
>>
>> I never followed up on it.  But if you think it might help with this problem, I could dust it off and try to finish it.
>>
>> Ken
> 
> I'm not familiar enough with the innards of rsync, sshd or cygwin to know how this would work.
> Is it possible to have a new CYGWIN environment option to switch the pipe behaviour without requiring changes to the ssh or rsync source code (and without breaking any existing stuff)?

My proposed change would be purely internal to Cygwin.  There should be no 
user-visible change in behavior except, perhaps, improved performance.  (And we 
don't know yet whether there would be improved performance.)  So I don't see a 
need for a new CYGWIN option.

Ken

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Mario Emmenlauer]

On 25.08.21 19:52, Ken Brown via Cygwin wrote:
> A couple years ago I had an idea for changing the pipe implementation to avoid overlapped I/O:
> 
>    https://cygwin.com/pipermail/cygwin-patches/2019q2/009393.html
>    https://cygwin.com/pipermail/cygwin-patches/2019q2/009423.html
> 
> I never followed up on it.  But if you think it might help with this problem, I could dust it off and try to finish it.

This sounds quite interesting! Do I assume correctly that these
changes may likely lead to a general improvement also in other
situations if the impact of the pipe is so big for this specific
case? Pipes are most likely used in quite a lot of scenarios?!

All the best,

     Mario Emmenlauer

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Ken Brown]

On 8/25/2021 4:33 PM, Mario Emmenlauer wrote:
> On 25.08.21 19:52, Ken Brown via Cygwin wrote:
>> A couple years ago I had an idea for changing the pipe implementation to avoid 
>> overlapped I/O:
>>
>>    https://cygwin.com/pipermail/cygwin-patches/2019q2/009393.html
>>    https://cygwin.com/pipermail/cygwin-patches/2019q2/009423.html
>>
>> I never followed up on it.  But if you think it might help with this problem, 
>> I could dust it off and try to finish it.
> 
> This sounds quite interesting! Do I assume correctly that these
> changes may likely lead to a general improvement also in other
> situations if the impact of the pipe is so big for this specific
> case? Pipes are most likely used in quite a lot of scenarios?!

I'm not prepared to make any predictions.  We won't really know without testing.

Ken

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Takashi Yano]

On Wed, 25 Aug 2021 13:52:19 -0400
Ken Brown wrote:
> On 8/25/2021 7:18 AM, Takashi Yano via Cygwin wrote:
> > On Tue, 24 Aug 2021 12:49:52 -0700
> > Chris Roehrig wrote:
> >> I have a network of Windows, Linux and Mac machines and I use rsync to synchronize various directories between them.
> >>
> >> I'm trying to figure out why my rsync transfers are so slow (<4 MB/s) only when the remote endpoint is Cygwin rsync over sshd (with both a Linux or Cygwin rsync client).   In all other scenarios, I get the full 100MB/s as expected from gigabit ethernet.  This has been an ongoing problem for me for a couple of years over several Windows and Cygwin versions, and I'd like to try to fix it.
> >>
> >> If I run rsync --daemon --no-detach under mintty in the foreground on the remote Windows endpoint,  I get the full 100 MB/s transfers, so it seems like it has something to do with rsync.exe running in the background under the cygrunsrv+sshd service (which was installed normally using ssh-host-config).
> >>
> >> If I do:
> >> 	pv /dev/zero | ssh $WINHOST "cat > /dev/null"
> >> or even
> >> 	pv /dev/urandom | ssh $WINHOST md5sum
> >> I also get the full 100 MB/s transfers, so it doesn't look like sshd itself is being throttled by bandwidth or CPU.
> >>
> >> The machines have less than 15% CPU utilization while transferring, with each of the 4 cores less than 30%, so it doesn't look to be CPU issue.
> >> In Task Manager, sshd.exe and rsync.exe seem to be running normally using only few percent CPU, and show Power Throttling=Disabled, Priority=Normal.   Setting their Priority to High doesn't seem to change things.
> >>
> >> Looking in Resource Monitor on the remote endpoint, the network usage is pretty much a flat horizontal line at about 18 Mbps (2.5 MB/s), so it sure looks to me as if rsync is somehow being bandwidth-throttled\b when run in the background under cygsshd.
> >>
> >> It's almost as if rsync has an implicit --bwlimit override when it is run from cygrunsrv+sshd (I've tried --bwlimit=0 on the client which makes no difference).
> >>
> >>
> >> Any ideas?    Not sure where to go from here.
> > 
> > In cygwin, just scp is very slow.
> > 
> > The transfer speed in my environment is as follows.
> > The tests were done with 100MB of test.dat file.
> > 
> > (1-1) From cygwin-PC,
> > [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:.
> > yano@linux-server's password:
> > test.dat                                      100%  100MB   4.0MB/s   00:24
> > [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat .
> > yano@linux-server's password:
> > test.dat                                      100%  100MB   8.0MB/s   00:12
> > 
> > (1-2) From linux-server,
> > yano@linux-server:~$ scp yano@cygwin-PC:test.dat .
> > yano@cygwin-PC's password:
> > test.dat                                      100%  100MB   4.0MB/s   00:24
> > yano@linux-server:~$ scp test.dat yano@cygwin-PC:.
> > yano@cygwin-PC's password:
> > test.dat                                      100%  100MB   4.1MB/s   00:24
> > 
> > 
> > I looked into this problem, and noticed that this is caused
> > by cygwin pipe implementation. Pipe in cygwin is configured
> > with FILE_FLAG_OVERLAPPED.
> > 
> > If the pipe is configured without FILE_FLAG_OVERLAPPED,
> > the transfer speed is much improved as follows.
> > 
> > 
> > (2-1) From cygwin-PC,
> > [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:.
> > yano@linux-server's password:
> > test.dat                                      100%  100MB  85.5MB/s   00:01
> > [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat .
> > yano@linux-server's password:
> > test.dat                                      100%  100MB  69.7MB/s   00:01
> > 
> > (2-2) From linux-server,
> > yano@linux-server:~$ scp yano@cygwin-PC:test.dat .
> > yano@cygwin-PC's password:
> > test.dat                                      100%  100MB  80.1MB/s   00:01
> > yano@linux-server:~$ scp test.dat yano@cygwin-PC:.
> > yano@cygwin-PC's password:
> > test.dat                                      100%  100MB  57.7MB/s   00:01
> > 
> > I am not sure why this happens and how to fix this.
> 
> A couple years ago I had an idea for changing the pipe implementation to avoid 
> overlapped I/O:
> 
>    https://cygwin.com/pipermail/cygwin-patches/2019q2/009393.html
>    https://cygwin.com/pipermail/cygwin-patches/2019q2/009423.html
> 
> I never followed up on it.  But if you think it might help with this problem, I 
> could dust it off and try to finish it.

Interesting.

It will be also helpfull for:
https://cygwin.com/pipermail/cygwin/2021-March/247987.html
which seems to be the same issue with
https://stackoverflow.com/questions/10385424/good-alternatives-to-cygwin-cygwin-doesnt-support-natively-support-win32-app

However, I wonder why scp dislikes overlapped I/O.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Ken Brown]

On 8/25/2021 5:29 PM, Takashi Yano wrote:
> On Wed, 25 Aug 2021 13:52:19 -0400
> Ken Brown wrote:
>> On 8/25/2021 7:18 AM, Takashi Yano via Cygwin wrote:
>>> On Tue, 24 Aug 2021 12:49:52 -0700
>>> Chris Roehrig wrote:
>>>> I have a network of Windows, Linux and Mac machines and I use rsync to synchronize various directories between them.
>>>>
>>>> I'm trying to figure out why my rsync transfers are so slow (<4 MB/s) only when the remote endpoint is Cygwin rsync over sshd (with both a Linux or Cygwin rsync client).   In all other scenarios, I get the full 100MB/s as expected from gigabit ethernet.  This has been an ongoing problem for me for a couple of years over several Windows and Cygwin versions, and I'd like to try to fix it.
>>>>
>>>> If I run rsync --daemon --no-detach under mintty in the foreground on the remote Windows endpoint,  I get the full 100 MB/s transfers, so it seems like it has something to do with rsync.exe running in the background under the cygrunsrv+sshd service (which was installed normally using ssh-host-config).
>>>>
>>>> If I do:
>>>> 	pv /dev/zero | ssh $WINHOST "cat > /dev/null"
>>>> or even
>>>> 	pv /dev/urandom | ssh $WINHOST md5sum
>>>> I also get the full 100 MB/s transfers, so it doesn't look like sshd itself is being throttled by bandwidth or CPU.
>>>>
>>>> The machines have less than 15% CPU utilization while transferring, with each of the 4 cores less than 30%, so it doesn't look to be CPU issue.
>>>> In Task Manager, sshd.exe and rsync.exe seem to be running normally using only few percent CPU, and show Power Throttling=Disabled, Priority=Normal.   Setting their Priority to High doesn't seem to change things.
>>>>
>>>> Looking in Resource Monitor on the remote endpoint, the network usage is pretty much a flat horizontal line at about 18 Mbps (2.5 MB/s), so it sure looks to me as if rsync is somehow being bandwidth-throttled\b when run in the background under cygsshd.
>>>>
>>>> It's almost as if rsync has an implicit --bwlimit override when it is run from cygrunsrv+sshd (I've tried --bwlimit=0 on the client which makes no difference).
>>>>
>>>>
>>>> Any ideas?    Not sure where to go from here.
>>>
>>> In cygwin, just scp is very slow.
>>>
>>> The transfer speed in my environment is as follows.
>>> The tests were done with 100MB of test.dat file.
>>>
>>> (1-1) From cygwin-PC,
>>> [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:.
>>> yano@linux-server's password:
>>> test.dat                                      100%  100MB   4.0MB/s   00:24
>>> [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat .
>>> yano@linux-server's password:
>>> test.dat                                      100%  100MB   8.0MB/s   00:12
>>>
>>> (1-2) From linux-server,
>>> yano@linux-server:~$ scp yano@cygwin-PC:test.dat .
>>> yano@cygwin-PC's password:
>>> test.dat                                      100%  100MB   4.0MB/s   00:24
>>> yano@linux-server:~$ scp test.dat yano@cygwin-PC:.
>>> yano@cygwin-PC's password:
>>> test.dat                                      100%  100MB   4.1MB/s   00:24
>>>
>>>
>>> I looked into this problem, and noticed that this is caused
>>> by cygwin pipe implementation. Pipe in cygwin is configured
>>> with FILE_FLAG_OVERLAPPED.
>>>
>>> If the pipe is configured without FILE_FLAG_OVERLAPPED,
>>> the transfer speed is much improved as follows.
>>>
>>>
>>> (2-1) From cygwin-PC,
>>> [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:.
>>> yano@linux-server's password:
>>> test.dat                                      100%  100MB  85.5MB/s   00:01
>>> [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat .
>>> yano@linux-server's password:
>>> test.dat                                      100%  100MB  69.7MB/s   00:01
>>>
>>> (2-2) From linux-server,
>>> yano@linux-server:~$ scp yano@cygwin-PC:test.dat .
>>> yano@cygwin-PC's password:
>>> test.dat                                      100%  100MB  80.1MB/s   00:01
>>> yano@linux-server:~$ scp test.dat yano@cygwin-PC:.
>>> yano@cygwin-PC's password:
>>> test.dat                                      100%  100MB  57.7MB/s   00:01
>>>
>>> I am not sure why this happens and how to fix this.
>>
>> A couple years ago I had an idea for changing the pipe implementation to avoid
>> overlapped I/O:
>>
>>     https://cygwin.com/pipermail/cygwin-patches/2019q2/009393.html
>>     https://cygwin.com/pipermail/cygwin-patches/2019q2/009423.html
>>
>> I never followed up on it.  But if you think it might help with this problem, I
>> could dust it off and try to finish it.
> 
> Interesting.
> 
> It will be also helpfull for:
> https://cygwin.com/pipermail/cygwin/2021-March/247987.html
> which seems to be the same issue with
> https://stackoverflow.com/questions/10385424/good-alternatives-to-cygwin-cygwin-doesnt-support-natively-support-win32-app
> 
> However, I wonder why scp dislikes overlapped I/O.

I agree that it would be good to understand this.  When I first proposed the 
change, I was thinking in terms of code simplification.  If it also improves 
performance (which we don't know yet), it becomes a higher priority, but in that 
case it would be nice to understand why it improves performance.

Ken

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Ken Brown]

On 8/26/2021 11:56 AM, Ken Brown via Cygwin wrote:
> On 8/25/2021 5:29 PM, Takashi Yano wrote:
>> On Wed, 25 Aug 2021 13:52:19 -0400
>> Ken Brown wrote:
>>> On 8/25/2021 7:18 AM, Takashi Yano via Cygwin wrote:
>>>> On Tue, 24 Aug 2021 12:49:52 -0700
>>>> Chris Roehrig wrote:
>>>>> I have a network of Windows, Linux and Mac machines and I use rsync to 
>>>>> synchronize various directories between them.
>>>>>
>>>>> I'm trying to figure out why my rsync transfers are so slow (<4 MB/s) only 
>>>>> when the remote endpoint is Cygwin rsync over sshd (with both a Linux or 
>>>>> Cygwin rsync client).   In all other scenarios, I get the full 100MB/s as 
>>>>> expected from gigabit ethernet.  This has been an ongoing problem for me 
>>>>> for a couple of years over several Windows and Cygwin versions, and I'd 
>>>>> like to try to fix it.
>>>>>
>>>>> If I run rsync --daemon --no-detach under mintty in the foreground on the 
>>>>> remote Windows endpoint,  I get the full 100 MB/s transfers, so it seems 
>>>>> like it has something to do with rsync.exe running in the background under 
>>>>> the cygrunsrv+sshd service (which was installed normally using 
>>>>> ssh-host-config).
>>>>>
>>>>> If I do:
>>>>>     pv /dev/zero | ssh $WINHOST "cat > /dev/null"
>>>>> or even
>>>>>     pv /dev/urandom | ssh $WINHOST md5sum
>>>>> I also get the full 100 MB/s transfers, so it doesn't look like sshd itself 
>>>>> is being throttled by bandwidth or CPU.
>>>>>
>>>>> The machines have less than 15% CPU utilization while transferring, with 
>>>>> each of the 4 cores less than 30%, so it doesn't look to be CPU issue.
>>>>> In Task Manager, sshd.exe and rsync.exe seem to be running normally using 
>>>>> only few percent CPU, and show Power Throttling=Disabled, 
>>>>> Priority=Normal.   Setting their Priority to High doesn't seem to change 
>>>>> things.
>>>>>
>>>>> Looking in Resource Monitor on the remote endpoint, the network usage is 
>>>>> pretty much a flat horizontal line at about 18 Mbps (2.5 MB/s), so it sure 
>>>>> looks to me as if rsync is somehow being bandwidth-throttled\b when run in 
>>>>> the background under cygsshd.
>>>>>
>>>>> It's almost as if rsync has an implicit --bwlimit override when it is run 
>>>>> from cygrunsrv+sshd (I've tried --bwlimit=0 on the client which makes no 
>>>>> difference).
>>>>>
>>>>>
>>>>> Any ideas?    Not sure where to go from here.
>>>>
>>>> In cygwin, just scp is very slow.
>>>>
>>>> The transfer speed in my environment is as follows.
>>>> The tests were done with 100MB of test.dat file.
>>>>
>>>> (1-1) From cygwin-PC,
>>>> [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:.
>>>> yano@linux-server's password:
>>>> test.dat                                      100%  100MB   4.0MB/s   00:24
>>>> [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat .
>>>> yano@linux-server's password:
>>>> test.dat                                      100%  100MB   8.0MB/s   00:12
>>>>
>>>> (1-2) From linux-server,
>>>> yano@linux-server:~$ scp yano@cygwin-PC:test.dat .
>>>> yano@cygwin-PC's password:
>>>> test.dat                                      100%  100MB   4.0MB/s   00:24
>>>> yano@linux-server:~$ scp test.dat yano@cygwin-PC:.
>>>> yano@cygwin-PC's password:
>>>> test.dat                                      100%  100MB   4.1MB/s   00:24
>>>>
>>>>
>>>> I looked into this problem, and noticed that this is caused
>>>> by cygwin pipe implementation. Pipe in cygwin is configured
>>>> with FILE_FLAG_OVERLAPPED.
>>>>
>>>> If the pipe is configured without FILE_FLAG_OVERLAPPED,
>>>> the transfer speed is much improved as follows.
>>>>
>>>>
>>>> (2-1) From cygwin-PC,
>>>> [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:.
>>>> yano@linux-server's password:
>>>> test.dat                                      100%  100MB  85.5MB/s   00:01
>>>> [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat .
>>>> yano@linux-server's password:
>>>> test.dat                                      100%  100MB  69.7MB/s   00:01
>>>>
>>>> (2-2) From linux-server,
>>>> yano@linux-server:~$ scp yano@cygwin-PC:test.dat .
>>>> yano@cygwin-PC's password:
>>>> test.dat                                      100%  100MB  80.1MB/s   00:01
>>>> yano@linux-server:~$ scp test.dat yano@cygwin-PC:.
>>>> yano@cygwin-PC's password:
>>>> test.dat                                      100%  100MB  57.7MB/s   00:01
>>>>
>>>> I am not sure why this happens and how to fix this.
>>>
>>> A couple years ago I had an idea for changing the pipe implementation to avoid
>>> overlapped I/O:
>>>
>>>     https://cygwin.com/pipermail/cygwin-patches/2019q2/009393.html
>>>     https://cygwin.com/pipermail/cygwin-patches/2019q2/009423.html
>>>
>>> I never followed up on it.  But if you think it might help with this problem, I
>>> could dust it off and try to finish it.
>>
>> Interesting.
>>
>> It will be also helpfull for:
>> https://cygwin.com/pipermail/cygwin/2021-March/247987.html
>> which seems to be the same issue with
>> https://stackoverflow.com/questions/10385424/good-alternatives-to-cygwin-cygwin-doesnt-support-natively-support-win32-app 
>>
>>
>> However, I wonder why scp dislikes overlapped I/O.
> 
> I agree that it would be good to understand this.  When I first proposed the 
> change, I was thinking in terms of code simplification.  If it also improves 
> performance (which we don't know yet), it becomes a higher priority, but in that 
> case it would be nice to understand why it improves performance.

Hi Takashi,

In case you want to try out my proposed change, I've just rebased the patches to 
the current master and pushed them to a new topic/pipe branch.

Ken

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Takashi Yano]

[-- Attachment #1: Type: text/plain, Size: 8082 bytes --]

On Thu, 26 Aug 2021 18:18:29 -0400
Ken Brown wrote:
> On 8/26/2021 11:56 AM, Ken Brown via Cygwin wrote:
> > On 8/25/2021 5:29 PM, Takashi Yano wrote:
> >> On Wed, 25 Aug 2021 13:52:19 -0400
> >> Ken Brown wrote:
> >>> On 8/25/2021 7:18 AM, Takashi Yano via Cygwin wrote:
> >>>> On Tue, 24 Aug 2021 12:49:52 -0700
> >>>> Chris Roehrig wrote:
> >>>>> I have a network of Windows, Linux and Mac machines and I use rsync to 
> >>>>> synchronize various directories between them.
> >>>>>
> >>>>> I'm trying to figure out why my rsync transfers are so slow (<4 MB/s) only 
> >>>>> when the remote endpoint is Cygwin rsync over sshd (with both a Linux or 
> >>>>> Cygwin rsync client).   In all other scenarios, I get the full 100MB/s as 
> >>>>> expected from gigabit ethernet.  This has been an ongoing problem for me 
> >>>>> for a couple of years over several Windows and Cygwin versions, and I'd 
> >>>>> like to try to fix it.
> >>>>>
> >>>>> If I run rsync --daemon --no-detach under mintty in the foreground on the 
> >>>>> remote Windows endpoint,  I get the full 100 MB/s transfers, so it seems 
> >>>>> like it has something to do with rsync.exe running in the background under 
> >>>>> the cygrunsrv+sshd service (which was installed normally using 
> >>>>> ssh-host-config).
> >>>>>
> >>>>> If I do:
> >>>>>     pv /dev/zero | ssh $WINHOST "cat > /dev/null"
> >>>>> or even
> >>>>>     pv /dev/urandom | ssh $WINHOST md5sum
> >>>>> I also get the full 100 MB/s transfers, so it doesn't look like sshd itself 
> >>>>> is being throttled by bandwidth or CPU.
> >>>>>
> >>>>> The machines have less than 15% CPU utilization while transferring, with 
> >>>>> each of the 4 cores less than 30%, so it doesn't look to be CPU issue.
> >>>>> In Task Manager, sshd.exe and rsync.exe seem to be running normally using 
> >>>>> only few percent CPU, and show Power Throttling=Disabled, 
> >>>>> Priority=Normal.   Setting their Priority to High doesn't seem to change 
> >>>>> things.
> >>>>>
> >>>>> Looking in Resource Monitor on the remote endpoint, the network usage is 
> >>>>> pretty much a flat horizontal line at about 18 Mbps (2.5 MB/s), so it sure 
> >>>>> looks to me as if rsync is somehow being bandwidth-throttled\b when run in 
> >>>>> the background under cygsshd.
> >>>>>
> >>>>> It's almost as if rsync has an implicit --bwlimit override when it is run 
> >>>>> from cygrunsrv+sshd (I've tried --bwlimit=0 on the client which makes no 
> >>>>> difference).
> >>>>>
> >>>>>
> >>>>> Any ideas?    Not sure where to go from here.
> >>>>
> >>>> In cygwin, just scp is very slow.
> >>>>
> >>>> The transfer speed in my environment is as follows.
> >>>> The tests were done with 100MB of test.dat file.
> >>>>
> >>>> (1-1) From cygwin-PC,
> >>>> [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:.
> >>>> yano@linux-server's password:
> >>>> test.dat                                      100%  100MB   4.0MB/s   00:24
> >>>> [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat .
> >>>> yano@linux-server's password:
> >>>> test.dat                                      100%  100MB   8.0MB/s   00:12
> >>>>
> >>>> (1-2) From linux-server,
> >>>> yano@linux-server:~$ scp yano@cygwin-PC:test.dat .
> >>>> yano@cygwin-PC's password:
> >>>> test.dat                                      100%  100MB   4.0MB/s   00:24
> >>>> yano@linux-server:~$ scp test.dat yano@cygwin-PC:.
> >>>> yano@cygwin-PC's password:
> >>>> test.dat                                      100%  100MB   4.1MB/s   00:24
> >>>>
> >>>>
> >>>> I looked into this problem, and noticed that this is caused
> >>>> by cygwin pipe implementation. Pipe in cygwin is configured
> >>>> with FILE_FLAG_OVERLAPPED.
> >>>>
> >>>> If the pipe is configured without FILE_FLAG_OVERLAPPED,
> >>>> the transfer speed is much improved as follows.
> >>>>
> >>>>
> >>>> (2-1) From cygwin-PC,
> >>>> [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:.
> >>>> yano@linux-server's password:
> >>>> test.dat                                      100%  100MB  85.5MB/s   00:01
> >>>> [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat .
> >>>> yano@linux-server's password:
> >>>> test.dat                                      100%  100MB  69.7MB/s   00:01
> >>>>
> >>>> (2-2) From linux-server,
> >>>> yano@linux-server:~$ scp yano@cygwin-PC:test.dat .
> >>>> yano@cygwin-PC's password:
> >>>> test.dat                                      100%  100MB  80.1MB/s   00:01
> >>>> yano@linux-server:~$ scp test.dat yano@cygwin-PC:.
> >>>> yano@cygwin-PC's password:
> >>>> test.dat                                      100%  100MB  57.7MB/s   00:01
> >>>>
> >>>> I am not sure why this happens and how to fix this.
> >>>
> >>> A couple years ago I had an idea for changing the pipe implementation to avoid
> >>> overlapped I/O:
> >>>
> >>>     https://cygwin.com/pipermail/cygwin-patches/2019q2/009393.html
> >>>     https://cygwin.com/pipermail/cygwin-patches/2019q2/009423.html
> >>>
> >>> I never followed up on it.  But if you think it might help with this problem, I
> >>> could dust it off and try to finish it.
> >>
> >> Interesting.
> >>
> >> It will be also helpfull for:
> >> https://cygwin.com/pipermail/cygwin/2021-March/247987.html
> >> which seems to be the same issue with
> >> https://stackoverflow.com/questions/10385424/good-alternatives-to-cygwin-cygwin-doesnt-support-natively-support-win32-app 
> >>
> >>
> >> However, I wonder why scp dislikes overlapped I/O.
> > 
> > I agree that it would be good to understand this.  When I first proposed the 
> > change, I was thinking in terms of code simplification.  If it also improves 
> > performance (which we don't know yet), it becomes a higher priority, but in that 
> > case it would be nice to understand why it improves performance.
> 
> Hi Takashi,
> 
> In case you want to try out my proposed change, I've just rebased the patches to 
> the current master and pushed them to a new topic/pipe branch.

Hi Ken,

Thanks much! I tested topic/pipe branch.

[yano@cygwin-PC ~]$ scp test.dat yano@linux-server:.
yano@linux-server's password:
test.dat                                      100%  100MB  95.9MB/s   00:01
[yano@cygwin-PC ~]$ scp yano@linux-server:test.dat .
yano@linux-server's password:
test.dat                                      100%  100MB   8.0MB/s   00:12

yano@linux-server:~$ scp yano@cygwin-PC:test.dat .
yano@cygwin-PC's password:
test.dat                                      100%  100MB 109.7MB/s   00:00
yano@linux-server:~$ scp test.dat yano@cygwin-PC:.
yano@cygwin-PC's password:
test.dat                                      100%  100MB  31.4MB/s   00:03

As shown above, outgoing transfer-rate has been improved upto near
theoretical limit. However, incoming transfer-rate is not improved
much.

I digged further and found the first patch attached solves the issue
as follows.

[yano@cygwin-PC ~]$ scp yano@linux-server:test.dat .
yano@linux-server's password:
test.dat                                      100%  100MB 112.8MB/s   00:00

yano@linux-server2:~$ scp test.dat yano@cygwin-PC:.
yano@cygwin-PC's password:
test.dat                                      100%  100MB 102.5MB/s   00:00


I also tested the case:
> >> https://cygwin.com/pipermail/cygwin/2021-March/247987.html
> >> which seems to be the same issue with
> >> https://stackoverflow.com/questions/10385424/good-alternatives-to-cygwin-cygwin-doesnt-support-natively-support-win32-app

Unfortunately, topic/pipe does not help.

I confirmed that applying the second patch attached, which reverts
to create() rather than nt_create(), and setting CYGWIN=pipe_byte
fixes the problem.

What do you think of this alternative implementation which does
not use nt_create()?

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

[-- Attachment #2: 0001-Cygwin-select-Improve-select-poll-response.patch --]
[-- Type: application/octet-stream, Size: 2460 bytes --]

From c2ececea731fb633a9e10666d7546e1c61c8c4f7 Mon Sep 17 00:00:00 2001
From: Takashi Yano <takashi.yano@nifty.ne.jp>
Date: Fri, 27 Aug 2021 19:54:41 +0900
Subject: [PATCH 1/2] Cygwin: select: Improve select/poll response.

---
 winsup/cygwin/select.cc | 32 ++++++++++++++++++++++++++++----
 1 file changed, 28 insertions(+), 4 deletions(-)

diff --git a/winsup/cygwin/select.cc b/winsup/cygwin/select.cc
index 8ad982c12..83e1c00e0 100644
--- a/winsup/cygwin/select.cc
+++ b/winsup/cygwin/select.cc
@@ -735,6 +735,7 @@ thread_pipe (void *arg)
   select_pipe_info *pi = (select_pipe_info *) arg;
   DWORD sleep_time = 0;
   bool looping = true;
+  DWORD t0 = GetTickCount ();
 
   while (looping)
     {
@@ -754,7 +755,12 @@ thread_pipe (void *arg)
 	break;
       cygwait (pi->bye, sleep_time >> 3);
       if (sleep_time < 80)
-	++sleep_time;
+	{
+	  DWORD t1 = GetTickCount ();
+	  if (t0 != t1)
+	    ++sleep_time;
+	  t0 = t1;
+	}
       if (pi->stop_thread)
 	break;
     }
@@ -930,6 +936,7 @@ thread_fifo (void *arg)
   select_fifo_info *pi = (select_fifo_info *) arg;
   DWORD sleep_time = 0;
   bool looping = true;
+  DWORD t0 = GetTickCount ();
 
   while (looping)
     {
@@ -949,7 +956,12 @@ thread_fifo (void *arg)
 	break;
       cygwait (pi->bye, sleep_time >> 3);
       if (sleep_time < 80)
-	++sleep_time;
+	{
+	  DWORD t1 = GetTickCount ();
+	  if (t0 != t1)
+	    ++sleep_time;
+	  t0 = t1;
+	}
       if (pi->stop_thread)
 	break;
     }
@@ -1125,6 +1137,7 @@ thread_console (void *arg)
   select_console_info *ci = (select_console_info *) arg;
   DWORD sleep_time = 0;
   bool looping = true;
+  DWORD t0 = GetTickCount ();
 
   while (looping)
     {
@@ -1144,7 +1157,12 @@ thread_console (void *arg)
 	break;
       cygwait (ci->bye, sleep_time >> 3);
       if (sleep_time < 80)
-	++sleep_time;
+	{
+	  DWORD t1 = GetTickCount ();
+	  if (t0 != t1)
+	    ++sleep_time;
+	  t0 = t1;
+	}
       if (ci->stop_thread)
 	break;
     }
@@ -1364,6 +1382,7 @@ thread_pty_slave (void *arg)
   select_pipe_info *pi = (select_pipe_info *) arg;
   DWORD sleep_time = 0;
   bool looping = true;
+  DWORD t0 = GetTickCount ();
 
   while (looping)
     {
@@ -1383,7 +1402,12 @@ thread_pty_slave (void *arg)
 	break;
       cygwait (pi->bye, sleep_time >> 3);
       if (sleep_time < 80)
-	++sleep_time;
+	{
+	  DWORD t1 = GetTickCount ();
+	  if (t0 != t1)
+	    ++sleep_time;
+	  t0 = t1;
+	}
       if (pi->stop_thread)
 	break;
     }
-- 
2.33.0


[-- Attachment #3: 0002-Cygwin-pipe-Revert-to-create-rather-than-nt_create.patch --]
[-- Type: application/octet-stream, Size: 2368 bytes --]

From 784892dc272a6b7899de5f89c65bb8cfab79591d Mon Sep 17 00:00:00 2001
From: Takashi Yano <takashi.yano@nifty.ne.jp>
Date: Fri, 27 Aug 2021 20:06:04 +0900
Subject: [PATCH 2/2] Cygwin: pipe: Revert to create() rather than nt_create().

---
 winsup/cygwin/fhandler_pipe.cc | 24 +++++++++++++++++++++++-
 1 file changed, 23 insertions(+), 1 deletion(-)

diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index 2dec0a848..b3af4a0a0 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -234,6 +234,24 @@ fhandler_pipe::raw_read (void *ptr, size_t& len)
       return;
     }
 
+  if (is_nonblocking ())
+    {
+      DWORD n;
+      BOOL ret = PeekNamedPipe (get_handle (), NULL, 0, NULL, &n, NULL);
+      if (!ret)
+	{
+	  __seterrno ();
+	  len = (size_t) -1;
+	  return;
+	}
+      if (n == 0)
+	{
+	  set_errno (EAGAIN);
+	  len = (size_t) -1;
+	  return;
+	}
+    }
+
   do
     {
       len = orig_len;
@@ -571,6 +589,7 @@ fhandler_pipe::create (LPSECURITY_ATTRIBUTES sa_ptr, PHANDLE r, PHANDLE w,
   return 0;
 }
 
+#if 0
 /* The next version of fhandler_pipe::create used to call the previous
    version.  But the read handle created by the latter doesn't have
    FILE_WRITE_ATTRIBUTES access unless the pipe is created with
@@ -588,6 +607,7 @@ fhandler_pipe::create (LPSECURITY_ATTRIBUTES sa_ptr, PHANDLE r, PHANDLE w,
 
 static int nt_create (LPSECURITY_ATTRIBUTES, PHANDLE, PHANDLE, DWORD,
 		      int64_t *);
+#endif
 
 int
 fhandler_pipe::create (fhandler_pipe *fhs[2], unsigned psize, int mode)
@@ -597,7 +617,7 @@ fhandler_pipe::create (fhandler_pipe *fhs[2], unsigned psize, int mode)
   int res = -1;
   int64_t unique_id;
 
-  int ret = nt_create (sa, &r, &w, psize, &unique_id);
+  int ret = create (sa, &r, &w, psize, NULL, 0, &unique_id);
   if (ret)
     __seterrno_from_win_error (ret);
   else if ((fhs[0] = (fhandler_pipe *) build_fh_dev (*piper_dev)) == NULL)
@@ -624,6 +644,7 @@ fhandler_pipe::create (fhandler_pipe *fhs[2], unsigned psize, int mode)
   return res;
 }
 
+#if 0
 static int
 nt_create (LPSECURITY_ATTRIBUTES sa_ptr, PHANDLE r, PHANDLE w,
 		DWORD psize, int64_t *unique_id)
@@ -755,6 +776,7 @@ nt_create (LPSECURITY_ATTRIBUTES sa_ptr, PHANDLE r, PHANDLE w,
   /* Success. */
   return 0;
 }
+#endif
 
 int
 fhandler_pipe::ioctl (unsigned int cmd, void *p)
-- 
2.33.0

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Ken Brown]

On 8/27/2021 7:24 AM, Takashi Yano wrote:
> On Thu, 26 Aug 2021 18:18:29 -0400
> Ken Brown wrote:
>> On 8/26/2021 11:56 AM, Ken Brown via Cygwin wrote:
[...]
>> In case you want to try out my proposed change, I've just rebased the patches to
>> the current master and pushed them to a new topic/pipe branch.
> 
> Hi Ken,
> 
> Thanks much! I tested topic/pipe branch.
> 
> [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:.
> yano@linux-server's password:
> test.dat                                      100%  100MB  95.9MB/s   00:01
> [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat .
> yano@linux-server's password:
> test.dat                                      100%  100MB   8.0MB/s   00:12
> 
> yano@linux-server:~$ scp yano@cygwin-PC:test.dat .
> yano@cygwin-PC's password:
> test.dat                                      100%  100MB 109.7MB/s   00:00
> yano@linux-server:~$ scp test.dat yano@cygwin-PC:.
> yano@cygwin-PC's password:
> test.dat                                      100%  100MB  31.4MB/s   00:03
> 
> As shown above, outgoing transfer-rate has been improved upto near
> theoretical limit. However, incoming transfer-rate is not improved
> much.
> 
> I digged further and found the first patch attached solves the issue
> as follows.
> 
> [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat .
> yano@linux-server's password:
> test.dat                                      100%  100MB 112.8MB/s   00:00
> 
> yano@linux-server2:~$ scp test.dat yano@cygwin-PC:.
> yano@cygwin-PC's password:
> test.dat                                      100%  100MB 102.5MB/s   00:00

Great!

> I also tested the case:
>>>> https://cygwin.com/pipermail/cygwin/2021-March/247987.html
>>>> which seems to be the same issue with
>>>> https://stackoverflow.com/questions/10385424/good-alternatives-to-cygwin-cygwin-doesnt-support-natively-support-win32-app
> 
> Unfortunately, topic/pipe does not help.
> 
> I confirmed that applying the second patch attached, which reverts
> to create() rather than nt_create(), and setting CYGWIN=pipe_byte
> fixes the problem.
> 
> What do you think of this alternative implementation which does
> not use nt_create()?

Two years ago I thought I needed nt_create to avoid problems when calling 
set_pipe_non_blocking.  Are you saying that's not an issue?  Is 
set_pipe_non_blocking unnecessary?  Is that the point of your modification to 
raw_read?

Ken

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Takashi Yano]

On Fri, 27 Aug 2021 12:00:50 -0400
Ken Brown wrote:

> On 8/27/2021 7:24 AM, Takashi Yano wrote:
> > On Thu, 26 Aug 2021 18:18:29 -0400
> > Ken Brown wrote:
> >> On 8/26/2021 11:56 AM, Ken Brown via Cygwin wrote:
> [...]
> >> In case you want to try out my proposed change, I've just rebased the patches to
> >> the current master and pushed them to a new topic/pipe branch.
> > 
> > Hi Ken,
> > 
> > Thanks much! I tested topic/pipe branch.
> > 
> > [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:.
> > yano@linux-server's password:
> > test.dat                                      100%  100MB  95.9MB/s   00:01
> > [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat .
> > yano@linux-server's password:
> > test.dat                                      100%  100MB   8.0MB/s   00:12
> > 
> > yano@linux-server:~$ scp yano@cygwin-PC:test.dat .
> > yano@cygwin-PC's password:
> > test.dat                                      100%  100MB 109.7MB/s   00:00
> > yano@linux-server:~$ scp test.dat yano@cygwin-PC:.
> > yano@cygwin-PC's password:
> > test.dat                                      100%  100MB  31.4MB/s   00:03
> > 
> > As shown above, outgoing transfer-rate has been improved upto near
> > theoretical limit. However, incoming transfer-rate is not improved
> > much.
> > 
> > I digged further and found the first patch attached solves the issue
> > as follows.
> > 
> > [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat .
> > yano@linux-server's password:
> > test.dat                                      100%  100MB 112.8MB/s   00:00
> > 
> > yano@linux-server2:~$ scp test.dat yano@cygwin-PC:.
> > yano@cygwin-PC's password:
> > test.dat                                      100%  100MB 102.5MB/s   00:00
> 
> Great!
> 
> > I also tested the case:
> >>>> https://cygwin.com/pipermail/cygwin/2021-March/247987.html
> >>>> which seems to be the same issue with
> >>>> https://stackoverflow.com/questions/10385424/good-alternatives-to-cygwin-cygwin-doesnt-support-natively-support-win32-app
> > 
> > Unfortunately, topic/pipe does not help.
> > 
> > I confirmed that applying the second patch attached, which reverts
> > to create() rather than nt_create(), and setting CYGWIN=pipe_byte
> > fixes the problem.
> > 
> > What do you think of this alternative implementation which does
> > not use nt_create()?
> 
> Two years ago I thought I needed nt_create to avoid problems when calling 
> set_pipe_non_blocking.  Are you saying that's not an issue?  Is 
> set_pipe_non_blocking unnecessary?  Is that the point of your modification to 
> raw_read?

Yes. Instead of making windows read function itself non-blocking,
it is possible to check if the pipe can be read before read using
PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is
returned.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Takashi Yano]

On Sat, 28 Aug 2021 02:21:11 +0900
Takashi Yano wrote:

> On Fri, 27 Aug 2021 12:00:50 -0400
> Ken Brown wrote:
> 
> > On 8/27/2021 7:24 AM, Takashi Yano wrote:
> > > On Thu, 26 Aug 2021 18:18:29 -0400
> > > Ken Brown wrote:
> > >> On 8/26/2021 11:56 AM, Ken Brown via Cygwin wrote:
> > [...]
> > >> In case you want to try out my proposed change, I've just rebased the patches to
> > >> the current master and pushed them to a new topic/pipe branch.
> > > 
> > > Hi Ken,
> > > 
> > > Thanks much! I tested topic/pipe branch.
> > > 
> > > [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:.
> > > yano@linux-server's password:
> > > test.dat                                      100%  100MB  95.9MB/s   00:01
> > > [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat .
> > > yano@linux-server's password:
> > > test.dat                                      100%  100MB   8.0MB/s   00:12
> > > 
> > > yano@linux-server:~$ scp yano@cygwin-PC:test.dat .
> > > yano@cygwin-PC's password:
> > > test.dat                                      100%  100MB 109.7MB/s   00:00
> > > yano@linux-server:~$ scp test.dat yano@cygwin-PC:.
> > > yano@cygwin-PC's password:
> > > test.dat                                      100%  100MB  31.4MB/s   00:03
> > > 
> > > As shown above, outgoing transfer-rate has been improved upto near
> > > theoretical limit. However, incoming transfer-rate is not improved
> > > much.
> > > 
> > > I digged further and found the first patch attached solves the issue
> > > as follows.
> > > 
> > > [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat .
> > > yano@linux-server's password:
> > > test.dat                                      100%  100MB 112.8MB/s   00:00
> > > 
> > > yano@linux-server2:~$ scp test.dat yano@cygwin-PC:.
> > > yano@cygwin-PC's password:
> > > test.dat                                      100%  100MB 102.5MB/s   00:00
> > 
> > Great!
> > 
> > > I also tested the case:
> > >>>> https://cygwin.com/pipermail/cygwin/2021-March/247987.html
> > >>>> which seems to be the same issue with
> > >>>> https://stackoverflow.com/questions/10385424/good-alternatives-to-cygwin-cygwin-doesnt-support-natively-support-win32-app
> > > 
> > > Unfortunately, topic/pipe does not help.
> > > 
> > > I confirmed that applying the second patch attached, which reverts
> > > to create() rather than nt_create(), and setting CYGWIN=pipe_byte
> > > fixes the problem.
> > > 
> > > What do you think of this alternative implementation which does
> > > not use nt_create()?
> > 
> > Two years ago I thought I needed nt_create to avoid problems when calling 
> > set_pipe_non_blocking.  Are you saying that's not an issue?  Is 
> > set_pipe_non_blocking unnecessary?  Is that the point of your modification to 
> > raw_read?
> 
> Yes. Instead of making windows read function itself non-blocking,
> it is possible to check if the pipe can be read before read using
> PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is
> returned.

As for writint to pipe, set_pipe_non_blocking seems to take effect
and be necessary.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Takashi Yano]

On Sat, 28 Aug 2021 11:00:24 +0900
Takashi Yano wrote:
> On Sat, 28 Aug 2021 02:21:11 +0900
> Takashi Yano wrote:
> 
> > On Fri, 27 Aug 2021 12:00:50 -0400
> > Ken Brown wrote:
> > 
> > > On 8/27/2021 7:24 AM, Takashi Yano wrote:
> > > > On Thu, 26 Aug 2021 18:18:29 -0400
> > > > Ken Brown wrote:
> > > >> On 8/26/2021 11:56 AM, Ken Brown via Cygwin wrote:
> > > [...]
> > > >> In case you want to try out my proposed change, I've just rebased the patches to
> > > >> the current master and pushed them to a new topic/pipe branch.
> > > > 
> > > > Hi Ken,
> > > > 
> > > > Thanks much! I tested topic/pipe branch.
> > > > 
> > > > [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:.
> > > > yano@linux-server's password:
> > > > test.dat                                      100%  100MB  95.9MB/s   00:01
> > > > [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat .
> > > > yano@linux-server's password:
> > > > test.dat                                      100%  100MB   8.0MB/s   00:12
> > > > 
> > > > yano@linux-server:~$ scp yano@cygwin-PC:test.dat .
> > > > yano@cygwin-PC's password:
> > > > test.dat                                      100%  100MB 109.7MB/s   00:00
> > > > yano@linux-server:~$ scp test.dat yano@cygwin-PC:.
> > > > yano@cygwin-PC's password:
> > > > test.dat                                      100%  100MB  31.4MB/s   00:03
> > > > 
> > > > As shown above, outgoing transfer-rate has been improved upto near
> > > > theoretical limit. However, incoming transfer-rate is not improved
> > > > much.
> > > > 
> > > > I digged further and found the first patch attached solves the issue
> > > > as follows.
> > > > 
> > > > [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat .
> > > > yano@linux-server's password:
> > > > test.dat                                      100%  100MB 112.8MB/s   00:00
> > > > 
> > > > yano@linux-server2:~$ scp test.dat yano@cygwin-PC:.
> > > > yano@cygwin-PC's password:
> > > > test.dat                                      100%  100MB 102.5MB/s   00:00
> > > 
> > > Great!
> > > 
> > > > I also tested the case:
> > > >>>> https://cygwin.com/pipermail/cygwin/2021-March/247987.html
> > > >>>> which seems to be the same issue with
> > > >>>> https://stackoverflow.com/questions/10385424/good-alternatives-to-cygwin-cygwin-doesnt-support-natively-support-win32-app
> > > > 
> > > > Unfortunately, topic/pipe does not help.
> > > > 
> > > > I confirmed that applying the second patch attached, which reverts
> > > > to create() rather than nt_create(), and setting CYGWIN=pipe_byte
> > > > fixes the problem.
> > > > 
> > > > What do you think of this alternative implementation which does
> > > > not use nt_create()?
> > > 
> > > Two years ago I thought I needed nt_create to avoid problems when calling 
> > > set_pipe_non_blocking.  Are you saying that's not an issue?  Is 
> > > set_pipe_non_blocking unnecessary?  Is that the point of your modification to 
> > > raw_read?
> > 
> > Yes. Instead of making windows read function itself non-blocking,
> > it is possible to check if the pipe can be read before read using
> > PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is
> > returned.
> 
> As for writint to pipe, set_pipe_non_blocking seems to take effect
> and be necessary.

So, the following patch seems to be appropriate.

diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index b3af4a0a0..82f5de3f4 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -48,6 +48,8 @@ fhandler_pipe::set_pipe_non_blocking (bool nonblocking)
   IO_STATUS_BLOCK io;
   FILE_PIPE_INFORMATION fpi;

+  if ((openflags & O_ACCMODE) == O_RDONLY)
+    return; /* Do nothing for read pipe */
   fpi.ReadMode = FILE_PIPE_BYTE_STREAM_MODE;
   fpi.CompletionMode = nonblocking ? FILE_PIPE_COMPLETE_OPERATION
     : FILE_PIPE_QUEUE_OPERATION;

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Corinna Vinschen]

On Aug 28 02:21, Takashi Yano via Cygwin wrote:
> On Fri, 27 Aug 2021 12:00:50 -0400
> Ken Brown wrote:
> > Two years ago I thought I needed nt_create to avoid problems when calling 
> > set_pipe_non_blocking.  Are you saying that's not an issue?  Is 
> > set_pipe_non_blocking unnecessary?  Is that the point of your modification to 
> > raw_read?
> 
> Yes. Instead of making windows read function itself non-blocking,
> it is possible to check if the pipe can be read before read using
> PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is
> returned.

The problem is this:

  if (PeekNamedPipe())
    ReadFile(blocking);

is not atomic.  I. e., if PeekNamedPipe succeeds, nothing keeps another
thread from draining the pipe between the PeekNamedPipe and the ReadFile
call.  And as soon as ReadFile runs, it hangs indefinitely and we can't
stop it via a signal.

Is a blocking ReadFile actually faster than a non-blocking read?  Or
does it mainly depend on BYTE vs. MESSAGE mode?

What if the pipe is created non-blocking and stays non-blocking all the
time and uses BYTE mode all the time?  Just as sockets, it would always
only emulate blocking mode.  Wouldn't that drop code size a lot and fix
most problems?


Corinna

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Takashi Yano]

On Sat, 28 Aug 2021 10:43:27 +0200
Corinna Vinschen wrote:
> On Aug 28 02:21, Takashi Yano via Cygwin wrote:
> > On Fri, 27 Aug 2021 12:00:50 -0400
> > Ken Brown wrote:
> > > Two years ago I thought I needed nt_create to avoid problems when calling 
> > > set_pipe_non_blocking.  Are you saying that's not an issue?  Is 
> > > set_pipe_non_blocking unnecessary?  Is that the point of your modification to 
> > > raw_read?
> > 
> > Yes. Instead of making windows read function itself non-blocking,
> > it is possible to check if the pipe can be read before read using
> > PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is
> > returned.
> 
> The problem is this:
> 
>   if (PeekNamedPipe())
>     ReadFile(blocking);
> 
> is not atomic.  I. e., if PeekNamedPipe succeeds, nothing keeps another
> thread from draining the pipe between the PeekNamedPipe and the ReadFile
> call.  And as soon as ReadFile runs, it hangs indefinitely and we can't
> stop it via a signal.

Hmm, you are right. Mutex guard seems to be necessary like pty code
if we go this way.

> Is a blocking ReadFile actually faster than a non-blocking read?  Or
> does it mainly depend on BYTE vs. MESSAGE mode?

Actually, I don't think so. Perhaps it is not essential problem of
overlapped I/O but something is wrong with current pipe code.

> What if the pipe is created non-blocking and stays non-blocking all the
> time and uses BYTE mode all the time?  Just as sockets, it would always
> only emulate blocking mode.  Wouldn't that drop code size a lot and fix
> most problems?

If 'non-blocking' means overlapped I/O, only the problem will be:
https://cygwin.com/pipermail/cygwin/2021-March/247987.html


-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Corinna Vinschen]

On Aug 28 18:41, Takashi Yano via Cygwin wrote:
> On Sat, 28 Aug 2021 10:43:27 +0200
> Corinna Vinschen wrote:
> > On Aug 28 02:21, Takashi Yano via Cygwin wrote:
> > > On Fri, 27 Aug 2021 12:00:50 -0400
> > > Ken Brown wrote:
> > > > Two years ago I thought I needed nt_create to avoid problems when calling 
> > > > set_pipe_non_blocking.  Are you saying that's not an issue?  Is 
> > > > set_pipe_non_blocking unnecessary?  Is that the point of your modification to 
> > > > raw_read?
> > > 
> > > Yes. Instead of making windows read function itself non-blocking,
> > > it is possible to check if the pipe can be read before read using
> > > PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is
> > > returned.
> > 
> > The problem is this:
> > 
> >   if (PeekNamedPipe())
> >     ReadFile(blocking);
> > 
> > is not atomic.  I. e., if PeekNamedPipe succeeds, nothing keeps another
> > thread from draining the pipe between the PeekNamedPipe and the ReadFile
> > call.  And as soon as ReadFile runs, it hangs indefinitely and we can't
> > stop it via a signal.
> 
> Hmm, you are right. Mutex guard seems to be necessary like pty code
> if we go this way.
> 
> > Is a blocking ReadFile actually faster than a non-blocking read?  Or
> > does it mainly depend on BYTE vs. MESSAGE mode?
> 
> Actually, I don't think so. Perhaps it is not essential problem of
> overlapped I/O but something is wrong with current pipe code.
> 
> > What if the pipe is created non-blocking and stays non-blocking all the
> > time and uses BYTE mode all the time?  Just as sockets, it would always
> > only emulate blocking mode.  Wouldn't that drop code size a lot and fix
> > most problems?
> 
> If 'non-blocking' means overlapped I/O, only the problem will be:
> https://cygwin.com/pipermail/cygwin/2021-March/247987.html

Sorry if that wasn't clear, but I was not talking about overlapped I/O,
which we should get rid off, but of real non-blocking mode, which
Windows pipes are fortunately capable of.


Corinna

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Takashi Yano]

On Sat, 28 Aug 2021 13:58:08 +0200
Corinna Vinschen wrote:
> On Aug 28 18:41, Takashi Yano via Cygwin wrote:
> > On Sat, 28 Aug 2021 10:43:27 +0200
> > Corinna Vinschen wrote:
> > > On Aug 28 02:21, Takashi Yano via Cygwin wrote:
> > > > On Fri, 27 Aug 2021 12:00:50 -0400
> > > > Ken Brown wrote:
> > > > > Two years ago I thought I needed nt_create to avoid problems when calling 
> > > > > set_pipe_non_blocking.  Are you saying that's not an issue?  Is 
> > > > > set_pipe_non_blocking unnecessary?  Is that the point of your modification to 
> > > > > raw_read?
> > > > 
> > > > Yes. Instead of making windows read function itself non-blocking,
> > > > it is possible to check if the pipe can be read before read using
> > > > PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is
> > > > returned.
> > > 
> > > The problem is this:
> > > 
> > >   if (PeekNamedPipe())
> > >     ReadFile(blocking);
> > > 
> > > is not atomic.  I. e., if PeekNamedPipe succeeds, nothing keeps another
> > > thread from draining the pipe between the PeekNamedPipe and the ReadFile
> > > call.  And as soon as ReadFile runs, it hangs indefinitely and we can't
> > > stop it via a signal.
> > 
> > Hmm, you are right. Mutex guard seems to be necessary like pty code
> > if we go this way.
> > 
> > > Is a blocking ReadFile actually faster than a non-blocking read?  Or
> > > does it mainly depend on BYTE vs. MESSAGE mode?
> > 
> > Actually, I don't think so. Perhaps it is not essential problem of
> > overlapped I/O but something is wrong with current pipe code.
> > 
> > > What if the pipe is created non-blocking and stays non-blocking all the
> > > time and uses BYTE mode all the time?  Just as sockets, it would always
> > > only emulate blocking mode.  Wouldn't that drop code size a lot and fix
> > > most problems?
> > 
> > If 'non-blocking' means overlapped I/O, only the problem will be:
> > https://cygwin.com/pipermail/cygwin/2021-March/247987.html
> 
> Sorry if that wasn't clear, but I was not talking about overlapped I/O,
> which we should get rid off, but of real non-blocking mode, which
> Windows pipes are fortunately capable of.

Do you mean, PIPE_NOWAIT flag? If this flags is specified in
the read pipe, non-cygwin apps cannot read the pipe correctly.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Ken Brown]

On 8/28/2021 11:43 AM, Takashi Yano via Cygwin wrote:
> On Sat, 28 Aug 2021 13:58:08 +0200
> Corinna Vinschen wrote:
>> On Aug 28 18:41, Takashi Yano via Cygwin wrote:
>>> On Sat, 28 Aug 2021 10:43:27 +0200
>>> Corinna Vinschen wrote:
>>>> On Aug 28 02:21, Takashi Yano via Cygwin wrote:
>>>>> On Fri, 27 Aug 2021 12:00:50 -0400
>>>>> Ken Brown wrote:
>>>>>> Two years ago I thought I needed nt_create to avoid problems when calling
>>>>>> set_pipe_non_blocking.  Are you saying that's not an issue?  Is
>>>>>> set_pipe_non_blocking unnecessary?  Is that the point of your modification to
>>>>>> raw_read?
>>>>>
>>>>> Yes. Instead of making windows read function itself non-blocking,
>>>>> it is possible to check if the pipe can be read before read using
>>>>> PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is
>>>>> returned.
>>>>
>>>> The problem is this:
>>>>
>>>>    if (PeekNamedPipe())
>>>>      ReadFile(blocking);
>>>>
>>>> is not atomic.  I. e., if PeekNamedPipe succeeds, nothing keeps another
>>>> thread from draining the pipe between the PeekNamedPipe and the ReadFile
>>>> call.  And as soon as ReadFile runs, it hangs indefinitely and we can't
>>>> stop it via a signal.
>>>
>>> Hmm, you are right. Mutex guard seems to be necessary like pty code
>>> if we go this way.
>>>
>>>> Is a blocking ReadFile actually faster than a non-blocking read?  Or
>>>> does it mainly depend on BYTE vs. MESSAGE mode?
>>>
>>> Actually, I don't think so. Perhaps it is not essential problem of
>>> overlapped I/O but something is wrong with current pipe code.
>>>
>>>> What if the pipe is created non-blocking and stays non-blocking all the
>>>> time and uses BYTE mode all the time?  Just as sockets, it would always
>>>> only emulate blocking mode.  Wouldn't that drop code size a lot and fix
>>>> most problems?
>>>
>>> If 'non-blocking' means overlapped I/O, only the problem will be:
>>> https://cygwin.com/pipermail/cygwin/2021-March/247987.html
>>
>> Sorry if that wasn't clear, but I was not talking about overlapped I/O,
>> which we should get rid off, but of real non-blocking mode, which
>> Windows pipes are fortunately capable of.
> 
> Do you mean, PIPE_NOWAIT flag? If this flags is specified in
> the read pipe, non-cygwin apps cannot read the pipe correctly.

While waiting for Corinna's response to this, I have one more question.  Do you 
understand why nt_create() failed and you had to revert to create()?  Was it an 
access problem because nt_create requested FILE_WRITE_ATTRIBUTES?  Or did I make 
some careless mistake in writing nt_create?

Ken

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Takashi Yano]

Hi Ken,

On Sat, 28 Aug 2021 16:55:52 -0400
Ken Brown wrote:
> On 8/28/2021 11:43 AM, Takashi Yano via Cygwin wrote:
> > On Sat, 28 Aug 2021 13:58:08 +0200
> > Corinna Vinschen wrote:
> >> On Aug 28 18:41, Takashi Yano via Cygwin wrote:
> >>> On Sat, 28 Aug 2021 10:43:27 +0200
> >>> Corinna Vinschen wrote:
> >>>> On Aug 28 02:21, Takashi Yano via Cygwin wrote:
> >>>>> On Fri, 27 Aug 2021 12:00:50 -0400
> >>>>> Ken Brown wrote:
> >>>>>> Two years ago I thought I needed nt_create to avoid problems when calling
> >>>>>> set_pipe_non_blocking.  Are you saying that's not an issue?  Is
> >>>>>> set_pipe_non_blocking unnecessary?  Is that the point of your modification to
> >>>>>> raw_read?
> >>>>>
> >>>>> Yes. Instead of making windows read function itself non-blocking,
> >>>>> it is possible to check if the pipe can be read before read using
> >>>>> PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is
> >>>>> returned.
> >>>>
> >>>> The problem is this:
> >>>>
> >>>>    if (PeekNamedPipe())
> >>>>      ReadFile(blocking);
> >>>>
> >>>> is not atomic.  I. e., if PeekNamedPipe succeeds, nothing keeps another
> >>>> thread from draining the pipe between the PeekNamedPipe and the ReadFile
> >>>> call.  And as soon as ReadFile runs, it hangs indefinitely and we can't
> >>>> stop it via a signal.
> >>>
> >>> Hmm, you are right. Mutex guard seems to be necessary like pty code
> >>> if we go this way.
> >>>
> >>>> Is a blocking ReadFile actually faster than a non-blocking read?  Or
> >>>> does it mainly depend on BYTE vs. MESSAGE mode?
> >>>
> >>> Actually, I don't think so. Perhaps it is not essential problem of
> >>> overlapped I/O but something is wrong with current pipe code.
> >>>
> >>>> What if the pipe is created non-blocking and stays non-blocking all the
> >>>> time and uses BYTE mode all the time?  Just as sockets, it would always
> >>>> only emulate blocking mode.  Wouldn't that drop code size a lot and fix
> >>>> most problems?
> >>>
> >>> If 'non-blocking' means overlapped I/O, only the problem will be:
> >>> https://cygwin.com/pipermail/cygwin/2021-March/247987.html
> >>
> >> Sorry if that wasn't clear, but I was not talking about overlapped I/O,
> >> which we should get rid off, but of real non-blocking mode, which
> >> Windows pipes are fortunately capable of.
> > 
> > Do you mean, PIPE_NOWAIT flag? If this flags is specified in
> > the read pipe, non-cygwin apps cannot read the pipe correctly.
> 
> While waiting for Corinna's response to this, I have one more question.  Do you 
> understand why nt_create() failed and you had to revert to create()?  Was it an 
> access problem because nt_create requested FILE_WRITE_ATTRIBUTES?  Or did I make 
> some careless mistake in writing nt_create?

I am sorry but no. I don't understand why piping C# program via
the pipe created by nt_create() has the issue. I tried to change
setup parameters in nt_create(), however, I did not succeed it to
work. I also couldn't find any mistake in nt_create() so far.

Win32 programs which use ReadFile() and WriteFile() work even
with the pipe created by nt_create() as well as overlapped I/O.

What does C# program differ from legacy win32 program at all?

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Ken Brown]

On 8/29/2021 4:41 AM, Takashi Yano wrote:
> Hi Ken,
> 
> On Sat, 28 Aug 2021 16:55:52 -0400
> Ken Brown wrote:
>> On 8/28/2021 11:43 AM, Takashi Yano via Cygwin wrote:
>>> On Sat, 28 Aug 2021 13:58:08 +0200
>>> Corinna Vinschen wrote:
>>>> On Aug 28 18:41, Takashi Yano via Cygwin wrote:
>>>>> On Sat, 28 Aug 2021 10:43:27 +0200
>>>>> Corinna Vinschen wrote:
>>>>>> On Aug 28 02:21, Takashi Yano via Cygwin wrote:
>>>>>>> On Fri, 27 Aug 2021 12:00:50 -0400
>>>>>>> Ken Brown wrote:
>>>>>>>> Two years ago I thought I needed nt_create to avoid problems when calling
>>>>>>>> set_pipe_non_blocking.  Are you saying that's not an issue?  Is
>>>>>>>> set_pipe_non_blocking unnecessary?  Is that the point of your modification to
>>>>>>>> raw_read?
>>>>>>>
>>>>>>> Yes. Instead of making windows read function itself non-blocking,
>>>>>>> it is possible to check if the pipe can be read before read using
>>>>>>> PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is
>>>>>>> returned.
>>>>>>
>>>>>> The problem is this:
>>>>>>
>>>>>>     if (PeekNamedPipe())
>>>>>>       ReadFile(blocking);
>>>>>>
>>>>>> is not atomic.  I. e., if PeekNamedPipe succeeds, nothing keeps another
>>>>>> thread from draining the pipe between the PeekNamedPipe and the ReadFile
>>>>>> call.  And as soon as ReadFile runs, it hangs indefinitely and we can't
>>>>>> stop it via a signal.
>>>>>
>>>>> Hmm, you are right. Mutex guard seems to be necessary like pty code
>>>>> if we go this way.
>>>>>
>>>>>> Is a blocking ReadFile actually faster than a non-blocking read?  Or
>>>>>> does it mainly depend on BYTE vs. MESSAGE mode?
>>>>>
>>>>> Actually, I don't think so. Perhaps it is not essential problem of
>>>>> overlapped I/O but something is wrong with current pipe code.
>>>>>
>>>>>> What if the pipe is created non-blocking and stays non-blocking all the
>>>>>> time and uses BYTE mode all the time?  Just as sockets, it would always
>>>>>> only emulate blocking mode.  Wouldn't that drop code size a lot and fix
>>>>>> most problems?
>>>>>
>>>>> If 'non-blocking' means overlapped I/O, only the problem will be:
>>>>> https://cygwin.com/pipermail/cygwin/2021-March/247987.html
>>>>
>>>> Sorry if that wasn't clear, but I was not talking about overlapped I/O,
>>>> which we should get rid off, but of real non-blocking mode, which
>>>> Windows pipes are fortunately capable of.
>>>
>>> Do you mean, PIPE_NOWAIT flag? If this flags is specified in
>>> the read pipe, non-cygwin apps cannot read the pipe correctly.
>>
>> While waiting for Corinna's response to this, I have one more question.  Do you
>> understand why nt_create() failed and you had to revert to create()?  Was it an
>> access problem because nt_create requested FILE_WRITE_ATTRIBUTES?  Or did I make
>> some careless mistake in writing nt_create?
> 
> I am sorry but no. I don't understand why piping C# program via
> the pipe created by nt_create() has the issue. I tried to change
> setup parameters in nt_create(), however, I did not succeed it to
> work. I also couldn't find any mistake in nt_create() so far.
> 
> Win32 programs which use ReadFile() and WriteFile() work even
> with the pipe created by nt_create() as well as overlapped I/O.
> 
> What does C# program differ from legacy win32 program at all?

I don't know.

By the way, when I introduced nt_create(), my preference would have been to 
simply change create() to use the NT API, but I was afraid to do that because I 
didn't want to take a chance on breaking something.  That's still my preference, 
if we can find a way to work around this problem with C# programs.

Ken

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Corinna Vinschen]

On Aug 29 00:43, Takashi Yano via Cygwin wrote:
> On Sat, 28 Aug 2021 13:58:08 +0200
> Corinna Vinschen wrote:
> > On Aug 28 18:41, Takashi Yano via Cygwin wrote:
> > > On Sat, 28 Aug 2021 10:43:27 +0200
> > > Corinna Vinschen wrote:
> > > > On Aug 28 02:21, Takashi Yano via Cygwin wrote:
> > > > > On Fri, 27 Aug 2021 12:00:50 -0400
> > > > > Ken Brown wrote:
> > > > > > Two years ago I thought I needed nt_create to avoid problems when calling 
> > > > > > set_pipe_non_blocking.  Are you saying that's not an issue?  Is 
> > > > > > set_pipe_non_blocking unnecessary?  Is that the point of your modification to 
> > > > > > raw_read?
> > > > > 
> > > > > Yes. Instead of making windows read function itself non-blocking,
> > > > > it is possible to check if the pipe can be read before read using
> > > > > PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is
> > > > > returned.
> > > > 
> > > > The problem is this:
> > > > 
> > > >   if (PeekNamedPipe())
> > > >     ReadFile(blocking);
> > > > 
> > > > is not atomic.  I. e., if PeekNamedPipe succeeds, nothing keeps another
> > > > thread from draining the pipe between the PeekNamedPipe and the ReadFile
> > > > call.  And as soon as ReadFile runs, it hangs indefinitely and we can't
> > > > stop it via a signal.
> > > 
> > > Hmm, you are right. Mutex guard seems to be necessary like pty code
> > > if we go this way.
> > > 
> > > > Is a blocking ReadFile actually faster than a non-blocking read?  Or
> > > > does it mainly depend on BYTE vs. MESSAGE mode?
> > > 
> > > Actually, I don't think so. Perhaps it is not essential problem of
> > > overlapped I/O but something is wrong with current pipe code.
> > > 
> > > > What if the pipe is created non-blocking and stays non-blocking all the
> > > > time and uses BYTE mode all the time?  Just as sockets, it would always
> > > > only emulate blocking mode.  Wouldn't that drop code size a lot and fix
> > > > most problems?
> > > 
> > > If 'non-blocking' means overlapped I/O, only the problem will be:
> > > https://cygwin.com/pipermail/cygwin/2021-March/247987.html
> > 
> > Sorry if that wasn't clear, but I was not talking about overlapped I/O,
> > which we should get rid off, but of real non-blocking mode, which
> > Windows pipes are fortunately capable of.
> 
> Do you mean, PIPE_NOWAIT flag? If this flags is specified in
> the read pipe, non-cygwin apps cannot read the pipe correctly.

Do you mean after execve?  If the child is a non-Cygwin process, we
could run a loop over the stdio descriptors in child_info_spawn::worker
setting pipes to blocking.

But that's a corner problem, easily solved.  It's more important that it
works nicely for Cygwin processes.


Corinna

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Takashi Yano]

[-- Attachment #1: Type: text/plain, Size: 1555 bytes --]

On Sat, 28 Aug 2021 18:41:02 +0900
Takashi Yano wrote:
> On Sat, 28 Aug 2021 10:43:27 +0200
> Corinna Vinschen wrote:
> > On Aug 28 02:21, Takashi Yano via Cygwin wrote:
> > > On Fri, 27 Aug 2021 12:00:50 -0400
> > > Ken Brown wrote:
> > > > Two years ago I thought I needed nt_create to avoid problems when calling 
> > > > set_pipe_non_blocking.  Are you saying that's not an issue?  Is 
> > > > set_pipe_non_blocking unnecessary?  Is that the point of your modification to 
> > > > raw_read?
> > > 
> > > Yes. Instead of making windows read function itself non-blocking,
> > > it is possible to check if the pipe can be read before read using
> > > PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is
> > > returned.
> > 
> > The problem is this:
> > 
> >   if (PeekNamedPipe())
> >     ReadFile(blocking);
> > 
> > is not atomic.  I. e., if PeekNamedPipe succeeds, nothing keeps another
> > thread from draining the pipe between the PeekNamedPipe and the ReadFile
> > call.  And as soon as ReadFile runs, it hangs indefinitely and we can't
> > stop it via a signal.
> 
> Hmm, you are right. Mutex guard seems to be necessary like pty code
> if we go this way.

I have found that set_pipe_non_blocking() succeeds for both read and
write pipes if the write pipe is created by CreateNamedPipe() and the
read pipe is created by CreateFile() contrary to the current create()
code. Therefore, not only nt_create() but also PeekNamedPipe() become
unnecessary.

Please see the revised patch attached.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

[-- Attachment #2: v2-0002-Cygwin-pipe-Revert-to-create-rather-than-nt_creat.patch --]
[-- Type: application/octet-stream, Size: 8881 bytes --]

From 3f4ed10f86c88e1194a5f11201a8e088ce53d127 Mon Sep 17 00:00:00 2001
From: Takashi Yano <takashi.yano@nifty.ne.jp>
Date: Sun, 29 Aug 2021 17:23:57 +0900
Subject: [PATCH v2 2/2] Cygwin: pipe: Revert to create() rather than
 nt_create().

---
 winsup/cygwin/fhandler_pipe.cc | 192 ++++-----------------------------
 1 file changed, 21 insertions(+), 171 deletions(-)

diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc
index 2dec0a848..a6ece17a6 100644
--- a/winsup/cygwin/fhandler_pipe.cc
+++ b/winsup/cygwin/fhandler_pipe.cc
@@ -463,13 +463,13 @@ fhandler_pipe::create (LPSECURITY_ATTRIBUTES sa_ptr, PHANDLE r, PHANDLE w,
   if (name)
     len += __small_sprintf (pipename + len, "%s", name);
 
-  open_mode |= PIPE_ACCESS_INBOUND | FILE_FLAG_FIRST_PIPE_INSTANCE;
+  open_mode |= PIPE_ACCESS_OUTBOUND | FILE_FLAG_FIRST_PIPE_INSTANCE;
 
   /* Retry CreateNamedPipe as long as the pipe name is in use.
      Retrying will probably never be necessary, but we want
      to be as robust as possible.  */
   DWORD err = 0;
-  while (r && !*r)
+  while (w && !*w)
     {
       static volatile ULONG pipe_unique_id;
       if (!name)
@@ -498,12 +498,12 @@ fhandler_pipe::create (LPSECURITY_ATTRIBUTES sa_ptr, PHANDLE r, PHANDLE w,
 	 definitely required for pty handling since fhandler_pty_master
 	 writes to the pipe in chunks, terminated by newline when CANON mode
 	 is specified.  */
-      *r = CreateNamedPipe (pipename, open_mode, pipe_mode, 1, psize,
+      *w = CreateNamedPipe (pipename, open_mode, pipe_mode, 1, psize,
 			   psize, NMPWAIT_USE_DEFAULT_WAIT, sa_ptr);
 
-      if (*r != INVALID_HANDLE_VALUE)
+      if (*w != INVALID_HANDLE_VALUE)
 	{
-	  debug_printf ("pipe read handle %p", *r);
+	  debug_printf ("pipe write handle %p", *w);
 	  err = 0;
 	  break;
 	}
@@ -516,14 +516,14 @@ fhandler_pipe::create (LPSECURITY_ATTRIBUTES sa_ptr, PHANDLE r, PHANDLE w,
 	     Pick a new name and retry.  */
 	  debug_printf ("pipe busy", !name ? ", retrying" : "");
 	  if (!name)
-	    *r = NULL;
+	    *w = NULL;
 	  break;
 	case ERROR_ACCESS_DENIED:
 	  /* The pipe is already open with incompatible parameters.
 	     Pick a new name and retry.  */
 	  debug_printf ("pipe access denied%s", !name ? ", retrying" : "");
 	  if (!name)
-	    *r = NULL;
+	    *w = NULL;
 	  break;
 	default:
 	  {
@@ -535,60 +535,42 @@ fhandler_pipe::create (LPSECURITY_ATTRIBUTES sa_ptr, PHANDLE r, PHANDLE w,
 
   if (err)
     {
-      *r = NULL;
+      *w = NULL;
       return err;
     }
 
-  if (!w)
-    debug_printf ("pipe write handle NULL");
+  if (!r)
+    debug_printf ("pipe read handle NULL");
   else
     {
       debug_printf ("CreateFile: name %s", pipename);
 
-      /* Open the named pipe for writing.
-	 Be sure to permit FILE_READ_ATTRIBUTES access.  */
-      DWORD access = GENERIC_WRITE | FILE_READ_ATTRIBUTES;
+      /* Open the named pipe for reading.
+	 Be sure to permit FILE_WRITE_ATTRIBUTES access.  */
+      DWORD access = GENERIC_READ | FILE_WRITE_ATTRIBUTES;
       if ((open_mode & PIPE_ACCESS_DUPLEX) == PIPE_ACCESS_DUPLEX)
-	access |= GENERIC_READ | FILE_WRITE_ATTRIBUTES;
-      *w = CreateFile (pipename, access, 0, sa_ptr, OPEN_EXISTING,
+	access |= GENERIC_WRITE | FILE_READ_ATTRIBUTES;
+      *r = CreateFile (pipename, access, 0, sa_ptr, OPEN_EXISTING,
 		      open_mode & FILE_FLAG_OVERLAPPED, 0);
 
-      if (!*w || *w == INVALID_HANDLE_VALUE)
+      if (!*r || *r == INVALID_HANDLE_VALUE)
 	{
 	  /* Failure. */
 	  DWORD err = GetLastError ();
 	  debug_printf ("CreateFile failed, r %p, %E", r);
-	  if (r)
-	    CloseHandle (*r);
-	  *w = NULL;
+	  if (w)
+	    CloseHandle (*w);
+	  *r = NULL;
 	  return err;
 	}
 
-      debug_printf ("pipe write handle %p", *w);
+      debug_printf ("pipe read handle %p", *r);
     }
 
   /* Success. */
   return 0;
 }
 
-/* The next version of fhandler_pipe::create used to call the previous
-   version.  But the read handle created by the latter doesn't have
-   FILE_WRITE_ATTRIBUTES access unless the pipe is created with
-   PIPE_ACCESS_DUPLEX, and it doesn't seem possible to add that access
-   right.  This causes set_pipe_non_blocking to fail.
-
-   To fix this we will define a helper function 'nt_create' that is
-   similar to the above fhandler_pipe::create but uses
-   NtCreateNamedPipeFile instead of CreateNamedPipe; this gives more
-   flexibility in setting the access rights.  We will use this helper
-   function in the version of fhandler_pipe::create below, which
-   suffices for all of our uses of set_pipe_non_blocking.  For
-   simplicity, nt_create will omit the 'open_mode' and 'name'
-   parameters, which aren't needed for our purposes.  */
-
-static int nt_create (LPSECURITY_ATTRIBUTES, PHANDLE, PHANDLE, DWORD,
-		      int64_t *);
-
 int
 fhandler_pipe::create (fhandler_pipe *fhs[2], unsigned psize, int mode)
 {
@@ -597,7 +579,7 @@ fhandler_pipe::create (fhandler_pipe *fhs[2], unsigned psize, int mode)
   int res = -1;
   int64_t unique_id;
 
-  int ret = nt_create (sa, &r, &w, psize, &unique_id);
+  int ret = create (sa, &r, &w, psize, NULL, 0, &unique_id);
   if (ret)
     __seterrno_from_win_error (ret);
   else if ((fhs[0] = (fhandler_pipe *) build_fh_dev (*piper_dev)) == NULL)
@@ -624,138 +606,6 @@ fhandler_pipe::create (fhandler_pipe *fhs[2], unsigned psize, int mode)
   return res;
 }
 
-static int
-nt_create (LPSECURITY_ATTRIBUTES sa_ptr, PHANDLE r, PHANDLE w,
-		DWORD psize, int64_t *unique_id)
-{
-  NTSTATUS status;
-  HANDLE npfsh;
-  ACCESS_MASK access;
-  OBJECT_ATTRIBUTES attr;
-  IO_STATUS_BLOCK io;
-  LARGE_INTEGER timeout;
-
-  /* Default to error. */
-  if (r)
-    *r = NULL;
-  if (w)
-    *w = NULL;
-
-  status = fhandler_base::npfs_handle (npfsh);
-  if (!NT_SUCCESS (status))
-    {
-      __seterrno_from_nt_status (status);
-      return GetLastError ();
-    }
-
-  /* Ensure that there is enough pipe buffer space for atomic writes.  */
-  if (!psize)
-    psize = DEFAULT_PIPEBUFSIZE;
-
-  UNICODE_STRING pipename;
-  WCHAR pipename_buf[MAX_PATH];
-  size_t len = __small_swprintf (pipename_buf, L"%S-%u-",
-				 &cygheap->installation_key,
-				 GetCurrentProcessId ());
-
-  access = GENERIC_READ | FILE_WRITE_ATTRIBUTES;
-
-  ULONG pipe_type = pipe_byte ? FILE_PIPE_BYTE_STREAM_TYPE
-    : FILE_PIPE_MESSAGE_TYPE;
-
-  /* Retry NtCreateNamedPipeFile as long as the pipe name is in use.
-     Retrying will probably never be necessary, but we want
-     to be as robust as possible.  */
-  DWORD err = 0;
-  while (r && !*r)
-    {
-      static volatile ULONG pipe_unique_id;
-      LONG id = InterlockedIncrement ((LONG *) &pipe_unique_id);
-      __small_swprintf (pipename_buf + len, L"pipe-nt-%p", id);
-      if (unique_id)
-	*unique_id = ((int64_t) id << 32 | GetCurrentProcessId ());
-
-      debug_printf ("name %W, size %u, mode %s", pipename_buf, psize,
-		    (pipe_type & FILE_PIPE_MESSAGE_TYPE)
-		    ? "PIPE_TYPE_MESSAGE" : "PIPE_TYPE_BYTE");
-
-      RtlInitUnicodeString (&pipename, pipename_buf);
-
-      InitializeObjectAttributes (&attr, &pipename,
-				  sa_ptr->bInheritHandle ? OBJ_INHERIT : 0,
-				  npfsh, sa_ptr->lpSecurityDescriptor);
-
-      timeout.QuadPart = -500000;
-      status = NtCreateNamedPipeFile (r, access, &attr, &io,
-				      FILE_SHARE_READ | FILE_SHARE_WRITE,
-				      FILE_CREATE, 0, pipe_type,
-				      FILE_PIPE_BYTE_STREAM_MODE,
-				      0, 1, psize, psize, &timeout);
-
-      if (NT_SUCCESS (status))
-	{
-	  debug_printf ("pipe read handle %p", *r);
-	  err = 0;
-	  break;
-	}
-
-      switch (status)
-	{
-	case STATUS_PIPE_BUSY:
-	case STATUS_INSTANCE_NOT_AVAILABLE:
-	case STATUS_PIPE_NOT_AVAILABLE:
-	  /* The pipe is already open with compatible parameters.
-	     Pick a new name and retry.  */
-	  debug_printf ("pipe busy, retrying");
-	  *r = NULL;
-	  break;
-	case STATUS_ACCESS_DENIED:
-	  /* The pipe is already open with incompatible parameters.
-	     Pick a new name and retry.  */
-	  debug_printf ("pipe access denied, retrying");
-	  *r = NULL;
-	  break;
-	default:
-	  {
-	    __seterrno_from_nt_status (status);
-	    err = GetLastError ();
-	    debug_printf ("failed, %E");
-	    *r = INVALID_HANDLE_VALUE;
-	  }
-	}
-    }
-
-  if (err)
-    {
-      *r = NULL;
-      return err;
-    }
-
-  if (!w)
-    debug_printf ("pipe write handle NULL");
-  else
-    {
-      debug_printf ("NtOpenFile: name %S", &pipename);
-
-      access = GENERIC_WRITE | FILE_READ_ATTRIBUTES;
-      status = NtOpenFile (w, access, &attr, &io, 0, 0);
-      if (!NT_SUCCESS (status))
-	{
-	  DWORD err = GetLastError ();
-	  debug_printf ("NtOpenFile failed, r %p, %E", r);
-	  if (r)
-	    CloseHandle (*r);
-	  *w = NULL;
-	  return err;
-	}
-
-      debug_printf ("pipe write handle %p", *w);
-    }
-
-  /* Success. */
-  return 0;
-}
-
 int
 fhandler_pipe::ioctl (unsigned int cmd, void *p)
 {
-- 
2.33.0

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Ken Brown]

On 8/29/2021 5:07 AM, Takashi Yano via Cygwin wrote:
> On Sat, 28 Aug 2021 18:41:02 +0900
> Takashi Yano wrote:
>> On Sat, 28 Aug 2021 10:43:27 +0200
>> Corinna Vinschen wrote:
>>> On Aug 28 02:21, Takashi Yano via Cygwin wrote:
>>>> On Fri, 27 Aug 2021 12:00:50 -0400
>>>> Ken Brown wrote:
>>>>> Two years ago I thought I needed nt_create to avoid problems when calling
>>>>> set_pipe_non_blocking.  Are you saying that's not an issue?  Is
>>>>> set_pipe_non_blocking unnecessary?  Is that the point of your modification to
>>>>> raw_read?
>>>>
>>>> Yes. Instead of making windows read function itself non-blocking,
>>>> it is possible to check if the pipe can be read before read using
>>>> PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is
>>>> returned.
>>>
>>> The problem is this:
>>>
>>>    if (PeekNamedPipe())
>>>      ReadFile(blocking);
>>>
>>> is not atomic.  I. e., if PeekNamedPipe succeeds, nothing keeps another
>>> thread from draining the pipe between the PeekNamedPipe and the ReadFile
>>> call.  And as soon as ReadFile runs, it hangs indefinitely and we can't
>>> stop it via a signal.
>>
>> Hmm, you are right. Mutex guard seems to be necessary like pty code
>> if we go this way.
> 
> I have found that set_pipe_non_blocking() succeeds for both read and
> write pipes if the write pipe is created by CreateNamedPipe() and the
> read pipe is created by CreateFile() contrary to the current create()
> code. Therefore, not only nt_create() but also PeekNamedPipe() become
> unnecessary.
> 
> Please see the revised patch attached.

That's a great idea.

I've applied your two patches to the topic/pipe branch.  I also rebased it and 
did a forced push in order to bring in Corinna's loader script fix.  So you'll 
have to do 'git fetch' and 'git rebase --hard origin/topic/pipe'.

Does this now fix all known problems with pipes?

Corinna, do you still see any benefit to switching to PIPE_NOWAIT?  AFAICT, it 
wouldn't decrease the code size at this point, so the only question is whether 
it might improve performance.

If you think it's worth trying, I'd be glad to code it up on a new branch, and 
we could compare the two.

Aside from that, I'm wondering how and when to merge the new pipe implementation 
to master.  It obviously needs much more widespread testing than it's gotten so 
far.  I'm a little nervous about it because I haven't thought about the details 
for two years, and no one other than me has tested it until a few days ago.

Ken

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Chris Roehrig]

I'd be happy to test it out if you can give me some commands to git it and build it to replace my existing stock Cygwin installation.
(Or it is just the cygwin.dll I'd need to replace?)

My daily backup scripts use a lot of pipes and named pipes.

-- Chris



On Sun Aug 29 2021, at 8:57 AM, Ken Brown via Cygwin <cygwin@cygwin.com> wrote:
> 
> Aside from that, I'm wondering how and when to merge the new pipe implementation to master.  It obviously needs much more widespread testing than it's gotten so far.  I'm a little nervous about it because I haven't thought about the details for two years, and no one other than me has tested it until a few days ago.
> 
> Ken
> 
> -- 
> Problem reports:      https://cygwin.com/problems.html
> FAQ:                  https://cygwin.com/faq/
> Documentation:        https://cygwin.com/docs.html
> Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Ken Brown]

On 8/29/2021 3:24 PM, Chris Roehrig wrote:
> I'd be happy to test it out if you can give me some commands to git it and build it to replace my existing stock Cygwin installation.
> (Or it is just the cygwin.dll I'd need to replace?)
> 
> My daily backup scripts use a lot of pipes and named pipes.

It's just cygwin1.dll that you'd need to replace.  Instructions for building the 
latter can be found at

   https://www.cygwin.com/faq.html#faq.programming.building-cygwin

but they're out of date in two respects:

1. Currently the build does *not* ignore errors in building the documentation.

2. In addition to the packages listed in the FAQ, you'll need the following:

   texlive-collection-latexrecommended
   texlive-collection-fontsrecommended
   texlive-collection-pictures
   python38-lxml
   python38-ply

These instructions will produce cygwin1.dll in the bin subdirectory of your 
install directory, built from the git master branch.

If you want to try the topic/pipe branch, just go to the newlib-cygwin git repo 
that you cloned, issue the command 'git checkout topic/pipe', and rebuild.

I'll be glad to help if you run into problems.

Ken

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Chris Roehrig]

I got it to build and tried out the topic/pipe branch (checked out on Monday around 4:30pm PDT):

1.  I didn't see any improvement in my sshd+rsync time, still 3-4 MB/sec.

2.  I get the following error from procps:   procps:ps/output.c:2195: please report this bug
(I also get this using the main branch build).

I first updated my cygwin normally using setup-x86_64.exe, then built and copied only the cygwin1.dll into /bin from the install/bin directory (quitting all cygwin tasks first).

On Sun Aug 29 2021, at 3:24 PM, Ken Brown via Cygwin <cygwin@cygwin.com> wrote:

> On 8/29/2021 3:24 PM, Chris Roehrig wrote:
>> I'd be happy to test it out if you can give me some commands to git it and build it to replace my existing stock Cygwin installation.
>> (Or it is just the cygwin.dll I'd need to replace?)
>> My daily backup scripts use a lot of pipes and named pipes.
> 
> It's just cygwin1.dll that you'd need to replace.  Instructions for building the latter can be found at
> 
>  https://www.cygwin.com/faq.html#faq.programming.building-cygwin
> 
> but they're out of date in two respects:
> 
> 1. Currently the build does *not* ignore errors in building the documentation.
> 
> 2. In addition to the packages listed in the FAQ, you'll need the following:
> 
>  texlive-collection-latexrecommended
>  texlive-collection-fontsrecommended
>  texlive-collection-pictures
>  python38-lxml
>  python38-ply
> 
> These instructions will produce cygwin1.dll in the bin subdirectory of your install directory, built from the git master branch.
> 
> If you want to try the topic/pipe branch, just go to the newlib-cygwin git repo that you cloned, issue the command 'git checkout topic/pipe', and rebuild.
> 
> I'll be glad to help if you run into problems.
> 
> Ken
> 
> -- 
> Problem reports:      https://cygwin.com/problems.html
> FAQ:                  https://cygwin.com/faq/
> Documentation:        https://cygwin.com/docs.html
> Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Ken Brown]

On 8/30/2021 7:58 PM, Chris Roehrig wrote:
> I got it to build and tried out the topic/pipe branch (checked out on Monday around 4:30pm PDT):
> 
> 1.  I didn't see any improvement in my sshd+rsync time, still 3-4 MB/sec.
> 
> 2.  I get the following error from procps:   procps:ps/output.c:2195: please report this bug
> (I also get this using the main branch build).
> 
> I first updated my cygwin normally using setup-x86_64.exe, then built and copied only the cygwin1.dll into /bin from the install/bin directory (quitting all cygwin tasks first).

Thanks for testing.

We're still working on this.  (The discussion has moved to the cygwin-developers 
mailing list.)  I'll let you know when it's stable, and maybe you can try again.

As to the procps error, can you give details so that someone can try to 
reproduce it and debug it?

Ken

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Chris Roehrig]

Thanks, I did some more tests:
	scp also shows no improvement with topic/pipe.    I tried running cygsshd with CYGWIN=pipe_byte as well as empty (in the registry HKLM/SYSTEM/CurrentControlSet/Services/cygsshd/Parameters/Environment/), using net stop cygsshd + net start cygsshd to restart it.   Not sure if that CYGWIN only applies to cygsshd or to all cygwin tasks.

I get the procps error just running 'procps' command with no args from mintty (/usr/bin/procps).     Looking at the procps-ng source (procps-ng 3.3.17-1 is what setup_x86_64 reports as my installed package), it appears to be a mmap() failure.

I'm wondering if I built/installed cygwin1.dll correctly because I don't get the procps error using the latest stock cygwin1.dll as installed by setup_x86_64.

-- Chris



On Tue Aug 31 2021, at 12:05 PM, Ken Brown via Cygwin <cygwin@cygwin.com> wrote:

> On 8/30/2021 7:58 PM, Chris Roehrig wrote:
>> I got it to build and tried out the topic/pipe branch (checked out on Monday around 4:30pm PDT):
>> 1.  I didn't see any improvement in my sshd+rsync time, still 3-4 MB/sec.
>> 2.  I get the following error from procps:   procps:ps/output.c:2195: please report this bug
>> (I also get this using the main branch build).
>> I first updated my cygwin normally using setup-x86_64.exe, then built and copied only the cygwin1.dll into /bin from the install/bin directory (quitting all cygwin tasks first).
> 
> Thanks for testing.
> 
> We're still working on this.  (The discussion has moved to the cygwin-developers mailing list.)  I'll let you know when it's stable, and maybe you can try again.
> 
> As to the procps error, can you give details so that someone can try to reproduce it and debug it?
> 
> Ken
> 
> -- 
> Problem reports:      https://cygwin.com/problems.html
> FAQ:                  https://cygwin.com/faq/
> Documentation:        https://cygwin.com/docs.html
> Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Chris Roehrig]

I did a 'git pull' of the latest topic/pipe and rebuilt and I now do indeed get 100MB/s transfers using both rsync and scp (without pipe_byte).
(It turns out last time I forgot 'make install'  -- Doh!)

I still get the procps error however.


On Tue Aug 31 2021, at 12:53 PM, Chris Roehrig <croehrig@house.org> wrote:

> Thanks, I did some more tests:
> 	scp also shows no improvement with topic/pipe.    I tried running cygsshd with CYGWIN=pipe_byte as well as empty (in the registry HKLM/SYSTEM/CurrentControlSet/Services/cygsshd/Parameters/Environment/), using net stop cygsshd + net start cygsshd to restart it.   Not sure if that CYGWIN only applies to cygsshd or to all cygwin tasks.
> 
> I get the procps error just running 'procps' command with no args from mintty (/usr/bin/procps).     Looking at the procps-ng source (procps-ng 3.3.17-1 is what setup_x86_64 reports as my installed package), it appears to be a mmap() failure.
> 
> I'm wondering if I built/installed cygwin1.dll correctly because I don't get the procps error using the latest stock cygwin1.dll as installed by setup_x86_64.
> 
> -- Chris
> 
> 
> 
> On Tue Aug 31 2021, at 12:05 PM, Ken Brown via Cygwin <cygwin@cygwin.com> wrote:
> 
>> On 8/30/2021 7:58 PM, Chris Roehrig wrote:
>>> I got it to build and tried out the topic/pipe branch (checked out on Monday around 4:30pm PDT):
>>> 1.  I didn't see any improvement in my sshd+rsync time, still 3-4 MB/sec.
>>> 2.  I get the following error from procps:   procps:ps/output.c:2195: please report this bug
>>> (I also get this using the main branch build).
>>> I first updated my cygwin normally using setup-x86_64.exe, then built and copied only the cygwin1.dll into /bin from the install/bin directory (quitting all cygwin tasks first).
>> 
>> Thanks for testing.
>> 
>> We're still working on this.  (The discussion has moved to the cygwin-developers mailing list.)  I'll let you know when it's stable, and maybe you can try again.
>> 
>> As to the procps error, can you give details so that someone can try to reproduce it and debug it?
>> 
>> Ken
>> 
>> -- 
>> Problem reports:      https://cygwin.com/problems.html
>> FAQ:                  https://cygwin.com/faq/
>> Documentation:        https://cygwin.com/docs.html
>> Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple
> 
> 
> -- 
> Problem reports:      https://cygwin.com/problems.html
> FAQ:                  https://cygwin.com/faq/
> Documentation:        https://cygwin.com/docs.html
> Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Brian Inglis]

I've found you must also copy the matching cygwin-console-helper.exe 
for everything to work correctly!

On 2021-08-31 14:23, Chris Roehrig wrote:
> I did a 'git pull' of the latest topic/pipe and rebuilt and I now do indeed get 100MB/s transfers using both rsync and scp (without pipe_byte).
> (It turns out last time I forgot 'make install'  -- Doh!)
> 
> I still get the procps error however.

> On Tue Aug 31 2021, at 12:53 PM, Chris Roehrig wrote:
>> Thanks, I did some more tests:
>> 	scp also shows no improvement with topic/pipe.    I tried running cygsshd with CYGWIN=pipe_byte as well as empty (in the registry HKLM/SYSTEM/CurrentControlSet/Services/cygsshd/Parameters/Environment/), using net stop cygsshd + net start cygsshd to restart it.   Not sure if that CYGWIN only applies to cygsshd or to all cygwin tasks.
>>
>> I get the procps error just running 'procps' command with no args from mintty (/usr/bin/procps).     Looking at the procps-ng source (procps-ng 3.3.17-1 is what setup_x86_64 reports as my installed package), it appears to be a mmap() failure.
>>
>> I'm wondering if I built/installed cygwin1.dll correctly because I don't get the procps error using the latest stock cygwin1.dll as installed by setup_

>> On Tue Aug 31 2021, at 12:05 PM, Ken Brown via Cygwin wrote:
>>> On 8/30/2021 7:58 PM, Chris Roehrig wrote:
>>>> I got it to build and tried out the topic/pipe branch (checked out on Monday around 4:30pm PDT):
>>>> 1.  I didn't see any improvement in my sshd+rsync time, still 3-4 MB/sec.
>>>> 2.  I get the following error from procps:   procps:ps/output.c:2195: please report this bug
>>>> (I also get this using the main branch build).
>>>> I first updated my cygwin normally using setup-x86_64.exe, then built and copied only the cygwin1.dll into /bin from the install/bin directory (quitting all cygwin tasks first).
>>>
>>> Thanks for testing.
>>>
>>> We're still working on this.  (The discussion has moved to the cygwin-developers mailing list.)  I'll let you know when it's stable, and maybe you can try again.
>>>
>>> As to the procps error, can you give details so that someone can try to reproduce it and debug it?

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Chris Roehrig]

I rebuild procps 3.3.17.29-2480 from source and it appears to work, so maybe the stock procps package is incompatibility with the current master branch.  (However, I built it against the stock /usr/include, not the current branch...)

I first needed to 'make /proc/libprocps.la', and there was an undefined siginfo_t.si_int in ./lib/test_process.c which I just commented-out to get it to build.

-- Chris

On Tue Aug 31 2021, at 1:23 PM, Chris Roehrig <croehrig@house.org> wrote:

> I did a 'git pull' of the latest topic/pipe and rebuilt and I now do indeed get 100MB/s transfers using both rsync and scp (without pipe_byte).
> (It turns out last time I forgot 'make install'  -- Doh!)
> 
> I still get the procps error however.
> 
> 
> On Tue Aug 31 2021, at 12:53 PM, Chris Roehrig <croehrig@house.org> wrote:
> 
>> Thanks, I did some more tests:
>> 	scp also shows no improvement with topic/pipe.    I tried running cygsshd with CYGWIN=pipe_byte as well as empty (in the registry HKLM/SYSTEM/CurrentControlSet/Services/cygsshd/Parameters/Environment/), using net stop cygsshd + net start cygsshd to restart it.   Not sure if that CYGWIN only applies to cygsshd or to all cygwin tasks.
>> 
>> I get the procps error just running 'procps' command with no args from mintty (/usr/bin/procps).     Looking at the procps-ng source (procps-ng 3.3.17-1 is what setup_x86_64 reports as my installed package), it appears to be a mmap() failure.
>> 
>> I'm wondering if I built/installed cygwin1.dll correctly because I don't get the procps error using the latest stock cygwin1.dll as installed by setup_x86_64.
>> 
>> -- Chris
>> 
>> 
>> 
>> On Tue Aug 31 2021, at 12:05 PM, Ken Brown via Cygwin <cygwin@cygwin.com> wrote:
>> 
>>> On 8/30/2021 7:58 PM, Chris Roehrig wrote:
>>>> I got it to build and tried out the topic/pipe branch (checked out on Monday around 4:30pm PDT):
>>>> 1.  I didn't see any improvement in my sshd+rsync time, still 3-4 MB/sec.
>>>> 2.  I get the following error from procps:   procps:ps/output.c:2195: please report this bug
>>>> (I also get this using the main branch build).
>>>> I first updated my cygwin normally using setup-x86_64.exe, then built and copied only the cygwin1.dll into /bin from the install/bin directory (quitting all cygwin tasks first).
>>> 
>>> Thanks for testing.
>>> 
>>> We're still working on this.  (The discussion has moved to the cygwin-developers mailing list.)  I'll let you know when it's stable, and maybe you can try again.
>>> 
>>> As to the procps error, can you give details so that someone can try to reproduce it and debug it?
>>> 
>>> Ken
>>> 
>>> -- 
>>> Problem reports:      https://cygwin.com/problems.html
>>> FAQ:                  https://cygwin.com/faq/
>>> Documentation:        https://cygwin.com/docs.html
>>> Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple
>> 
>> 
>> -- 
>> Problem reports:      https://cygwin.com/problems.html
>> FAQ:                  https://cygwin.com/faq/
>> Documentation:        https://cygwin.com/docs.html
>> Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple
> 
> 
> -- 
> Problem reports:      https://cygwin.com/problems.html
> FAQ:                  https://cygwin.com/faq/
> Documentation:        https://cygwin.com/docs.html
> Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Ken Brown]

On 9/1/2021 5:11 PM, Chris Roehrig wrote:
> I rebuild procps 3.3.17.29-2480 from source and it appears to work, so maybe the stock procps package is incompatibility with the current master branch.

Maybe, but it could also be a Cygwin bug.  I'll do a bisection of the Cygwin 
sources to see if I can track it down.

Ken

P.S.  The custom on this list is not to top-post.  Thanks.

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Chris Roehrig]

On Thu Sep 2 2021, at 8:25 AM, Ken Brown via Cygwin <cygwin@cygwin.com> wrote:
> On 9/1/2021 5:11 PM, Chris Roehrig wrote:
>> I rebuild procps 3.3.17.29-2480 from source and it appears to work, so maybe the stock procps package is incompatible with the current master branch.
> 
> Maybe, but it could also be a Cygwin bug.  I'll do a bisection of the Cygwin sources to see if I can track it down.

I did some more tests and it still doesn't completely work:
	procps -ef		# works
	procps -eo user,stime,tty,time,args	# works
	procps -eo pid  # fails with that same mmap() error

I also rebuilt it all using cygport and it gives the same error (pscommand.exe with no args).

> P.S.  The custom on this list is not to top-post.  Thanks.

Sorry, I didn't realize there was an etiquette.    Fixed.

-- Chris

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Ken Brown]

On 9/2/2021 3:03 PM, Chris Roehrig wrote:
> 
> On Thu Sep 2 2021, at 8:25 AM, Ken Brown via Cygwin <cygwin@cygwin.com> wrote:
>> On 9/1/2021 5:11 PM, Chris Roehrig wrote:
>>> I rebuild procps 3.3.17.29-2480 from source and it appears to work, so maybe the stock procps package is incompatible with the current master branch.
>>
>> Maybe, but it could also be a Cygwin bug.  I'll do a bisection of the Cygwin sources to see if I can track it down.
> 
> I did some more tests and it still doesn't completely work:
> 	procps -ef		# works
> 	procps -eo user,stime,tty,time,args	# works
> 	procps -eo pid  # fails with that same mmap() error
> 
> I also rebuilt it all using cygport and it gives the same error (pscommand.exe with no args).

I'm still seeing the error too, and I'm even seeing it if I rebuild Cygwin 3.2.0 
with the current gcc.  I don't know what the problem is.

>> P.S.  The custom on this list is not to top-post.  Thanks.
> 
> Sorry, I didn't realize there was an etiquette.    Fixed.

Thanks!

Ken

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Corinna Vinschen]

[resent, this time with the ML in To]

On Sep  2 12:03, Chris Roehrig wrote:
> 
> On Thu Sep 2 2021, at 8:25 AM, Ken Brown via Cygwin <cygwin@cygwin.com> wrote:
> > On 9/1/2021 5:11 PM, Chris Roehrig wrote:
> >> I rebuild procps 3.3.17.29-2480 from source and it appears to work, so maybe the stock procps package is incompatible with the current master branch.
> > 
> > Maybe, but it could also be a Cygwin bug.  I'll do a bisection of the Cygwin sources to see if I can track it down.
> 
> I did some more tests and it still doesn't completely work:
> 	procps -ef		# works
> 	procps -eo user,stime,tty,time,args	# works
> 	procps -eo pid  # fails with that same mmap() error
> 
> I also rebuilt it all using cygport and it gives the same error (pscommand.exe with no args).

Since you're building Cygwin by yourself anyway, can you do me a favor
and try this:

$ git revert 2f05de4dbf9c

and see if that fixes your issue?


Thanks,
Corinna

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Chris Roehrig]

On Fri Sep 3 2021, at 12:55 PM, Corinna Vinschen <corinna-cygwin@cygwin.com> wrote:

> [resent, this time with the ML in To]
> 
> On Sep  2 12:03, Chris Roehrig wrote:
>> 
>> On Thu Sep 2 2021, at 8:25 AM, Ken Brown via Cygwin <cygwin@cygwin.com> wrote:
>>> On 9/1/2021 5:11 PM, Chris Roehrig wrote:
>>>> I rebuild procps 3.3.17.29-2480 from source and it appears to work, so maybe the stock procps package is incompatible with the current master branch.
>>> 
>>> Maybe, but it could also be a Cygwin bug.  I'll do a bisection of the Cygwin sources to see if I can track it down.
>> 
>> I did some more tests and it still doesn't completely work:
>> 	procps -ef		# works
>> 	procps -eo user,stime,tty,time,args	# works
>> 	procps -eo pid  # fails with that same mmap() error
>> 
>> I also rebuilt it all using cygport and it gives the same error (pscommand.exe with no args).
> 
> Since you're building Cygwin by yourself anyway, can you do me a favor
> and try this:
> 
> $ git revert 2f05de4dbf9c
> 
> and see if that fixes your issue?

I got procps working I think (both with and without the revert).

I think the problem might just be that I wasn't also copying the rebuilt /bin/cygprocps-8.dll to match the /bin/procps.exe.    There's some tricky renaming that make install does, so I did 'make install DESTDIR=/tmp/install' and copied just those two files.     I'm guessing it would all work on a properly fully installed build.

-- Chris

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Achim Gratz]

Am 03.09.2021 um 22:59 schrieb Chris Roehrig:
> I got procps working I think (both with and without the revert).

That likely wasn't what Corinna wanted to know, though.

Please re-install the procps-ng, cygwin and cygwin-devel packages from 
setup (and revert any other alterations you may have made).  Then check 
that the error has gone away, rebuild and install the new cygwin1.dll 
with and without the revert mentioned by Corinna and check for the error 
under these two conditions as well.

-- 
Achim.

(on the road :-)

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Brian Inglis]

On 2021-09-03 14:59, Chris Roehrig wrote:
> On Fri Sep 3 2021, at 12:55 PM, Corinna Vinschen wrote:
>> On Sep  2 12:03, Chris Roehrig wrote:
>>> On Thu Sep 2 2021, at 8:25 AM, Ken Brown via Cygwin wrote:
>>>> On 9/1/2021 5:11 PM, Chris Roehrig wrote:
>>>>> I rebuild procps 3.3.17.29-2480 from source and it appears to
>>>>> work, so maybe the stock procps package is incompatible with
>>>>> the current master branch.

>>>> Maybe, but it could also be a Cygwin bug.  I'll do a bisection 
>>>> of the Cygwin sources to see if I can track it down.

>>> I did some more tests and it still doesn't completely work:
>>> 	procps -ef		# works
>>> 	procps -eo user,stime,tty,time,args	# works
>>> 	procps -eo pid  # fails with that same mmap() error
>>> I also rebuilt it all using cygport and it gives the same error 
>>> (pscommand.exe with no args).

>> Since you're building Cygwin by yourself anyway, can you do me a
>> favor and try this:
>> $ git revert 2f05de4dbf9c
>> and see if that fixes your issue?

> I got procps working I think (both with and without the revert). I
> think the problem might just be that I wasn't also copying the 
> rebuilt /bin/cygprocps-8.dll to match the /bin/procps.exe. There's
> some tricky renaming that make install does, so I did 'make install 
> DESTDIR=/tmp/install' and copied just those two files. I'm guessing
> it would all work on a properly fully installed build.

If you are building packages in the same way as scallywag with:

	$ cygport PKG.cygport download all check

then to install all binary packages for dogfooding:

$ for tar in PKG-VER-1.ARCH/dist/PKG/**/*PKG*-VER.tar.*z*
   do
	tar -x -C / -f $tar	# * -C / installs usr/... under /usr/
   done

for example, I just did:

$ for tar in \
	dash-0.5.11.5-1.x86_64/dist/dash/**/*dash*-0.5.11.5-1.tar.*z*
   do
	tar -x -C / -f $tar
   done


-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Achim Gratz]

04.09.2021 18:45, Brian Inglis:
[...]
> then to install all binary packages for dogfooding:

Would you please stop telling folks to do things that potentially breaks 
their systems?

There are quite a few more steps to take if you want to emulate what 
setup does.  Then again you cannot do that on a live Cygwin system 
unless you _really_ know what exactly it is you're doing and which 
packages can or cannot be touched that way.


-- 
Achim.

(on the road :-)

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Brian Inglis]

On 2021-09-05 02:18, Achim Gratz wrote:
> 04.09.2021 18:45, Brian Inglis:
> [...]
>> then to install all binary packages for dogfooding:
> 
> Would you please stop telling folks to do things that potentially breaks 
> their systems?
> 
> There are quite a few more steps to take if you want to emulate what 
> setup does.  Then again you cannot do that on a live Cygwin system 
> unless you _really_ know what exactly it is you're doing and which 
> packages can or cannot be touched that way.

This is a downside of having only a general and not also a "technical" 
mailing list to discuss and diagnose "technical" issues and approaches.
The suggestion was intended as a tip to ensure *complete* locally 
rebuilt package contents are installed, by folks who are already having 
issues with their installation, rebuilding Cygwin and packages, and 
replacing the Cygwin dll etc. and packages to diagnose those issues. So 
hopefully they are also aware of preremove and postinstall scripts, in 
packages where they are provided, and the /etc/setup/PKG.lst.gz 
manifests for files to be removed before installation, and provided 
after installation. For the latter I would expect technical folks to 
make use of them but leave them to Cygwin Setup programs for officially 
released package upgrades.

[Also some of us do not have the appropriate technical background and 
expertise to get a Cygwin overlay mirror working reliably from the 
available instructions (if it was easy enough, someone would have 
provided a local-overlay-mirror package to do it years ago), nor systems 
fast enough to run Cygwin Setup for every package install.
Due to Windows "improvements", my system upgraded a few years ago is 
just as "fast" as the 10 year old system it replaced. ;^>
Cygwin Setup upgrades can take as long as Windows Update installing the 
latest patches.
Some of my approaches, suggestions, and participation here are coloured 
by those limitations (also by expensive, slow, uncompetitive networks).]

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Achim Gratz]

05.09.2021 17:11, Brian Inglis:

> The suggestion was intended as a tip to ensure *complete* locally 
> rebuilt package contents are installed,

Setup has its "from_cwd" installation mode for precisely that reason: 
installing a local package without the need to create a full install 
hierarchy and setup.ini files.

> Due to Windows "improvements", my system upgraded a few years ago is 
> just as "fast" as the 10 year old system it replaced. ;^>
> Cygwin Setup upgrades can take as long as Windows Update installing the 
> latest patches.

If you haven't already, put your Cygwin installation on SSD and exclude 
the Cygwin root (and local mirror directory) from real-time virus scans. 
  I do monthly updates on slow machines (CPU is a 1.8GHz IvyBridge 
Celeron) and that takes around two to five minutes depending on how 
large the update is in any given month (the man-db update happens in the 
background, so is not part of these figures).


-- 
Achim.

(on the road :-)

mmap failure [was: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?]

[Ken Brown]

I've reduced the procps failure to the following test case:

$ cat mmap_test.c
#include <unistd.h>
#include <stdio.h>
#include <sys/mman.h>

int
main ()
{
   void *addr;
   int page_size = getpagesize ();

   addr = mmap (0, page_size, PROT_READ | PROT_WRITE,
                MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
   if (addr == MAP_FAILED)
     perror ("mmap");
}

$ gcc mmap_test.c

$ ./a
mmap: Invalid argument

This happens if I use cygwin1.dll built from the current git master.  But it 
also happens if I build from cygwin-3_2_0-release with the recent patches 
applied that make Cygwin build without error:

0e12b4939 Cygwin: C++17: register keyword is deprecated
a7340e0c0 Cygwin: dumper: fix up GCC pragma for g++ 11.2
2a212c086 Cygwin: workaround a g++ 11.2 initialization bug
9e3f1737e Cygwin: loader script: add DWARF 5 sections
bdfd2b004 Cygwin: testsuite: avoid "conflicting types" gcc warning
6fc498e2e strstr: avoid warnings
26da270b2 ldexp/ldexpf: avoid assembler warning
edce2a557 Cygwin: fix declaration of RtlInitEmptyUnicodeString

So there appears to be something wrong with cygwin1.dll built with the current 
build tools (gcc 11.2.0, binutils 2.37, not sure what else is relevant).

Ken

Re: mmap failure [was: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?]

[Ken Brown]

On 9/4/2021 6:37 PM, Ken Brown via Cygwin wrote:
> I've reduced the procps failure to the following test case:
> 
> $ cat mmap_test.c
> #include <unistd.h>
> #include <stdio.h>
> #include <sys/mman.h>
> 
> int
> main ()
> {
>    void *addr;
>    int page_size = getpagesize ();
> 
>    addr = mmap (0, page_size, PROT_READ | PROT_WRITE,
>                 MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
>    if (addr == MAP_FAILED)
>      perror ("mmap");
> }
> 
> $ gcc mmap_test.c
> 
> $ ./a
> mmap: Invalid argument
> 
> This happens if I use cygwin1.dll built from the current git master.  But it 
> also happens if I build from cygwin-3_2_0-release with the recent patches 
> applied that make Cygwin build without error:
> 
> 0e12b4939 Cygwin: C++17: register keyword is deprecated
> a7340e0c0 Cygwin: dumper: fix up GCC pragma for g++ 11.2
> 2a212c086 Cygwin: workaround a g++ 11.2 initialization bug
> 9e3f1737e Cygwin: loader script: add DWARF 5 sections
> bdfd2b004 Cygwin: testsuite: avoid "conflicting types" gcc warning
> 6fc498e2e strstr: avoid warnings
> 26da270b2 ldexp/ldexpf: avoid assembler warning
> edce2a557 Cygwin: fix declaration of RtlInitEmptyUnicodeString

Sorry, those commit ids are wrong.  They're what I got after rebasing the master 
branch so that those commits come first.

> So there appears to be something wrong with cygwin1.dll built with the current 
> build tools (gcc 11.2.0, binutils 2.37, not sure what else is relevant).
> 
> Ken
> 

Re: mmap failure [was: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?]

[Ken Brown]

On 9/4/2021 6:54 PM, Ken Brown via Cygwin wrote:
> On 9/4/2021 6:37 PM, Ken Brown via Cygwin wrote:
>> I've reduced the procps failure to the following test case:
>>
>> $ cat mmap_test.c
>> #include <unistd.h>
>> #include <stdio.h>
>> #include <sys/mman.h>
>>
>> int
>> main ()
>> {
>>    void *addr;
>>    int page_size = getpagesize ();
>>
>>    addr = mmap (0, page_size, PROT_READ | PROT_WRITE,
>>                 MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
>>    if (addr == MAP_FAILED)
>>      perror ("mmap");
>> }
>>
>> $ gcc mmap_test.c
>>
>> $ ./a
>> mmap: Invalid argument
>>
>> This happens if I use cygwin1.dll built from the current git master.  But it 
>> also happens if I build from cygwin-3_2_0-release with the recent patches 
>> applied that make Cygwin build without error:
>>
>> 0e12b4939 Cygwin: C++17: register keyword is deprecated
>> a7340e0c0 Cygwin: dumper: fix up GCC pragma for g++ 11.2
>> 2a212c086 Cygwin: workaround a g++ 11.2 initialization bug
>> 9e3f1737e Cygwin: loader script: add DWARF 5 sections
>> bdfd2b004 Cygwin: testsuite: avoid "conflicting types" gcc warning
>> 6fc498e2e strstr: avoid warnings
>> 26da270b2 ldexp/ldexpf: avoid assembler warning
>> edce2a557 Cygwin: fix declaration of RtlInitEmptyUnicodeString
> 
> Sorry, those commit ids are wrong.  They're what I got after rebasing the master 
> branch so that those commits come first.

[Pressed Send too soon.]

Here are the correct commits:

8169e39ab Cygwin: C++17: register keyword is deprecated
3ca80b360 Cygwin: dumper: fix up GCC pragma for g++ 11.2
bdb7991db Cygwin: workaround a g++ 11.2 initialization bug
801120c1f Cygwin: loader script: add DWARF 5 sections
d5cc66426 Cygwin: testsuite: avoid "conflicting types" gcc warning
c2fe205b5 strstr: avoid warnings
76c2c7a89 ldexp/ldexpf: avoid assembler warning
eeeb5650c Cygwin: fix declaration of RtlInitEmptyUnicodeString

> 
>> So there appears to be something wrong with cygwin1.dll built with the current 
>> build tools (gcc 11.2.0, binutils 2.37, not sure what else is relevant).
>>
>> Ken
>>
> 

Re: mmap failure [was: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?]

[Ken Brown]

On 9/4/2021 6:58 PM, Ken Brown via Cygwin wrote:
> On 9/4/2021 6:54 PM, Ken Brown via Cygwin wrote:
>> On 9/4/2021 6:37 PM, Ken Brown via Cygwin wrote:
>>> I've reduced the procps failure to the following test case:
>>>
>>> $ cat mmap_test.c
>>> #include <unistd.h>
>>> #include <stdio.h>
>>> #include <sys/mman.h>
>>>
>>> int
>>> main ()
>>> {
>>>    void *addr;
>>>    int page_size = getpagesize ();
>>>
>>>    addr = mmap (0, page_size, PROT_READ | PROT_WRITE,
>>>                 MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
>>>    if (addr == MAP_FAILED)
>>>      perror ("mmap");
>>> }
>>>
>>> $ gcc mmap_test.c
>>>
>>> $ ./a
>>> mmap: Invalid argument
>>>
>>> This happens if I use cygwin1.dll built from the current git master.  But it 
>>> also happens if I build from cygwin-3_2_0-release with the recent patches 
>>> applied that make Cygwin build without error:
>>>
>>> 0e12b4939 Cygwin: C++17: register keyword is deprecated
>>> a7340e0c0 Cygwin: dumper: fix up GCC pragma for g++ 11.2
>>> 2a212c086 Cygwin: workaround a g++ 11.2 initialization bug
>>> 9e3f1737e Cygwin: loader script: add DWARF 5 sections
>>> bdfd2b004 Cygwin: testsuite: avoid "conflicting types" gcc warning
>>> 6fc498e2e strstr: avoid warnings
>>> 26da270b2 ldexp/ldexpf: avoid assembler warning
>>> edce2a557 Cygwin: fix declaration of RtlInitEmptyUnicodeString
>>
>> Sorry, those commit ids are wrong.  They're what I got after rebasing the 
>> master branch so that those commits come first.
> 
> [Pressed Send too soon.]
> 
> Here are the correct commits:
> 
> 8169e39ab Cygwin: C++17: register keyword is deprecated
> 3ca80b360 Cygwin: dumper: fix up GCC pragma for g++ 11.2
> bdb7991db Cygwin: workaround a g++ 11.2 initialization bug
> 801120c1f Cygwin: loader script: add DWARF 5 sections
> d5cc66426 Cygwin: testsuite: avoid "conflicting types" gcc warning
> c2fe205b5 strstr: avoid warnings
> 76c2c7a89 ldexp/ldexpf: avoid assembler warning
> eeeb5650c Cygwin: fix declaration of RtlInitEmptyUnicodeString
> 
>>
>>> So there appears to be something wrong with cygwin1.dll built with the 
>>> current build tools (gcc 11.2.0, binutils 2.37, not sure what else is relevant).

Wait a minute...I'll bet this is related to the MEM_EXTENDED_PARAMETER 
initialization problem that was dealt with in commit bdb7991db.

Ken

Re: mmap failure [was: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?]

[Ken Brown]

On 9/4/2021 8:04 PM, Ken Brown via Cygwin wrote:
> On 9/4/2021 6:58 PM, Ken Brown via Cygwin wrote:
>> On 9/4/2021 6:54 PM, Ken Brown via Cygwin wrote:
>>> On 9/4/2021 6:37 PM, Ken Brown via Cygwin wrote:
>>>> I've reduced the procps failure to the following test case:
>>>>
>>>> $ cat mmap_test.c
>>>> #include <unistd.h>
>>>> #include <stdio.h>
>>>> #include <sys/mman.h>
>>>>
>>>> int
>>>> main ()
>>>> {
>>>>    void *addr;
>>>>    int page_size = getpagesize ();
>>>>
>>>>    addr = mmap (0, page_size, PROT_READ | PROT_WRITE,
>>>>                 MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
>>>>    if (addr == MAP_FAILED)
>>>>      perror ("mmap");
>>>> }
>>>>
>>>> $ gcc mmap_test.c
>>>>
>>>> $ ./a
>>>> mmap: Invalid argument
>>>>
>>>> This happens if I use cygwin1.dll built from the current git master.  But it 
>>>> also happens if I build from cygwin-3_2_0-release with the recent patches 
>>>> applied that make Cygwin build without error:
>>>>
>>>> 0e12b4939 Cygwin: C++17: register keyword is deprecated
>>>> a7340e0c0 Cygwin: dumper: fix up GCC pragma for g++ 11.2
>>>> 2a212c086 Cygwin: workaround a g++ 11.2 initialization bug
>>>> 9e3f1737e Cygwin: loader script: add DWARF 5 sections
>>>> bdfd2b004 Cygwin: testsuite: avoid "conflicting types" gcc warning
>>>> 6fc498e2e strstr: avoid warnings
>>>> 26da270b2 ldexp/ldexpf: avoid assembler warning
>>>> edce2a557 Cygwin: fix declaration of RtlInitEmptyUnicodeString
>>>
>>> Sorry, those commit ids are wrong.  They're what I got after rebasing the 
>>> master branch so that those commits come first.
>>
>> [Pressed Send too soon.]
>>
>> Here are the correct commits:
>>
>> 8169e39ab Cygwin: C++17: register keyword is deprecated
>> 3ca80b360 Cygwin: dumper: fix up GCC pragma for g++ 11.2
>> bdb7991db Cygwin: workaround a g++ 11.2 initialization bug
>> 801120c1f Cygwin: loader script: add DWARF 5 sections
>> d5cc66426 Cygwin: testsuite: avoid "conflicting types" gcc warning
>> c2fe205b5 strstr: avoid warnings
>> 76c2c7a89 ldexp/ldexpf: avoid assembler warning
>> eeeb5650c Cygwin: fix declaration of RtlInitEmptyUnicodeString
>>
>>>
>>>> So there appears to be something wrong with cygwin1.dll built with the 
>>>> current build tools (gcc 11.2.0, binutils 2.37, not sure what else is 
>>>> relevant).
> 
> Wait a minute...I'll bet this is related to the MEM_EXTENDED_PARAMETER 
> initialization problem that was dealt with in commit bdb7991db.

More data: When I run the test case under gdb, it succeeds.  When I run it under 
strace, I see VirtualAlloc2 in fhandler_dev_zero::mmap failing with windows 
error 87.

Ken

Re: mmap failure [was: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?]

[Corinna Vinschen]

On Sep  5 09:24, Ken Brown via Cygwin wrote:
> On 9/4/2021 8:04 PM, Ken Brown via Cygwin wrote:
> > On 9/4/2021 6:58 PM, Ken Brown via Cygwin wrote:
> > > Here are the correct commits:
> > > 
> > > 8169e39ab Cygwin: C++17: register keyword is deprecated
> > > 3ca80b360 Cygwin: dumper: fix up GCC pragma for g++ 11.2
> > > bdb7991db Cygwin: workaround a g++ 11.2 initialization bug
> > > 801120c1f Cygwin: loader script: add DWARF 5 sections
> > > d5cc66426 Cygwin: testsuite: avoid "conflicting types" gcc warning
> > > c2fe205b5 strstr: avoid warnings
> > > 76c2c7a89 ldexp/ldexpf: avoid assembler warning
> > > eeeb5650c Cygwin: fix declaration of RtlInitEmptyUnicodeString
> > > 
> > > > 
> > > > > So there appears to be something wrong with cygwin1.dll
> > > > > built with the current build tools (gcc 11.2.0, binutils
> > > > > 2.37, not sure what else is relevant).
> > 
> > Wait a minute...I'll bet this is related to the MEM_EXTENDED_PARAMETER
> > initialization problem that was dealt with in commit bdb7991db.
> 
> More data: When I run the test case under gdb, it succeeds.  When I run it
> under strace, I see VirtualAlloc2 in fhandler_dev_zero::mmap failing with
> windows error 87.

Are the const's I added to the MEM_EXTENDED_PARAMETER data invalid,
perhaps?


Corinna

Re: mmap failure [was: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?]

[Ken Brown]

On 9/6/2021 11:32 AM, Corinna Vinschen via Cygwin wrote:
> On Sep  5 09:24, Ken Brown via Cygwin wrote:
>> On 9/4/2021 8:04 PM, Ken Brown via Cygwin wrote:
>>> On 9/4/2021 6:58 PM, Ken Brown via Cygwin wrote:
>>>> Here are the correct commits:
>>>>
>>>> 8169e39ab Cygwin: C++17: register keyword is deprecated
>>>> 3ca80b360 Cygwin: dumper: fix up GCC pragma for g++ 11.2
>>>> bdb7991db Cygwin: workaround a g++ 11.2 initialization bug
>>>> 801120c1f Cygwin: loader script: add DWARF 5 sections
>>>> d5cc66426 Cygwin: testsuite: avoid "conflicting types" gcc warning
>>>> c2fe205b5 strstr: avoid warnings
>>>> 76c2c7a89 ldexp/ldexpf: avoid assembler warning
>>>> eeeb5650c Cygwin: fix declaration of RtlInitEmptyUnicodeString
>>>>
>>>>>
>>>>>> So there appears to be something wrong with cygwin1.dll
>>>>>> built with the current build tools (gcc 11.2.0, binutils
>>>>>> 2.37, not sure what else is relevant).
>>>
>>> Wait a minute...I'll bet this is related to the MEM_EXTENDED_PARAMETER
>>> initialization problem that was dealt with in commit bdb7991db.
>>
>> More data: When I run the test case under gdb, it succeeds.  When I run it
>> under strace, I see VirtualAlloc2 in fhandler_dev_zero::mmap failing with
>> windows error 87.
> 
> Are the const's I added to the MEM_EXTENDED_PARAMETER data invalid,
> perhaps?

I tried removing them, and I got the same error.  I also tried removing static, 
and I tried removing both static and const.

Ken

Re: mmap failure [was: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?]

[Ken Brown]

On 9/6/2021 1:12 PM, Ken Brown via Cygwin wrote:
> On 9/6/2021 11:32 AM, Corinna Vinschen via Cygwin wrote:
>> On Sep  5 09:24, Ken Brown via Cygwin wrote:
>>> On 9/4/2021 8:04 PM, Ken Brown via Cygwin wrote:
>>>> On 9/4/2021 6:58 PM, Ken Brown via Cygwin wrote:
>>>>> Here are the correct commits:
>>>>>
>>>>> 8169e39ab Cygwin: C++17: register keyword is deprecated
>>>>> 3ca80b360 Cygwin: dumper: fix up GCC pragma for g++ 11.2
>>>>> bdb7991db Cygwin: workaround a g++ 11.2 initialization bug
>>>>> 801120c1f Cygwin: loader script: add DWARF 5 sections
>>>>> d5cc66426 Cygwin: testsuite: avoid "conflicting types" gcc warning
>>>>> c2fe205b5 strstr: avoid warnings
>>>>> 76c2c7a89 ldexp/ldexpf: avoid assembler warning
>>>>> eeeb5650c Cygwin: fix declaration of RtlInitEmptyUnicodeString
>>>>>
>>>>>>
>>>>>>> So there appears to be something wrong with cygwin1.dll
>>>>>>> built with the current build tools (gcc 11.2.0, binutils
>>>>>>> 2.37, not sure what else is relevant).
>>>>
>>>> Wait a minute...I'll bet this is related to the MEM_EXTENDED_PARAMETER
>>>> initialization problem that was dealt with in commit bdb7991db.
>>>
>>> More data: When I run the test case under gdb, it succeeds.  When I run it
>>> under strace, I see VirtualAlloc2 in fhandler_dev_zero::mmap failing with
>>> windows error 87.
>>
>> Are the const's I added to the MEM_EXTENDED_PARAMETER data invalid,
>> perhaps?
> 
> I tried removing them, and I got the same error.  I also tried removing static, 
> and I tried removing both static and const.

BTW, when I reported that the test case succeeds under gdb, that only happens 
when I build the test case without optimization.  If I build with -O2, it fails 
under gdb also.  [In all my tests, I built cygwin1.dll without optimization.] 
This makes no sense to me at all.

Ken

Re: mmap failure [was: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?]

[Eliot Moss]

On 9/6/2021 1:38 PM, Ken Brown via Cygwin wrote:
> On 9/6/2021 1:12 PM, Ken Brown via Cygwin wrote:
>> On 9/6/2021 11:32 AM, Corinna Vinschen via Cygwin wrote:
>>> On Sep  5 09:24, Ken Brown via Cygwin wrote:
>>>> On 9/4/2021 8:04 PM, Ken Brown via Cygwin wrote:
>>>>> On 9/4/2021 6:58 PM, Ken Brown via Cygwin wrote:
>>>>>> Here are the correct commits:
>>>>>>
>>>>>> 8169e39ab Cygwin: C++17: register keyword is deprecated
>>>>>> 3ca80b360 Cygwin: dumper: fix up GCC pragma for g++ 11.2
>>>>>> bdb7991db Cygwin: workaround a g++ 11.2 initialization bug
>>>>>> 801120c1f Cygwin: loader script: add DWARF 5 sections
>>>>>> d5cc66426 Cygwin: testsuite: avoid "conflicting types" gcc warning
>>>>>> c2fe205b5 strstr: avoid warnings
>>>>>> 76c2c7a89 ldexp/ldexpf: avoid assembler warning
>>>>>> eeeb5650c Cygwin: fix declaration of RtlInitEmptyUnicodeString
>>>>>>
>>>>>>>
>>>>>>>> So there appears to be something wrong with cygwin1.dll
>>>>>>>> built with the current build tools (gcc 11.2.0, binutils
>>>>>>>> 2.37, not sure what else is relevant).
>>>>>
>>>>> Wait a minute...I'll bet this is related to the MEM_EXTENDED_PARAMETER
>>>>> initialization problem that was dealt with in commit bdb7991db.
>>>>
>>>> More data: When I run the test case under gdb, it succeeds.  When I run it
>>>> under strace, I see VirtualAlloc2 in fhandler_dev_zero::mmap failing with
>>>> windows error 87.
>>>
>>> Are the const's I added to the MEM_EXTENDED_PARAMETER data invalid,
>>> perhaps?
>>
>> I tried removing them, and I got the same error.  I also tried removing static, and I tried 
>> removing both static and const.
> 
> BTW, when I reported that the test case succeeds under gdb, that only happens when I build the test 
> case without optimization.  If I build with -O2, it fails under gdb also.  [In all my tests, I built 
> cygwin1.dll without optimization.] This makes no sense to me at all.

There can be a number of possibilities, but I wonder about a variable
uninitialized along some path.  By accident, the contents may have a
non-failing value with -O0 by the situation can be different for -O2.
If you're dealing with concurrency and such, then -O0 and -O2 can act
differently with respect to races.  Just some thoughts ...

Best - Eliot Moss

Re: mmap failure [was: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?]

[Corinna Vinschen]

On Sep  6 13:38, Ken Brown via Cygwin wrote:
> On 9/6/2021 1:12 PM, Ken Brown via Cygwin wrote:
> > On 9/6/2021 11:32 AM, Corinna Vinschen via Cygwin wrote:
> > > On Sep  5 09:24, Ken Brown via Cygwin wrote:
> > > > On 9/4/2021 8:04 PM, Ken Brown via Cygwin wrote:
> > > > > On 9/4/2021 6:58 PM, Ken Brown via Cygwin wrote:
> > > > > > Here are the correct commits:
> > > > > > 
> > > > > > 8169e39ab Cygwin: C++17: register keyword is deprecated
> > > > > > 3ca80b360 Cygwin: dumper: fix up GCC pragma for g++ 11.2
> > > > > > bdb7991db Cygwin: workaround a g++ 11.2 initialization bug
> > > > > > 801120c1f Cygwin: loader script: add DWARF 5 sections
> > > > > > d5cc66426 Cygwin: testsuite: avoid "conflicting types" gcc warning
> > > > > > c2fe205b5 strstr: avoid warnings
> > > > > > 76c2c7a89 ldexp/ldexpf: avoid assembler warning
> > > > > > eeeb5650c Cygwin: fix declaration of RtlInitEmptyUnicodeString
> > > > > > 
> > > > > > > 
> > > > > > > > So there appears to be something wrong with cygwin1.dll
> > > > > > > > built with the current build tools (gcc 11.2.0, binutils
> > > > > > > > 2.37, not sure what else is relevant).
> > > > > 
> > > > > Wait a minute...I'll bet this is related to the MEM_EXTENDED_PARAMETER
> > > > > initialization problem that was dealt with in commit bdb7991db.
> > > > 
> > > > More data: When I run the test case under gdb, it succeeds.  When I run it
> > > > under strace, I see VirtualAlloc2 in fhandler_dev_zero::mmap failing with
> > > > windows error 87.
> > > 
> > > Are the const's I added to the MEM_EXTENDED_PARAMETER data invalid,
> > > perhaps?
> > 
> > I tried removing them, and I got the same error.  I also tried removing
> > static, and I tried removing both static and const.
> 
> BTW, when I reported that the test case succeeds under gdb, that only
> happens when I build the test case without optimization.  If I build with
> -O2, it fails under gdb also.  [In all my tests, I built cygwin1.dll without
> optimization.] This makes no sense to me at all.

Good hint.  I found the culprit.  With optimization, the code doesn't
set the "Reserved" bits in the first struct of MEM_EXTENDED_PARAMETER
to 0.  This is at least required by the VirtualAlloc2 function, though.
Needless to say that this behaviour isn't documented...

I'll push a patch shortly.


Thanks,
Corinna

Re: mmap failure [was: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?]

[Corinna Vinschen]

On Sep  6 19:59, Corinna Vinschen via Cygwin wrote:
> On Sep  6 13:38, Ken Brown via Cygwin wrote:
> > On 9/6/2021 1:12 PM, Ken Brown via Cygwin wrote:
> > > On 9/6/2021 11:32 AM, Corinna Vinschen via Cygwin wrote:
> > > > On Sep  5 09:24, Ken Brown via Cygwin wrote:
> > > > > On 9/4/2021 8:04 PM, Ken Brown via Cygwin wrote:
> > > > > > On 9/4/2021 6:58 PM, Ken Brown via Cygwin wrote:
> > > > > > > Here are the correct commits:
> > > > > > > 
> > > > > > > 8169e39ab Cygwin: C++17: register keyword is deprecated
> > > > > > > 3ca80b360 Cygwin: dumper: fix up GCC pragma for g++ 11.2
> > > > > > > bdb7991db Cygwin: workaround a g++ 11.2 initialization bug
> > > > > > > 801120c1f Cygwin: loader script: add DWARF 5 sections
> > > > > > > d5cc66426 Cygwin: testsuite: avoid "conflicting types" gcc warning
> > > > > > > c2fe205b5 strstr: avoid warnings
> > > > > > > 76c2c7a89 ldexp/ldexpf: avoid assembler warning
> > > > > > > eeeb5650c Cygwin: fix declaration of RtlInitEmptyUnicodeString
> > > > > > > 
> > > > > > > > 
> > > > > > > > > So there appears to be something wrong with cygwin1.dll
> > > > > > > > > built with the current build tools (gcc 11.2.0, binutils
> > > > > > > > > 2.37, not sure what else is relevant).
> > > > > > 
> > > > > > Wait a minute...I'll bet this is related to the MEM_EXTENDED_PARAMETER
> > > > > > initialization problem that was dealt with in commit bdb7991db.
> > > > > 
> > > > > More data: When I run the test case under gdb, it succeeds.  When I run it
> > > > > under strace, I see VirtualAlloc2 in fhandler_dev_zero::mmap failing with
> > > > > windows error 87.
> > > > 
> > > > Are the const's I added to the MEM_EXTENDED_PARAMETER data invalid,
> > > > perhaps?
> > > 
> > > I tried removing them, and I got the same error.  I also tried removing
> > > static, and I tried removing both static and const.
> > 
> > BTW, when I reported that the test case succeeds under gdb, that only
> > happens when I build the test case without optimization.  If I build with
> > -O2, it fails under gdb also.  [In all my tests, I built cygwin1.dll without
> > optimization.] This makes no sense to me at all.
> 
> Good hint.  I found the culprit.  With optimization, the code doesn't
> set the "Reserved" bits in the first struct of MEM_EXTENDED_PARAMETER
> to 0.

No, wait.  I get what you say.  The optimzation settings of the test
case should have no influence on the code inside the DLL.  That doesn't
make sense for sure.  However, I ran the testcase under GDB, I could
reproduce the issue, and I could fix it by setting mmap_ext.Reserved = 0;
Go figure!


Corinna

Re: mmap failure [was: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?]

[Ken Brown]

On 9/6/2021 2:07 PM, Corinna Vinschen via Cygwin wrote:
> On Sep  6 19:59, Corinna Vinschen via Cygwin wrote:
>> On Sep  6 13:38, Ken Brown via Cygwin wrote:
>>> On 9/6/2021 1:12 PM, Ken Brown via Cygwin wrote:
>>>> On 9/6/2021 11:32 AM, Corinna Vinschen via Cygwin wrote:
>>>>> On Sep  5 09:24, Ken Brown via Cygwin wrote:
>>>>>> On 9/4/2021 8:04 PM, Ken Brown via Cygwin wrote:
>>>>>>> On 9/4/2021 6:58 PM, Ken Brown via Cygwin wrote:
>>>>>>>> Here are the correct commits:
>>>>>>>>
>>>>>>>> 8169e39ab Cygwin: C++17: register keyword is deprecated
>>>>>>>> 3ca80b360 Cygwin: dumper: fix up GCC pragma for g++ 11.2
>>>>>>>> bdb7991db Cygwin: workaround a g++ 11.2 initialization bug
>>>>>>>> 801120c1f Cygwin: loader script: add DWARF 5 sections
>>>>>>>> d5cc66426 Cygwin: testsuite: avoid "conflicting types" gcc warning
>>>>>>>> c2fe205b5 strstr: avoid warnings
>>>>>>>> 76c2c7a89 ldexp/ldexpf: avoid assembler warning
>>>>>>>> eeeb5650c Cygwin: fix declaration of RtlInitEmptyUnicodeString
>>>>>>>>
>>>>>>>>>
>>>>>>>>>> So there appears to be something wrong with cygwin1.dll
>>>>>>>>>> built with the current build tools (gcc 11.2.0, binutils
>>>>>>>>>> 2.37, not sure what else is relevant).
>>>>>>>
>>>>>>> Wait a minute...I'll bet this is related to the MEM_EXTENDED_PARAMETER
>>>>>>> initialization problem that was dealt with in commit bdb7991db.
>>>>>>
>>>>>> More data: When I run the test case under gdb, it succeeds.  When I run it
>>>>>> under strace, I see VirtualAlloc2 in fhandler_dev_zero::mmap failing with
>>>>>> windows error 87.
>>>>>
>>>>> Are the const's I added to the MEM_EXTENDED_PARAMETER data invalid,
>>>>> perhaps?
>>>>
>>>> I tried removing them, and I got the same error.  I also tried removing
>>>> static, and I tried removing both static and const.
>>>
>>> BTW, when I reported that the test case succeeds under gdb, that only
>>> happens when I build the test case without optimization.  If I build with
>>> -O2, it fails under gdb also.  [In all my tests, I built cygwin1.dll without
>>> optimization.] This makes no sense to me at all.
>>
>> Good hint.  I found the culprit.  With optimization, the code doesn't
>> set the "Reserved" bits in the first struct of MEM_EXTENDED_PARAMETER
>> to 0.
> 
> No, wait.  I get what you say.  The optimzation settings of the test
> case should have no influence on the code inside the DLL.  That doesn't
> make sense for sure.  However, I ran the testcase under GDB, I could
> reproduce the issue, and I could fix it by setting mmap_ext.Reserved = 0;
> Go figure!

I don't get it, but I can confirm that the problem is fixed.

Thanks.

Ken

Re: mmap failure [was: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?]

[Peter Dons Tychsen]

Hi there,

On Mon, 2021-09-06 at 14:40 -0400, Ken Brown via Cygwin wrote:
> > No, wait.  I get what you say.  The optimzation settings of the test
> > case should have no influence on the code inside the DLL.  That
> > doesn't
> > make sense for sure.  However, I ran the testcase under GDB, I could
> > reproduce the issue, and I could fix it by setting mmap_ext.Reserved
> > = 0;
> > Go figure!
> 
> I don't get it, but I can confirm that the problem is fixed.

That sounds a bit like a voodoo fix, that could quickly regress again.

Here is my 2 cents:

Currently the mmap_ext structure is setup like this:

 215       MEM_EXTENDED_PARAMETER mmap_ext = {
 216         .Type = MemExtendedParameterAddressRequirements,
 217         .Pointer = (PVOID) &mmap_req
 218       };

This means that all other entries in the struct are zero at
initialization as described here:
https://en.cppreference.com/w/c/language/struct_initialization

So if you set "mmap_ext.Reserved = 0" again after that its a double
failure. 

1) The compiler should ignore it as it is redundent.
2) It makes no sense, as it is already zero :-)

Since it is not ignored, the compiler clearly puts in code to to
reinitialize the variable (which some compilers do not optimize for).

The reason this makes it "work" it probably because of some other stack
curruption that is going on which then disappears because of the code
moving around a bit due to the new line. My best guess would be that
other fun stuff in the same location would also "fix" the problem.

These are not the droids you are looking for. The real problem is
elsewhere, and is likely due to some stack-smashing going on. This is
also likely why recompiling the test program makes a difference as that
changes what goes on the variable stack. When the code moves around
again (e.g. new compiler version), it could break again.

Looking at the test program -02 vs -O0:

pushq	%rbp
.seh_pushreg	%rbp
movq	%rsp, %rbp
.seh_setframe	%rbp, 0
subq	$64, %rsp
.seh_stackalloc	64
.seh_endprologue

vs

subq	$56, %rsp
.seh_stackalloc	56

Which gives a different stack layout. I think the problem must be in
the start of mmap() or subsequent calls like CreateMapping() and
MapView(). Something smashes or affects the stack.

Thanks,

/pedro

Re: mmap failure [was: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?]

[Peter Dons Tychsen]

Hi there,

On Mon, 2021-09-06 at 14:40 -0400, Ken Brown via Cygwin wrote:
> > No, wait.  I get what you say.  The optimzation settings of the test
> > case should have no influence on the code inside the DLL.  That
> > doesn't
> > make sense for sure.  However, I ran the testcase under GDB, I could
> > reproduce the issue, and I could fix it by setting mmap_ext.Reserved
> > = 0;
> > Go figure!
> 
> I don't get it, but I can confirm that the problem is fixed.

That sounds a bit like a voodoo fix, that could quickly regress again.

Here is my 2 cents:

Currently the mmap_ext structure is setup like this:

 215       MEM_EXTENDED_PARAMETER mmap_ext = {
 216         .Type = MemExtendedParameterAddressRequirements,
 217         .Pointer = (PVOID) &mmap_req
 218       };

This means that all other entries in the struct are zero at
initialization as described here:
https://en.cppreference.com/w/c/language/struct_initialization

So if you set "mmap_ext.Reserved = 0" again after that its a double
failure. 

1) The compiler should ignore it as it is redundent.
2) It makes no sense, as it is already zero :-)

Since it is not ignored, the compiler clearly puts in code to to
reinitialize the variable (which some compilers do not optimize for).

The reason this makes it "work" it probably because of some other stack
curruption that is going on which then disappears because of the code
moving around a bit due to the new line. My best guess would be that
other fun stuff in the same location would also "fix" the problem.

These are not the droids you are looking for. The real problem is
elsewhere, and is likely due to some stack-smashing going on. This is
also likely why recompiling the test program makes a difference as that
changes what goes on the variable stack. When the code moves around
again (e.g. new compiler version), it could break again.

Looking at the test program -02 vs -O0:

pushq	%rbp
.seh_pushreg	%rbp
movq	%rsp, %rbp
.seh_setframe	%rbp, 0
subq	$64, %rsp
.seh_stackalloc	64
.seh_endprologue

vs

subq	$56, %rsp
.seh_stackalloc	56

Which gives a different stack layout. I think the problem must be in
the start of mmap() or subsequent calls like CreateMapping() and
MapView(). Something smashes or affects the stack.

Thanks,

/pedro

Re: mmap failure [was: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?]

[Ken Brown]

On 9/6/2021 4:54 PM, Peter Dons Tychsen wrote:
> Hi there,
> 
> On Mon, 2021-09-06 at 14:40 -0400, Ken Brown via Cygwin wrote:
>>> No, wait.  I get what you say.  The optimzation settings of the test
>>> case should have no influence on the code inside the DLL.  That
>>> doesn't
>>> make sense for sure.  However, I ran the testcase under GDB, I could
>>> reproduce the issue, and I could fix it by setting mmap_ext.Reserved
>>> = 0;
>>> Go figure!
>>
>> I don't get it, but I can confirm that the problem is fixed.
> 
> That sounds a bit like a voodoo fix, that could quickly regress again.
> 
> Here is my 2 cents:
> 
> Currently the mmap_ext structure is setup like this:
> 
>   215       MEM_EXTENDED_PARAMETER mmap_ext = {
>   216         .Type = MemExtendedParameterAddressRequirements,
>   217         .Pointer = (PVOID) &mmap_req
>   218       };
> 
> This means that all other entries in the struct are zero at
> initialization as described here:
> https://en.cppreference.com/w/c/language/struct_initialization
> 
> So if you set "mmap_ext.Reserved = 0" again after that its a double
> failure.

You're looking at the wrong source code.  The bug didn't occur until the code 
was changed to do the following:

       /* g++ 11.2 workaround: don't use initializer */
       MEM_EXTENDED_PARAMETER mmap_ext;
       mmap_ext.Type = MemExtendedParameterAddressRequirements;
       mmap_ext.Pointer = (PVOID) &mmap_req;

This left mmap_ext.Reserved uninitialized, which Corinna has now fixed.

Ken

Re: mmap failure [was: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?]

[Ken Brown]

On 9/6/2021 5:24 PM, Ken Brown via Cygwin wrote:
> On 9/6/2021 4:54 PM, Peter Dons Tychsen wrote:
>> Hi there,
>>
>> On Mon, 2021-09-06 at 14:40 -0400, Ken Brown via Cygwin wrote:
>>>> No, wait.  I get what you say.  The optimzation settings of the test
>>>> case should have no influence on the code inside the DLL.  That
>>>> doesn't
>>>> make sense for sure.  However, I ran the testcase under GDB, I could
>>>> reproduce the issue, and I could fix it by setting mmap_ext.Reserved
>>>> = 0;
>>>> Go figure!
>>>
>>> I don't get it, but I can confirm that the problem is fixed.
>>
>> That sounds a bit like a voodoo fix, that could quickly regress again.
>>
>> Here is my 2 cents:
>>
>> Currently the mmap_ext structure is setup like this:
>>
>>   215       MEM_EXTENDED_PARAMETER mmap_ext = {
>>   216         .Type = MemExtendedParameterAddressRequirements,
>>   217         .Pointer = (PVOID) &mmap_req
>>   218       };
>>
>> This means that all other entries in the struct are zero at
>> initialization as described here:
>> https://en.cppreference.com/w/c/language/struct_initialization
>>
>> So if you set "mmap_ext.Reserved = 0" again after that its a double
>> failure.
> 
> You're looking at the wrong source code.  The bug didn't occur until the code 
> was changed to do the following:
> 
>        /* g++ 11.2 workaround: don't use initializer */
>        MEM_EXTENDED_PARAMETER mmap_ext;
>        mmap_ext.Type = MemExtendedParameterAddressRequirements;
>        mmap_ext.Pointer = (PVOID) &mmap_req;
> 
> This left mmap_ext.Reserved uninitialized, which Corinna has now fixed.

I should add that when I said "I don't get it", I wasn't referring to Corinna's 
fix.  I was referring to the fact that the bug did *not* occur when the 
unoptimized build was run under gdb.

Ken

Re: mmap failure [was: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?]

[Brian Inglis]

On 2021-09-06 15:24, Ken Brown via Cygwin wrote:
> On 9/6/2021 4:54 PM, Peter Dons Tychsen wrote:
>> Hi there,
>>
>> On Mon, 2021-09-06 at 14:40 -0400, Ken Brown via Cygwin wrote:
>>>> No, wait.  I get what you say.  The optimzation settings of the test
>>>> case should have no influence on the code inside the DLL.  That
>>>> doesn't
>>>> make sense for sure.  However, I ran the testcase under GDB, I could
>>>> reproduce the issue, and I could fix it by setting mmap_ext.Reserved
>>>> = 0;
>>>> Go figure!
>>>
>>> I don't get it, but I can confirm that the problem is fixed.
>>
>> That sounds a bit like a voodoo fix, that could quickly regress again.
>>
>> Here is my 2 cents:
>>
>> Currently the mmap_ext structure is setup like this:
>>
>>   215       MEM_EXTENDED_PARAMETER mmap_ext = {
>>   216         .Type = MemExtendedParameterAddressRequirements,
>>   217         .Pointer = (PVOID) &mmap_req
>>   218       };
>>
>> This means that all other entries in the struct are zero at
>> initialization as described here:
>> https://en.cppreference.com/w/c/language/struct_initialization
>>
>> So if you set "mmap_ext.Reserved = 0" again after that its a double
>> failure.
> 
> You're looking at the wrong source code.  The bug didn't occur until the 
> code was changed to do the following:
> 
>        /* g++ 11.2 workaround: don't use initializer */
>        MEM_EXTENDED_PARAMETER mmap_ext;
>        mmap_ext.Type = MemExtendedParameterAddressRequirements;
>        mmap_ext.Pointer = (PVOID) &mmap_req;
> 
> This left mmap_ext.Reserved uninitialized, which Corinna has now fixed.

With undocumented structure member initialization an issue, maybe better 
to future proof using e.g.

	MEM_EXTENDED_PARAMETER mmap_ext = { 0 }; // or memset or bzero
	...

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]

Re: mmap failure [was: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?]

[Ken Brown]

On 9/6/2021 11:34 PM, Brian Inglis wrote:
> On 2021-09-06 15:24, Ken Brown via Cygwin wrote:
>> On 9/6/2021 4:54 PM, Peter Dons Tychsen wrote:
>>> Hi there,
>>>
>>> On Mon, 2021-09-06 at 14:40 -0400, Ken Brown via Cygwin wrote:
>>>>> No, wait.  I get what you say.  The optimzation settings of the test
>>>>> case should have no influence on the code inside the DLL.  That
>>>>> doesn't
>>>>> make sense for sure.  However, I ran the testcase under GDB, I could
>>>>> reproduce the issue, and I could fix it by setting mmap_ext.Reserved
>>>>> = 0;
>>>>> Go figure!
>>>>
>>>> I don't get it, but I can confirm that the problem is fixed.
>>>
>>> That sounds a bit like a voodoo fix, that could quickly regress again.
>>>
>>> Here is my 2 cents:
>>>
>>> Currently the mmap_ext structure is setup like this:
>>>
>>>   215       MEM_EXTENDED_PARAMETER mmap_ext = {
>>>   216         .Type = MemExtendedParameterAddressRequirements,
>>>   217         .Pointer = (PVOID) &mmap_req
>>>   218       };
>>>
>>> This means that all other entries in the struct are zero at
>>> initialization as described here:
>>> https://en.cppreference.com/w/c/language/struct_initialization
>>>
>>> So if you set "mmap_ext.Reserved = 0" again after that its a double
>>> failure.
>>
>> You're looking at the wrong source code.  The bug didn't occur until the code 
>> was changed to do the following:
>>
>>        /* g++ 11.2 workaround: don't use initializer */
>>        MEM_EXTENDED_PARAMETER mmap_ext;
>>        mmap_ext.Type = MemExtendedParameterAddressRequirements;
>>        mmap_ext.Pointer = (PVOID) &mmap_req;
>>
>> This left mmap_ext.Reserved uninitialized, which Corinna has now fixed.
> 
> With undocumented structure member initialization an issue, maybe better to 
> future proof using e.g.
> 
>      MEM_EXTENDED_PARAMETER mmap_ext = { 0 }; // or memset or bzero

I don't see what this would accomplish.  We're already initializing every member 
after Corinna's last patch.

Ken

RE: mmap failure [was: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?]

[Lavrentiev, Anton (NIH/NLM/NCBI) [C]]

> >
> > With undocumented structure member initialization an issue, maybe better to
> > future proof using e.g.
> >
> >      MEM_EXTENDED_PARAMETER mmap_ext = { 0 }; // or memset or bzero
> 
> I don't see what this would accomplish.  We're already initializing every member
> after Corinna's last patch.
> 

Well, if one day MS decides to use some of the Reserved field by splinting off a new
named field out of that bulk of bits, the code will be in trouble again...
While with memset, bzero or C-style initializer, everything is guaranteed to be zapped with 0.

Anton Lavrentiev
Contractor NIH/NLM/NCBI

Re: mmap failure [was: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?]

[Ken Brown]

On 9/7/2021 5:52 PM, Lavrentiev, Anton (NIH/NLM/NCBI) [C] wrote:
>>>
>>> With undocumented structure member initialization an issue, maybe better to
>>> future proof using e.g.
>>>
>>>       MEM_EXTENDED_PARAMETER mmap_ext = { 0 }; // or memset or bzero
>>
>> I don't see what this would accomplish.  We're already initializing every member
>> after Corinna's last patch.
>>
> 
> Well, if one day MS decides to use some of the Reserved field by splinting off a new
> named field out of that bulk of bits, the code will be in trouble again...
> While with memset, bzero or C-style initializer, everything is guaranteed to be zapped with 0.

MS can't add a new named field to a documented struct without breaking a lot of 
code.  I think it's extremely unlikely that they would do that.  On the other 
hand, I think it's very likely that a reader of the Cygwin code would be 
confused by code that initializes a struct to 0 prior to assigning a value to 
every member.

Ken

Re: mmap failure [was: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?]

[Sam Edge]

On 07/09/2021 23:44, Ken Brown via Cygwin wrote:
 >
 > MS can't add a new named field to a documented struct without
breaking a lot of code.  I think it's extremely unlikely that they would
do that.  On the other hand, I think it's very likely that a reader of
the Cygwin code would be confused by code that initializes a struct to 0
prior to assigning a value to every member.

Hi Ken.

It is common practice when following a coding standard (especially those
for safety or security sensitive code) to initialize each and every
automatic C/C++ variable at the point of definition even if it is later
assigned a value. The '= {0}' mechanism for structures is a common way
of ensuring this is the case for all fields in a structure and should be
familiar to most developers.

This ensures deterministic behaviour instead of bugs that come and go
depending upon what was on the stack or in the register before the
variable is used. It also helps when reviewing code.

A compiler or static analyser cannot always determine whether a variable
is used before it's assigned so getting into the habit of doing this
saves a lot of grief down the line, as demonstrated by this thread!

--
Sam "Here Endeth The Lesson" Edge

Re: mmap failure [was: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?]

[Corinna Vinschen]

On Sep  6 21:34, Brian Inglis wrote:
> On 2021-09-06 15:24, Ken Brown via Cygwin wrote:
> > On 9/6/2021 4:54 PM, Peter Dons Tychsen wrote:
> > > Hi there,
> > > 
> > > On Mon, 2021-09-06 at 14:40 -0400, Ken Brown via Cygwin wrote:
> > > > > No, wait.  I get what you say.  The optimzation settings of the test
> > > > > case should have no influence on the code inside the DLL.  That
> > > > > doesn't
> > > > > make sense for sure.  However, I ran the testcase under GDB, I could
> > > > > reproduce the issue, and I could fix it by setting mmap_ext.Reserved
> > > > > = 0;
> > > > > Go figure!
> > > > 
> > > > I don't get it, but I can confirm that the problem is fixed.
> > > 
> > > That sounds a bit like a voodoo fix, that could quickly regress again.
> > > 
> > > Here is my 2 cents:
> > > 
> > > Currently the mmap_ext structure is setup like this:
> > > 
> > >   215       MEM_EXTENDED_PARAMETER mmap_ext = {
> > >   216         .Type = MemExtendedParameterAddressRequirements,
> > >   217         .Pointer = (PVOID) &mmap_req
> > >   218       };
> > > 
> > > This means that all other entries in the struct are zero at
> > > initialization as described here:
> > > https://en.cppreference.com/w/c/language/struct_initialization
> > > 
> > > So if you set "mmap_ext.Reserved = 0" again after that its a double
> > > failure.
> > 
> > You're looking at the wrong source code.  The bug didn't occur until the
> > code was changed to do the following:
> > 
> >        /* g++ 11.2 workaround: don't use initializer */
> >        MEM_EXTENDED_PARAMETER mmap_ext;
> >        mmap_ext.Type = MemExtendedParameterAddressRequirements;
> >        mmap_ext.Pointer = (PVOID) &mmap_req;
> > 
> > This left mmap_ext.Reserved uninitialized, which Corinna has now fixed.
> 
> With undocumented structure member initialization an issue, maybe better to
> future proof using e.g.
> 
> 	MEM_EXTENDED_PARAMETER mmap_ext = { 0 }; // or memset or bzero
> 	...

You're right.  The bits in Reserved are just that, reserved.  So MSFT
may decide to use some of the  bits for other purposes, should the need
arise.  I've fixed that in git master.


Thanks,
Corinna

Re: mmap failure [was: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?]

[Peter Dons Tychsen]

Hi Ken,

On Mon, 2021-09-06 at 17:24 -0400, Ken Brown via Cygwin wrote:
> You're looking at the wrong source code.  The bug didn't occur until
> the code 
> was changed to do the following:

You are right. I do not know why i looked at an old checkout of the
code. Shame on me! Sorry for wasting your time. Maybe i should stop
looking at code after 24:00. Bad idea! :-).

So with my new shiny view, the fix makes perfectly sense. Great!

/pedro

Re: mmap failure [was: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?]

[Brian Inglis]

On 2021-09-04 16:37, Ken Brown via Cygwin wrote:
> I've reduced the procps failure to the following test case:
> 
> $ cat mmap_test.c
> #include <unistd.h>
> #include <stdio.h>
> #include <sys/mman.h>
> 
> int
> main ()
> {
>    void *addr;
>    int page_size = getpagesize ();
> 
>    addr = mmap (0, page_size, PROT_READ | PROT_WRITE,
>                 MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
>    if (addr == MAP_FAILED)
>      perror ("mmap");
> }
> 
> $ gcc mmap_test.c
> 
> $ ./a
> mmap: Invalid argument

Check on Linux and retry on Cygwin without MAP_PRIVATE?
MAP_PRIVATE implies an fd but MAP_ANON does not, and MAP_ANON is in 
Linux(2)/Unix? not POSIX(3p).

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Takashi Yano]

[-- Attachment #1: Type: text/plain, Size: 2804 bytes --]

On Sun, 29 Aug 2021 11:57:04 -0400
Ken Brown wrote:
> On 8/29/2021 5:07 AM, Takashi Yano via Cygwin wrote:
> > On Sat, 28 Aug 2021 18:41:02 +0900
> > Takashi Yano wrote:
> >> On Sat, 28 Aug 2021 10:43:27 +0200
> >> Corinna Vinschen wrote:
> >>> On Aug 28 02:21, Takashi Yano via Cygwin wrote:
> >>>> On Fri, 27 Aug 2021 12:00:50 -0400
> >>>> Ken Brown wrote:
> >>>>> Two years ago I thought I needed nt_create to avoid problems when calling
> >>>>> set_pipe_non_blocking.  Are you saying that's not an issue?  Is
> >>>>> set_pipe_non_blocking unnecessary?  Is that the point of your modification to
> >>>>> raw_read?
> >>>>
> >>>> Yes. Instead of making windows read function itself non-blocking,
> >>>> it is possible to check if the pipe can be read before read using
> >>>> PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is
> >>>> returned.
> >>>
> >>> The problem is this:
> >>>
> >>>    if (PeekNamedPipe())
> >>>      ReadFile(blocking);
> >>>
> >>> is not atomic.  I. e., if PeekNamedPipe succeeds, nothing keeps another
> >>> thread from draining the pipe between the PeekNamedPipe and the ReadFile
> >>> call.  And as soon as ReadFile runs, it hangs indefinitely and we can't
> >>> stop it via a signal.
> >>
> >> Hmm, you are right. Mutex guard seems to be necessary like pty code
> >> if we go this way.
> > 
> > I have found that set_pipe_non_blocking() succeeds for both read and
> > write pipes if the write pipe is created by CreateNamedPipe() and the
> > read pipe is created by CreateFile() contrary to the current create()
> > code. Therefore, not only nt_create() but also PeekNamedPipe() become
> > unnecessary.
> > 
> > Please see the revised patch attached.
> 
> That's a great idea.
> 
> I've applied your two patches to the topic/pipe branch.  I also rebased it and 
> did a forced push in order to bring in Corinna's loader script fix.  So you'll 
> have to do 'git fetch' and 'git rebase --hard origin/topic/pipe'.
> 
> Does this now fix all known problems with pipes?

Only the small thing remaining is pipe mode. If the pipe mode
is changed to byte mode, the following issue will be solved.
https://cygwin.com/pipermail/cygwin/2021-March/247987.html

How about the simple patch attached?

The comment in pipe code says:
     Note that the write side of the pipe is opened as PIPE_TYPE_MESSAGE.
     This *seems* to more closely mimic Linux pipe behavior and is
     definitely required for pty handling since fhandler_pty_master
     writes to the pipe in chunks, terminated by newline when CANON mode
     is specified.

This mentions about pty behaiviour in canonical mode, however, the
pty pipe is created as message mode even with this patch. Are there
any other reasons that message mode is preferred for pipe?

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

[-- Attachment #2: 0003-Cygwin-pipe-Set-byte-mode-as-default-rather-than-mes.patch --]
[-- Type: application/octet-stream, Size: 1091 bytes --]

From ec2d43f25f7cf4fc3b01bd2e15e3b48e3c9461d1 Mon Sep 17 00:00:00 2001
From: Takashi Yano <takashi.yano@nifty.ne.jp>
Date: Mon, 30 Aug 2021 04:12:56 +0900
Subject: [PATCH 3/3] Cygwin: pipe: Set byte mode as default rather than
 message mode.

Currently, the pipe mode is message by default. This causes a issue
with C# (.NET) programs if they are piped. With this patch, default
pipe mode is changed to byte mode. The pipe mode can be revert to
message mode by setting CYGWIN=nopipe_byte.

Addresses: https://cygwin.com/pipermail/cygwin/2021-March/247987.html
---
 winsup/cygwin/globals.cc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/winsup/cygwin/globals.cc b/winsup/cygwin/globals.cc
index 48fb312de..3534381ec 100644
--- a/winsup/cygwin/globals.cc
+++ b/winsup/cygwin/globals.cc
@@ -69,7 +69,7 @@ int NO_COPY dynamically_loaded;
 /* Some CYGWIN environment variable variables. */
 bool allow_glob = true;
 bool ignore_case_with_glob;
-bool pipe_byte;
+bool pipe_byte = true;
 bool reset_com;
 bool wincmdln;
 winsym_t allow_winsymlinks = WSYM_default;
-- 
2.33.0

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Ken Brown]

On 8/29/2021 3:37 PM, Takashi Yano wrote:
> On Sun, 29 Aug 2021 11:57:04 -0400
> Ken Brown wrote:
>> On 8/29/2021 5:07 AM, Takashi Yano via Cygwin wrote:
>>> On Sat, 28 Aug 2021 18:41:02 +0900
>>> Takashi Yano wrote:
>>>> On Sat, 28 Aug 2021 10:43:27 +0200
>>>> Corinna Vinschen wrote:
>>>>> On Aug 28 02:21, Takashi Yano via Cygwin wrote:
>>>>>> On Fri, 27 Aug 2021 12:00:50 -0400
>>>>>> Ken Brown wrote:
>>>>>>> Two years ago I thought I needed nt_create to avoid problems when calling
>>>>>>> set_pipe_non_blocking.  Are you saying that's not an issue?  Is
>>>>>>> set_pipe_non_blocking unnecessary?  Is that the point of your modification to
>>>>>>> raw_read?
>>>>>>
>>>>>> Yes. Instead of making windows read function itself non-blocking,
>>>>>> it is possible to check if the pipe can be read before read using
>>>>>> PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is
>>>>>> returned.
>>>>>
>>>>> The problem is this:
>>>>>
>>>>>     if (PeekNamedPipe())
>>>>>       ReadFile(blocking);
>>>>>
>>>>> is not atomic.  I. e., if PeekNamedPipe succeeds, nothing keeps another
>>>>> thread from draining the pipe between the PeekNamedPipe and the ReadFile
>>>>> call.  And as soon as ReadFile runs, it hangs indefinitely and we can't
>>>>> stop it via a signal.
>>>>
>>>> Hmm, you are right. Mutex guard seems to be necessary like pty code
>>>> if we go this way.
>>>
>>> I have found that set_pipe_non_blocking() succeeds for both read and
>>> write pipes if the write pipe is created by CreateNamedPipe() and the
>>> read pipe is created by CreateFile() contrary to the current create()
>>> code. Therefore, not only nt_create() but also PeekNamedPipe() become
>>> unnecessary.
>>>
>>> Please see the revised patch attached.
>>
>> That's a great idea.
>>
>> I've applied your two patches to the topic/pipe branch.  I also rebased it and
>> did a forced push in order to bring in Corinna's loader script fix.  So you'll
>> have to do 'git fetch' and 'git rebase --hard origin/topic/pipe'.
>>
>> Does this now fix all known problems with pipes?
> 
> Only the small thing remaining is pipe mode. If the pipe mode
> is changed to byte mode, the following issue will be solved.
> https://cygwin.com/pipermail/cygwin/2021-March/247987.html
> 
> How about the simple patch attached?
> 
> The comment in pipe code says:
>       Note that the write side of the pipe is opened as PIPE_TYPE_MESSAGE.
>       This *seems* to more closely mimic Linux pipe behavior and is
>       definitely required for pty handling since fhandler_pty_master
>       writes to the pipe in chunks, terminated by newline when CANON mode
>       is specified.
> 
> This mentions about pty behaiviour in canonical mode, however, the
> pty pipe is created as message mode even with this patch. Are there
> any other reasons that message mode is preferred for pipe?

No idea.  All I remember is that there was a lot of discussion around the time 
that it was decided to use PIPE_TYPE_MESSAGE by default.  Corinna probably 
remembers the reasons.

Ken

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Ken Brown]

On 8/29/2021 5:07 AM, Takashi Yano via Cygwin wrote:
> On Sat, 28 Aug 2021 18:41:02 +0900
> Takashi Yano wrote:
>> On Sat, 28 Aug 2021 10:43:27 +0200
>> Corinna Vinschen wrote:
>>> On Aug 28 02:21, Takashi Yano via Cygwin wrote:
>>>> On Fri, 27 Aug 2021 12:00:50 -0400
>>>> Ken Brown wrote:
>>>>> Two years ago I thought I needed nt_create to avoid problems when calling
>>>>> set_pipe_non_blocking.  Are you saying that's not an issue?  Is
>>>>> set_pipe_non_blocking unnecessary?  Is that the point of your modification to
>>>>> raw_read?
>>>>
>>>> Yes. Instead of making windows read function itself non-blocking,
>>>> it is possible to check if the pipe can be read before read using
>>>> PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is
>>>> returned.
>>>
>>> The problem is this:
>>>
>>>    if (PeekNamedPipe())
>>>      ReadFile(blocking);
>>>
>>> is not atomic.  I. e., if PeekNamedPipe succeeds, nothing keeps another
>>> thread from draining the pipe between the PeekNamedPipe and the ReadFile
>>> call.  And as soon as ReadFile runs, it hangs indefinitely and we can't
>>> stop it via a signal.
>>
>> Hmm, you are right. Mutex guard seems to be necessary like pty code
>> if we go this way.
> 
> I have found that set_pipe_non_blocking() succeeds for both read and
> write pipes if the write pipe is created by CreateNamedPipe() and the
> read pipe is created by CreateFile() contrary to the current create()
> code. Therefore, not only nt_create() but also PeekNamedPipe() become
> unnecessary.
> 
> Please see the revised patch attached.

I haven't had a chance to test this myself yet, but occurs to me that we might 
have a different problem after this patch: Does the write handle that we get 
from CreateNamedPipe() have FILE_READ_ATTRIBUTES access?

Ken

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Takashi Yano]

On Sun, 29 Aug 2021 17:04:56 -0400
Ken Brown wrote:
> On 8/29/2021 5:07 AM, Takashi Yano via Cygwin wrote:
> > On Sat, 28 Aug 2021 18:41:02 +0900
> > Takashi Yano wrote:
> >> On Sat, 28 Aug 2021 10:43:27 +0200
> >> Corinna Vinschen wrote:
> >>> On Aug 28 02:21, Takashi Yano via Cygwin wrote:
> >>>> On Fri, 27 Aug 2021 12:00:50 -0400
> >>>> Ken Brown wrote:
> >>>>> Two years ago I thought I needed nt_create to avoid problems when calling
> >>>>> set_pipe_non_blocking.  Are you saying that's not an issue?  Is
> >>>>> set_pipe_non_blocking unnecessary?  Is that the point of your modification to
> >>>>> raw_read?
> >>>>
> >>>> Yes. Instead of making windows read function itself non-blocking,
> >>>> it is possible to check if the pipe can be read before read using
> >>>> PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is
> >>>> returned.
> >>>
> >>> The problem is this:
> >>>
> >>>    if (PeekNamedPipe())
> >>>      ReadFile(blocking);
> >>>
> >>> is not atomic.  I. e., if PeekNamedPipe succeeds, nothing keeps another
> >>> thread from draining the pipe between the PeekNamedPipe and the ReadFile
> >>> call.  And as soon as ReadFile runs, it hangs indefinitely and we can't
> >>> stop it via a signal.
> >>
> >> Hmm, you are right. Mutex guard seems to be necessary like pty code
> >> if we go this way.
> > 
> > I have found that set_pipe_non_blocking() succeeds for both read and
> > write pipes if the write pipe is created by CreateNamedPipe() and the
> > read pipe is created by CreateFile() contrary to the current create()
> > code. Therefore, not only nt_create() but also PeekNamedPipe() become
> > unnecessary.
> > 
> > Please see the revised patch attached.
> 
> I haven't had a chance to test this myself yet, but occurs to me that we might 
> have a different problem after this patch: Does the write handle that we get 
> from CreateNamedPipe() have FILE_READ_ATTRIBUTES access?

I have just checked this, and the answer is "No". Due to this problem,
NtQueryInformationFile() call in select() fails on the write pipe.

It seems that we need more consideration...

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Takashi Yano]

On Mon, 30 Aug 2021 09:13:14 +0900
Takashi Yano wrote:
> On Sun, 29 Aug 2021 17:04:56 -0400
> Ken Brown wrote:
> > On 8/29/2021 5:07 AM, Takashi Yano via Cygwin wrote:
> > > On Sat, 28 Aug 2021 18:41:02 +0900
> > > Takashi Yano wrote:
> > >> On Sat, 28 Aug 2021 10:43:27 +0200
> > >> Corinna Vinschen wrote:
> > >>> On Aug 28 02:21, Takashi Yano via Cygwin wrote:
> > >>>> On Fri, 27 Aug 2021 12:00:50 -0400
> > >>>> Ken Brown wrote:
> > >>>>> Two years ago I thought I needed nt_create to avoid problems when calling
> > >>>>> set_pipe_non_blocking.  Are you saying that's not an issue?  Is
> > >>>>> set_pipe_non_blocking unnecessary?  Is that the point of your modification to
> > >>>>> raw_read?
> > >>>>
> > >>>> Yes. Instead of making windows read function itself non-blocking,
> > >>>> it is possible to check if the pipe can be read before read using
> > >>>> PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is
> > >>>> returned.
> > >>>
> > >>> The problem is this:
> > >>>
> > >>>    if (PeekNamedPipe())
> > >>>      ReadFile(blocking);
> > >>>
> > >>> is not atomic.  I. e., if PeekNamedPipe succeeds, nothing keeps another
> > >>> thread from draining the pipe between the PeekNamedPipe and the ReadFile
> > >>> call.  And as soon as ReadFile runs, it hangs indefinitely and we can't
> > >>> stop it via a signal.
> > >>
> > >> Hmm, you are right. Mutex guard seems to be necessary like pty code
> > >> if we go this way.
> > > 
> > > I have found that set_pipe_non_blocking() succeeds for both read and
> > > write pipes if the write pipe is created by CreateNamedPipe() and the
> > > read pipe is created by CreateFile() contrary to the current create()
> > > code. Therefore, not only nt_create() but also PeekNamedPipe() become
> > > unnecessary.
> > > 
> > > Please see the revised patch attached.
> > 
> > I haven't had a chance to test this myself yet, but occurs to me that we might 
> > have a different problem after this patch: Does the write handle that we get 
> > from CreateNamedPipe() have FILE_READ_ATTRIBUTES access?
> 
> I have just checked this, and the answer is "No". Due to this problem,
> NtQueryInformationFile() call in select() fails on the write pipe.
> 
> It seems that we need more consideration...

We have two easy options:
1) Configure the pipe with PIPE_ACCESS_DUPLEX.
2) Use nt_create() again and forget C# program issue.



Even without this problem, select() for writing pipe has a bug
and does not wrok as expected. The following patch seems to be
needed.

diff --git a/winsup/cygwin/select.cc b/winsup/cygwin/select.cc
index 83e1c00e0..ac2fd227e 100644
--- a/winsup/cygwin/select.cc
+++ b/winsup/cygwin/select.cc
@@ -612,7 +612,6 @@ pipe_data_available (int fd, fhandler_base *fh, HANDLE h, bool writing)
           that.  This means that a pipe could still block since you could
           be trying to write more to the pipe than is available in the
           buffer but that is the hazard of select().  */
-      fpli.WriteQuotaAvailable = fpli.OutboundQuota - fpli.ReadDataAvailable;
       if (fpli.WriteQuotaAvailable > 0)
        {
          paranoid_printf ("fd %d, %s, write: size %u, avail %u", fd,

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Ken Brown]

On 8/29/2021 8:22 PM, Takashi Yano via Cygwin wrote:
> On Mon, 30 Aug 2021 09:13:14 +0900
> Takashi Yano wrote:
>> On Sun, 29 Aug 2021 17:04:56 -0400
>> Ken Brown wrote:
>>> On 8/29/2021 5:07 AM, Takashi Yano via Cygwin wrote:
>>>> On Sat, 28 Aug 2021 18:41:02 +0900
>>>> Takashi Yano wrote:
>>>>> On Sat, 28 Aug 2021 10:43:27 +0200
>>>>> Corinna Vinschen wrote:
>>>>>> On Aug 28 02:21, Takashi Yano via Cygwin wrote:
>>>>>>> On Fri, 27 Aug 2021 12:00:50 -0400
>>>>>>> Ken Brown wrote:
>>>>>>>> Two years ago I thought I needed nt_create to avoid problems when calling
>>>>>>>> set_pipe_non_blocking.  Are you saying that's not an issue?  Is
>>>>>>>> set_pipe_non_blocking unnecessary?  Is that the point of your modification to
>>>>>>>> raw_read?
>>>>>>>
>>>>>>> Yes. Instead of making windows read function itself non-blocking,
>>>>>>> it is possible to check if the pipe can be read before read using
>>>>>>> PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is
>>>>>>> returned.
>>>>>>
>>>>>> The problem is this:
>>>>>>
>>>>>>     if (PeekNamedPipe())
>>>>>>       ReadFile(blocking);
>>>>>>
>>>>>> is not atomic.  I. e., if PeekNamedPipe succeeds, nothing keeps another
>>>>>> thread from draining the pipe between the PeekNamedPipe and the ReadFile
>>>>>> call.  And as soon as ReadFile runs, it hangs indefinitely and we can't
>>>>>> stop it via a signal.
>>>>>
>>>>> Hmm, you are right. Mutex guard seems to be necessary like pty code
>>>>> if we go this way.
>>>>
>>>> I have found that set_pipe_non_blocking() succeeds for both read and
>>>> write pipes if the write pipe is created by CreateNamedPipe() and the
>>>> read pipe is created by CreateFile() contrary to the current create()
>>>> code. Therefore, not only nt_create() but also PeekNamedPipe() become
>>>> unnecessary.
>>>>
>>>> Please see the revised patch attached.
>>>
>>> I haven't had a chance to test this myself yet, but occurs to me that we might
>>> have a different problem after this patch: Does the write handle that we get
>>> from CreateNamedPipe() have FILE_READ_ATTRIBUTES access?
>>
>> I have just checked this, and the answer is "No". Due to this problem,
>> NtQueryInformationFile() call in select() fails on the write pipe.
>>
>> It seems that we need more consideration...
> 
> We have two easy options:
> 1) Configure the pipe with PIPE_ACCESS_DUPLEX.
> 2) Use nt_create() again and forget C# program issue.

I vote for 2), but let's see what Corinna thinks.

> Even without this problem, select() for writing pipe has a bug
> and does not wrok as expected. The following patch seems to be
> needed.
> 
> diff --git a/winsup/cygwin/select.cc b/winsup/cygwin/select.cc
> index 83e1c00e0..ac2fd227e 100644
> --- a/winsup/cygwin/select.cc
> +++ b/winsup/cygwin/select.cc
> @@ -612,7 +612,6 @@ pipe_data_available (int fd, fhandler_base *fh, HANDLE h, bool writing)
>             that.  This means that a pipe could still block since you could
>             be trying to write more to the pipe than is available in the
>             buffer but that is the hazard of select().  */
> -      fpli.WriteQuotaAvailable = fpli.OutboundQuota - fpli.ReadDataAvailable;
>         if (fpli.WriteQuotaAvailable > 0)
>          {
>            paranoid_printf ("fd %d, %s, write: size %u, avail %u", fd,
> 

I agree.

Ken

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Takashi Yano]

On Sun, 29 Aug 2021 22:15:29 -0400
Ken Brown wrote:
> On 8/29/2021 8:22 PM, Takashi Yano via Cygwin wrote:
> > On Mon, 30 Aug 2021 09:13:14 +0900
> > Takashi Yano wrote:
> >> On Sun, 29 Aug 2021 17:04:56 -0400
> >> Ken Brown wrote:
> >>> On 8/29/2021 5:07 AM, Takashi Yano via Cygwin wrote:
> >>>> On Sat, 28 Aug 2021 18:41:02 +0900
> >>>> Takashi Yano wrote:
> >>>>> On Sat, 28 Aug 2021 10:43:27 +0200
> >>>>> Corinna Vinschen wrote:
> >>>>>> On Aug 28 02:21, Takashi Yano via Cygwin wrote:
> >>>>>>> On Fri, 27 Aug 2021 12:00:50 -0400
> >>>>>>> Ken Brown wrote:
> >>>>>>>> Two years ago I thought I needed nt_create to avoid problems when calling
> >>>>>>>> set_pipe_non_blocking.  Are you saying that's not an issue?  Is
> >>>>>>>> set_pipe_non_blocking unnecessary?  Is that the point of your modification to
> >>>>>>>> raw_read?
> >>>>>>>
> >>>>>>> Yes. Instead of making windows read function itself non-blocking,
> >>>>>>> it is possible to check if the pipe can be read before read using
> >>>>>>> PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is
> >>>>>>> returned.
> >>>>>>
> >>>>>> The problem is this:
> >>>>>>
> >>>>>>     if (PeekNamedPipe())
> >>>>>>       ReadFile(blocking);
> >>>>>>
> >>>>>> is not atomic.  I. e., if PeekNamedPipe succeeds, nothing keeps another
> >>>>>> thread from draining the pipe between the PeekNamedPipe and the ReadFile
> >>>>>> call.  And as soon as ReadFile runs, it hangs indefinitely and we can't
> >>>>>> stop it via a signal.
> >>>>>
> >>>>> Hmm, you are right. Mutex guard seems to be necessary like pty code
> >>>>> if we go this way.
> >>>>
> >>>> I have found that set_pipe_non_blocking() succeeds for both read and
> >>>> write pipes if the write pipe is created by CreateNamedPipe() and the
> >>>> read pipe is created by CreateFile() contrary to the current create()
> >>>> code. Therefore, not only nt_create() but also PeekNamedPipe() become
> >>>> unnecessary.
> >>>>
> >>>> Please see the revised patch attached.
> >>>
> >>> I haven't had a chance to test this myself yet, but occurs to me that we might
> >>> have a different problem after this patch: Does the write handle that we get
> >>> from CreateNamedPipe() have FILE_READ_ATTRIBUTES access?
> >>
> >> I have just checked this, and the answer is "No". Due to this problem,
> >> NtQueryInformationFile() call in select() fails on the write pipe.
> >>
> >> It seems that we need more consideration...
> > 
> > We have two easy options:
> > 1) Configure the pipe with PIPE_ACCESS_DUPLEX.
> > 2) Use nt_create() again and forget C# program issue.
> 
> I vote for 2), but let's see what Corinna thinks.

BTW. what's wrong if just:

static int
nt_create (LPSECURITY_ATTRIBUTES sa_ptr, PHANDLE r, PHANDLE w,
                DWORD psize, int64_t *unique_id)
{
  if (r && w)
    {
      static volatile ULONG pipe_unique_id;
      LONG id = InterlockedIncrement ((LONG *) &pipe_unique_id);
      if (unique_id)
        *unique_id = ((int64_t) id << 32 | GetCurrentProcessId ());
      if (!CreatePipe (r, w, sa_ptr, psize))
        {
          *r = *w = NULL;
          return GetLastError ();
        }
    }
  return 0;
}

?

In my environment, I cannot find any defects.
- No performance degradation.
- set_pipe_non_blocking() works for both read and write pipes.
- NtQueryInformationFile() in select() works for both r/w pipes.
- Piping C# program works.

Is naming the pipe really necessary?

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Ken Brown]

On 8/28/2021 4:43 AM, Corinna Vinschen via Cygwin wrote:
> On Aug 28 02:21, Takashi Yano via Cygwin wrote:
>> On Fri, 27 Aug 2021 12:00:50 -0400
>> Ken Brown wrote:
>>> Two years ago I thought I needed nt_create to avoid problems when calling
>>> set_pipe_non_blocking.  Are you saying that's not an issue?  Is
>>> set_pipe_non_blocking unnecessary?  Is that the point of your modification to
>>> raw_read?
>>
>> Yes. Instead of making windows read function itself non-blocking,
>> it is possible to check if the pipe can be read before read using
>> PeekNamedPipe(). If the pipe cannot be read right now, EAGAIN is
>> returned.
> 
> The problem is this:
> 
>    if (PeekNamedPipe())
>      ReadFile(blocking);
> 
> is not atomic.  I. e., if PeekNamedPipe succeeds, nothing keeps another
> thread from draining the pipe between the PeekNamedPipe and the ReadFile
> call.  And as soon as ReadFile runs, it hangs indefinitely and we can't
> stop it via a signal.
> 
> Is a blocking ReadFile actually faster than a non-blocking read?  Or
> does it mainly depend on BYTE vs. MESSAGE mode?
> 
> What if the pipe is created non-blocking and stays non-blocking all the
> time and uses BYTE mode all the time?  Just as sockets, it would always
> only emulate blocking mode.  Wouldn't that drop code size a lot and fix
> most problems?

It would certainly reduce the code size.  I think it's worth a try, and we can 
see if any problems remain.

Ken

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Keith Christian]

I've been following his thread with interest both here and on the Cygwin
developers list.  I, too rsync between Cygwin and Linux machines.

Lots of good debate showing the genius of the folks that support the cygwin
infrastructure.

Looks like the lively discussion on both lists has stopped.  What is the
final consensus?

Should we look for a new updates to the cygwin dll, cygrunsrv, ssh, rsync,
etc?

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Ken Brown]

On 9/16/2021 6:00 PM, Keith Christian via Cygwin wrote:
> I've been following his thread with interest both here and on the Cygwin
> developers list.  I, too rsync between Cygwin and Linux machines.
> 
> Lots of good debate showing the genius of the folks that support the cygwin
> infrastructure.
> 
> Looks like the lively discussion on both lists has stopped.  What is the
> final consensus?
> 
> Should we look for a new updates to the cygwin dll, cygrunsrv, ssh, rsync,
> etc?

All that's needed is an update to the cygwin package.  There should be a test 
release coming very soon, possibly tomorrow if nothing unexpected shows up.

Ken

Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled?

[Keith Christian]

Ken,

Thank you for the reply, this is great news!

Looking forward to speedier rsyncs with Cygwin.

------------  Keith


On Thu, Sep 16, 2021, 16:49 Ken Brown via Cygwin <cygwin@cygwin.com> wrote:

> On 9/16/2021 6:00 PM, Keith Christian via Cygwin wrote:
> > I've been following his thread with interest both here and on the Cygwin
> > developers list.  I, too rsync between Cygwin and Linux machines.
> >
> > Lots of good debate showing the genius of the folks that support the
> cygwin
> > infrastructure.
> >
> > Looks like the lively discussion on both lists has stopped.  What is the
> > final consensus?
> >
> > Should we look for a new updates to the cygwin dll, cygrunsrv, ssh,
> rsync,
> > etc?
>
> All that's needed is an update to the cygwin package.  There should be a
> test
> release coming very soon, possibly tomorrow if nothing unexpected shows up.
>
> Ken
>
> --
> Problem reports:      https://cygwin.com/problems.html
> FAQ:                  https://cygwin.com/faq/
> Documentation:        https://cygwin.com/docs.html
> Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple
>