From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2126.outbound.protection.outlook.com [40.107.237.126]) by sourceware.org (Postfix) with ESMTPS id 26F683858D3C for ; Thu, 26 Aug 2021 15:56:10 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 26F683858D3C ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=HLcr4TSaN5aE00/iYIW69IowMxlyCZOIlhAYenhi8XDB8CROzVFz+pBes5gqpaeFVjsxrtja9bIhotMzJQUWQ9CdETCTLWFwPZzIBA8L6kqJrdF/fbGKMp9a0pOQODXRSP6RwOpjcFOoGttAY+3xrqW9za3gIZgOzryOwm44C9lSU82Lac+bShlOtxOXHqbELFvdRwcaLV0c5wjfs/PAS4ylxz8lXz69w+K/dR+OqjcZff6OPQ57mxGhJkAabnlF1PjBo13Yy5gOvBDHC0GwtzEnlUsGjOeO6ZOcuCbBZXlAGePzlX/s5V7lRFSRk3KQjfX+Sk8+JJpOXOHXZaiK/A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=j9nMTQf6UA8g3fai0Ysks0W4w3akjEoKeSkQ0eGILqk=; b=QdrQR4UfoTFxzvLRtcDQXh0BUe4oGP1XcneIQrzTheOcvCwknubkoWxeXCUo0wXXM1F4oL0uCtY0CbAvPhWJCuNtd1Ups2/JZJJw0i4qr/xcH5Nh8KLT+vrMTcaXjH5HXrCa0ZHGWsaPLq8GiA4e4UUn3gnIwBgyKfD8tceiFiWv6w85B2w3fkCxtvk0Kdyfkpry2AeBb1eJ7KznZAzQYV5ZhpT8kudd2LOpKFSK0cTY8C6Q5VujJR62ezZYF9HIqJ7g71GLsPsX7rDXmg/jFCGmJ57QYns77WiYSgO8yfHbUoi683dqPgLpwqtvnuVf5l+0IWsEg+CBd/qZkJYTEA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=cornell.edu; dmarc=pass action=none header.from=cornell.edu; dkim=pass header.d=cornell.edu; arc=none Received: from BN7PR04MB4388.namprd04.prod.outlook.com (2603:10b6:406:f8::19) by BN6PR04MB0899.namprd04.prod.outlook.com (2603:10b6:405:40::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4436.19; Thu, 26 Aug 2021 15:56:09 +0000 Received: from BN7PR04MB4388.namprd04.prod.outlook.com ([fe80::8d9d:335b:d8e4:79fe]) by BN7PR04MB4388.namprd04.prod.outlook.com ([fe80::8d9d:335b:d8e4:79fe%7]) with mapi id 15.20.4436.027; Thu, 26 Aug 2021 15:56:09 +0000 Subject: Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled? To: cygwin@cygwin.com References: <41A583E1-C8E7-42AB-9F24-EEC33A41EC60@house.org> <20210825201845.07b6400b79dc5558a7761efe@nifty.ne.jp> <20210826062934.54f2f2216021c095bb7ba13b@nifty.ne.jp> From: Ken Brown Message-ID: Date: Thu, 26 Aug 2021 11:56:07 -0400 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.13.0 In-Reply-To: <20210826062934.54f2f2216021c095bb7ba13b@nifty.ne.jp> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-ClientProxiedBy: CH2PR07CA0055.namprd07.prod.outlook.com (2603:10b6:610:5b::29) To BN7PR04MB4388.namprd04.prod.outlook.com (2603:10b6:406:f8::19) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from [IPv6:2603:7081:7e3f:3419:b191:35f5:6937:52a] (2603:7081:7e3f:3419:b191:35f5:6937:52a) by CH2PR07CA0055.namprd07.prod.outlook.com (2603:10b6:610:5b::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4457.18 via Frontend Transport; Thu, 26 Aug 2021 15:56:08 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 0a15499c-7ca7-4aca-1442-08d968aa0502 X-MS-TrafficTypeDiagnostic: BN6PR04MB0899: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:3968; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: j6LG3n3QPbD/gFBRTOWIZHkgT5dZT9ymIl4aRepbpwPKkLFxvilWHn0accZb8LcDwHFfB0/I7f8v+WXkw1727OXLChVSpsm4F+5tEESkP1yqQuNd+aoR1Bolqekr3YKRnTVTeE6CVRb0Yvk+ihq76GGABO98ckxSbh5z+J630ZExI6nDaX8C7Lnj5vEmTVkTBZi83YXFFlxI8cgeL1zgRzVYJo/o1b+Ie3m0J71iMqtWJjn1ohbx7UKl0IMXQHrxWgwHDfLGBe73ZZtZ587wrtwuZObTKBWaBlxY6rsx9+O1ajXiQ8lPS/qQ0aiLQseC8wyN+wjh7xyheibDhgYheK7OFeENRktes/iIo8qlbUOHyFV21GCpgThTPmpMtKGOyimHObRVd2e2QpZ+jSMIHGc0PgBoVmXd68Jpkjm4/IlVFHrL970SiEf2bliZsaA37kgQ1UrXZ1xKACM9FND2s45Ut9Wo44IkQPYZgh3VaoJslU1vktetp0BqbgBrVIvtaL5LcuNev8O2nCyAuw8DkUwfEXNNAXxWfRx1BtvW90iFlME+ZUIl57MxaqvfZxGfFtiZCdCEktEWu4yRv2YClEEh/IExBwFEx+c3kmwDfBWcQtUao23tg0jpP7eRO+l/DG9CDcg41C6vu0bcXq8JTJ38Ai1ytn5I0Q06ZKQjTVfEGBDIUVCgp4DaI7lGGDjxGXxRSuL9L0vLx54Y1WdvGAN2ORB3vkgvCNjxrNew+0RzKJvKsReGWCcgCxUegSsADJehoHG5xS2Ubatq++3YeCw5/8RECqKy8Xn7cfiSqK0QdG8dDbs2K0VFNthd3H6iO+LMQU15gYAI51t3RdcI7w== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BN7PR04MB4388.namprd04.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(396003)(136003)(346002)(366004)(376002)(39860400002)(66946007)(8936002)(186003)(478600001)(75432002)(6916009)(66556008)(2616005)(66476007)(6486002)(36756003)(31686004)(83380400001)(31696002)(316002)(38100700002)(966005)(786003)(53546011)(8676002)(86362001)(2906002)(5660300002)(45980500001)(43740500002)(460985005); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?Windows-1252?Q?sNJ8zvEqY+MzxQBXnr2nS4piH7P2v09HVrBSAYP2whWF7cq7p4B5S6dp?= =?Windows-1252?Q?5kth+67kzuBlSb313zMI0qtyyKwWPzoAznjqZgH4eR6vHIHDuc8PFZEt?= =?Windows-1252?Q?sr8KqTN/XjVmgyHNYXKVri/2ChVahf8WyQxPSLv8odTijTsAMksIt8ka?= =?Windows-1252?Q?+27x44R6B33c8Qc+onu/5x1fF6XrXWAs2YZf++VhwxWMCYKUUixxVv1/?= =?Windows-1252?Q?gTlOgs6e7235cSy+B+HsMa60HTanx2QjbAnzbOeDrQ48pAf/dTobpgW2?= =?Windows-1252?Q?D5Kxm+4x1lsY0WnXcl+yHZuXUSP1PC7+GCT6PFj+4zKfjw0aasO9m6L+?= =?Windows-1252?Q?KkKpmdCWXIfqTXZfwXia4NGxcacKMCpgDUO7HQVjY2rQQGaBTeOxGFeP?= =?Windows-1252?Q?V++dtNVeGb4ZNVtk/kvVyPVJt0reKRmXF99gfLDnDRBDI14eBwQN36W3?= =?Windows-1252?Q?c7nZiuFE9XKGk+5WlFtHrXPrGADOEsDTTdU+8vhphWlXwfhX5W5148rL?= =?Windows-1252?Q?wE1+aHkmztbsIAjhPtzzL4wusHmS4pEEVK5LM0Ve0wBLRSZ7XLsF5Ryt?= =?Windows-1252?Q?FzwY+mI6/q57Q8RzT07Tx3U88N51PQSQ4rf6K8yzm0HcAVY/bXK3J4I2?= =?Windows-1252?Q?dt3I1Zv0A3rE6dFNocKt6lJRXKXHGbHwz2sRufuAD7du/YSu66xiKPYW?= =?Windows-1252?Q?Np3bh/p67L2RggXy9+Xcnd8cIAxySr6PNjdefXz2sw9CEC7gEN8rDLlG?= =?Windows-1252?Q?hcQa2UKBTGjs+fyxJicxj9K6hrRm9/2asxx7nT+KsQrDtjZ4xT8ZluX3?= =?Windows-1252?Q?XB4ZFqBnyO9A2+kt3WXMRmfBQ+jdveXJ+fQ5Lq5gNCPGk3Qxf7jEyy9r?= =?Windows-1252?Q?yiv1Zwq3L9Ds8z4pW55zh8Y1YAwNSKWbGeW/Vz98aErGZwhwDxpPBgwd?= =?Windows-1252?Q?sHafLSqvJVChDBOy5NgiBzjtYalu7aK7EvwJ3K6Oeh8eoO6DbONjG0My?= =?Windows-1252?Q?F7TQuLkiYN2VfeOQDFs8gF3LO8n+CjOXPnulHF7M5TomlHrsLAEJCmRd?= =?Windows-1252?Q?yLTYla9TGZydhlqso6uVyfnofevfm1CQd4Rd56XfHU7SHtTGD6gsmlgP?= =?Windows-1252?Q?r6NA1DoyDiiQP0bISNk32vuKa1qXT3sH/7NGuWlXGp6fH5FH+WpWQKaG?= =?Windows-1252?Q?ca3+qp+TTHxvDD9rChT62Nwl02IVwhbVZ0hb3psAEjBXqLdSEkHbRN1T?= =?Windows-1252?Q?BlLcwND3POowdTMPzAn4lBKGqIXQjEaE+bgk9VdvZ/4Wt5V2yLSWo3Dr?= =?Windows-1252?Q?HrY5ZFhxxnoShiM0OFMEfvZ8fyh8YzYP3R4g5nnv3tf9OjZzm7tAKu0q?= =?Windows-1252?Q?Hcn+BgLfqCFLLG757mpDA8TbIVwGGx8tqH2Oni3jWA3SwaL+Kqo1DP/l?= =?Windows-1252?Q?pTYwcDs1+ZaTcwfmzwHr5tfR0BJb5isMaP75gg5hCluRwolZh/AStKX9?= =?Windows-1252?Q?1uIysW+m?= X-OriginatorOrg: cornell.edu X-MS-Exchange-CrossTenant-Network-Message-Id: 0a15499c-7ca7-4aca-1442-08d968aa0502 X-MS-Exchange-CrossTenant-AuthSource: BN7PR04MB4388.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Aug 2021 15:56:08.9006 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 5d7e4366-1b9b-45cf-8e79-b14b27df46e1 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: SAlIfgwFKvkLwVhICBiG5xrO7ewZ4W1ic5yJLWdkrFUaNlp6DiewC9zp3uupnc+8VBW+QDM/EEvyrpopTf9qkg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR04MB0899 X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, MSGID_FROM_MTA_HEADER, NICE_REPLY_A, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_PASS, TXREP, T_SPF_HELO_TEMPERROR autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: cygwin@cygwin.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: General Cygwin discussions and problem reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Aug 2021 15:56:20 -0000 On 8/25/2021 5:29 PM, Takashi Yano wrote: > On Wed, 25 Aug 2021 13:52:19 -0400 > Ken Brown wrote: >> On 8/25/2021 7:18 AM, Takashi Yano via Cygwin wrote: >>> On Tue, 24 Aug 2021 12:49:52 -0700 >>> Chris Roehrig wrote: >>>> I have a network of Windows, Linux and Mac machines and I use rsync to synchronize various directories between them. >>>> >>>> I'm trying to figure out why my rsync transfers are so slow (<4 MB/s) only when the remote endpoint is Cygwin rsync over sshd (with both a Linux or Cygwin rsync client). In all other scenarios, I get the full 100MB/s as expected from gigabit ethernet. This has been an ongoing problem for me for a couple of years over several Windows and Cygwin versions, and I'd like to try to fix it. >>>> >>>> If I run rsync --daemon --no-detach under mintty in the foreground on the remote Windows endpoint, I get the full 100 MB/s transfers, so it seems like it has something to do with rsync.exe running in the background under the cygrunsrv+sshd service (which was installed normally using ssh-host-config). >>>> >>>> If I do: >>>> pv /dev/zero | ssh $WINHOST "cat > /dev/null" >>>> or even >>>> pv /dev/urandom | ssh $WINHOST md5sum >>>> I also get the full 100 MB/s transfers, so it doesn't look like sshd itself is being throttled by bandwidth or CPU. >>>> >>>> The machines have less than 15% CPU utilization while transferring, with each of the 4 cores less than 30%, so it doesn't look to be CPU issue. >>>> In Task Manager, sshd.exe and rsync.exe seem to be running normally using only few percent CPU, and show Power Throttling=Disabled, Priority=Normal. Setting their Priority to High doesn't seem to change things. >>>> >>>> Looking in Resource Monitor on the remote endpoint, the network usage is pretty much a flat horizontal line at about 18 Mbps (2.5 MB/s), so it sure looks to me as if rsync is somehow being bandwidth-throttled when run in the background under cygsshd. >>>> >>>> It's almost as if rsync has an implicit --bwlimit override when it is run from cygrunsrv+sshd (I've tried --bwlimit=0 on the client which makes no difference). >>>> >>>> >>>> Any ideas? Not sure where to go from here. >>> >>> In cygwin, just scp is very slow. >>> >>> The transfer speed in my environment is as follows. >>> The tests were done with 100MB of test.dat file. >>> >>> (1-1) From cygwin-PC, >>> [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:. >>> yano@linux-server's password: >>> test.dat 100% 100MB 4.0MB/s 00:24 >>> [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat . >>> yano@linux-server's password: >>> test.dat 100% 100MB 8.0MB/s 00:12 >>> >>> (1-2) From linux-server, >>> yano@linux-server:~$ scp yano@cygwin-PC:test.dat . >>> yano@cygwin-PC's password: >>> test.dat 100% 100MB 4.0MB/s 00:24 >>> yano@linux-server:~$ scp test.dat yano@cygwin-PC:. >>> yano@cygwin-PC's password: >>> test.dat 100% 100MB 4.1MB/s 00:24 >>> >>> >>> I looked into this problem, and noticed that this is caused >>> by cygwin pipe implementation. Pipe in cygwin is configured >>> with FILE_FLAG_OVERLAPPED. >>> >>> If the pipe is configured without FILE_FLAG_OVERLAPPED, >>> the transfer speed is much improved as follows. >>> >>> >>> (2-1) From cygwin-PC, >>> [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:. >>> yano@linux-server's password: >>> test.dat 100% 100MB 85.5MB/s 00:01 >>> [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat . >>> yano@linux-server's password: >>> test.dat 100% 100MB 69.7MB/s 00:01 >>> >>> (2-2) From linux-server, >>> yano@linux-server:~$ scp yano@cygwin-PC:test.dat . >>> yano@cygwin-PC's password: >>> test.dat 100% 100MB 80.1MB/s 00:01 >>> yano@linux-server:~$ scp test.dat yano@cygwin-PC:. >>> yano@cygwin-PC's password: >>> test.dat 100% 100MB 57.7MB/s 00:01 >>> >>> I am not sure why this happens and how to fix this. >> >> A couple years ago I had an idea for changing the pipe implementation to avoid >> overlapped I/O: >> >> https://cygwin.com/pipermail/cygwin-patches/2019q2/009393.html >> https://cygwin.com/pipermail/cygwin-patches/2019q2/009423.html >> >> I never followed up on it. But if you think it might help with this problem, I >> could dust it off and try to finish it. > > Interesting. > > It will be also helpfull for: > https://cygwin.com/pipermail/cygwin/2021-March/247987.html > which seems to be the same issue with > https://stackoverflow.com/questions/10385424/good-alternatives-to-cygwin-cygwin-doesnt-support-natively-support-win32-app > > However, I wonder why scp dislikes overlapped I/O. I agree that it would be good to understand this. When I first proposed the change, I was thinking in terms of code simplification. If it also improves performance (which we don't know yet), it becomes a higher priority, but in that case it would be nice to understand why it improves performance. Ken