From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2104.outbound.protection.outlook.com [40.107.244.104]) by sourceware.org (Postfix) with ESMTPS id 8040B385840C for ; Sat, 11 Sep 2021 02:17:26 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 8040B385840C Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=cornell.edu Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=cornell.edu ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=V2ZdcbF5iddmhvOIf12PB4++2BdCwf2X7gxSakd6/8E3/R+yCyH4l5r9DaEf3X9NeLlh1JDibf4Q5AxcOB5Vp0ouJDY6JYiUS5uYNwSu1tysmUMgnzwsBL8mU/pJOzJPS9QWuLBK846HeLtl+0M2gcSavj3oDcSnSvhvAQeKMhg/8N2vySo2x1WIVgZzlMK25aPMWvzuMNKPby6B0piZjCOUsTg0sSzKEb6JkTkgVVmYjcRY3rIfcVzPZAbR9Y8pjqgZfdlGK/OvxBgf9pVw8BHtwmLeCPx44IzcY8jnsrZ51RqKikMji1tCCQRHSkOxWcQ6a0j9P2+jqLKxYPVemw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=j4IvxBlKMhgJ88AZtXMCaL0mL4BwfEfp7DXkq3jucu4=; b=YzbLZBUUghUoQO5t2ndJ0YtQae6m1OdqDhTjxAjCCYT9HM6eL8Tfdk8lwoO6oKh4DURLHJpJFKxjPMJWXq9rRskVwhEAUJFazcUBU/9QR3cU5JO/XgI89nNrf1L8BAKBz8F4xubSCXNdVoKUbzkYqqddNxDbO31XIb977FbXkrom+LwtIgwPs7tO+lw2Qq79IVE4ms1j28lo2ioJojvybDNFK5stGdbtOfF4w5AQ4gbpN1C4jwBo8psMaZepdYiPv3vRxvlGf40vx9FBJeb5j7sF+BlJFVU0sk0S7ePDWfSOhsPIiXoPOu3bPoKSxv/cLJEbfDhlNarnjx/I8K+wsQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=cornell.edu; dmarc=pass action=none header.from=cornell.edu; dkim=pass header.d=cornell.edu; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cornell.edu; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=j4IvxBlKMhgJ88AZtXMCaL0mL4BwfEfp7DXkq3jucu4=; b=dgh3ULsdyDal/WQEeELy8tu0bLSeOQJcapYc/vFVKvHSHJ23+H4NHAam5ihbxyxFnjhXJ4bwPBJ8vBY0Mi/2zUxghXCEzg1oQq9hdNCNxgE2AzPtffeRnBSQiJisHEBnjX+e9CevR+K20z8015p8uX5cZh1A3wKmqclX/0VdFsU= Authentication-Results: cygwin.com; dkim=none (message not signed) header.d=none;cygwin.com; dmarc=none action=none header.from=cornell.edu; Received: from BN7PR04MB4388.namprd04.prod.outlook.com (2603:10b6:406:f8::19) by BN7PR04MB5345.namprd04.prod.outlook.com (2603:10b6:408:9::32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4500.17; Sat, 11 Sep 2021 02:17:23 +0000 Received: from BN7PR04MB4388.namprd04.prod.outlook.com ([fe80::5113:e84a:b38a:7a66]) by BN7PR04MB4388.namprd04.prod.outlook.com ([fe80::5113:e84a:b38a:7a66%6]) with mapi id 15.20.4500.018; Sat, 11 Sep 2021 02:17:23 +0000 Subject: Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled? To: cygwin-developers@cygwin.com References: <41A583E1-C8E7-42AB-9F24-EEC33A41EC60@house.org> <20210825201845.07b6400b79dc5558a7761efe@nifty.ne.jp> <20210826062934.54f2f2216021c095bb7ba13b@nifty.ne.jp> <3b560051-ab27-f392-ca4b-d1fd9b5733b0@cornell.edu> <20210827202440.47706fc2fc07c5e9a1bc0047@nifty.ne.jp> <20210907122631.65452be8d021ec72259431d5@nifty.ne.jp> <20210909124115.555c6be15d675500617d284a@nifty.ne.jp> <20210909170549.506cc3c1f6029d904fece6dd@nifty.ne.jp> <20210909211940.51ef391e27d43f0421962cb8@nifty.ne.jp> <20210909214246.cd1ff1a3062fea27e51ad4ae@nifty.ne.jp> <33386baf-3b2d-d57f-2ad3-1bd328ed7935@cornell.edu> <20210911075734.aaf37697ba7db2ad14d911a3@nifty.ne.jp> From: Ken Brown Message-ID: Date: Fri, 10 Sep 2021 22:17:21 -0400 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.13.0 In-Reply-To: <20210911075734.aaf37697ba7db2ad14d911a3@nifty.ne.jp> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-ClientProxiedBy: CH0PR03CA0087.namprd03.prod.outlook.com (2603:10b6:610:cc::32) To BN7PR04MB4388.namprd04.prod.outlook.com (2603:10b6:406:f8::19) MIME-Version: 1.0 Received: from [192.168.1.211] (74.69.128.111) by CH0PR03CA0087.namprd03.prod.outlook.com (2603:10b6:610:cc::32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4500.14 via Frontend Transport; Sat, 11 Sep 2021 02:17:22 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 538ae2b7-e1d2-4661-d8fe-08d974ca4a35 X-MS-TrafficTypeDiagnostic: BN7PR04MB5345: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:9508; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: NGED3w3/PX8r9/0ueXIk11TbsmN7z1142laL1AGnFBHZaGca9A/1xcLzIsXQN6W4bwOiFDx1bzaau0f+JMrsRu+Az1X6SRcSeSUsaoVc8Iwc/4wHnrGlZfvhQbKYDB2dTsB1oolLF5mMU+/7GYIaG/6QRMFO0TAhWFSSlznBTZU7IUdJxvWD4KZ3b6XYcMtfM7oDb+cLZAKiAL/+O82JKZA/fpPPxKOlJ3ms4v68eSpwFj2EMRQbne4he33hEYNPOU1ilqrBjyPnYFoMfWkaQHgjwrLIHL+kghE/scnOrHOkZEjF33BeF4iOMr9VoUT7eLx/AI/bTq4vCY5uJJyHW3omRgydOkdb08u8DYkpmSySe1R1zxtzOyeI+3PAvLxFgume/jHjI/J2iBUT8dWFu+zEIQjtfGqNw8g6JdPEUk4+ZhnHkavFxn4ST1NtxSExPxRZQyztO64D7gdTVo8hXnWlwN9dNOmddhwXVMmo6Y9/3IMw/ExwtiOtUNXJQMUYBfDu9OKyc17PeG0igfnTH7jK797AJBUPYnVZE3x+CXlUtoujvA3RobxnDKnuJssrhGfXtLDsAXGwpWQx7sdeKfLFiJGpgHBQ1clPXD1TfHhSWeo3k1VY9zhrq8g2PDcz3/Nn9MyRVihY0v+YCnWcIgf68aDbSAlvUdBeUEcuZ1XP7Z7C8/YyP0A1ztbE8ELgUXK7pEbeuqEGKtdWZcXAB/FnVEF0tpQqemBNW927tow= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BN7PR04MB4388.namprd04.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(2616005)(38100700002)(956004)(66946007)(186003)(53546011)(31686004)(26005)(8936002)(8676002)(508600001)(66476007)(75432002)(6916009)(66556008)(2906002)(83380400001)(31696002)(316002)(16576012)(36756003)(6486002)(86362001)(5660300002)(43740500002)(45980500001); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?Windows-1252?Q?/49zQfW6lm5ulB912YLU/P2CHt9CNkxHZzSWV1W3c3/dWpMwuCKDlUPI?= =?Windows-1252?Q?oyGKtBGbgEizm7Reyxll6J83o5e/r+0n1v0bGpHQUMfwuEKGFx2BP/gc?= =?Windows-1252?Q?HQjri1Z1r5rBgCOyUp51dg2RWlqpxoB8DaFq08LmEIaZJ3kEY50WT6YI?= =?Windows-1252?Q?1Wv0JAnfdWv8043/mSEdnJAJ7tMS0Rih8JFsoxXREYvPnONqSwApXRmn?= =?Windows-1252?Q?xWn2lhDfOayT32NcrzKnC641J/1JkYfQxP11j0yLyaPc/EhAr/xKpvi1?= =?Windows-1252?Q?H1GAKPVlmDh0lZvybIk//OWlSbus6ZQCLUNek6YppcRz19PecEoBIu4F?= =?Windows-1252?Q?V8Z16p7QdRSo33246kx/y6/DLWWUMzQgGUA8OYwRM6uWUG7B2abo0vPN?= =?Windows-1252?Q?LVKl6vXbcEIEnkQZjwON2IB75tTTYus+tmZMyxrzgSEtiNVx/b4mMES7?= =?Windows-1252?Q?ZsuFPptjAGfzgpjGQ+G7z3K9TywncfxwRBuSMGLL6I7u3g3xE5vyzv6K?= =?Windows-1252?Q?S5hqwTJd9liIPMAhFp+Sn70tKwrNGoOQKJ5DT+gSWnOapKcURtzKvT6B?= =?Windows-1252?Q?HiNWI0HtFTkjop9MzKD052cDKDgooawaCynzu79KxPc/+vPK+GgKfY8w?= =?Windows-1252?Q?TSzEPF6Z5r6UTj4p6xlKQX85JJRIhor8QLvdaka/TYAJIolkwmgAQP0J?= =?Windows-1252?Q?KJdtLJvAtG2UK3RX8/iYgRD0DigS2NTbe8n/5dPpRdk5BHQKVMxTks1c?= =?Windows-1252?Q?8CGJCxeo1ifWA1XINESzxjCNIYzlZNxdM28ej1ecxy0e6vlKEOlzQu4o?= =?Windows-1252?Q?A9CRa7HfVokxTT4Eo/QpDv4SUIyrTn7UtHU7WirCST07kjpNC0p75mbs?= =?Windows-1252?Q?nkPNyMu2RbXT5oMn2v6E4U8GoNrg70qMKkSs+QQuMIcksfYXdkxxIkYP?= =?Windows-1252?Q?EgJmOKJH63TS5+wvm/ZNt+h4uU+sK7jrGWEaMAeGCO5z4EIMC2KNmsYm?= =?Windows-1252?Q?DdNm5QI9ogu5Y5vm8kU2xWxA2q+LXDqpjJpwkBT/OIB/ZHbxJJ4/oNSr?= =?Windows-1252?Q?78bsRhITk1Z0CuQTK1YDQ7E2zGTqhwpMLBbyyrrgV+AnVJNYaRu8Cc8G?= =?Windows-1252?Q?Pw2sjRKFhdN3CjOvmtH75/ZE4aPGAXpcGEPVDUgHVbUZjmM1JwktZxts?= =?Windows-1252?Q?7XKGtP1ChFnK60yL/YOQb3fDwektPBypCbeFg7hHMHDrwhfWQdnx/qby?= =?Windows-1252?Q?Hhyb+3bk3F0ByRu/wrkxzoMqDZoG9Xrq0oqkrUNDPdgbwbsEhxZ5qOPx?= =?Windows-1252?Q?60u9TmI0EsWvcTksZp4ECzqZpM2BJiQTxWmxgxvghqOubTEWhHaMot+O?= =?Windows-1252?Q?jdlux/+i0ckNEZQxDTf9uv2Uwz68qQLXmhFldxwxu9eZYjho/FfxVI9H?= X-OriginatorOrg: cornell.edu X-MS-Exchange-CrossTenant-Network-Message-Id: 538ae2b7-e1d2-4661-d8fe-08d974ca4a35 X-MS-Exchange-CrossTenant-AuthSource: BN7PR04MB4388.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Sep 2021 02:17:22.8843 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 5d7e4366-1b9b-45cf-8e79-b14b27df46e1 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: xn8OAl650mOBJxCAn+LYp6n+dAj5gt2q2EqpBmt4it6jansXTRi5NE07tXVqdr2uMBcDQLVHb6h6bnbz0VpKCA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN7PR04MB5345 X-Spam-Status: No, score=-10.7 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, MSGID_FROM_MTA_HEADER, NICE_REPLY_A, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_HELO_PASS, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: cygwin-developers@cygwin.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Cygwin core component developers mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Sep 2021 02:17:28 -0000 On 9/10/2021 6:57 PM, Takashi Yano wrote: > On Fri, 10 Sep 2021 11:17:58 -0400 > Ken Brown wrote: >> I've rerun your test with the latest version, and the test results are similar. >> I've also run a suite of fifo tests that I've accumulated, and they all pass >> also, so I pushed your patch. >> >> I think we're in pretty good shape now. The only detail remaining, AFAIK, is >> how to best avoid a deadlock if the pipe has been created by a non-Cygwin >> process. I've proposed a timeout, but maybe there's a better idea. > > I am not pretty sure what is the problem, but is not the following > patch enough? > > diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h > index d309be2f7..13fba9a14 100644 > --- a/winsup/cygwin/fhandler.h > +++ b/winsup/cygwin/fhandler.h > @@ -1205,6 +1205,7 @@ public: > select_record *select_except (select_stuff *); > char *get_proc_fd_name (char *buf); > int open (int flags, mode_t mode = 0); > + void open_setup (int flags); > void fixup_after_fork (HANDLE); > int dup (fhandler_base *child, int); > int close (); > diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc > index 6994a5dce..d84e6ad84 100644 > --- a/winsup/cygwin/fhandler_pipe.cc > +++ b/winsup/cygwin/fhandler_pipe.cc > @@ -191,6 +191,17 @@ out: > return 0; > } > > +void > +fhandler_pipe::open_setup (int flags) > +{ > + fhandler_base::open_setup (flags); > + if (get_dev () == FH_PIPER && !read_mtx) > + { > + SECURITY_ATTRIBUTES *sa = sec_none_cloexec (flags); > + read_mtx = CreateMutexW (sa, FALSE, NULL); > + } > +} > + > off_t > fhandler_pipe::lseek (off_t offset, int whence) > { > > > AFAIK, another problem remaining is: > > On Mon, 6 Sep 2021 14:49:55 +0200 > Corinna Vinschen wrote: >> - What about calling select for writing on pipes read by non-Cygwin >> processes? In that case, we still can't rely on WriteQuotaAvailable, >> just as before. This is the problem I was talking about. In this case the non-Cygwin process might have a large pending read, so that the Cygwin process calling select on the write side will see WriteQuotaAvailable == 0. This could lead to a deadlock with the Cygwin process waiting for write ready while the non-Cygwin process is blocked trying to read. My suggestion is that we impose a timeout in this situation, after which select reports write ready. Ken