From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2106.outbound.protection.outlook.com [40.107.243.106]) by sourceware.org (Postfix) with ESMTPS id DEB0D3857C50 for ; Thu, 2 Sep 2021 19:34:47 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org DEB0D3857C50 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=cornell.edu Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=cornell.edu ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=e1TOrGOWM66+hFBSu3IfgIXGz4FMylyh4O0eRisRzbIir3/F/Raaiko7BOspT7JO2lzhDid7g4YhcL/XDMK/wQiXeSJqhYicZmN6U1jYGWIg7RZanzxXUT5VROxOWkIaaIy1iG5g81rEjygxgxGCjEUfzSfZq+i6aQhkFXnFICJJ2Qyfv8AYY9Jr30LPsGABXtiXAzvTrdoyRL3feLnQtTOAC4m816ualu+Du1LH+YlhC2VMg8eZgum3OXS8fgSX8sWxazviP7oc56VQqnHDLnU5FgMkyJQtG4HzmRA5wUl30MDapfPof5I2SaTohv+8FvqNLYFh59xejs8RgiY6PA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=qoU/dYx9GLf5z49lcp/2GFa5/42kbYHhjERJoGVc9Us=; b=h1mcC+t75lcG6696w8liSXoZxQVIcA/qW0RgaN5BBxqAOaGl3jNrSdMP+S9yd7NgQ0t7vI+tEiy6mGRC9i1g3xcL+OW4pF68ZlaglR1kZJ+fLkLSRoVsS7JnLsy8c2WEbpxGugLJlJyjsO9aHzDRa7F8JbmykKxYw2Q1tns3CSD+Y0WOxKB3oiGxi3QkBCDBj9b6j/HCsiWjpYkeiWnEO3Ql1N90yrn8Cm+goF1WzyWMYvIeQklFCLeE+wJ3tdB5SHyqOuh+sYyN7v+KfQobFymvi5A/9skO+6Z/Ba1uzLyRVaFQLHcBATcbvUTuWKQjsnynxdAJVuaUP1wLfzLDCg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=cornell.edu; dmarc=pass action=none header.from=cornell.edu; dkim=pass header.d=cornell.edu; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cornell.edu; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=qoU/dYx9GLf5z49lcp/2GFa5/42kbYHhjERJoGVc9Us=; b=SAaCVkRTOk6NpyWo+3cmmja8YL4jjVpponD/Jm4PpDJf1umX78L6SD95vXsUxX9G2VyT+99PqcnbfDKKZzZQBtqKl5V8ytvE/K8+meD8gUn2wxG2oLZ4M6tFxOGFnBfzfx5z9PiQzrEyyZT7ZBd5a0jhdbbO9gO/D8o/0fct2Fc= Authentication-Results: cygwin.com; dkim=none (message not signed) header.d=none;cygwin.com; dmarc=none action=none header.from=cornell.edu; Received: from BN7PR04MB4388.namprd04.prod.outlook.com (2603:10b6:406:f8::19) by BN7PR04MB3953.namprd04.prod.outlook.com (2603:10b6:406:c9::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4478.20; Thu, 2 Sep 2021 19:34:46 +0000 Received: from BN7PR04MB4388.namprd04.prod.outlook.com ([fe80::b510:889b:1fd0:d80e]) by BN7PR04MB4388.namprd04.prod.outlook.com ([fe80::b510:889b:1fd0:d80e%7]) with mapi id 15.20.4478.020; Thu, 2 Sep 2021 19:34:46 +0000 Subject: Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled? To: cygwin-developers@cygwin.com References: <20210901080220.ee4a5bfbea62cc1ae0a9598e@nifty.ne.jp> <20210901091652.6bf3cccbcaed4a22f6ffa6b0@nifty.ne.jp> <20210901172339.1039604b7067e0492534a20f@nifty.ne.jp> <24138e20-aa97-cfea-bf48-198fc67755ea@cornell.edu> <9ba687eb-f4a0-18f8-b10b-76e7e51e123e@cornell.edu> <152bfc0c-2f72-c684-6fc5-aa7c36c136b8@cornell.edu> From: Ken Brown Message-ID: Date: Thu, 2 Sep 2021 15:34:44 -0400 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.13.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-ClientProxiedBy: CH2PR12CA0030.namprd12.prod.outlook.com (2603:10b6:610:57::40) To BN7PR04MB4388.namprd04.prod.outlook.com (2603:10b6:406:f8::19) MIME-Version: 1.0 Received: from [IPv6:2603:7081:7e3f:3419:3d1d:1494:5af9:3061] (2603:7081:7e3f:3419:3d1d:1494:5af9:3061) by CH2PR12CA0030.namprd12.prod.outlook.com (2603:10b6:610:57::40) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4478.17 via Frontend Transport; Thu, 2 Sep 2021 19:34:46 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: cecf8699-001a-4290-ba5d-08d96e48b894 X-MS-TrafficTypeDiagnostic: BN7PR04MB3953: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:9508; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 66Kkv7LcbbRbr+Q+ikHlQB62T/U6WdeqUTyerFd6R7aEJNhXZHlneVFWg6KMikV6btbSKxY+DgAHdKhQNOOPnI9CMcVpRAtpxjcpo1c+BJkuzVROfj6DhltYIGCPdBXFEp5AnS/w63i58VvR5QLVDGh13+T+aQRIVJscHAAiqQ49GxJuW6KBvzy///OgmaTBa28TN8eRf3IaHyNbqktjpsTWxhKKLUhQ2YdJG0yD5O7QhNmQVQSOfH105M/Gch/HQGtbLKBD8uxkA9LgpRt+5fLz+9S1JTXEUjMdiTLYL8fsFqDpx05P4ulXpaA1s0gugLxD1UaECpkoIJIBxzCzU9rXW32OpdtSIFk4qMAAltMUs+tZG8Z5Z5Q/GSoeBlLn40cK2kdgA8q3fFFCX7pkS/zBlvxtjURHjPXa91M2O4dXiSDuS154naNW3mJaazJ0gmCmh3cvqmV3KKon1etvmLUaOZI04f7pGsuCW7m+UGYyuUdDreSGF/QJB8DMD6nZHrePbCrWqeyku/PLIow1oOKdfgdWDAgMykSspaGQLzZB81hhMLc10og9iwFSPFI4DJlfW4TZn6URDR7NU/uIoiqbZckS3HsJ7qI/Z12MVdjcuWkCTJ4yQHfdoVwrm1aWb3VBu1da1NcihXBIzmJ9+/uJ2tWzP1VgKlW4495MEaQUbSMn0e1IAiUXnh8SAKtDoN4NdYl2n7zQ0rJERIhxxnJRWoM1FPYEPXWjvWSXQdg= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BN7PR04MB4388.namprd04.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(376002)(136003)(366004)(346002)(39860400002)(396003)(38100700002)(66476007)(53546011)(66946007)(186003)(66556008)(5660300002)(83380400001)(36756003)(31696002)(86362001)(8676002)(316002)(6916009)(2616005)(8936002)(75432002)(6486002)(2906002)(31686004)(478600001)(45980500001)(43740500002); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?Windows-1252?Q?lAuRmgvGsUGG84OHxyBF473y5wvPrn3pw9pmcogUZHedCfj9DvOQyk0d?= =?Windows-1252?Q?9DNoSBfB8aLGUF7mK4kJVgwJByCjj6OQKBOgco3kQxJUZIKBPfIp4Wq4?= =?Windows-1252?Q?mxPEhnuKFjJhrP3MlC9jJ09GTRt+r6DiYu09NQ1I40GXSIvHDm+pmIo6?= =?Windows-1252?Q?z2G7qec+pw9+Z2vOTFEkG+HFzhrZ4dWzib56F3CK9Tx7TbJHkhcTxzeM?= =?Windows-1252?Q?lMVIhtMcynD7jUDU1iFK6uHxS6XSy5NIFmOoXz0JjJK2DlblOFyuSa0c?= =?Windows-1252?Q?yskCsehZlfse+uQYexxua93qHOQJ4HiaGuGtIe0wifvi78tw0HZYDiXM?= =?Windows-1252?Q?oF3XUC1d+YR5syaIvvuBchMEALfbXCVWkJyESoGV3a8/+V3G1TSZHwbs?= =?Windows-1252?Q?BYyoogcwfVzFsx6D23gUjtwY+xpfNcIHoEumTLM/VMpVMZYoHrIH0CeG?= =?Windows-1252?Q?Deb+rJEDn3Lwk+LEBLKLbKS8mXMyH16rA1ulrtQhWExbdMKhGHwm7Bj+?= =?Windows-1252?Q?TWwIVbhzezXdXjTBo5lu8pWlIABXmhpYL13wuvMXLTa2GnEwE9AftovE?= =?Windows-1252?Q?houNI+uKK29m5JXQFcTM7G5UxIuCr4X58YFzSieI5K6Y25Niu+OnqZCf?= =?Windows-1252?Q?UqWvlHQBUSQG5Wf9szWXPCKWpCu20Og7tVNjEdoT1ROw/6kw1uymFOL+?= =?Windows-1252?Q?uxQ/mObMSYAYIJxzMvOfc8eyqFnPvjmKSiQULLQrnSZQ5MkbU6yavTm7?= =?Windows-1252?Q?5CGuWTVRTNLkUZGuiRGFhFEa65QpD/UgzS4k57Ppzcglq1n0EHFZB3L3?= =?Windows-1252?Q?xpATQ50+MQnn1qsUj5WMDoLhzMhBWVYTGKZrXEM/lJPgYb2Q6giZYRXB?= =?Windows-1252?Q?rG4WAUk+quDoKpDqYpg+Aoh6aXDwvBFItzwsl27VrElWuVhH+hRUsv2l?= =?Windows-1252?Q?GRfa9gmw9sggiG12YvSqYhh00LlRa7KczX4MZihC/Y19P+joxHfBsVO9?= =?Windows-1252?Q?2BqLs2Ps1XPjDXUXjX1oN7YlkHFRd8Q+vZNyI3riiZelyJDWzSRYFB8H?= =?Windows-1252?Q?WXMhaaMf67KQcZPctjkJQJ69Bb2B2UgSRwW7PMe6tNLE1uE7b0Hj1mRN?= =?Windows-1252?Q?zI+Fc4/QtEXYWnG1UOheQYIrvjeNPOaTa7efe1tFijk4U4po3gtti377?= =?Windows-1252?Q?HZeRIIsSjAMB+zIj46xsaIzmCYG8UpUFU0W5lAETDAwy3/1FHDt6i/O7?= =?Windows-1252?Q?5KljLrKJ7Hw0BjhyvNOLiMHwrLl0N8pOzbriW7bLiqB1eaxEAxCAkS17?= =?Windows-1252?Q?yV5xymV+OXr+Bh+ABk/B1ePBsw+n1QhBzRqWjMtgDacaJs4zWumEqpm6?= =?Windows-1252?Q?Ta8gxdnMFz5p/UAq149yySOHwxmy+fPTtmXWN0OWv/B4ACBS1NAwQT5B?= =?Windows-1252?Q?H7CmG78ZHjIHXCvXiQ0S+96hdJTiANJjcW2RUR6YCRZqoa3fBXiBOCJh?= =?Windows-1252?Q?QttmcNqL?= X-OriginatorOrg: cornell.edu X-MS-Exchange-CrossTenant-Network-Message-Id: cecf8699-001a-4290-ba5d-08d96e48b894 X-MS-Exchange-CrossTenant-AuthSource: BN7PR04MB4388.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Sep 2021 19:34:46.5399 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 5d7e4366-1b9b-45cf-8e79-b14b27df46e1 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: L37ip+Mu8jDQhz6J+c0a4WXdNOoKi62SKBUHrEhAg3VYR729BebnBFQdlqOLtwMTVEdsNBwjeD4R0bDF+4mFBw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN7PR04MB3953 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, MSGID_FROM_MTA_HEADER, NICE_REPLY_A, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_HELO_PASS, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: cygwin-developers@cygwin.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Cygwin core component developers mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Sep 2021 19:34:50 -0000 On 9/2/2021 3:00 PM, Corinna Vinschen wrote: > On Sep 2 09:01, Ken Brown wrote: >> On 9/2/2021 4:17 AM, Corinna Vinschen wrote: >>> What if the readers never request more than, say, 50 or even 25% of the >>> available buffer space? Our buffer is 64K and there's no guarantee that >>> any read > PIPE_BUF (== 4K) is atomic anyway. This can work without >>> having to check the other side of the pipe. Something like this, >>> ignoring border cases: >>> >>> pipe::create() >>> { >>> [...] >>> mutex = CreateMutex(); >>> } >>> >>> pipe::raw_read(char *buf, size_t num_requested) >>> { >>> if (blocking) >>> { >>> WFSO(mutex); >>> NtQueryInformationFile(FilePipeLocalInformation); >>> if (!fpli.ReadDataAvailable >>> && num_requested > fpli.InboundQuota / 4) >>> num_requested = fpli.InboundQuota / 4; >>> NtReadFile(pipe, buf, num_requested); >>> ReleaseMutex(mutex); >>> } >>> } >>> >>> It's not entirely foolproof, but it should fix 99% of the cases. >> >> I like it! >> >> Do you think there's anything we can or should do to avoid a deadlock in the >> rare cases where this fails? The only thing I can think of immediately is >> to always impose a timeout if select is called with infinite timeout on the >> write side of a pipe, after which we report that the pipe is write ready. >> After all, we've lived since 2008 with a bug that caused select to *always* >> report write ready. > > Indeed. Hmm. What timeout are you thinking of? Seconds? Minutes? Probably seconds (maybe 20, like the connect timeout for sockets?), but it's hard to know without seeing naturally occurring cases where this happens. >> Alternatively, we could just wait and see if there's an actual use case in >> which someone encounters a deadlock. > > Or that. Fixing up select isn't too hard in that case, I guess. I think this would be my preference, at least during testing (if we can persuade people to test it). Ken