From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM02-BN1-obe.outbound.protection.outlook.com (mail-bn1nam07on2129.outbound.protection.outlook.com [40.107.212.129]) by sourceware.org (Postfix) with ESMTPS id 286453858408 for ; Sun, 12 Sep 2021 15:10:57 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 286453858408 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=cornell.edu Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=cornell.edu ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=S009cVHEh2XF1zHPf3O7RJfLLqRj3pLRE7d7FvTLek5scMG3OAH2UMKijTY05aTu1ikQ8U3HJ2wiEMDKIdzTJrDitlClXgP4LMK4Mh7zM9KiiMPYIZx/4rJLFp1IGz1vucBB1GXUp/7j839Pj78fQE24zif3LN9viPDoyEEOitKgPXWXddnI1wMoa6tQtvdsUxgmx/DdbZHJHbXa7O0PT5OIemWmOkblylupt50QdiSGfrmcbZup5+lzskueOR0J7mBKONCsBtGi6bY6TWb0AiPi99o5LA1gB23hU7cWEotx/iWSY1au609I2JWAJpsIZqvnNCvg+hmu2Sqac5pfVw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=1zh1otvOES6kquE2vCmL8ZN4kEsxgsHdEylxj99OhXk=; b=UFtxlUdA0ZcLhJGwUW9e/OTOnwvfRa2Kg53HYjhZnTbSDiM2tZGxSxtZ3jCxuWPM/WWRyrCh+PNAtWQcKPItSE85mGHVAvHT51zlfMvnRCCalHuvijergO7x3PsHLP8k4D1c7M8y2D8Bcy9dOyeGoTsLpjaGWyPyiD4eewFe/rV1j8BYY9urz3X8Cci4JWSrk4TCFwxqmD0iVH5+GuFEGsYFQINMyDJFlVR0GC9MskcVh72N1JCX8Wpeng9kiOxSrxOrlc+fDrgzb3m4U+ERxkSu5b1jD0w7QTdeGmCBE22Pua9CKiUOat3UoCDBqYgTDXMs6aLiJ5zTwtGAyT2GhA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=cornell.edu; dmarc=pass action=none header.from=cornell.edu; dkim=pass header.d=cornell.edu; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cornell.edu; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1zh1otvOES6kquE2vCmL8ZN4kEsxgsHdEylxj99OhXk=; b=Ps9NON89nrz1SFzjy+Y6t//3RzffM87nTR9DxtePoIIM8Lcbo52h7oDtA6g/MNbHeeEDD9/e7WdiE1vpExui929xOgZC4N11s0zIEbPsqcF4VDP0WjKaDM7H2qfk+wfb02ax/shdiQwkJDtLKLGtroR2ObIbeyKzEdjACs9JBd4= Authentication-Results: cygwin.com; dkim=none (message not signed) header.d=none;cygwin.com; dmarc=none action=none header.from=cornell.edu; Received: from BN7PR04MB4388.namprd04.prod.outlook.com (2603:10b6:406:f8::19) by BN7PR04MB3954.namprd04.prod.outlook.com (2603:10b6:406:be::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4500.14; Sun, 12 Sep 2021 15:10:55 +0000 Received: from BN7PR04MB4388.namprd04.prod.outlook.com ([fe80::5113:e84a:b38a:7a66]) by BN7PR04MB4388.namprd04.prod.outlook.com ([fe80::5113:e84a:b38a:7a66%6]) with mapi id 15.20.4500.018; Sun, 12 Sep 2021 15:10:55 +0000 Subject: Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled? To: cygwin-developers@cygwin.com References: <41A583E1-C8E7-42AB-9F24-EEC33A41EC60@house.org> <20210826062934.54f2f2216021c095bb7ba13b@nifty.ne.jp> <3b560051-ab27-f392-ca4b-d1fd9b5733b0@cornell.edu> <20210827202440.47706fc2fc07c5e9a1bc0047@nifty.ne.jp> <20210907122631.65452be8d021ec72259431d5@nifty.ne.jp> <20210909124115.555c6be15d675500617d284a@nifty.ne.jp> <20210909170549.506cc3c1f6029d904fece6dd@nifty.ne.jp> <20210909211940.51ef391e27d43f0421962cb8@nifty.ne.jp> <20210909214246.cd1ff1a3062fea27e51ad4ae@nifty.ne.jp> <33386baf-3b2d-d57f-2ad3-1bd328ed7935@cornell.edu> <20210911075734.aaf37697ba7db2ad14d911a3@nifty.ne.jp> <20210911113517.f74fc3ac1971bbf04c7a9bd1@nifty.ne.jp> <695ce1f4-4f7d-f3f3-6dd3-087467d67b28@cornell.edu> <20210912174849.3d38107568065a95aeb19c7c@nifty.ne.jp> <20210912200423.667e40eb1adc52461bbefa20@nifty.ne.jp> From: Ken Brown Message-ID: Date: Sun, 12 Sep 2021 11:10:54 -0400 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.13.0 In-Reply-To: <20210912200423.667e40eb1adc52461bbefa20@nifty.ne.jp> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-ClientProxiedBy: BL1PR13CA0217.namprd13.prod.outlook.com (2603:10b6:208:2bf::12) To BN7PR04MB4388.namprd04.prod.outlook.com (2603:10b6:406:f8::19) MIME-Version: 1.0 Received: from [192.168.1.211] (74.69.128.111) by BL1PR13CA0217.namprd13.prod.outlook.com (2603:10b6:208:2bf::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4523.9 via Frontend Transport; Sun, 12 Sep 2021 15:10:54 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 7a364b5a-d990-4eed-8d72-08d975ff848a X-MS-TrafficTypeDiagnostic: BN7PR04MB3954: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8273; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: wXKSKMxe/e0l/wMM56fi0qJMFpn1hcx48QUCpcUWUqUypfTt2g17nb7/AvDtoV4hcJFAQFn6TZyREK1OdTVYxPVzVtha38vZR9rXv/0wAS0LBUxkSoRbIaffJGX7t8OiqpDh+/dh38zGAaWvImhBUMCeD1+PGad7Ye70T9J65P5xoKAbQ+kW7c2ZY4IeODT2/uNKXMgHqt3XfgPsE3W7wRT9muYykNHKHrUvIlPqRxtAY0WiLpCGeugq9wNuVipEwhjE6rS3+tyjXJiB4WC5Ptqi55enbJaFLS9+gPqbvCP9QGW3RJM/t4Ma8/pnsO4kUGp2KdPrGWPj4Uwhx7Mk2Q0h0EnI2HPz3NkKeVUEuM6sL/GHnuO//moloYYqLPLmzS9x9gttKgnOaV/i/2ftkUKiQF+YBsuAyIggG+Yuvm+ANqV87qP0EidMi8Bet7mHHlZLWtXBq+i7TbRCmIBsZM3+u3RZV1I7TGDYO6T+pij0NRzn3O8ggEtO1ghSqpiCy1+JnmZJziZUO4rNY7ketXLqOsd4U8PQ5nqpGjcnKa3mPlbh2EU/SST6RzEvF3sDcRexzxk0iwAL8AcXDOXw7rrU4OMGhuyVV4Cn4EG7oCQTkdewXTngiarZlsCoEQoQcP8nxIc5rs7pfwooTx8XYa9XkUKmAjBXYSZp15+mOJk+QGFHXn5wQAISjpFdEwHaCH0EbvKnNMW6z5iOfq16avjb+uV2AGtjyD1IPbDzJRvxOsWqkQSQCctwEWweRbDPGsZ2ru5VzYNsi+jyA7yUtg== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BN7PR04MB4388.namprd04.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(8936002)(26005)(186003)(66946007)(36756003)(53546011)(2906002)(86362001)(8676002)(83380400001)(75432002)(2616005)(316002)(508600001)(6916009)(6486002)(66476007)(66556008)(5660300002)(31686004)(31696002)(956004)(16576012)(38100700002)(45980500001)(43740500002)(460985005); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?Windows-1252?Q?MQSzsHV2mE6V4trSBGrLR0/MoK9xRD2pXaSP8JXTWH8VRdxZEoFEYrek?= =?Windows-1252?Q?Rru6ILDaung2WGjf152ubYQv2QrT4F9ar3IrNVQtHd6pW08SaY39WzUr?= =?Windows-1252?Q?mK8jxTxW8jCrLB3n5FfwGXwGsHip4H1R17ijmaWA2tMBpFuGoLXPRog6?= =?Windows-1252?Q?m7ne5IXPPo6nn0T+mGrE5cSfKtjpqs0kzWcZGzWlF781ehWNPnJjclKj?= =?Windows-1252?Q?4OFbOIxB5np+7tvfk0tKlTF8sGxCRiK5Y30nnqbmLUTWA5X2psx8qPpT?= =?Windows-1252?Q?aSdhgTvxnO/L/r+VuSqnW+rQGkLGkV0MGjZUVpwNoomMfDP9N2eixTlY?= =?Windows-1252?Q?TOg1hmNgqTkdHWBCWoamyIeEgthkHZpQ0yoHEYpepRITBg6Ga3i4k9gc?= =?Windows-1252?Q?VKF/MLTTDe55TKoNWzHrbDRF4wVnyquz5fbvC3IWPphPTj7oUz7YRfbk?= =?Windows-1252?Q?Z7AiU6D5BxK0MYBlvss30afVWuChUDiqIXhEowkXzr+epeleJjRAq/lS?= =?Windows-1252?Q?FKhSt53uC1Zk0kJRYAoh6HEzkfzZQN91qxgyUzDoH4Uy9vhkD+u0Wssc?= =?Windows-1252?Q?gFvPfdfcKA+UWOmk6aX5b1imMbvmJM5aN3coiRyI+316YK01xMXOF1iX?= =?Windows-1252?Q?7FiJj2V5jSyHAkAI2v6jLhGpuoEx4nRZcyTZN3p7xOBa6v7gRXfwjsqz?= =?Windows-1252?Q?D15+t01lj/gAMcd2jtF2/cPl/+pQQUsuc9Zzv+fqP2CNS6ZaqFBu2GiX?= =?Windows-1252?Q?2xccpulELYy9AcKZM7VUz7PJFtHLBQB3bpqcoTw/rNmikHbZgGEOrx66?= =?Windows-1252?Q?DPoh6JlZ3jvD90MPWz+JTPvMXsG6MxOn4SbCcxGZ3XypC2ISdrlCiTvZ?= =?Windows-1252?Q?x1wH9F/y0qYkBr6HhOczom/uh88N3Lhhepub+ImOwOQ7v6Af6hi0bkSn?= =?Windows-1252?Q?q7HwVh4CzUksBsbjp/4vbq6kAgwhf85UcviY6paOGB/Hl/cGyJcQxhYv?= =?Windows-1252?Q?4RdAIb8Yar2H+1vMS3bOmoSt+d25tbzKyBQjswG1aDltd+TCzQHbIk1w?= =?Windows-1252?Q?Z3oy9UWpIu7ueAhm3cbZhekN+8rDVC9qvwXilMq/wd9I1KlGVZcIid3L?= =?Windows-1252?Q?PYzYSmMOi03r1ztYK7MqMrcAwQEX06bIgHhyIuJmkN5y/r617ROgJ/0C?= =?Windows-1252?Q?lC48/YrJzqoBbYLEWqxhfjB62Suwa3puRPiLVWjxjjbfpRaRbBS5CHZX?= =?Windows-1252?Q?pTASPgXgepZyvy1lkEDVw92TO1o4k3Hhpk1x++kr7Px5z6UPS9Xk7keI?= =?Windows-1252?Q?NaUKj0H6CUPqBfCYktjjVS5hZ1wsfprfJtHfhHPLuOyyJIo+Z/wHMdCj?= =?Windows-1252?Q?68/3M5k2EZeKbFnuoxxB6EPNswG7K4+ktmHjUDo/Y2WGRKbXMB03TIi6?= X-OriginatorOrg: cornell.edu X-MS-Exchange-CrossTenant-Network-Message-Id: 7a364b5a-d990-4eed-8d72-08d975ff848a X-MS-Exchange-CrossTenant-AuthSource: BN7PR04MB4388.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Sep 2021 15:10:55.2187 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 5d7e4366-1b9b-45cf-8e79-b14b27df46e1 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: LTtgBxMlZzVoXpR7hQvCeSK0irxgSZt29Lf36J4aRHXNDnbX1KrgKnJt1W/SfQM/RuYqo3nN2fD3nL1v+xATFw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN7PR04MB3954 X-Spam-Status: No, score=-11.5 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, MSGID_FROM_MTA_HEADER, NICE_REPLY_A, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_HELO_PASS, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: cygwin-developers@cygwin.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Cygwin core component developers mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Sep 2021 15:10:58 -0000 On 9/12/2021 7:04 AM, Takashi Yano wrote: > On Sun, 12 Sep 2021 17:48:49 +0900 > Takashi Yano wrote: >> On Sat, 11 Sep 2021 09:12:02 -0400 >> Ken Brown wrote: >>> On 9/10/2021 10:35 PM, Takashi Yano wrote: >>>> On Fri, 10 Sep 2021 22:17:21 -0400 >>>> Ken Brown wrote: >>>>> On 9/10/2021 6:57 PM, Takashi Yano wrote: >>>>>> On Fri, 10 Sep 2021 11:17:58 -0400 >>>>>> Ken Brown wrote: >>>>>>> I've rerun your test with the latest version, and the test results are similar. >>>>>>> I've also run a suite of fifo tests that I've accumulated, and they all pass >>>>>>> also, so I pushed your patch. >>>>>>> >>>>>>> I think we're in pretty good shape now. The only detail remaining, AFAIK, is >>>>>>> how to best avoid a deadlock if the pipe has been created by a non-Cygwin >>>>>>> process. I've proposed a timeout, but maybe there's a better idea. >>>>>> >>>>>> I am not pretty sure what is the problem, but is not the following >>>>>> patch enough? >>>>>> >>>>>> diff --git a/winsup/cygwin/fhandler.h b/winsup/cygwin/fhandler.h >>>>>> index d309be2f7..13fba9a14 100644 >>>>>> --- a/winsup/cygwin/fhandler.h >>>>>> +++ b/winsup/cygwin/fhandler.h >>>>>> @@ -1205,6 +1205,7 @@ public: >>>>>> select_record *select_except (select_stuff *); >>>>>> char *get_proc_fd_name (char *buf); >>>>>> int open (int flags, mode_t mode = 0); >>>>>> + void open_setup (int flags); >>>>>> void fixup_after_fork (HANDLE); >>>>>> int dup (fhandler_base *child, int); >>>>>> int close (); >>>>>> diff --git a/winsup/cygwin/fhandler_pipe.cc b/winsup/cygwin/fhandler_pipe.cc >>>>>> index 6994a5dce..d84e6ad84 100644 >>>>>> --- a/winsup/cygwin/fhandler_pipe.cc >>>>>> +++ b/winsup/cygwin/fhandler_pipe.cc >>>>>> @@ -191,6 +191,17 @@ out: >>>>>> return 0; >>>>>> } >>>>>> >>>>>> +void >>>>>> +fhandler_pipe::open_setup (int flags) >>>>>> +{ >>>>>> + fhandler_base::open_setup (flags); >>>>>> + if (get_dev () == FH_PIPER && !read_mtx) >>>>>> + { >>>>>> + SECURITY_ATTRIBUTES *sa = sec_none_cloexec (flags); >>>>>> + read_mtx = CreateMutexW (sa, FALSE, NULL); >>>>>> + } >>>>>> +} >>>>>> + >>>>>> off_t >>>>>> fhandler_pipe::lseek (off_t offset, int whence) >>>>>> { >>>>>> >>>>>> >>>>>> AFAIK, another problem remaining is: >>>>>> >>>>>> On Mon, 6 Sep 2021 14:49:55 +0200 >>>>>> Corinna Vinschen wrote: >>>>>>> - What about calling select for writing on pipes read by non-Cygwin >>>>>>> processes? In that case, we still can't rely on WriteQuotaAvailable, >>>>>>> just as before. >>>>> >>>>> This is the problem I was talking about. In this case the non-Cygwin process >>>>> might have a large pending read, so that the Cygwin process calling select on >>>>> the write side will see WriteQuotaAvailable == 0. This could lead to a deadlock >>>>> with the Cygwin process waiting for write ready while the non-Cygwin process is >>>>> blocked trying to read. >>>> >>>> Then, the above patch is for another issue. >>>> The problem happes when: >>>> 1) Start command prompt. >>>> 2) Run 'echo AAAAAAAAAAAA | \cygwin64\bin\cat >>>> This causes hang up in cat. In this case, pipe is created by cmd.exe. >>>> Therefore, read_mtx is not created. >>> >>> Confirmed, and your patch fixes it. Maybe you should check for error in the >>> call to CreateMutexW and print a debug message in that case. >>> >>>>> My suggestion is that we impose a timeout in this situation, after which select >>>>> reports write ready. >>>> >>>> Keeping read handle in write pipe (Corinna's query_hdl) causes problem >>>> that write side cannot detect close on read side. >>>> Is it possible to open read handle temporally when pipe_data_available() >>>> is called? >>> >>> That would be nice, but I have no idea how you could do that. >> >> Hmm. Then, what about PoC code attached? This returns to Corinna's >> query_hdl, and counts read/write handles to detect closing reader side. >> >> If the number of read handles is equal to number of write handles, >> only the pairs of write handle and query_hdl are alive. So, read pipe >> supposed to be closed. >> >> This patch depends another patch I posted a few hours ago. > > Revised a bit. I don't see how this solves the problem. In the case we were worried about where we have a non-Cygwin reader, the writer has no query_hdl, and you're just always reporting write ready, aren't you? Or am I missing something? Ken