From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2093.outbound.protection.outlook.com [40.107.92.93]) by sourceware.org (Postfix) with ESMTPS id A1F773858415 for ; Wed, 1 Sep 2021 23:02:48 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org A1F773858415 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=cornell.edu Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=cornell.edu ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=FD1tkrImfYHavjZ45uuDQp3oL6EZkueUIu8NeJFD0vUcJQy12wi43at6o+7EBzV0JmmzzFKs/5bFhu6yMoVRbXxuuOQE5oErm2Jj0TAIfVdGKsCZ1sFPEQta0Oiqs+tpo6HjK5M7UHaxqs9CQee0zlsCtUvjrGz12G2KZ4wJ9PpH+PpqLQqCLnAHnfKlV8Ksm9H82T/UsyY0Xnt7ZffNj+UpbLv4q+Ll3qg+oIhz21tX1wH/BDZbT24zMZZoLk3xMUh+7qwvYMLnVCsKe1T319SA8zI9XRxP/omJaxiccVs9ppTmRE1RroZ3JQd711mtGak3U0f2Dv0bfwNAE9a/Xg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=J28MkvpqYCPBpUg24Tkp5o/yOfNYC8DLLt6NwISZJHQ=; b=ZEpffn01tAmiSTEQZIvtcEd0rODdw0qdfY9FDYtjclLiBOOQvHKY/GaoOcki/9LcK3bFY+Lw/XanR/JVL4wrgKspNvo+V7ioaOw0cqyJUBy+jrra+SiXM28PvaUYFl8bZgOscIaxp6WAiZqCtM0YGEMChpqTnTSujJOCAj/iAmzPrPhF72zZ6iXNE4P2ERdyYnv/XKTAArTQ3LmWm+PFwhCyMWSBhENFco/izjZf8ThOWp2/80AvEPeuW8HsIyrVmWwNdqfKOyMieIxb9mqm/2DcKM3I08/DNoE7j2d9uWox+TZw3GloYy0yPI30ZF/cvPE370p7nlSvjaPwlO9a2w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=cornell.edu; dmarc=pass action=none header.from=cornell.edu; dkim=pass header.d=cornell.edu; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cornell.edu; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=J28MkvpqYCPBpUg24Tkp5o/yOfNYC8DLLt6NwISZJHQ=; b=NYW9jtnnZ+LcBs+32JICSqh0f4elZiccG2dYefBchEzMi+8ujVpeIVb+WsBPH+eDuKIakWXnTQnUJL0QSaAJJQ0z5ngCCmCC9zARqYkrdjA/bZMXHZt9KR5R2u2vM10syVeeo10m43Z3cSVQFFhlxy0UVoqwRCsrDnTzTQ2ZSa8= Authentication-Results: cygwin.com; dkim=none (message not signed) header.d=none;cygwin.com; dmarc=none action=none header.from=cornell.edu; Received: from BN7PR04MB4388.namprd04.prod.outlook.com (2603:10b6:406:f8::19) by BN3PR04MB2324.namprd04.prod.outlook.com (2603:10b6:400:1::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4457.23; Wed, 1 Sep 2021 23:02:47 +0000 Received: from BN7PR04MB4388.namprd04.prod.outlook.com ([fe80::b510:889b:1fd0:d80e]) by BN7PR04MB4388.namprd04.prod.outlook.com ([fe80::b510:889b:1fd0:d80e%7]) with mapi id 15.20.4478.020; Wed, 1 Sep 2021 23:02:47 +0000 Subject: Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled? To: cygwin-developers@cygwin.com References: <583ca127-02e7-6b3c-3732-6478c0f862e3@cornell.edu> <20210901080220.ee4a5bfbea62cc1ae0a9598e@nifty.ne.jp> <20210901091652.6bf3cccbcaed4a22f6ffa6b0@nifty.ne.jp> <20210901172339.1039604b7067e0492534a20f@nifty.ne.jp> <24138e20-aa97-cfea-bf48-198fc67755ea@cornell.edu> From: Ken Brown Message-ID: <9ba687eb-f4a0-18f8-b10b-76e7e51e123e@cornell.edu> Date: Wed, 1 Sep 2021 19:02:45 -0400 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.13.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-ClientProxiedBy: CH2PR02CA0009.namprd02.prod.outlook.com (2603:10b6:610:4e::19) To BN7PR04MB4388.namprd04.prod.outlook.com (2603:10b6:406:f8::19) MIME-Version: 1.0 Received: from [IPv6:2603:7081:7e3f:3419:dd6:ee3e:5092:8b08] (2603:7081:7e3f:3419:dd6:ee3e:5092:8b08) by CH2PR02CA0009.namprd02.prod.outlook.com (2603:10b6:610:4e::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4478.19 via Frontend Transport; Wed, 1 Sep 2021 23:02:46 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 196ce1cf-66cf-466d-01de-08d96d9c9d2c X-MS-TrafficTypeDiagnostic: BN3PR04MB2324: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:10000; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 1AsdJFnFZ36rnwIvzY/rBcaqvEs4ViTJnAlwKTA+eYizUK2zuRIj3KVWKF2ZD1hATDKHdMsBphLa4G1LLvnS58aYaUucNLDTeLuHJArkRVpfh4JNZG3FdxuSiuBcTEYifyeIl/hcYnCvuneZdlvBC+YfAdubyz82Y6dUXQ7kOBJd8ExUTZTGk9OqwpIx634wM0vPsFQKdhypE5UNQMN7MbTrpoOydDxZ1BC7iAc4X1HO1uyMY5MxRO0tdp+kdlztGDnCJm65bjHiqWwOA9tlBDH0Yx7Aw34+OCtSN3hEqhaQ6iC73t3eGfU2SS5gDjl8Y9iErGGq05BboKHOj5JzRGoIGs0FisSgGrjK/74s1CUPH1Q7gR+lKYWKJjKas6emrYCv3UqvJ88taboL8f8vX+bxc+NYVcIBet6xAN5jve44Myk1uHv5RhbZI29QsuJmY9ecmOUNMEdPDR+yiBMYt1Y11SYxfYYnghNbK6AIC4hONKeqtDs9N9h8r+pxJteALL6fJqeEMr1JGh6nfWJRZVVvjepktVm3DLVuz9A/dfUzZ9FvQLPIJ099HMWRaE0nt/8Nh8S2jRNCB/EBO7UqZidyLJ3HViyK4lZ7gYmWM7LBMSeZIOblxRb6c5gWKq7c2Q1OJ97OihzKOQtXuJWfbz5QynHS6JrUIgIlcyKO6vf73jm9QKskQnpWxhYj2PqmsTTLH5Wd1iXkZe206zscDtoE/B5WprA7qKhOoF0EEP2q5CoV5AuDasaV+WUi/SnJrm5ZYa2vd2CuxGVaaWg3HwCvRjqJ1nKPPz7gK8SdiOyB6t9BCzFI9nZ7OWWME1lE X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BN7PR04MB4388.namprd04.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(376002)(366004)(39860400002)(346002)(136003)(396003)(36756003)(186003)(75432002)(316002)(31696002)(5660300002)(6916009)(83380400001)(31686004)(38100700002)(8936002)(86362001)(2906002)(966005)(66556008)(53546011)(66946007)(6486002)(66476007)(478600001)(2616005)(8676002)(43740500002)(45980500001); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?Windows-1252?Q?XTwAv5Gf9rhZ+JtKNJGVbyQ3rEpEf8ibdfGzQFTNQ0hmScyl2D0jfkHy?= =?Windows-1252?Q?/7k+7prfG6AqmyDgGW3N9krFauXB4WxQPSZYgJxLIpWguhl332f9UWXl?= =?Windows-1252?Q?mCTYQZ6d7pOfzkjDQhbbEIAjmHXsOPTotNaHQ9rCRiAGMifLnzQgMP4S?= =?Windows-1252?Q?vfUVfIiZzl+efGRT4RJgtWxQVQ3ojm7A83M5eOAFTplRAAlp21nY8hQf?= =?Windows-1252?Q?7IZLvwjvMd9CLQI1H9M8WOq8km6xYGEORFJLC2LJKPgPozNhPMd/v0Xz?= =?Windows-1252?Q?PgPjzZxU8tlvPrzSsOJ8R8f14+6Qq+eIoTbp6Uyl78SThMI9VV62kFr3?= =?Windows-1252?Q?NoNahNJBwYtgZJ4YQOXl9w75X3OuCimKvdA5Kxc2YZxLCuJLGHXPBFTe?= =?Windows-1252?Q?ruYyzlsAziiQWpeiGcOsBqCbXZAfJ+EZ3YZup9TwhMizX3tKVohzoepy?= =?Windows-1252?Q?XtSJrr1y/naEdgLVcKy7UJ70YVspa5wslx4NpxRoT1yiN+Qx+5BYAVDi?= =?Windows-1252?Q?0V/JbDA0vxy2WogqW/m7wl3QE0N3TAInrYphiumGJqwVgAPpOObdb3FS?= =?Windows-1252?Q?HOk1TkWMeXcMnGcw4JejqlTcVCpIZ2uaNdM9R+iAKo/hw4zW9hoSVBb3?= =?Windows-1252?Q?O569lsUR1N5+ZYSh99iR60AIsvP03mE/7N2CtAh1mMl6BqojjjrOMDEb?= =?Windows-1252?Q?+M2UXC0IcAlewa/xsQjOP7eo7+XtsBBuRl3IU59UZRWLJbc98XrvVyZr?= =?Windows-1252?Q?lqtIRLJSVw1I8ySEiAQlzBqIHBZ9wGoJwMPzOfhfxIiGIpzdXa0Xe+k9?= =?Windows-1252?Q?nxuKGOfUYfpr2Il56450MOIJBsiv41AAlrl79WjmQIvcbRHmDzld7IXi?= =?Windows-1252?Q?+ICVzVT7dcq9KC9jf1rPXbTzurcL4+Q+boGBH85ZHYlHSsAgu/oR2tUz?= =?Windows-1252?Q?AbRHgXvDkNYWlVCX8c8gOs7jkjA8zzvBvn0jOePb7yyYidbTNXBc/nPh?= =?Windows-1252?Q?ikq1GyJCfhu8fTdqq0GdGFuW4GGyiB4zihqp44up0rckvmNJmNylhnQo?= =?Windows-1252?Q?d8HDHgn09fNd7oOPWqKs1TgiKbLieZ2xv4QahsJOyEM2bmWQUKdlb3Mi?= =?Windows-1252?Q?ElLKxvFbHt1TUgrrCBCQNGvYB9udumKsEdFMwblf9f7EgmnWnSLY/pyn?= =?Windows-1252?Q?TaUHtGxGQxUaEcd0ib8zogpZnzTBzLaW6nr4j97DR8BYVRQWf47dyK7I?= =?Windows-1252?Q?7pzzpLX4D0ACxrFOvse338MsNlv6Sg55HDHpr7OoAXGlQSU8CeAuAcvP?= =?Windows-1252?Q?O/jDpgJaI7yXviQ5BA+svVwieU0oolA6oRpgqx3HQIJ/QvFMx3mbtbAT?= =?Windows-1252?Q?j3WsXJUjZQSZU7IwOdMENR3Hm5/2am6UCsalxSNujTngrG0rTX8Iq0rm?= =?Windows-1252?Q?FCm9vPa/Y2pnDYuDphB1cwKr2NSjZJCladS6a3RahrGBrLXdPemaA5v/?= =?Windows-1252?Q?M8baKzjV?= X-OriginatorOrg: cornell.edu X-MS-Exchange-CrossTenant-Network-Message-Id: 196ce1cf-66cf-466d-01de-08d96d9c9d2c X-MS-Exchange-CrossTenant-AuthSource: BN7PR04MB4388.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Sep 2021 23:02:47.0627 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 5d7e4366-1b9b-45cf-8e79-b14b27df46e1 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 3e2l00b00iC3t9B5lJ7w2zEWK7jDHI1wZ4wcVaL4kvuJUL6WEqFF28rF73xQuEoq4vAIj1ijEwf8Zu/JQN4SOQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN3PR04MB2324 X-Spam-Status: No, score=-3.6 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, MSGID_FROM_MTA_HEADER, NICE_REPLY_A, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_HELO_PASS, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: cygwin-developers@cygwin.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Cygwin core component developers mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Sep 2021 23:02:59 -0000 On 9/1/2021 9:52 AM, Corinna Vinschen wrote: > On Sep 1 08:56, Ken Brown wrote: >> On 9/1/2021 4:46 AM, Corinna Vinschen wrote: >>> On Sep 1 17:23, Takashi Yano wrote: >>>> On Wed, 1 Sep 2021 10:07:48 +0200 >>>> Corinna Vinschen wrote: >>>>> On Sep 1 09:16, Takashi Yano wrote: >>>>>> On Wed, 1 Sep 2021 08:02:20 +0900 >>>>>> Takashi Yano wrote: >>>>>>> On Tue, 31 Aug 2021 17:50:14 +0200 >>>>>>> Corinna Vinschen wrote: >>>>>>>> So for the time being I suggest the below patch on top of topic/pipe. >>>>>>>> It contains everything we discussed so far. >>>>>>> >>>>>>> One more thing. 'git log' cannot stop normally with 'q' with your patch. >>>>>> >>>>>> The same happes with 'yes |less'. >>>>>> >>>>>> The cause is that write side cannot detect closing read side because >>>>>> query_hdl (read handle) is still opened. >>>>> >>>>> Oh >>>>> >>>>> my >>>>> >>>>> god. >>>>> >>>>> >>>>> That kills the entire idea of keeping the read handle :( >>>> >>>> One idea is: >>>> >>>> Count read handle and write handle opned using NtQueryObject(). >>>> If the numbers of opened handle are equal each other, only >>>> the write side (pair of write handle and query_hdl) is alive. >>>> In this case, write() returns error. >>>> If read side is alive, number of read handles is greater than >>>> number of write handles. >>> >>> Interesting idea. But where do you do the count? The event object >>> will not get signalled, so WFMO will not return when performing a >>> blocking write. >> >> What if we create an event that we signal every time a reader closes, and we >> add that to the events that WFMO is waiting for? >> >> If this doesn't work for some reason, a different (but more complicated) >> idea is to keep a count of the number of open readers in shared memory. >> When this is 0, write returns an error. I'm thinking of shared memory as in >> topic/af_unix (which I copied in the fifo implementation), but maybe >> something simpler would work since we only have a single variable to keep >> track of. > > Great idea that. What we need would be some semaphore upside down. > One that can be used to count items and which is signalled if it's > down to zero. Here's an idea (untested), based on https://stackoverflow.com/questions/6559854/is-there-something-opposite-to-semaphore: We create an ordinary Windows semaphore and use it to count the readers: It starts at 0, we increment it by calling ReleaseSemaphore when a reader is opened (by fhandler_pipe::create, fork/exec, dup), and we decrement it by calling WFSO when a reader closes. When we decrement it, we test whether it's been reduced to 0. We do this by calling ReleaseSemaphore and using its lpPreviousCount argument. We also create an event that we can use to make WFMO return during a blocking write. We signal this event if a reader closes and we've discovered that there are no more readers. In this case we cancel the pending write [*] and return an error. I'm sure I've overlooked something, but does this seem feasible? Ken [*] I don't know offhand if Windows provides a way to cancel a pending write. If not, we could use query_hdl to drain the pipe.