From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on20708.outbound.protection.outlook.com [IPv6:2a01:111:f400:7e8a::708]) by sourceware.org (Postfix) with ESMTPS id 4B5DA3858C60 for ; Mon, 20 Sep 2021 21:09:59 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 4B5DA3858C60 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=cornell.edu Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=cornell.edu ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=SyYfWVkDRi7Hvf6Wonpz5fObjRirAnRyo7HAq/aS2070pU5WchR60/E3EgQiqYn5tTIitAYe2RrBkpgv4w4uXLWo/Y4jp99k5ylQLEsqEFgWUpzhbch4iicNykFh8L6l99gPzkQHDkGw+auU214w5QlG2SyE9ZJvvDVKhRap5vd1qoJiLBUAYXjRg5iZXvbEFvz/ed4DrR2fawp7chX3VCmITeLaZZgViVunU55ltZpCwcSbpLXIb71NqJwcxJLCitM8praLiq8VFzCfRvyFtxLfed2opf//z3hKxF2jT97jQMbbTr/48XxGWeZ6MZePm015t0Q/c3D89fd9uRrvyw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=gKHzEm+ltS1ytcuZtpAlPaepMbe+B0+S22ncwHB1uRI=; b=ha8zZ1V4oZ6/jBi3h6OwBsesxSGhe6fdlGL2SifvKitRq394gC7OshgVjXjr0mPY2H0mPwDUi6gIkDb3atnDqg/SA6839tjNOWA38JmtxwI3UeSZxBYHEG+B5XNQYMdhe8JGF75rbTBrmP6fI9cieFVo0/AqAQVsuCJhlkWB/p+lMEx1W2x6Cdwfat2VahuR6nmozExQ4igUXUAv5XvGxU4db+mQiLGMG/H1OCJw77Hqasu4E2ysKxdKLd1+xGahJYFsKhVkVRJ60ydHQhBNPPwIPbl+BWhUP4sH09Pa0MWPZRzDwbqdZFMS73oVAbm9ConX6kOSfW5S+cIle5OdMw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=cornell.edu; dmarc=pass action=none header.from=cornell.edu; dkim=pass header.d=cornell.edu; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cornell.edu; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=gKHzEm+ltS1ytcuZtpAlPaepMbe+B0+S22ncwHB1uRI=; b=VhA4OGt03GZvWxSm4WhkB7D1D1xCWkUVQuPzhxEpA31bPj28PZLzH5G6uz8eYGUtJcbrfFePjlCdW2fXFerA/iVldsJ7hF55vNFForkEiZm5x60uxbaYFJwfhwS9kthVc8JsIBnvVXUwxfD0tjRAQoJ3ywHJptoF10d0Pfc0Okk= Authentication-Results: cygwin.com; dkim=none (message not signed) header.d=none;cygwin.com; dmarc=none action=none header.from=cornell.edu; Received: from BN7PR04MB4388.namprd04.prod.outlook.com (2603:10b6:406:f8::19) by BN7PR04MB4258.namprd04.prod.outlook.com (2603:10b6:406:fa::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4523.16; Mon, 20 Sep 2021 21:09:49 +0000 Received: from BN7PR04MB4388.namprd04.prod.outlook.com ([fe80::5113:e84a:b38a:7a66]) by BN7PR04MB4388.namprd04.prod.outlook.com ([fe80::5113:e84a:b38a:7a66%6]) with mapi id 15.20.4523.018; Mon, 20 Sep 2021 21:09:49 +0000 Subject: Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled? To: cygwin-developers@cygwin.com References: <20210907122631.65452be8d021ec72259431d5@nifty.ne.jp> <20210909124115.555c6be15d675500617d284a@nifty.ne.jp> <20210909170549.506cc3c1f6029d904fece6dd@nifty.ne.jp> <20210909211940.51ef391e27d43f0421962cb8@nifty.ne.jp> <20210909214246.cd1ff1a3062fea27e51ad4ae@nifty.ne.jp> <33386baf-3b2d-d57f-2ad3-1bd328ed7935@cornell.edu> <20210911075734.aaf37697ba7db2ad14d911a3@nifty.ne.jp> <20210911113517.f74fc3ac1971bbf04c7a9bd1@nifty.ne.jp> <20210920215229.cea5722b2c2fcd7e03917108@nifty.ne.jp> <00f4eca0-2b38-f4bb-98f2-e99e7ee941f5@cornell.edu> From: Ken Brown Message-ID: Date: Mon, 20 Sep 2021 17:09:48 -0400 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.14.0 In-Reply-To: <00f4eca0-2b38-f4bb-98f2-e99e7ee941f5@cornell.edu> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-ClientProxiedBy: CH2PR18CA0058.namprd18.prod.outlook.com (2603:10b6:610:55::38) To BN7PR04MB4388.namprd04.prod.outlook.com (2603:10b6:406:f8::19) MIME-Version: 1.0 Received: from [IPv6:2603:7081:7e3f:3419:d571:af58:205e:a58] (2603:7081:7e3f:3419:d571:af58:205e:a58) by CH2PR18CA0058.namprd18.prod.outlook.com (2603:10b6:610:55::38) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4523.14 via Frontend Transport; Mon, 20 Sep 2021 21:09:49 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 20fb6e92-c899-4f73-4003-08d97c7afb6b X-MS-TrafficTypeDiagnostic: BN7PR04MB4258: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:7691; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: cUyPCMpsdbZTZ+Svf1P4LZfWRrkxCvHx4k7L6XRMIDb/8Rps87vkXceozENlOCnlXlskHIxm5WqVTFzBFN9KQhQ+ASKDGrD2C4nXC8SRTwX4GA2SPOcbHWL1z02jKevMfMP0eZQByUEku9VCcmPVWSQie5dbwKW6JWvpWKwwde0l3j2a4hMMXmJ9qDtRl/uRwR5SNK21PZkgMFeW8XckAxslSUnCOmRfVKSnDzk7II/GLvo/dtokCunOCEqoHA3b/LAxA2+CRmQu+LV7bEu0KAuiQr7GvLnpr0+mffrVGyuI5wowZpG3+pV2dSJF2Z7+2hxBRvspAFVlP3hbhlbkKHVdCQEvIHYbaeuDCEhNZx2mmQO+RU6j0BLnv7y34J0SPkfIxefX+85YNY1oRCqVtX+Ebz+EtqkBA3rfaHRZBqGWb58ddt3qQmtQeQfllCrK1ForACO1zuLCQE+PokAumV/txWl3wS7zPsUfIZ/6xoPoFsn2YjB+zmkHujh4G42+YUCkRVD4XUQy62USdeJ/AWPeAKuPCLYvmkaaC6Ll7ArjDFQDbVsjE2Z6R354Y1Rvc1ZK0hJKO4krLIo1bGZgoiVxwdXD6E8ood0wLzUhKyXM/cnKGFK1+RgU1YTYiwqj7I84/0F29ZKhqMvoe+hcq6CslzggAZCgwi25MCkhcaDeeI+jOTkjATSOBASZmxFnmS57BSZ0QEuD9xiDhKIgZ0DoE7lLR321T1AIgvNXf18= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BN7PR04MB4388.namprd04.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(396003)(346002)(366004)(136003)(39860400002)(376002)(75432002)(66476007)(86362001)(2616005)(66946007)(8676002)(36756003)(31686004)(2906002)(66556008)(53546011)(8936002)(478600001)(83380400001)(6486002)(38100700002)(186003)(31696002)(5660300002)(6916009)(316002)(45980500001)(43740500002); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?Windows-1252?Q?gwQVMZBhRz7lTPoR5kSgEH9ESqxJuCIc2oavpItwFRlmNPcmYto5XmX9?= =?Windows-1252?Q?PZBbYTnjIo4VWEA2M5qWNtNWNdC3BlUesaxQeb/AJdgLScD8OHbdd68w?= =?Windows-1252?Q?9hNv5OCP4P6P73z4Y8iQj0yBbZtPBrWTEzgAdboc0+ZYcJlNUcdWrfD+?= =?Windows-1252?Q?q/wYRtV6b0U7mb0HeEA3EAfcSz5yksgWKbPbr66vzq3jQwQd7pq4ELgc?= =?Windows-1252?Q?Xs5PAdCCnG5vFbIV2/pSBnVGEEhfa1jPbiH/kQbZbnOu71f1xPyysdo2?= =?Windows-1252?Q?vbbubZuTTNp5H2MEAE23tavzTtiLLqtRYmoqK6vZbGrFv0j0mQcur1uv?= =?Windows-1252?Q?KQHFLj8o34YVGcvgUTnrkKOGPoF/dCVSFNNWP+opTxCcueFSezrnsYlZ?= =?Windows-1252?Q?LZKqH8zXho9mxuV5vv+vOn8Kx79v8rlyCTC4XzIoizt6FesmkDJ0zxMS?= =?Windows-1252?Q?EafZOuOoP1mWxJemdiCs76Cl1cL42muLYtKi1KP0EW6qjhgylc6HUklZ?= =?Windows-1252?Q?1azlpRw+AX+qr7gtVUGtF4eg3vGsc8VY15j3CEunCgOK5gEujEqmkU0J?= =?Windows-1252?Q?Scw4ETCVimDCQsJyGP2+j07JiMgyz58AMmdqGdflZBwcoZnbPwEI1FHp?= =?Windows-1252?Q?FwDc9SCF5/5ydnpCl0hXubHX3iKUVwWmcfnbOenSBRv9Eywm/o71Dy4O?= =?Windows-1252?Q?g4CiD7wTMNsrbnEqiMesgDOcahwra+Bq3C2I7aAqotUtQcHhR3STo9b1?= =?Windows-1252?Q?RpXUldgGYy1ulhv2EZnGYhtWcWLberzBLSpaOar/gdgyWMS0hMSKIqJW?= =?Windows-1252?Q?YVYGNmay8Kyng/pMrv9kUxgR/2qaT4BBOlXyABO0cyak4xTPOei1675O?= =?Windows-1252?Q?aeZX+bFPQx7gXzNhPH0N+O+NDNWZeXnb8EHzcuiDRew97l9Ys1Bwg/jt?= =?Windows-1252?Q?F9/Be4laBc2wgF7QQuAM7UcvASWTxFnkgs36nKjt3R64LsBQv0LVcOgD?= =?Windows-1252?Q?DtYLFCy5kqHgQvRLauHMCmlJfecXrUDYvy2pJjl2VKRdPqu1/CYtid4a?= =?Windows-1252?Q?dXY7xkbAvvEAAszbcWz3WvypuJgHtRGOX/kRZlpSD8T0Cj8HOaSfwFt+?= =?Windows-1252?Q?oys+rm1pmg5bcuPzJt37Eui1kGjIXRaHTbM48OOHgnI+qItNgoAX79r+?= =?Windows-1252?Q?cKj1hWEMeWm1xIm9RVWBs2Leg/k6R7FgHq7Y9D57IZ08Ks1iOAyOLaOx?= =?Windows-1252?Q?Uc5D9ceK2pyPcW9uoTUNoBZPhC5FWRc9xAP7Guwj+dAWBEiURdCboGjT?= =?Windows-1252?Q?0X60x4p598HdFennl+gMt9eyM1NURLx5R6GTLt95hGgYL7lMc/Hc8g4d?= =?Windows-1252?Q?fZ/tL3UDr7Cd++1D/wkgSOGypNmt51a/QAOFKK9TMIkBe8v8Pt+4xwg1?= =?Windows-1252?Q?1wEEibJiNJbckwMKPGByXyInruMgCDcRjMfMkNtnrzVv4iBlhrgPOwJz?= =?Windows-1252?Q?sP/8ZRSC?= X-OriginatorOrg: cornell.edu X-MS-Exchange-CrossTenant-Network-Message-Id: 20fb6e92-c899-4f73-4003-08d97c7afb6b X-MS-Exchange-CrossTenant-AuthSource: BN7PR04MB4388.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Sep 2021 21:09:49.7489 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 5d7e4366-1b9b-45cf-8e79-b14b27df46e1 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: R416i7bpRq5tdzCes2OtaAsGCCKuYJ3TxszbOsZWmZYfh2DVjk0ElYTNYTF4t+QcM+tg4qowhvdI0rqLzgJTow== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN7PR04MB4258 X-Spam-Status: No, score=-0.2 required=5.0 tests=BAYES_00, DKIM_INVALID, DKIM_SIGNED, KAM_DMARC_STATUS, MSGID_FROM_MTA_HEADER, NICE_REPLY_A, RCVD_IN_DNSWL_NONE, SPF_HELO_PASS, SPF_PASS, TXREP autolearn=no autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: cygwin-developers@cygwin.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Cygwin core component developers mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Sep 2021 21:10:00 -0000 On 9/20/2021 3:14 PM, Ken Brown wrote: > Hi Takashi, > > On 9/20/2021 8:52 AM, Takashi Yano wrote: >> On Mon, 13 Sep 2021 11:07:42 +0200 >> Corinna Vinschen wrote: >>> On Sep 11 11:35, Takashi Yano wrote: >>>> Keeping read handle in write pipe (Corinna's query_hdl) causes problem >>>> that write side cannot detect close on read side. >>>> Is it possible to open read handle temporally when pipe_data_available() >>>> is called? >>> >>> 1. You would have to know which process keeps the other side of the pipe. >>> 2. You would have to have the permission to open the other process to >>>     duplicate the pipe into your own process >>> 3. You would have to know the HANDLE value of the read side of your pipe >>>     in that other process. >>> >>> Point 1 is (kind of) doable using GetNamedPipeClientProcessId or >>> GetNamedPipeServerProcessId.  ZIt's not clear how reliable these >>> functions are, given that both pipe sides are created by the same >>> process and then usually inherited by two child processes communicating >>> over that pipe. >>> >>> Point 2 is most of the time the case, especially when talking with >>> native processes. >>> >>> Point 3 requires some sort of IPC. >>> >>> Having said that, I think this is too complicated. >> >> In the current implementation, there is a potential risk that >> write side fails to detect the closure of read side if more than >> one writers exist and one of them is non-cygwin process. >> >> Therefore, I had tried to implement the above idea (tentative >> query handle). Finally, I could convince myself to some extent, >> so I attach a patch for that. >> >> As for point 1 and 3, I used NtQuerySystemInformation() with >> SystemHandleInformation. I also used NtQueryObject() with >> ObjectNameInformation in order to know which handle is the >> other side of the pipe. As for point 2, it is not possible to >> open the process which is running as a service, so the current >> query_hdl is used for such process as an exception. >> >> Ken, WDYT of this implementation? > > This looks like a great idea!  I'll review it in detail later today or tomorrow. I only saw one thing that doesn't look right to me: > + /* Check object name */ > + status = NtQueryObject (h, ObjectNameInformation, ntfn, 65536, &len); > + if (!NT_SUCCESS (status)) > + goto close_handle; > + ntfn->Name.Buffer[ntfn->Name.Length / sizeof (WCHAR)] = L'\0'; > + if (wcscmp (name, ntfn->Name.Buffer) == 0) > + { > + query_hdl_proc = proc; > + query_hdl_value = (HANDLE)(intptr_t) shi->Handles[i].HandleValue; > + qh = h; > + break; > + } > +close_handle: > + CloseHandle (h); > +close_proc: > + CloseHandle (proc); Doesn't this mean that query_hdl_proc is not a valid handle any more? So the attempt to reduce overhead at the beginning of tentative_query_hdl() will never work. Other than that, I still think it looks great. But it's complicated enough that I think Corinna should review it too when she returns. (It also uses Windows functions that I have no experience with.) Nevertheless, I'm tempted to push it so that it can get testing, even if Corinna changes it or reverts it later. Ken