From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2104.outbound.protection.outlook.com [40.107.94.104]) by sourceware.org (Postfix) with ESMTPS id 8265A384A880 for ; Fri, 3 Sep 2021 12:13:05 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 8265A384A880 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=cornell.edu Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=cornell.edu ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=caQ7naoJ5fNvMBWJz5is2rF6Xo02IWM/N9wuMWcjZtM2ckuXOpZKpiRanHx05z5Y/STsbPnuN6obbjrmkVzHgfe9qqfuFnCf9bWLxUKfuB0WczV0gDMDTqTZjn9fzdA4da/Qsw5MlABWqMwgULFLCbnhwr+oGVy9OEi0urlff9d5pajVAYf1oqcWleiMpd1AXDt3C7uExWReGW9HnpVmbCegRTUi09bUcrN7lTi/wtFyTLmSByMGF3AlkGpSPdthgJttu15JB+oInaPfd3GLB1IsHF1TmJqEZMBsCo6MTyLNJXxXfH6/47+/gvomweAF3btQLmlC1WimWfFkYD7q8A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=fD7/hr+4Ss8ToBrk/IPF5nlCJ3jDqWnvaH9B7qPME/s=; b=bXPVm2TwcbDsEsfqSgly0N6/cDeVS071Wdmv6I6bjznAhdbAiUR0z8o7Qyd10LyzDiGRd3SKT7PY9zYv9Vef81MCftmxu+FrqU+rOCXfgwV97qptOENJR8T4IvDviEIhm+ffCz2KhPP3MVgyvwa+0Ag/nBKZO6sFx3WD+ZpBDFWyKBfk28xcXPe3ABAKXpecUPKkgOyCAHVBCvaRUL3tIz0Y17jtA55fTJ5kQSyMyQx/3PJCyUMWcw7yhwGowdm6sop8H43i7i0gRyKAq6vZSWT9R0vX0YFpQdVnH8Sf1e3qvlXO4Splk9+AQbT+ZsggNPgicp/xPWzS7MQwCGAGnQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=cornell.edu; dmarc=pass action=none header.from=cornell.edu; dkim=pass header.d=cornell.edu; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cornell.edu; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=fD7/hr+4Ss8ToBrk/IPF5nlCJ3jDqWnvaH9B7qPME/s=; b=DzNTY2qUaRooch25Ix6qpsAvAjJW562zU9PNXqD7Zluj+lvn8ynFFhsxWHPROoM0WoZ/Z8NI+MDD/TPeFDLvMwOMaGDf16o6NCNJq0ehL9xE/pFMA23LHCgFjW01s7R5pCy14TOBactHSyLDNQulPVZNMpqw1/3f7MHjCW8r9Kc= Authentication-Results: cygwin.com; dkim=none (message not signed) header.d=none;cygwin.com; dmarc=none action=none header.from=cornell.edu; Received: from BN7PR04MB4388.namprd04.prod.outlook.com (2603:10b6:406:f8::19) by BN6PR04MB0354.namprd04.prod.outlook.com (2603:10b6:404:99::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4478.17; Fri, 3 Sep 2021 12:13:04 +0000 Received: from BN7PR04MB4388.namprd04.prod.outlook.com ([fe80::b510:889b:1fd0:d80e]) by BN7PR04MB4388.namprd04.prod.outlook.com ([fe80::b510:889b:1fd0:d80e%7]) with mapi id 15.20.4478.020; Fri, 3 Sep 2021 12:13:03 +0000 Subject: Re: cygrunsrv + sshd + rsync = 20 times too slow -- throttled? To: cygwin-developers@cygwin.com References: <24138e20-aa97-cfea-bf48-198fc67755ea@cornell.edu> <9ba687eb-f4a0-18f8-b10b-76e7e51e123e@cornell.edu> <152bfc0c-2f72-c684-6fc5-aa7c36c136b8@cornell.edu> <20210903190046.663c60fb11c936e344821383@nifty.ne.jp> <20210903191340.c28ae366e79ca14799bacc1f@nifty.ne.jp> From: Ken Brown Message-ID: <03d54bd7-0bd1-0e46-828e-53bb1389e8d3@cornell.edu> Date: Fri, 3 Sep 2021 08:13:01 -0400 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.13.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-ClientProxiedBy: CH0PR04CA0104.namprd04.prod.outlook.com (2603:10b6:610:75::19) To BN7PR04MB4388.namprd04.prod.outlook.com (2603:10b6:406:f8::19) MIME-Version: 1.0 Received: from [IPv6:2603:7081:7e3f:3419:a4be:f19a:7570:4bb4] (2603:7081:7e3f:3419:a4be:f19a:7570:4bb4) by CH0PR04CA0104.namprd04.prod.outlook.com (2603:10b6:610:75::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4478.19 via Frontend Transport; Fri, 3 Sep 2021 12:13:03 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: a3c46cef-bc7d-4529-af11-08d96ed42e0e X-MS-TrafficTypeDiagnostic: BN6PR04MB0354: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:10000; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: dbWRZS4upD7Pi6+1RqmJrobK6KMGa6V2GrHdpnNvwVvgC1+ELMvQLFP5e74SgmWVAOeX3TceuzBpp8p9h2ki10qaqryisKv1+NzbspIi8y2/LgB1/GhpVvkn9jHqA3wtIVh51VLpGNApKVBILn7HvLSO3b9c2qHpGuTl6Vy+VvgTkKWdOFjDw4zl5buTNNhsiyrg+ubxNdV0POA/Pw0aeiieIBj5DpnksQ5iNjEQ3iS2audKwhL7bqK+Xa5y7nxk/r1It8jLIl+a2ZWPsusEKnRQulMQaOVT3/Mfn2ibe4aii897JrwzVLSdaoivbEmeA5sehCE82EGHeWq1NYUzByZ/rGIbJ4BAdfaJYBxE/mZvrMC3gdW2cpoIF6pFxEG37Od/0krj07t8djLUeCGQDQXf4rDeolUtfaOvvHOtQoi/tPjrKjzjIXoWQ8tRdFJkUem+ZRJHseyUrPlmKUGvwHt1JVpozV/TqsJBxelW/BCKjPvSD/nJNwWszkHc0go5f3uAo6LEjuYoxSI+opGFlWJrDmSuI9QhxWsSAfP0z1nIBGU4ep+JkkwcvgE6bO60S0KshU65IKDQ9DkVXVeTyruJYX6RLUSCWG9vYwMnsWC8i25H+OTJ05AEJMqoikwQM9zShtz2Ug3SIGKqQRtECmkIyrypFnnllC4ou+eFDCt+DK8RZ8q25fBLce34XzV8TPjJeUt0ZXaRI9k27LKnM5tpW+eHuwTdD4uGdIik+nY= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BN7PR04MB4388.namprd04.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(346002)(366004)(39860400002)(136003)(376002)(396003)(86362001)(31696002)(66946007)(66476007)(66556008)(2906002)(2616005)(478600001)(83380400001)(186003)(8676002)(38100700002)(75432002)(8936002)(5660300002)(316002)(36756003)(6486002)(53546011)(31686004)(6916009)(45980500001)(43740500002); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?Windows-1252?Q?b9xYZjJVH8Em6LHzBlYCyd44OE3jHT/Jo92Heiq7IgXym1u9N8a4apBc?= =?Windows-1252?Q?hFbgOlIXW87uhsiYUaa5Mh/p6nX7KsQGAPOb/rn/1Dm8AfdM3XK7NE2S?= =?Windows-1252?Q?91giAWAXAFnPcDBVLPtAdV37pnyNbymSR4YKSP37FBfztomogiIapp+7?= =?Windows-1252?Q?jXnYlNKUnplzvILRNgaWSYdJFok7IUCTdY+7uTFjUfuL56oBKgrhYLiw?= =?Windows-1252?Q?vQcapxTZPQepKZzX75aR1jLGAYu8cIqRjdenR0lXXXRiZ6SLvyZdOd9o?= =?Windows-1252?Q?/aQRWmOmPgJC9OMzu7XjTFew6jD1a2LGY7lHwxLogYTxqy4Oq9Kgu+1N?= =?Windows-1252?Q?dWiCQrcLv0CXEVxUrJrBQdlF9YzuMnCFFHMRvMbDTQN68C0Wf25/4XR1?= =?Windows-1252?Q?LmuDRZwRKzKDh59odKA9yESuZl6WjFmSzRaPEwSsKKmx/fB2f4Xse9t1?= =?Windows-1252?Q?cGW+YOsXiZ3xsQIm+Yvbmo0JE1UKE8+ZSqgVp73gUUUmz/AUOfYw1VnG?= =?Windows-1252?Q?yY39OEVsVELzllH5Mib39vdpuGCNGhWdNd5ZyrrBPgoBlp41fTemMbp5?= =?Windows-1252?Q?RIuW27gIiaHQqDEK1gXghIIznS1Bl60mjy73X4VIBZF1b6wfaClqlOUk?= =?Windows-1252?Q?JLFMjgLnz+2W7dE/fKty+3py1B9nxqMesCt5fu/39S8UKIyxtww5cDz6?= =?Windows-1252?Q?HJLG1b4hzEKcz9GkyPCRXK1ug/htHLlFuNrnlDB8/HxiuDxrhpi4jRYr?= =?Windows-1252?Q?ayvprovDphcoq71tFT0WFuzMfQ2P+NMaS3mRaWm5zGTCZeqNe7qYRDEf?= =?Windows-1252?Q?NEDi6uSEIKDqitFZWpZMQSnhvq8jImLrCqHToB5wA8IKsdsjdGjaTLzG?= =?Windows-1252?Q?psZQAUtHz4n0aO/DF2X8KFtafoORZpDnhN72zdljPoIUw6D6b8rRKrjB?= =?Windows-1252?Q?bptzt20D6G5h6q6ecLpTyVN00QGHocRg4IfsHe46U/cKmjnRddwaTHcn?= =?Windows-1252?Q?mi8F09oGI1qg5gf5QGMWMrsWYGcia87xUNPlNQhN7jNjRtO8C4GpKOKW?= =?Windows-1252?Q?ACRGwXvB5qLyXM7XuEcMYTUBu0GtI+IFsrG7DO1LEwcBD6YQcUF3d0az?= =?Windows-1252?Q?Vz7IJ8xMEJUDPpklzFt91W0dVYOaNbVsss4+R3zzRTXybtUFj1ShgrKG?= =?Windows-1252?Q?q+wvnyZGbcegKoe5vmcYJGNWFlDGwKIxtWF3mxCLU9XPuQDUUqNlDWuF?= =?Windows-1252?Q?mr1PxByE6XIKhs+qS84r9O4VQcLhx5OXv1DWzG2iIpvaWSE1J7OFpcOM?= =?Windows-1252?Q?ki8iJNtmMrR8W28sfoFY1CGUDJftRj/XWnUgQRnpTLolly/QAzHPOTzt?= =?Windows-1252?Q?b3SGsjyZNLDfftgYhoAMGrKTbxUIY9hW42dDnl7CYK+28HcKnyTimt+V?= =?Windows-1252?Q?o0pQ9BZXsmEzFYbXxGc7YaJa1CR3CRYbl3XW/J1RopQUmZheLACjfCyO?= =?Windows-1252?Q?pe7BC3Qj?= X-OriginatorOrg: cornell.edu X-MS-Exchange-CrossTenant-Network-Message-Id: a3c46cef-bc7d-4529-af11-08d96ed42e0e X-MS-Exchange-CrossTenant-AuthSource: BN7PR04MB4388.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Sep 2021 12:13:03.6431 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 5d7e4366-1b9b-45cf-8e79-b14b27df46e1 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: K97zV/2yNuN4BXSxXrX/UFtyd7raY6bgyaO2FiPAHOIWRfxOk5CFJ1ALTdKeiAx+Dd142Df7GukRpGSwx3Sxow== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR04MB0354 X-Spam-Status: No, score=-3.3 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, JMQ_SPF_NEUTRAL, MSGID_FROM_MTA_HEADER, NICE_REPLY_A, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_HELO_PASS, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: cygwin-developers@cygwin.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Cygwin core component developers mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Sep 2021 12:13:07 -0000 On 9/3/2021 7:41 AM, Corinna Vinschen wrote: > On Sep 3 13:31, Corinna Vinschen wrote: >> On Sep 3 19:13, Takashi Yano wrote: >>> On Fri, 3 Sep 2021 19:00:46 +0900 >>> Takashi Yano wrote: >>>> On Thu, 2 Sep 2021 21:35:21 +0200 >>>> Corinna Vinschen wrote: >>>>> On Sep 2 21:00, Corinna Vinschen wrote: >>>>> It's getting too late again. I drop off for tonight, but I attached >>>>> my POC code I have so far. It also adds the snippets from my previous >>>>> patch which fixes stuff Takashi found during testing. It also fixes >>>>> something which looks like a bug in raw_write: >>>>> >>>>> - ptr = ((char *) ptr) + chunk; >>>>> + ptr = ((char *) ptr) + nbytes_now; >>>>> >>>>> Incrementing ptr by chunk bytes while only nbytes_now have been written >>>>> looks incorrect. >>>>> >>>>> As for the reader, it makes the # of bytes to read dependent on the >>>>> number of reader handles. I don't know if that's such a bright idea, >>>>> but this can be changed easily. >>>>> >>>>> Anyway, this runs all my testcases successfully but they are anything >>>>> but thorough. >>>>> >>>>> Patch relativ to topic/pipe attached. Would you both mind to take a >>>>> scrutinizing look? >>>> >>>> Thanks. >>>> >>>> Your code seems that read() returns only the partial data even >>>> if the pipe stil has more data. Is this by design? >>>> >>>> This happes in both blocking and non-blocking case. >>> >>> The patch attached seems to fix the issue. >> >> I'm sorry, but I don't see what your patch is supposed to do in the >> first place. What I see is that it now calls NtQueryInformationFile >> even in the non-blocking case, which is not supposed to happen. >> >> I'm a bit puzzled what the actual bug is. >> >> The code changing len is only called if there's no data in the pipe. >> In that case we only request a partial buffer so as not to block >> the writer on select. >> >> If there *is* data in the pipe, it will just go straight to the >> NtReadFile code without changing len. >> >> Where's the mistake? > > Oh, wait. Do you mean, if we only request less than len bytes, but > after NtReadFile there's still data in the buffer, we should try to > deplete the buffer up to len bytes in a subsequent NtReadFile? > > I thought this is unnecessary, actually, because of POSIX: > > The standard developers considered adding atomicity requirements to a > pipe or FIFO, but recognized that due to the nature of pipes and FIFOs > there could be no guarantee of atomicity of reads of {PIPE_BUF} or any > other size that would be an aid to applications portability. I agree. I think if read returns less than what was requested, it's up to the caller to call read again if desired. One tiny thing I noticed: get_obj_handle_count can return 0. So the line reader_count = get_obj_handle_count (get_handle ()); should be reader_count = get_obj_handle_count (get_handle ()) ?: 1; Or else get_obj_handle_count should be changed to return 1 instead of 0 if NtQueryObject fails. Ken